ML Data preparation
ml prep matrixml prep matrixml prep vectorml prep columnsml prep imputeml prep imputeml prep impute_knnml prep impute_knnml prep impute_iterativeml prep impute_iterativeml prep splitml prep split_stratifiedml prep kfoldml prep one_hotml prep label_encodeml prep polynomialml prep discretizeml prep variance_thresholdml prep select_k_bestml prep corr_matrixml prep relieffml prep relieffml prep cfsml prep cfsml prep rfeml prep rfeml prep smoteml prep smoteml prep tfidfml prep tfidfml prep wrapperml prep wrapper
ml prep matrix <json_rows> <json_fields>
Description
Build the matrix X from a JSON array of objects (ex: the result of sql parse) and a list of fields; missing values become null
Build the matrix X from a JSON array of objects (ex: the result of sql parse) and a list of fields; missing values become null
Parameters
json_rows: A JSON array of objects - String - required
json_fields: The fields to keep, in this order - String - required
json_rows: A JSON array of objects - String - required
json_fields: The fields to keep, in this order - String - required
admin
ml prep matrix "[{\"temp\": 21.5, \"hum\": 40, \"ok\": 1}, {\"temp\": \"23\", \"hum\": null, \"ok\": 0}]" "[\"temp\", \"hum\"]"
mentdb
[[21.5, 40], [23, null]]
Complete: a table with a text column (region) -> matrix -> model -> prediction of new rows
admin
#The rows of a table (sql parse, csv parse or json): age, visits, region (text) and the column to predict, churn; -> "[rows]" "[{\"age\": 23, \"visits\": 2, \"region\": \"north\", \"churn\": 1}, {\"age\": 45, \"visits\": 12, \"region\": \"south\", \"churn\": 0}, {\"age\": 31, \"visits\": 3, \"region\": \"west\", \"churn\": 1}, {\"age\": 52, \"visits\": 15, \"region\": \"north\", \"churn\": 0}, {\"age\": 27, \"visits\": 1, \"region\": \"south\", \"churn\": 1}, {\"age\": 39, \"visits\": 9, \"region\": \"west\", \"churn\": 1}, {\"age\": 61, \"visits\": 11, \"region\": \"north\", \"churn\": 0}, {\"age\": 22, \"visits\": 4, \"region\": \"west\", \"churn\": 0}, {\"age\": 48, \"visits\": 8, \"region\": \"south\", \"churn\": 0}, {\"age\": 35, \"visits\": 6, \"region\": \"north\", \"churn\": 0}, {\"age\": 57, \"visits\": 14, \"region\": \"west\", \"churn\": 0}, {\"age\": 29, \"visits\": 5, \"region\": \"south\", \"churn\": 1}]"; #The region becomes 3 columns 0/1, the categories are given so that the new rows get the same columns; -> "[fields]" "[\"age\", \"visits\", \"region\"]"; -> "[enc]" "{\"one_hot\": {\"region\": [\"north\", \"south\", \"west\"]}}"; -> "[X]" (ml prep matrix [rows] [fields] [enc]); -> "[y]" (ml prep vector [rows] "churn"); ml train "tuto_churn" "logistic" [X] [y]; #Two new customers, encoded the same way; -> "[new]" (ml prep matrix "[{\"age\": 26, \"visits\": 2, \"region\": \"west\"}, {\"age\": 50, \"visits\": 13, \"region\": \"north\"}]" [fields] [enc]); -> "[p]" (ml model predict_proba "tuto_churn" [new]); ml model delete "tuto_churn"; [p]
mentdb
{"classes": [0, 1], "proba": [[0.0800226516167637, 0.9199773483832363], [0.9881054581469852, 0.011894541853014763]]}
ml prep matrix <json_rows> <json_fields> <json_params>
Description
Build X with text columns: one_hot gives one 0/1 column per category of the listed fields, in place of the field ({"one_hot": ["field"]}, the categories taken from the rows, sorted, or {"one_hot": {"field": ["category", ...]}} to get the same columns for the training and the new rows; an unknown category gives 0 everywhere)
Build X with text columns: one_hot gives one 0/1 column per category of the listed fields, in place of the field ({"one_hot": ["field"]}, the categories taken from the rows, sorted, or {"one_hot": {"field": ["category", ...]}} to get the same columns for the training and the new rows; an unknown category gives 0 everywhere)
Parameters
json_rows: A JSON array of objects - String - required
json_fields: The fields to keep, in this order - String - required
json_params: The parameters - String - required
json_rows: A JSON array of objects - String - required
json_fields: The fields to keep, in this order - String - required
json_params: The parameters - String - required
admin
ml prep matrix "[{\"age\": 25, \"region\": \"north\"}, {\"age\": 40, \"region\": \"south\"}, {\"age\": 33, \"region\": null}]" "[\"age\", \"region\"]" "{\"one_hot\": [\"region\"]}"
mentdb
[[25, 1, 0], [40, 0, 1], [33, null, null]]
Fixed categories (the same columns for the new rows)
admin
ml prep matrix "[{\"age\": 25, \"region\": \"west\"}]" "[\"age\", \"region\"]" "{\"one_hot\": {\"region\": [\"north\", \"south\", \"west\"]}}"
mentdb
[[25, 0, 0, 1]]
Often used with
ml fit · ml model delete · ml model predict · ml model transform · ml prep impute · ml prep vector · ml train
ml fit · ml model delete · ml model predict · ml model transform · ml prep impute · ml prep vector · ml train
ml prep vector <json_rows> <field>
Description
Get the values of one field from a JSON array of objects (the labels or the target)
Get the values of one field from a JSON array of objects (the labels or the target)
Parameters
json_rows: A JSON array of objects - String - required
field: The field - String - required
json_rows: A JSON array of objects - String - required
field: The field - String - required
admin
ml prep vector "[{\"temp\": 21.5, \"ok\": \"yes\"}, {\"temp\": 23, \"ok\": \"no\"}]" "ok"
mentdb
["yes", "no"]
Often used with
ml fit · ml model delete · ml model predict · ml model transform · ml prep impute · ml prep matrix · ml train
ml fit · ml model delete · ml model predict · ml model transform · ml prep impute · ml prep matrix · ml train
ml prep columns <json_X> <json_columns>
Description
Keep some columns of X (indexes from 0)
Keep some columns of X (indexes from 0)
Parameters
json_X: The rows - String - required
json_columns: The column indexes - String - required
json_X: The rows - String - required
json_columns: The column indexes - String - required
admin
ml prep columns "[[1, 2, 3], [4, 5, 6]]" "[2, 0]"
mentdb
[[3, 1], [6, 4]]
Used in use cases (2 recipes)
Data scientist › Select the useful variables of a regression
Data scientist › Features that only work together
Data scientist › Select the useful variables of a regression
Data scientist › Features that only work together
Often used with
ml cross_validate · ml prep select_k_best · ml prep relieff · ml prep rfe · ml prep variance_threshold
ml cross_validate · ml prep select_k_best · ml prep relieff · ml prep rfe · ml prep variance_threshold
ml prep impute <json_X> <strategy>
Description
Replace the missing values (null) of each column by its mean, median or mode
Replace the missing values (null) of each column by its mean, median or mode
Parameters
json_X: The rows - String - required
strategy: mean, median or mode - String - required
json_X: The rows - String - required
strategy: mean, median or mode - String - required
admin
ml prep impute "[[1, null], [3, 4], [null, 8]]" "mean"
mentdb
[[1, 6], [3, 4], [2, 8]]
admin
ml prep impute "[[1, 2], [1, null], [7, 2], [null, 5]]" "mode"
mentdb
[[1, 2], [1, 2], [7, 2], [1, 5]]
ml prep impute <json_X> <strategy> <value>
Description
Replace the missing values (null) by a constant
Replace the missing values (null) by a constant
Parameters
json_X: The rows - String - required
strategy: constant - String - required
value: The constant - Number - required
json_X: The rows - String - required
strategy: constant - String - required
value: The constant - Number - required
admin
ml prep impute "[[1, null], [null, 4]]" "constant" 0
mentdb
[[1, 0], [0, 4]]
Often used with
ml fit · ml model delete · ml model predict · ml model transform · ml prep matrix · ml prep vector · ml train
ml fit · ml model delete · ml model predict · ml model transform · ml prep matrix · ml prep vector · ml train
ml prep impute_knn <json_X>
Description
Replace the missing values (null) by the mean of the 5 nearest rows (distance on the other columns, scaled)
Replace the missing values (null) by the mean of the 5 nearest rows (distance on the other columns, scaled)
Parameters
json_X: The rows - String - required
json_X: The rows - String - required
admin
ml prep impute_knn "[[1, 2, 3], [1.1, 2.1, null], [5, 6, 7], [5.2, null, 7.1], [0.9, 1.9, 2.9]]"
mentdb
[[1, 2, 3], [1.1, 2.1, 5], [5, 6, 7], [5.2, 3, 7.1], [0.9, 1.9, 2.9]]
ml prep impute_knn <json_X> <json_params>
Description
KNN imputation with k neighbors
KNN imputation with k neighbors
Parameters
json_X: The rows - String - required
json_params: {"k": 2} - String - required
json_X: The rows - String - required
json_params: {"k": 2} - String - required
admin
ml prep impute_knn "[[1, 2, 3], [1.1, 2.1, null], [5, 6, 7], [5.2, null, 7.1], [0.9, 1.9, 2.9]]" "{\"k\": 2}"
mentdb
[[1, 2, 3], [1.1, 2.1, 2.95], [5, 6, 7], [5.2, 4, 7.1], [0.9, 1.9, 2.9]]
ml prep impute_iterative <json_X>
Description
Iterative imputation (MICE-like): each column with missing values is predicted by a ridge regression on the other columns, 10 rounds
Iterative imputation (MICE-like): each column with missing values is predicted by a ridge regression on the other columns, 10 rounds
Parameters
json_X: The rows - String - required
json_X: The rows - String - required
admin
ml prep impute_iterative "[[1, 2], [2, 4], [3, null], [4, 8], [null, 10]]"
mentdb
[[1, 2], [2, 4], [3, 6.000285551064459], [4, 8], [4.999428734647764, 10]]
ml prep impute_iterative <json_X> <json_params>
Description
Iterative imputation with parameters: max_iter, alpha (ridge penalty)
Iterative imputation with parameters: max_iter, alpha (ridge penalty)
Parameters
json_X: The rows - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_params: The parameters - String - required
admin
ml prep impute_iterative "[[1, 2], [2, 4], [3, null], [4, 8], [null, 10]]" "{\"max_iter\": 20}"
mentdb
[[1, 2], [2, 4], [3, 6.000285551064726], [4, 8], [4.999428734647236, 10]]
ml prep split <json_X> <json_y> <test_size> <seed>
Description
Shuffle and split X and y into a training set and a test set
Shuffle and split X and y into a training set and a test set
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
test_size: The fraction of the test set, ex: 0.25 - Number - required
seed: The seed of the shuffle - Number - required
json_X: The rows - String - required
json_y: The labels or values - String - required
test_size: The fraction of the test set, ex: 0.25 - Number - required
seed: The seed of the shuffle - Number - required
admin
ml prep split "[[1], [2], [3], [4], [5], [6], [7], [8]]" "[0, 0, 0, 0, 1, 1, 1, 1]" 0.25 42
mentdb
{"X_train": [[4], [2], [5], [1], [8], [6]], "X_test": [[3], [7]], "y_train": [0, 0, 1, 0, 1, 1], "y_test": [0, 1], "train_index": [3, 1, 4, 0, 7, 5], "test_index": [2, 6]}
Complete: split -> train on the training set -> score on the test set
admin
-> "[X]" "[[1, 2], [2, 1], [1, 1], [2, 2], [8, 9], [9, 8], [8, 8], [9, 9], [1, 3], [9, 7]]"; -> "[y]" "[0, 0, 0, 0, 1, 1, 1, 1, 0, 1]"; json load "tuto_s" (ml prep split [X] [y] 0.3 42); ml train "tuto_knn" "knn_classifier" (json select "tuto_s" "/X_train") (json select "tuto_s" "/y_train") "{\"k\": 3}"; -> "[acc]" (ml model score "tuto_knn" (json select "tuto_s" "/X_test") (json select "tuto_s" "/y_test")); json unload "tuto_s"; ml model delete "tuto_knn"; concat "accuracy on the test set: " [acc]
mentdb
accuracy on the test set: 1
ml prep split_stratified <json_X> <json_y> <test_size> <seed>
Description
Split X and y keeping the proportion of each class in the training and test sets
Split X and y keeping the proportion of each class in the training and test sets
Parameters
json_X: The rows - String - required
json_y: The labels - String - required
test_size: The fraction of the test set - Number - required
seed: The seed of the shuffle - Number - required
json_X: The rows - String - required
json_y: The labels - String - required
test_size: The fraction of the test set - Number - required
seed: The seed of the shuffle - Number - required
admin
ml prep split_stratified "[[1], [2], [3], [4], [5], [6], [7], [8]]" "[0, 0, 0, 0, 1, 1, 1, 1]" 0.5 42
mentdb
{"X_train": [[2], [1], [8], [6]], "X_test": [[3], [7], [4], [5]], "y_train": [0, 0, 1, 1], "y_test": [0, 1, 0, 1], "train_index": [1, 0, 7, 5], "test_index": [2, 6, 3, 4]}
ml prep kfold <n> <k> <seed>
Description
Get the training and test indexes of a k-fold cross validation
Get the training and test indexes of a k-fold cross validation
Parameters
n: The number of rows - Number - required
k: The number of folds - Number - required
seed: The seed of the shuffle - Number - required
n: The number of rows - Number - required
k: The number of folds - Number - required
seed: The seed of the shuffle - Number - required
admin
ml prep kfold 6 3 42
mentdb
[{"train": [2, 3, 4, 5], "test": [0, 1]}, {"train": [0, 1, 2, 3], "test": [4, 5]}, {"train": [0, 1, 4, 5], "test": [2, 3]}]
ml prep one_hot <json_values>
Description
One-hot encoding of a categorical variable (one column per category)
One-hot encoding of a categorical variable (one column per category)
Parameters
json_values: The values - String - required
json_values: The values - String - required
admin
ml prep one_hot "[\"red\", \"green\", \"red\", \"blue\"]"
mentdb
{"categories": ["blue", "green", "red"], "matrix": [[0, 0, 1], [0, 1, 0], [0, 0, 1], [1, 0, 0]]}
ml prep label_encode <json_values>
Description
Encode the categories as integers (sorted categories)
Encode the categories as integers (sorted categories)
Parameters
json_values: The values - String - required
json_values: The values - String - required
admin
ml prep label_encode "[\"red\", \"green\", \"red\", \"blue\"]"
mentdb
{"classes": ["blue", "green", "red"], "codes": [2, 1, 2, 0]}
ml prep polynomial <json_X> <degree>
Description
Polynomial features: every product of the columns up to the degree (without the constant)
Polynomial features: every product of the columns up to the degree (without the constant)
Parameters
json_X: The rows - String - required
degree: The degree (1 to 6) - Number - required
json_X: The rows - String - required
degree: The degree (1 to 6) - Number - required
admin
ml prep polynomial "[[2, 3], [1, 4]]" 2
mentdb
{"names": ["x0", "x1", "x0*x0", "x0*x1", "x1*x1"], "X": [[2, 3, 4, 6, 9], [1, 4, 1, 4, 16]]}
ml prep discretize <json_values> <method> <bins>
Description
Binning of a variable: equal width or equal frequency bins
Binning of a variable: equal width or equal frequency bins
Parameters
json_values: The values - String - required
method: width or frequency - String - required
bins: The number of bins - Number - required
json_values: The values - String - required
method: width or frequency - String - required
bins: The number of bins - Number - required
admin
ml prep discretize "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]" "width" 3
mentdb
{"bins": [0, 0, 0, 0, 1, 1, 1, 2, 2, 2], "edges": [1, 4, 7, 10], "counts": [4, 3, 3]}
admin
ml prep discretize "[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]" "frequency" 2
mentdb
{"bins": [0, 0, 0, 0, 0, 1, 1, 1, 1, 1], "edges": [1, 6.5, 55], "counts": [5, 5]}
ml prep variance_threshold <json_X> <threshold>
Description
Keep the columns whose variance is greater than a threshold
Keep the columns whose variance is greater than a threshold
Parameters
json_X: The rows - String - required
threshold: The minimum variance - Number - required
json_X: The rows - String - required
threshold: The minimum variance - Number - required
admin
ml prep variance_threshold "[[1, 5, 0], [2, 5, 0], [3, 5, 1]]" 0.1
mentdb
{"variances": [0.6666666666666666, 0, 0.22222222222222224], "selected": [0, 2]}
ml prep select_k_best <json_X> <json_y> <k> <score>
Description
Univariate feature selection: score of each column and the k best ones.
Scores: f_regression (regression), f_classif (ANOVA), chi2 (non negative features), mutual_info.
Univariate feature selection: score of each column and the k best ones.
Scores: f_regression (regression), f_classif (ANOVA), chi2 (non negative features), mutual_info.
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
k: The number of columns to keep - Number - required
score: f_regression, f_classif, chi2 or mutual_info - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
k: The number of columns to keep - Number - required
score: f_regression, f_classif, chi2 or mutual_info - String - required
admin
ml prep select_k_best "[[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 5], [7, 8], [8, 8]]" "[6.1, 8.0, 11.2, 14.1, 18.3, 19.9, 24.8, 26.1]" 1 "f_regression"
mentdb
{"scores": [633.2391226294905, 254.11327943015198], "p_values": [2.593273471518387E-7, 3.869092470853985E-6], "selected": [0]}
admin
ml prep select_k_best "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]" "[0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]" 1 "f_classif"
mentdb
{"scores": [389.32620320855875, 590.8521739130479], "p_values": [1.8221560926789948E-9, 2.8377467042872695E-10], "selected": [1]}
Used in use cases (2 recipes)
Data scientist › Select the useful variables of a regression
Data scientist › Features that only work together
Data scientist › Select the useful variables of a regression
Data scientist › Features that only work together
Often used with
ml cross_validate · ml prep columns · ml prep relieff · ml prep rfe · ml prep variance_threshold
ml cross_validate · ml prep columns · ml prep relieff · ml prep rfe · ml prep variance_threshold
ml prep corr_matrix <json_X> <method>
Description
Correlation matrix of the columns (pearson or spearman)
Correlation matrix of the columns (pearson or spearman)
Parameters
json_X: The rows - String - required
method: pearson or spearman - String - required
json_X: The rows - String - required
method: pearson or spearman - String - required
admin
ml prep corr_matrix "[[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 5], [7, 8], [8, 8]]" "pearson"
mentdb
[[1, 0.9718701861963732], [0.9718701861963732, 1]]
ml prep relieff <json_X> <json_y>
Description
ReliefF feature weights (Kononenko): a feature is good when it separates the nearest rows of different classes and not the nearest rows of the same class (RReliefF for a numeric target).
Returns the scores, the ranking (best first) and the selected features (positive weight).
ReliefF feature weights (Kononenko): a feature is good when it separates the nearest rows of different classes and not the nearest rows of the same class (RReliefF for a numeric target).
Returns the scores, the ranking (best first) and the selected features (positive weight).
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
admin
ml prep relieff "[[2, 1], [3, 2], [6, 1], [7, 2], [8, 1], [6, 6], [7, 8], [2, 7], [3, 5], [1, 2], [9, 2], [8, 7], [4, 4], [6.5, 2.5], [2.5, 8]]" "[\"no\", \"no\", \"yes\", \"yes\", \"yes\", \"no\", \"no\", \"no\", \"no\", \"no\", \"yes\", \"no\", \"no\", \"yes\", \"no\"]"
mentdb
{"scores": [0.17189814814814797, 0.16275132275132273], "ranking": [0, 1], "selected": [0, 1], "task": "classification"}
ml prep relieff <json_X> <json_y> <json_params>
Description
ReliefF with parameters: k (neighbors, 10), samples (0 = all rows), n_features (0 = the positive weights), task (auto, classification, regression), seed
ReliefF with parameters: k (neighbors, 10), samples (0 = all rows), n_features (0 = the positive weights), task (auto, classification, regression), seed
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
admin
ml prep relieff "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]" "[0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]" "{\"k\": 3, \"n_features\": 1}"
mentdb
{"scores": [0.5423280423280423, 0.5411255411255411], "ranking": [0, 1], "selected": [0], "task": "classification"}
ml prep cfs <json_X> <json_y>
Description
Correlation-based feature selection (Hall): the subset whose features are correlated with the class but not with each other (symmetric uncertainty, best first search)
Correlation-based feature selection (Hall): the subset whose features are correlated with the class but not with each other (symmetric uncertainty, best first search)
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
admin
ml prep cfs "[[2, 1], [3, 2], [6, 1], [7, 2], [8, 1], [6, 6], [7, 8], [2, 7], [3, 5], [1, 2], [9, 2], [8, 7], [4, 4], [6.5, 2.5], [2.5, 8]]" "[\"no\", \"no\", \"yes\", \"yes\", \"yes\", \"no\", \"no\", \"no\", \"no\", \"no\", \"yes\", \"no\", \"no\", \"yes\", \"no\"]"
mentdb
{"selected": [0, 1], "merit": 0.2933464908142476, "feature_class": [0.24395936159002546, 0.2618116080253718]}
ml prep cfs <json_X> <json_y> <json_params>
Description
CFS with parameters: bins (discretization, 10), max_stale (stops after this number of expansions without improvement, 5)
CFS with parameters: bins (discretization, 10), max_stale (stops after this number of expansions without improvement, 5)
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
admin
ml prep cfs "[[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 5], [7, 8], [8, 8]]" "[6.1, 8.0, 11.2, 14.1, 18.3, 19.9, 24.8, 26.1]" "{\"bins\": 4}"
mentdb
{"selected": [0], "merit": 1, "feature_class": [1, 1]}
ml prep rfe <json_X> <json_y>
Description
Recursive feature elimination with a random forest: train, remove the least important feature, again, until half of the features are left.
Ranking: 1 = selected, 2 = the last removed...
Recursive feature elimination with a random forest: train, remove the least important feature, again, until half of the features are left.
Ranking: 1 = selected, 2 = the last removed...
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
admin
ml prep rfe "[[2, 1], [3, 2], [6, 1], [7, 2], [8, 1], [6, 6], [7, 8], [2, 7], [3, 5], [1, 2], [9, 2], [8, 7], [4, 4], [6.5, 2.5], [2.5, 8]]" "[\"no\", \"no\", \"yes\", \"yes\", \"yes\", \"no\", \"no\", \"no\", \"no\", \"no\", \"yes\", \"no\", \"no\", \"yes\", \"no\"]"
mentdb
{"selected": [0], "ranking": [1, 2]}
ml prep rfe <json_X> <json_y> <json_params>
Description
RFE with any supervised algorithm (its importance, or the permutation importance): algo, params, n_features, step, seed; with cv >= 2 every number of features is cross validated and the best one is kept
RFE with any supervised algorithm (its importance, or the permutation importance): algo, params, n_features, step, seed; with cv >= 2 every number of features is cross validated and the best one is kept
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
admin
ml prep rfe "[[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 5], [7, 8], [8, 8]]" "[6.1, 8.0, 11.2, 14.1, 18.3, 19.9, 24.8, 26.1]" "{\"algo\": \"linear\", \"n_features\": 1}"
mentdb
{"selected": [0], "ranking": [1, 2]}
Cross validated
admin
ml prep rfe "[[1, 2], [1.5, 1.8], [1.2, 2.2], [0.8, 1.9], [5, 8], [6, 8.5], [5.5, 7.5], [6.2, 8.1], [9, 1], [8.5, 1.5], [9.2, 0.8], [8.8, 1.2]]" "[0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]" "{\"algo\": \"naive_bayes\", \"cv\": 3}"
mentdb
{"selected": [1], "score": 1, "steps": [{"n_features": 2, "score": 1, "features": [0, 1]}, {"n_features": 1, "score": 1, "features": [1]}], "ranking": [2, 1]}
ml prep smote <json_X> <json_y>
Description
SMOTE oversampling (Chawla): synthetic rows of the minority classes between a row and one of its 5 nearest neighbors of the same class, until every class has as many rows as the majority class.
Returns X and y (original rows first) and the number of rows added per class. Use it on the training set only.
SMOTE oversampling (Chawla): synthetic rows of the minority classes between a row and one of its 5 nearest neighbors of the same class, until every class has as many rows as the majority class.
Returns X and y (original rows first) and the number of rows added per class. Use it on the training set only.
Parameters
json_X: The rows - String - required
json_y: The labels - String - required
json_X: The rows - String - required
json_y: The labels - String - required
admin
ml prep smote "[[1, 1], [1.2, 0.9], [0.8, 1.1], [5, 5], [5.1, 4.9], [4.9, 5.2], [5.2, 5.1], [4.8, 4.8]]" "[\"rare\", \"rare\", \"rare\", \"ok\", \"ok\", \"ok\", \"ok\", \"ok\"]"
mentdb
{"X": [[1, 1], [1.2, 0.9], [0.8, 1.1], [5, 5], [5.1, 4.9], [4.9, 5.2], [5.2, 5.1], [4.8, 4.8], [0.9890669474505679, 1.005466526274716], [1.138256108933468, 0.930871945533266]], "y": ["rare", "rare", "rare", "ok", "ok", "ok", "ok", "ok", "rare", "rare"], "added": {"rare": 2}}
ml prep smote <json_X> <json_y> <json_params>
Description
SMOTE with parameters: k (neighbors, 5), ratio (rows of a minority class / rows of the majority class, 1), classes (only these classes), seed
SMOTE with parameters: k (neighbors, 5), ratio (rows of a minority class / rows of the majority class, 1), classes (only these classes), seed
Parameters
json_X: The rows - String - required
json_y: The labels - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_y: The labels - String - required
json_params: The parameters - String - required
admin
ml prep smote "[[1, 1], [1.2, 0.9], [0.8, 1.1], [5, 5], [5.1, 4.9], [4.9, 5.2], [5.2, 5.1], [4.8, 4.8]]" "[0, 0, 0, 1, 1, 1, 1, 1]" "{\"k\": 2, \"ratio\": 0.8}"
mentdb
{"X": [[1, 1], [1.2, 0.9], [0.8, 1.1], [5, 5], [5.1, 4.9], [4.9, 5.2], [5.2, 5.1], [4.8, 4.8], [0.9890669474505679, 1.005466526274716]], "y": [0, 0, 0, 1, 1, 1, 1, 1, 0], "added": {"0": 1}}
ml prep tfidf <json_texts>
Description
TF-IDF vectorization of texts for the classification of texts with ml train: one column per word, idf = ln((1 + n) / (1 + df)) + 1, rows normalized (l2).
Returns X, the vocabulary and the idf.
TF-IDF vectorization of texts for the classification of texts with ml train: one column per word, idf = ln((1 + n) / (1 + df)) + 1, rows normalized (l2).
Returns X, the vocabulary and the idf.
Parameters
json_texts: A JSON array of texts - String - required
json_texts: A JSON array of texts - String - required
admin
ml prep tfidf "[\"pump leaking oil\", \"oil leak at the pump seal\", \"motor overheating\", \"high temperature on the motor\"]"
mentdb
{"X": [[0, 0, 0, 0.6676785446095399, 0, 0.5264054336099155, 0, 0, 0.5264054336099155, 0, 0, 0], [0.45338639737285463, 0, 0.45338639737285463, 0, 0, 0.3574550433419527, 0, 0, 0.3574550433419527, 0.45338639737285463, 0, 0.3574550433419527], [0, 0, 0, 0, 0.6191302964899972, 0, 0, 0.7852882757103967, 0, 0, 0, 0], [0, 0.4854606118156975, 0, 0, 0.3827427224171519, 0, 0.4854606118156975, 0, 0, 0, 0.4854606118156975, 0.3827427224171519]], "vocabulary": ["at", "high", "leak", "leaking", "motor", "oil", "on", "overheating", "pump", "seal", "temperature", "the"], "idf": [1.916290731874155, 1.916290731874155, 1.916290731874155, 1.916290731874155, 1.5108256237659907, 1.5108256237659907, 1.916290731874155, 1.916290731874155, 1.5108256237659907, 1.916290731874155, 1.916290731874155, 1.5108256237659907]}
ml prep tfidf <json_texts> <json_params>
Description
TF-IDF with parameters: lowercase (true), stop_words (none, english, french), ngram (1 to 5), min_df and max_df (count or fraction of the texts), max_features (the most frequent terms), sublinear_tf, binary, norm (l2, l1, none).
To vectorize new texts with the same columns, give the vocabulary and the idf of the first call.
TF-IDF with parameters: lowercase (true), stop_words (none, english, french), ngram (1 to 5), min_df and max_df (count or fraction of the texts), max_features (the most frequent terms), sublinear_tf, binary, norm (l2, l1, none).
To vectorize new texts with the same columns, give the vocabulary and the idf of the first call.
Parameters
json_texts: A JSON array of texts - String - required
json_params: The parameters - String - required
json_texts: A JSON array of texts - String - required
json_params: The parameters - String - required
admin
ml prep tfidf "[\"pump leaking oil\", \"oil leak at the pump seal\", \"motor overheating\", \"high temperature on the motor\"]" "{\"stop_words\": \"english\", \"ngram\": 2, \"min_df\": 2}"
mentdb
{"X": [[0, 0.7071067811865475, 0.7071067811865475], [0, 0.7071067811865475, 0.7071067811865475], [1, 0, 0], [1, 0, 0]], "vocabulary": ["motor", "oil", "pump"], "idf": [1.5108256237659907, 1.5108256237659907, 1.5108256237659907]}
New texts with the vocabulary of the training texts
admin
ml prep tfidf "[\"oil on the motor\"]" "{\"vocabulary\": [\"leak\", \"motor\", \"oil\", \"pump\"], \"idf\": [1.9163, 1.5108, 1.5108, 1.5108]}"
mentdb
{"X": [[0, 0.7071067811865476, 0.7071067811865476, 0]], "vocabulary": ["leak", "motor", "oil", "pump"], "idf": [1.9163, 1.5108, 1.5108, 1.5108]}
ml prep wrapper <json_X> <json_y>
Description
Wrapper feature selection: greedy forward search, each subset of features scored by a 5-fold cross validation of naive_bayes; stops when the score does not increase
Wrapper feature selection: greedy forward search, each subset of features scored by a 5-fold cross validation of naive_bayes; stops when the score does not increase
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
admin
ml prep wrapper "[[2, 1], [3, 2], [6, 1], [7, 2], [8, 1], [6, 6], [7, 8], [2, 7], [3, 5], [1, 2], [9, 2], [8, 7], [4, 4], [6.5, 2.5], [2.5, 8]]" "[\"no\", \"no\", \"yes\", \"yes\", \"yes\", \"no\", \"no\", \"no\", \"no\", \"no\", \"yes\", \"no\", \"no\", \"yes\", \"no\"]"
mentdb
{"selected": [0, 1], "score": 0.8, "metric": "accuracy", "steps": [{"features": [1], "score": 0.7333333333333333}, {"features": [0, 1], "score": 0.8}]}
ml prep wrapper <json_X> <json_y> <json_params>
Description
Wrapper with parameters: algo, params, direction (forward or backward), k (folds), min_gain, seed
Wrapper with parameters: algo, params, direction (forward or backward), k (folds), min_gain, seed
Parameters
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
json_X: The rows - String - required
json_y: The labels or values - String - required
json_params: The parameters - String - required
admin
ml prep wrapper "[[1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 5], [7, 8], [8, 8]]" "[6.1, 8.0, 11.2, 14.1, 18.3, 19.9, 24.8, 26.1]" "{\"algo\": \"linear\", \"direction\": \"backward\", \"k\": 4}"
mentdb
{"selected": [0, 1], "score": 0.9859252221512795, "metric": "r2", "steps": [{"features": [0, 1], "score": 0.9859252221512795}]}