percent_outliers_15iqr

numeric
v < (q25 − 1.5 *

iqr), v > (q75 +

1.5 * iqr)

percent_outliers_3iqr

numeric

percent_outliers_1_99

numeric

percent_outliers_3std

numeric
v < (mean − 3 *

std), v > (mean +

3 *sample_std))

has_outliers_15iqr

boolean

has_outliers_3iqr

boolean

has_outliers_1_99

boolean

has_outliers_3std

boolean

Distribution
normality_statistic

numeric
Check how differs

features

from a normal

distribution

(Skewtest, kurtosi

stest)

normality_p

numeric
P value

is_normal_5

boolean
normality_p < 0.05

is_normal_1

boolean
normality_p < 0.01

Statistical features
mean

numeric

normalized_mean

numeric

median

numeric

normalized_median

numeric

var

numeric

std

numeric

coeff_var

numeric
coefficient of

variation

min

numeric

max

numeric

range

numeric

normalized_range

numeric

entropy

numeric
a degree of

randomness or

uncertainty

gini

numeric
Gini coefficient of

an array (e.g., as

determined at

https://github.com/

oliviaguest/gini)

q25

numeric
25% percentile

q75

numeric
75% percentile

med_abs_dev

numeric
median absolute

deviation

avg_abs_dev

numeric
average absolute

deviation

quant_coeff_disp

numeric
(q75 − q25)/

(q75 + q25)

skewness

numeric
measure of the

asymmetry of the

probability

distribution

kurtosis

numeric
measure of the

“tailedness” of the

probability

distribution

moment_5

numeric
measure a

distribution

moment_6

numeric

moment_7

numeric

moment_8

numeric

moment_9

numeric

moment_10

numeric

TABLE 2

Feature Type
Pairwise Feature
Options for values
Type of data
Other Notes

Data Type Features
Pairwise _column_
Numerical-
categorical

data_type
categorical,

categorical-

numerical,

decimal-decimal,

etc.

Pairwise_ML_target
Numerical-Target,
categorical
One column is

decimal-Target,

target, the other

etc.

column is

something else

Shared data
has_shared_elements

boolean

features
num_shared_elements

numeric

percent_shared_elements

numeric

identical

boolean

has_shared_unique_

boolean

elements

num_shared_unique_

numeric

elements

percent_shared_unique_

numeric

elements

identical_unique

boolean

Numerical-
correlation_value

numeric

numerical pairwise
correlation_p

numeric

features
correlation_

boolean

significant_005

ks_statistic

numeric

ks_p

numeric

ks_significant_005

boolean

percent_range_overlap

numeric

has_range_overlap

numeric

String-String
chi_sq_statistic

numeric
chi-square statistic

pairwise features
chi_sq_p

numeric

chi_sq_significant_

numeric

005

Numerical-String
one_way_anova_

numeric
one-way ANOVA

pairwise features
statistic

one_way anova_p

numeric

one_way_anova_

boolean

significant_005

The code features 230 may include aspects of the code in relation to the visualizations. For example, the code features 230 may include an API call immediately before the visualization command to represent an explanation of functional blocks to be used in the ML pipeline. For example, the visualization may provide a visual explanation of why the immediately preceding functional block is used in the ML pipeline. As another example of the code features 230, the code features 230 may include an API call immediately after the visualization command to represent an exploration of the functional blocks to be used in the ML pipeline. For example, after observing the visualization of the precursor dataset 210, what is observed by the visualization leads to the choice of the next functional block in the ML pipeline. For example, after observing the visualization results of a boxplot showing the missing values, what is observed by the visualization leads to the choice of the next functional block in the ML pipeline, such as fillna( ) to facilitate filling the missing values.

The correlations 235 may represent relationships between the data features 225 and the visualizations 220, and the correlations 240 may represent relationships between the code features 230 and the visualizations 220.

The data feature based-rules 245 may represent a relationship between the date features 225 and the visualizations 220. The rule may include a conditional statement with an antecedent (the data feature) and a consequent (the visualization), with a numerical value representative of the probability that when the antecedent occurs, the consequent also occurs. Additionally or alternatively, the rule may account for lift, which may represent the likelihood of the consequent occurring with knowledge that the antecedent is present, rather than merely that it is present.

The code features based-rules 250 may be similar or comparable to the data feature based-rules 245, although based on the code features 230 rather than the data features 225.

The list of rules 255 may include rules that are stored or presented in a manner that is human readable. For example, the rules may be written with the antecedent, the consequent, and the confidence as a statement: If [antecedent] occurs, there is a [confidence] likelihood that [consequent] occurs. In such a statement, antecedent may be a data feature, a code feature, a combination of multiple data features, or a combination of multiple code features, consequent may include the visualizations (e.g., plot type and/or what data to plot), and confidence may represent the likelihood of the consequent occurring in the event of the antecedent occurring.

For each of FIGS. 3-6, the associated operations may be performed by any suitable system or device. For example, one or more operations of the operations depicted in FIGS. 3-6 may be performed by or directed for performance by the system 100, the visualization predictor 110, or other components of FIG. 1. Additionally or alternatively, the operations may be performed by a computing system such as the computing system 700 of FIG. 7.

Modifications, additions, or omissions may be made to FIGS. 3-6 without departing from the scope of the present disclosure. For example, the operations depicted in FIGS. 3-6 may include more or fewer operations than those illustrated and described in the present disclosure. Further, the order of the description of operations depicted in FIGS. 3-6 does not mean that the operations must be performed in the described order. In addition, in some instances, a same operation may be described with respect to different portions of the operations depicted in FIGS. 3-6, but in some instances may only be performed once and used for the different portions of the operations depicted in FIGS. 3-6.

FIG. 3 is a flowchart of an example method 300 of extracting coding features and data features from training machine learning pipelines, in accordance with one or more embodiments of the present disclosure.

At block 305, training ML pipelines may be selected as part of a training dataset. For example, a set of ML pipelines from Kaggle or GitHub that are highly rated and include visualizations may be selected.

At block 310, abstract syntax tree (AST) processing may be used to identify API calls in the code of the ML pipelines of the block 305. For example, the code of the ML pipelines may be parsed using AST to identify functional blocks as API calls within the ML pipeline.

At block 315, a mandatory argument may be identified for a plotting API call identified in the pipeline code. For example, if the API plotting call is plt.hist(y_classes), the code y_classes may be identified as the mandatory argument.

At block 320, an AST traceback may be used to see how the mandatory arguments are defined in the code. For example, the code may be parsed backwards from the plotting API to identify where the mandatory arguments are defined. Following the example above, the code may be parsed backwards from the plotting API to identify use of y_classes earlier than plt.hist(y_classes) in the code (e.g., the line y_classes=kmeans.predict(X) may be located just before the plotting command in the pipeline code).

At block 325, a determination may be made whether the mandatory argument of the block 315 is processed by an ML API in the ML pipeline before the plotting API call. If it is processed by an ML API, the method 300 may proceed to the block 330. If it is not processed by an ML API, the method 300 may proceed to the block 335. For the given example mentioned above, the ML API might be identified which is “Predict( )”.

At block 330, the given ML API before the plotting API may be added to generate rules. For example, a statement may be generated with the plotting API as the consequent and the ML APIs being the antecedent. In some embodiments, if multiple arguments are used by the plotting API, all of the arguments being processed by the plotting API may be included to generate the rules. In these and other embodiments, the given ML API occurring before the plotting API call may be added as a code feature related to the visualization being used for explanation of the ML pipeline.

At block 335, the pipeline code may be reversed.

At block 340, an AST traceback may be used to see how the mandatory arguments are defined in the code in reverse. For example, by reversing the code and then performing the AST traceback, the arguments used in the plotting API call may be identified after the plotting API call.

At block 345, a determination may be made whether the argument is processed by the ML API, and may be similar or comparable to the block 325. If it is processed by the ML API, the method 300 may proceed to the block 350. If the ML API is not processed by the ML API, the method 300 may proceed to the block 355.

At block 350, the given ML API after the plotting API call may be added to generate rules. For example, the given ML API occurring after the plotting API call may be added as a code feature related to the visualization being used in exploration of the ML pipeline.

At block 355, a determination may be made to check whether the argument is a column in the precursor training dataset. If the argument is such a column, the method 300 may proceed to the block 360. If the argument is not such a column, the method 300 may proceed to the block 365.

At block 360, the data features may be added for generating rules. For example, the data features of the column may be added in conjunction with the visualization to facilitate generation of a rule related to the data feature and the visualization.

At block 365, the argument from the plotting API and/or the plotting API itself may be discarded. Stated another way, the plotting API may not be used in generating rules to facilitate predicting visualizations for a new ML pipeline.

FIG. 4 is a flowchart of an example method 400 of generating a machine learning pipeline that include visualizations, in accordance with one or more embodiments of the present disclosure.

At block 405, a new ML pipeline may be obtained. For example, the new ML pipeline may be the ML pipeline 105 of FIG. 1.

At block 410, code and/or dataset features of the new ML pipeline may be extracted. For example, the extraction of the code and/or dataset features may be similar or comparable to the method 300 of FIG. 3. While FIG. 3 is described with reference to training ML pipelines and/or precursor datasets, the same or a similar process may be undertaken to identify code features and/or dataset features for the new ML pipeline.

At block 415, all the code and/or dataset features may be mapped to a given rule. For example, the code and/or data set features may be compared to the given rule. In some embodiments, the given rule may be selected as the first rule in a list of rules (such as the list of rules 120 of FIG. 1). In some embodiments, the rules may be listed in order of confidence and/or lift. As another example, one or more rules with lower confidence may be included earlier in the list such that rules with lower confidence may be considered and/or included. For example, a given ML API may have all rules associated with it included higher in the list, even if the confidence is low.

At block 420, a determination may be made whether the code and/or dataset features match the antecedent of the given rule. If the features do match the antecedent of the given rule, the method 400 may proceed to the block 425. If the features do not match the antecedent of the given rule, the method 400 may proceed to the block 430.

At block 425, the visualization associated with the given rule may be generated. For example, the visualization may be generated based on the API visualization call as the visualization command (e.g., what kind of plot is to be generated), and the parameters of the API visualization command may be based on the given rule. In some embodiments, the block 425 may include embedding the visualization and/or the visualization command including the parameters within the new ML pipeline. In these and other embodiments, the visualization commands may be inserted just before or just after the associated functional block (e.g., ML API command) based on the given rule.

At block 430, the method 400 may proceed to a next rule to consider for visualizations. For example, the method 400 may go to the next rule in the list of rules. After the block 430, the method 400 may return to the block 415 to map the features to the next rule proceeded to at the block 430.

At block 435, after generating the visualization at the block 425, a determination may be made whether the visualization has already been generated. If it has already been generated, the method 400 may proceed to the block 430 to move to the next rule. If it has not already been generated, the method 400 may proceed to the block 440.

At block 440, a determination may be made whether visualization constraints are satisfied. Such visualization constraints may include a threshold number of visualizations to include in a given ML pipeline (e.g., such as limiting a number of visualizations to three). As another example, such visualization constraints may include limits on a number of types or styles of visualizations, a limit on a number of visualizations of a certain data field or data type, or other similar or related constraints. If the visualization constraints are met, the method 400 may proceed to the block 450 to end.

At block 445, a determination may be made whether there are any remaining rules which have not been analyzed. If there are rules remaining, the method 400 may proceed to the block 430. If there are no rules remaining, the method 400 may proceed to the block 450.

At block 450, the method 400 may end.

FIG. 5 is a flowchart of an example method 500 of deriving rules related to visualizations, in accordance with one or more embodiments of the present disclosure.

At block 510, a training dataset may be obtained. For example, the training dataset may include a set of ML pipelines that are obtained that are highly rated and/or include visualizations. Such ML pipelines may be obtained from Kaggle, GitHub, or other project repositories.

At block 520, each ML pipeline of the training dataset may be mapped to a respective precursor training dataset. A given precursor dataset may be the dataset of information on which a respective ML pipeline of the training dataset is trained. For example, on the project page and/or in the code of the ML pipeline, the precursor training dataset may be identified.

At block 530, pairs of plotting APIs (e.g., visualization commands) and the column that it plots in the precursor training dataset may be retrieved. For example, the visualization command and the parameters thereof may be analyzed such that the data column that is visualized may be retrieved.

At block 540, data features for each data column may be computed. For example, the data features may correspond to aspects of the data in the columns and/or other aspects of the data, such as those described above in Tables 1 and 2.

At block 550, a data feature dataset may be built. For example, the data feature dataset may include a given plot API or other visualization command, the column(s) that are visualized, and/or the data features related to the visualized columns.

At block 560, features may be discretized for continuous numerical features. For example, some values in the columns may vary widely and include continuous numerical values. As shown in Table 1 and Table 2, there are many features in the data feature dataset that may be continuous numerical values. For such values, one or more approaches may be used to discretize the data such that, rather than a continuous set of numerals, discrete buckets or groups of values may be designated within which all of the data fits. For example, the continuous values may be discretized in an equal-width manner where a set number of buckets are generated and each bucket covers an equivalent numerical range. As another example, an equal-frequency approach may be undertaken where a set number of buckets is generated and covers a range such that each bucket includes an equivalent number of observations. As another example, a K-means approach (or other unsupervised machine learning clustering algorithms) may be undertaken to group the numerical features into different clusters. For example, where a set number of averages are determined, and buckets around the set number of averages may be utilized, or a set number of standard deviations from the average is used as one bucket, and outside of that range is used as two other buckets (above and below the central range).

By discretizing the data, the rules may more readily be applicable to new datasets. For example, the rules can then be applicable to a given range of values rather than trying to generate and/or apply rules from continuous values.

At block 570, the data feature dataset may be converted to an association rule mining (ARM) compatible format. For example, such an association rule may include an antecedent and the likelihood of a consequent based on the presence of the antecedent. In some embodiments, such rules may be included in an ARM training dataset, which may be based on the data feature dataset.

At block 580, the converted data feature dataset may be fit to the ARM processing to find out rules between the data features and the plots. For example, the ARM processing may analyze the variety of data feature dataset and/or the visualizations to generate rules that articulate the relationships between the various data features and the inclusion of the visualizations. Stated another way, the block 580 may analyze which data features repeatedly occurred in relation to a given visualization such that the relationship may be articulated as a rule (e.g., when a column of the data set included the target for which the ML pipeline was designed to solve, that column (the target) was nearly always plotted with a histogram of the value).

In some embodiments, the blocks 560, 570, and/or 580 may operate together to perform the ARM process to determine the rules that are to be generated from the data features.

FIG. 6 is a flowchart of another example method 600 of generating a machine learning pipeline that include visualizations, in accordance with one or more embodiments of the present disclosure.

At block 610, a ML pipeline may be obtained. For example, the ML pipeline may be a new ML pipeline generated by an automatic ML pipeline generating tool that generates a series of functional blocks to perform a ML task. As another example, the ML pipeline obtained at the block 610 may include a ML pipeline designed and/or generated manually by a data scientist or other user, and they may seek to add visualizations to enhance or improve their ML pipeline.

At block 620, the ML pipeline may be used as the input to a visualization predictor that is configured to output visualization commands. For example, the ML pipeline may be analyzed by the visualization predictor and the visualization predictor may output a series of visualization commands (e.g., the API call to invoke a particular visualization and/or the parameters on which the API call is to act). In some embodiments, the block 620 may include identifying a location within the ML pipeline where the visualization command is to be located (e.g., just before or just after the code feature with which the visualization command may be associated).

At block 630, the ML pipeline may be instantiated with the visualization commands embedded within the ML pipeline. For example, the visualization commands with the associated parameters may be embedded at the locations identified at the block 620 such that an entire ML pipeline with additional plotting API calls (e.g., visualization commands) included in the updated ML pipeline. In some embodiments, the updated ML pipeline may be stored as a Jupyter Notebook or other programmatic notebook, Python programming code including the visualization commands embedded within the code, or as any other computer-readable and/or human readable programming code with the visualization commands included.

FIG. 7 illustrates a block diagram of an example computing system 700, according to at least one embodiment of the present disclosure. The computing system 700 may be configured to implement or direct one or more operations associated with a visualization predictor (e.g., the system 100 and/or the visualization predictor of FIG. 1). The computing system 700 may include a processor 710, a memory 720, and a data storage 730. The processor 710, the memory 720