100 can additionally or alternatively leverage: a probabilistic module, heuristic module, deterministic module, or any other suitable module leveraging any other suitable computation method, machine learning method or combination thereof. However, any suitable machine learning approach can otherwise be incorporated in the system 100. Further, any suitable model (e.g., machine learning, non-machine learning, etc.) can be used in implementing the intelligent optimization platform 110 and/or other components of the system 100.

The system 100 may additionally include a surrogate model generator (implemented via one or more of the optimization models 140) that may be implemented by the intelligent optimization platform 110. Specifically, when an API request is received by the system 100 that requests a status or suggestions of a partially completed (or incomplete) optimization work request, the intelligent optimization platform 110 may function to identify candidate data points and other data (including suggested hyperparameter values and optimized hyperparameters values) generated by each of the plurality of intelligent queue worker machines 120 for responding to the partially completed optimization work request and further, may trigger one or more of the optimization models of the ensemble of optimization models to generate a surrogate (or proxy) model that can be used to test the uncertainty and/or the likelihood that a candidate data point would perform well in an external model. In one example, the system 100 may function to obtain hyperparameter values of a most recent job of a first intelligent queue worker machine implementing Hyperopt and cause one of the optimization models 140, such as MOE, to generate the surrogate model using the hyperparameter values to test how well the candidate hyperparameter value data points may perform.

The system 100 may also implement a ranking system 155 that functions to rank multiple suggestions for a given optimization work request (or across multiple optimization work requests for a given user) such that the suggestions having hyperparameter values most likely to perform the best can be passed or pulled via the API 105. The ranking system 155 may be implemented in any suitable manner including by the one or more optimization algorithms of the ensemble 140 that generated the suggestions. For instance, if MOE is used to generate a plurality of suggestions for responding to an optimization work request, the system 100 may function to use MOE to implement the ranking system 155.

It shall be noted that the sub-systems and components of the system 100 may be connected or placed in operable communication using any suitable network and any suitable manner. For instance, the components of the system 100 may be connected directly or indirectly over a network. The network may include any public (e.g., the Internet) or private network (e.g., intranet), a virtual private network, a wireless local area network, a local area network, a wide area network, a wireless wide area network, a global area network, a cellular network, any combination of the aforementioned and the like.

2. Method for Multiple Criteria Optimization Using an Intelligent Optimization Platform

2.1 Convex Frontier

As shown in FIG. 2, a method 200 for multi-criteria optimization by implementing an intelligent optimization platform includes receiving an optimization work request S210, configuring a scalarized function for multiple objective functions of a machine learning model S220, optimizing the scalarized function S230, identifying Pareto optimal solutions for the scalarized function of the machine learning model S240, and tuning and/or implementing the machine learning model S250.

The method 200 generally functions to enable a generation of Pareto optimal solution sets for optimizing competing performance objective functions of a machine learning model. For instance, it may be an objective or desire to improve, at a same time, one or more of an accuracy, an error rate, validation loss, sparsity, an efficiency (speed of prediction or inference), and/or the like for a given machine learning model. However, in many instances, if one performance objective, such as accuracy is improved, a second objective, such as predictive efficiency or speed, may be diminished or degraded for the machine learning model. Thus, in such instances, there may be significant tradeoffs in performance of disparate objectives when they are optimized independently.

Accordingly, in one or more embodiments, the method 200 may function to recognize competing performance objectives of a machine learning model in which an apparent inverse or divergent optimization relationship may exist between two distinct and/or competing performance objectives of the machine learning model. The method 200 may function to overcome this divergent optimization scenario by implementing one or more techniques that enables a joint optimization of competing performance objectives of a machine learning model that further enables a determination of hyperparameter values that optimizes both of the competing objective functions and consequently, the subject machine learning model along both of the competing performance objectives, as described in more detail below.

Additionally, the method 200 may configure an intelligent optimization platform, in response to input values provided via an intelligent Application Program Interface (API), such that the intelligent optimization platform generates multiple Pareto optimal suggestions for hyperparameter values of competing objective functions of a scalarized function. In the context of the present application, a Pareto optimal solution for a given scalarized function and/or convex combination for a machine learning model generally relates to a solution that is not strictly more optimal than another Pareto optimal solution. That is, the Pareto optimal solutions identified along the Pareto-efficient frontier define a set of solutions where a first and second competing objective may be improved without sacrificing either objective.

S210, which includes receiving an optimization work request, functions to receive an optimization work request comprising a multi-criteria optimization work request via an intelligent APL The multi-criteria optimization work request may be referred to herein as a multi-criteria tuning work request. Preferably, a multi-criteria optimization work request relates to an optimization request made to the intelligent optimization platform that requires the optimization of two or more criterion and/or two or more objective functions (or metrics) of a single machine learning model rather than an optimization of one objective function of a single model. The intelligent API may be implemented as a client application on a client device, such as a web browser, or any suitable interface accessible to a remote user system. Within the intelligent API, the remote user may be able to create or define the multi-criteria optimization work request (or experiment) by providing one or more details of the objective functions or performance objectives that a user desires to optimize, hyperparameters and/or other features of machine learning model along with constraints (e.g., optimization budgets, bounds, etc.) for performing the optimization trials by the intelligent optimization platform.

Accordingly, in some embodiments, a multi-criteria optimization work request may include an identification of at least a first metric (e.g., predictive accuracy) of a machine learning model and a second metric (e.g., predictive efficiency/speed) of the machine learning model that a user desires to optimize. In such embodiments, each of the first metric and the second metric may be represented as a two-dimensional objective function, e.g., ƒ₁(x, y) and ƒ₂(x, y), respectively. In one example, the variables x and y of each of the objective functions ƒ₁and ƒ₂represent prospective objective function values, which may both take values in a range in the continuous interval of 0 to 1 (i.e., [0, 1]), and “x” and “y” may represent possible hyperparameter values that may operate to optimize ƒ₁and ƒ₂, respectively. It shall be noted that, while in a preferred embodiment, the method 200 functions to implement two-dimensional objective functions, any value of multidimensional objective functions including, but not limited to, three-dimensional objective functions (e.g., x, y, z) may be implemented.

Optionally, S215, which includes preprocessing a multi-criteria optimization work request, functions to perform a preliminary evaluation of the two or more objective functions of the multi-criteria optimization work request to determine whether the two or more objective functions for a given machine learning model compete or diverge in performance when provided a random sample of objective function values for each of the two or more objective functions. In some embodiments, the random values for each of the two or more objective functions may be generated using a low-discrepancy optimization source that may function to populate and/or compute random possible values for each of the functions.

Accordingly, S215 may function to validate whether the objective functions identified within an optimization work request compete. That is, in some embodiments, S215 may function to validate the divergence or competition between distinct objective functions of a given model based on similarities or differences in the outputs of the distinct objective functions based on similar or same input. The input into the objective functions may be selected from possible hyperparameter values for each of the distinct objectives. In one or more embodiments, if S215 determines that two or more of the objective functions compete, S215 may function to trigger a signal for switching or selecting an optimization mode of an intelligent optimization platform that performs an optimization of the contending objective functions, as described herein.

S220, which includes computing a new joint objective function based on the two or more competing objective functions, preferably functions to generate a scalarized function based on the two or more competing objective functions identified in the multi-criteria optimization work request. As referred to herein, a scalarization of the two or more competing objective functions preferably relates to a creation of a single, joint objective function that combines the two or more competing objective functions of a given model in a manner that enables a joint and/or simultaneous optimization of each of the two or more competing objective functions using a single equation.

Referring again to the example above, the two example competing objective functions, ƒ₁(x, y) and ƒ₂(x, y), maybe scalarized into a single, joint objective function represented as g (x, y)=λ*ƒ₁(x, y)+(1−λ)*ƒ₂(x, y), where the new, joint objective function that is optimized may be g (x, y). Effectively, the scalarization combines at least two two-dimensional competing objective functions into a single, combined convex combination having two dimensions. The convex combination preferably defines an interplay between the two or more competing objective functions. Specifically, the interplay between the two or more competing objective functions revealed by the convex combination function may include a Pareto optimal solution along a frontier curve of points (i.e., possible solutions to the scalarized function) having generally a convex shape. Accordingly, the scalarized function g may sometimes be referred to herein as a convex combination of ƒ₁and ƒ₂. Still with respect to such example, a weighting or a tuning factor represented by lambda (λ) may be constrained as follows: 0<λ<1, in some embodiments. The lambda value λ, may function to set a hyperplane and/or direction of optimization. The lambda values may generally enable a sweeping search for Pareto optimal values or define regions of search. Accordingly, lambda may function as a weighting value that may enable greater or lesser optimization emphasis of either ƒ₁or ƒ₂when its value is increased or decreased. For example, in the scalarized function, g (x, y)=λ*ƒ₁(x, y)+(1−λ)*ƒ₂(x, y), a larger lambda value may indicate a greater emphasis on the objective function ƒ₁and a lesser emphasis on second objective function ƒ₂. Conversely, a smaller lambda value may indicate a lesser emphasis on ƒ₁and a greater emphasis on ƒ₂. Accordingly, during an optimization and/or a tuning of the scalarized function as described in more detail in S230, the method 200 may function to adjust values for k, together with providing values for “x” and “y”.

Once a scalarized function is defined for the two or more competing objective functions of a machine learning model, S220 may function to provide the scalarized function as optimization input into the one or more optimization sources of the intelligent optimization platform. That is, in one or more embodiments, the intelligent optimization platform may function to optimize the objective functions of the scalarization g (x, y). For instance, S220 may function to perform and/or execute tuning operations that operate to identify potential hyperparameter values (i.e., “x” and “y”) for a given scalarized function, as described in U.S. Pat. Nos. 10,217,061 and 10,282,237, which are both incorporated herein in their entireties by this reference.

S230, which includes optimizing the scalarized function, functions to configure the optimization settings of the intelligent optimization platform to enable optimization of the scalarized function for optimizing a given machine learning model. In some embodiments, a basic configuration of the intelligent optimization platform includes settings for optimization of a single objective function that may typically be a two-dimensional objective function. However, in the circumstances in which a multi-criteria optimization work request is received or detected (e.g., S215), S230 may function to switch or convert an optimization mode of the intelligent optimization platform from a first optimization mode (e.g., for optimizing a single objective function) to a second optimization mode for optimizing a generated scalarization function based on a multi-criteria optimization work request and/or based on a received optimization mode selecting or switching signal (as provided by S215).

S230 may additionally or alternatively include S231, which includes implementing a first optimization phase of the objective functions of the scalarized function of a machine learning model, functions to provide one or more parameters of the scalarized function as input into a first optimization source of the intelligent optimization platform. For instance, S231 may function to provide a minimum and maximum value for each of the two objective functions that define the scalarized function. In this first optimization phase, S231 preferably functions to explore a number of possible values for the objective functions of the scalarized function bounded between an upper and lower bound defined in the multi-criteria optimization work request. That is, S231 may function to (randomly) populate a field of potential values for each of the objective functions of the scalarized function according to one or more predetermined evaluation or testing constraints.

Accordingly, S231 may function to allocate a first portion (e.g., 20% of optimization budget) of an optimization budget (as further defined in the multi-criteria optimization work request) to a low-discrepancy optimization source or the like of the intelligent optimization platform. In turn, the low-discrepancy optimization source may function to generate random values for the objective functions (e.g., values of x and y for g (x, y)) of the scalarized function that, in some embodiments, may be represented along a two-dimensional plane (ƒ₁-ƒ₂), as shown by example, in FIG. 5. A random distribution of values for the objective functions of the scalarized function may, therefore, be identified in S231.

S230 may additionally or alternatively include S233, which includes implementing a second optimization phase of the objective functions of the scalarized function, functions to provide the scalarized function (as defined in S220) as input into a second optimization source of the intelligent optimization platform. In this second optimization phase, S233 may function to optimize values of the objective functions of the scalarized function by incrementally adjusting the scalarized function by changing a lambda value. That is, in one example, S233 may function to incrementally adjust values of lambda, k, in a sweeping fashion (or any suitable manner) between the constraints of 0 and 1 to generate Pareto optimal solutions sets along a Pareto-efficient frontier that may be illustrated as a convex frontier. In such example, as a lambda value of the scalarized function is incrementally adjusted or changed, S230 may function to use the second optimization source to identify and/or generate optimal hyperparameter values for the scalarized function at each given lambda setting and/or value. In this example, S230 may function to test each of the random objective function values generated by the low-discrepancy optimization source or the like within a space or region defined by a selected lambda value.

In this second optimization phase, S233 may first function to allocate a second portion (e.g., 60% of optimization budget) of an optimization budget to the second optimization source of the intelligent optimization. Preferably, S233 allocates a larger portion of the optimization budget to the second optimization phase relative to the first optimization phase (and a third optimization described further below). In some embodiments, a technical advantage of allocating a larger optimization (or testing) budget to the second optimization phase enables a well-developed Pareto-efficient frontier that represents a number of objective function values for the scalarized function that jointly optimizes each of the two competing objective functions. Accordingly, depending on a desired performance of a machine learning model, a selection of an ordered pair of objective function values along the Pareto-efficient frontier for the scalarized function should yield an optimal performance of each of the two competing objective functions of the machine learning model.

As shown in FIG. 5, S233 may function to identify a frontier (e.g., a Pareto optimal frontier) by dividing the lambda value of the scalarized function into equal parts (e.g., six equal parts or the like), in one or more embodiments, and sweeping the lambda values between each of the resulting sections of the Pareto optimal frontier. For example, if a range of lambda is between 0 and 1, [0,1], S233 may function to partition the values of lambda into four equal sections (e.g., 0-0.25, 0.26-0.50, 0.51-0.75, and. 76-1). Specifically, in some embodiments, once the range of lambda values of the scalarized function is divided into equal parts, S233 may function to incrementally adjust the lambda value of the scalarized function within each section or sub-range and use the second optimization source to generate or identify objective function values for the scalarized function within each distinct subsection. Accordingly, after each incremental adjustment of the lambda value of the scalarized function within a divided lambda segment, the adjusted scalarized function may be provided as input into the second optimization source for generating or identifying objective function values (i.e., new ordered pairs of x and y values for g (x, y)) or new points for the scalarized function. It shall be noted that while S233 may preferably function to subdivide the total range of lambda values into equal parts, S233 may alternatively divide the total range of lambda values in any suitable manner, including unequally, randomly, or other predetermined manner. For instance, in some embodiments, failure regions for a given scalarized function may be known. In such instance, less or no optimization resources may be allocated to such failure regions by diminishing a lambda-determined search region or by eliminating the region from an optimization search all together. That is, S233 may function to exclude one or more values for lambda in which it may be known or in which there is a probability that corresponding hyperparameter values associated with a region of search set by the one or more lambda values may fail to optimize the scalarized function.

In this second optimization phase, which may also be referred to herein as the lambda sweeping phase, S233 functions to define a frontier of Pareto optimal values for the scalarization function by sweeping (e.g., incrementally adjusting a lambda value from in a range of 0 to 1 or 1 to 0 or the like). Preferably, each point on the frontier includes an ordered pair of the objective functions achievable by the machine learning model. It shall be noted that the Pareto-optimal solutions may not necessarily be unique, as there can be multiple input combinations of objective function values x and y that achieve a desired accuracy of 0.8 and efficiency of 0.2, for example. In some embodiments, the resultant Pareto optimal frontier may typically define a convex arc with a plurality of Pareto optimal values defining the frontier and also, surrounding lower and upper sections of the frontier and with some Pareto optimal values falling on or around the frontier arc.

S230 may additionally or alternatively include S235, which includes implementing a third optimization phase of the objective functions of the scalarized function, functions to provide the scalarized function as input into the second optimization source of the intelligent optimization platform. In this third optimization phase, S235 may function to optimize objective function values of the scalarized function by using extreme values for lambda (e.g., o or near o values and 1 or near 1 values). That is, S235 may function to select and/or adjust the lambda values of the scalarized function to lambda values near or at an upper bound of lambda (e.g., 1) and lambda values near or at a lower bound of lambda (e.g., o). In this way, S235 may function to populate the two edges of the Pareto optimal frontier with optimized objective function values for the scalarized function. Accordingly, the best possible Pareto optimal solutions that best optimizes each of the respective two or more competing objective functions may be represented along the frontier.

S240, which includes identifying Pareto optimal solutions for the objective functions of the scalarized function, functions to construct a graphical representation based on a plurality of pairings of objective function values (e.g., x, y values for g (x, y)) generated for the scalarized function. The graphical representation preferably includes a distribution of points defined by the plurality of pairings of objective function values for the scalarized function. Additionally, or alternatively, the graphical representation of the generated objective function values for the scalarized function preferably has two dimensions in which a first axis may represent values for a first objective function (e.g., ƒ