122 for purposes of clarity and explanation. As illustrated in FIG. 1, entity profiles that store attributes and corresponding attributes values for one or more corresponding entities are stored in entity profile store 123. Training data, as described above, is stored in training data store 125.

In one or more embodiments, the ML application 104 and/or system 100 refers to hardware and/or software configured to perform operations described herein for generating a ranked array of user interface elements corresponding to entities using machine learning. The ML application 104 and/or system 100 also refers to processes for updating a ranking analysis (e.g., attribute weights) based on user-received re-ordering of graphical interface elements corresponding to the entities. Examples of operations for these processes are described below with reference to FIG. 2.

In an embodiment, the system 100 is implemented on one or more digital devices. The term “digital device” generally refers to any hardware device that includes a processor. A digital device may refer to a physical device executing an application or a virtual machine. Examples of digital devices include a computer, a tablet, a laptop, a desktop, a netbook, a server, a web server, a network policy server, a proxy server, a generic machine, a function-specific hardware device, a hardware router, a hardware switch, a hardware firewall, a hardware firewall, a hardware network address translator (NAT), a hardware load balancer, a mainframe, a television, a content receiver, a set-top box, a printer, a mobile handset, a smartphone, a personal digital assistant (“PDA”), a wireless receiver and/or transmitter, a base station, a communication management device, a router, a switch, a controller, an access point, and/or a client device.

3. Updating Machine Learning Training Data Using Graphical Inputs

FIG. 2 illustrates an example set of operations, referred to collectively as a method 200, for generating and rendering a ranked array of user interface elements, where individual user interface elements correspond to individual entities in accordance with one or more embodiments. Elements of the method 200 depicted in FIG. 2 are associated with updating a training data set with user input that re-orders one or more of the user interface elements corresponding to entities within the ranked list. One or more operations illustrated in FIG. 2 may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in FIG. 2 should not be construed as limiting the scope of one or more embodiments.

The method 200 begins by training a machine learning model to generate (or equivalently, compute) match scores by comparing entity attribute values to a set of requirements (operation 204). The system may train a machine learning model in one example by obtaining training data sets that include attributes and corresponding attribute values for one or more entities (operation 206). The training data sets may also include a set of requirements against which entity attribute values are compared (operation 206). The training data sets may also include a match score representing a degree of match between entity attribute values and the set of requirements (operation 206). The collective training data set of attributes, attribute values, and match scores may be used to train the machine learning model.

The term “entities” as used herein is a generic term meant to encompass any of a number of types of discrete subjects that may be compared to a set of requirements to rank the entities against the set of requirements. One example of an entity includes any type of applicant that is evaluated against a set of requirements. In one embodiment, an applicant-type of entity is an application for an employment or scholastic (e.g., a student) position at an employer or school, respectively. In both of these cases, an applicant is evaluated for suitability of the position (or academic institution seat) against a set of desired criteria (“requirements”). Another embodiment of an entity is that of a corporate organization submitting a response to a request for proposal (RFP). In this example, the one or more entities submitting responses to an RFP are evaluated as to their technical and financial competitiveness relative to a set of desired criteria. These examples of entities are presented for convenience of explanation. The techniques described herein may be applied to other types of entities.

A set of requirements used for training data (and for target data, described below) may be adapted according to the entities being ranked. At a generic level, a set of requirements may include financial requirements, performance requirements, types of experiences, experience levels, certifications (e.g., from accredited institutions or industry certification organizations), or other requirements. Specific embodiments of requirements may include attributes associated with an employment history (years of service, levels of responsibility, disciplinary record), academic credentials (years of schooling completed, degrees or certificates granted), capabilities (e.g., business functions such as billing, types of services or products provided), location, minimum and/or maximum bid price or estimated cost, and the like.

In some examples, the set of requirements may include a concise or well defined attribute and a corresponding attribute value (e.g., structured data). For example, an attribute for “minimum bid/maximum bid” may be associated with attribute values of “$100/$1000.” In addition to a concise and/or standardized attribute name, the system may also apply natural language techniques to identify attributes and associated attribute values for an entity even if the entity provides data in a narrative form (e.g., unstructured data). For example, a narrative description in a response to a request for proposal may state “our best price is $775 and we will not exceed that amount by any more than 10%.” The machine learning system, using natural language processing, topic extraction, and other machine learning training techniques, may correctly interpret this as attribute values of “$775/$850.”

The training data may include a “match score” that quantifies a degree of similarity between attribute values for an entity and a set of requirements. In some examples, the match score may be calculated using any of a variety of similarity scores that include, but are not limited to, cosine similarity techniques.

In some examples, the system generates a match score corresponding to an entity as a whole relative to the set of requirements. In some examples, the system may first generate match scores individually for particular attributes associated with an entity and corresponding requirements of the set of requirements. The system may then generate a match score for an entity as a whole using these individual match scores. In some examples, the match score for the entity as a whole may be generated by generating an average, a weighted average, or a sum of the match scores for the particular individual attributes.

Once the system has generated a match score, the system may apply a label to the training data set thereby quantifying the degree of match between entity attributes/attribute values and a corresponding set of requirements. The system may then train a machine learning model with the collective data of entity attributes and attribute values, the set of requirements, and the match score described above.

Once trained, the system may apply the trained machine learning model to target entity data to determine a match between a target entity and a target set of requirements (operation 208). The operation 208 is performed using techniques analogous to those described above for training a machine learning model. For example, the trained machine learning model may receive entity data and identify attributes and/or attribute values within the entity data. In examples in which the entity data is provided in an unstructured format, the system may apply natural language processing techniques to identify attributes and/or corresponding attribute values within the entity data. The system may analyze structured data (e.g., such as data submitted via a form) by abstracting values from corresponding attribute locations within the structure.

The system may then generate a match score for a particular entity in light of a target set of requirements (operation 208). The system may generate a match score (e.g., using a cosine similarity technique) for an entity as a whole relative to a set of requirements. In one example, this may be done by collectively analyzing a similarity between the collective attribute values and the collective corresponding set of requirements. In another example, the system may generate the match score by generating individual match scores between individual requirements in the set of requirements and corresponding entity attributes. Then, these individual match scores may be averaged, averaged with differing weights, and/or summed to generate an entity-level match score relative to the entire set of requirements. These calculations may, in some cases, not be used when the match score is a binary value. The system may repeat this process for some or all of the entities evaluated relative to the set of requirements.

The system may then generate a graphical user interface (GUI) that presents individual interface elements, each of which corresponds to an entity, positioned within the GUI based on, at least in part, an entity match score (operation 212). The GUI may have any of a number of configurations that are generically referred to as a “ranked array.” In one example of a ranked array, the individual interface elements that correspond to entities may be arranged in a list format in which a first interface element associated with a highest match score is at a top of the GUI list and a second interface element associated with a lowest match score is at a bottom of the GUI list (opposite the top). In another example of a ranked array, the individual interface elements may be arranged in a grid. In one embodiment, decade tiers of match scores (e.g., 90-100, 80-89, 70-79) are arranged in rows of the grid and individual units within the decade (e.g., 90-92, 93-94) are arranged in columns.

As described below in more detail, individual user interface elements may themselves include indications of a degree of a match. For example, one or more user interface elements may be colored (e.g., black, red) to indicate a binary match score of “no match.” In other examples, the system may generate shaded interface elements to indicate a degree of match. One extreme of shading (e.g., black) may indicate a binary match score of “no match” with the set of requirements and another extreme (e.g., white) may indicate a binary match score of “match” with the set of requirements. Shades of gray may indicate intervening levels of matching. This shading may be performed using colors as well (e.g., from red to green) and not merely grayscale shading.

In some example, one or more user interface elements may include one or both of a grayscale shading value (from white to black) and/or a color scheme to indicate an evaluation status of an entity. As presented above, one type of application for the techniques described herein is to rank different entities relative to a set of criteria so that, ultimately, a subset of the entities may be selected. Selection processes often include multiple stages of further inquiring and/or negotiation before a disposition (mutual agreement, acceptance, denial) is reached. In some examples, a grayscale shading value and/or a color scheme may be used to indicate a stage of a selection process. For example, a white background in an interface element may indicate that an entity remains a candidate and no additional inquiry has been performed. A gray background in an interface element may indicate that an entity has been contacted and discussions are pending. A green background in an interface element may indicate that an entity has been accepted or approved. A black background in an interface element may indicate that an entity has been rejected.

After presenting a GUI as described above in the operation 212, the system may receive user input that re-orders one or more user interface elements corresponding to one or more entities within the GUI (operation 216). More specifically, the system may receive input that moves one interface element from a first location within the GUI (e.g., the ranked list or grid) to a second, different location within the GUI. Locations within the GUI are associated with match scores. For example, a second match score for the first user interface element moved to the second location may be generated based on a match score associated with one or more neighboring user interface elements. If the second location for the first user interface element is between two other user interface elements, the second match score for the first user interface element may be an average value of the match scores of the neighboring user interface elements. In another example, the second match score for the first user interface element may be based on a cosine similarity score with one or more neighboring user interface elements. In still other embodiments, the system may re-calculate attribute weights based on the re-located interface element and apply the re-calculated weights to a revised match score.

The system may then generate new training data that reflects the second (i.e., revised) match score for the first entity that is based on the second location for the corresponding user interface element (operation 220). In some examples, the system may update a corresponding label in the training data with the second match score that corresponds to the second location of the first user interface element.

In some examples, the system is configured to accommodate individual preferences. For example, different users of the system may have corresponding different user profiles that store corresponding revisions to the training data. While the system may be trained with a common training data set, different user inputs provided by different users may be stored in corresponding user profiles. These may be used to re-train or otherwise modify the calculations by which different attributes are weighted to form a ranked array of entities.

Regardless, the updates to the training data may be used to re-train a machine learning model so that rankings associated with the new training data are reflected in future computations (operation 224). The same training techniques described above in the context of. operations 204 and 206 may be applied to operation 224.

Alternatively, if no input is received, the system may use the current training (operation 230). In some examples, the system may interpret not receiving input as data and increase weight of a training data set relative to any new training data sets received. This technique (or other similar techniques) may be used to reflect the accuracy of an existing training data set (used for a number of analyses without change) in light of subsequently received new training data sets.

4. Example Embodiment

Detailed examples are described below for purposes of clarity. Components and/or operations described below should be understood as one specific example which may not be applicable to certain embodiments. Accordingly, components and/or operations described below should not be construed as limiting the scope of any of the claims.

FIG. 3 illustrates an example graphical user interface (GUI) 300 that includes a number of user interface elements 304-380. Each of the user interface elements 304-380 correspond to an entity and are arranged in the GUI 300 in a ranked array that reflects a match score between attribute values of the entity and a set of requirements.

In this example, the ranked array of user interface elements 304-380 is arranged in a grid format. In this example format, a user may identify a percentage match of a particular entity and/or patterns of matches within a group of ranked entities with a glance. Rows 384A-384D (collectively 384) identify decade “tiers” of a percentage match. Columns 388A-388D (collectively 388) identify a finer graduation of match within the tiers. As described above, the match scores are generated based on entity attributes in light of a set of requirements.

As illustrated, row 384A identifies user interface elements 304, 308, 312, 316, and 320 as having a match score from 90% to 100%. Row 388A indicates that user interface element 304 has a match score from 90% to 92%. Row 388B indicates that user interface element 308 has a match score from 93% to 94%. Row 388C indicates that user interface element 312 has a match score from 95% to 96%. Row 388D indicates that user interface element 316 has a match score from 97% to 98%. Row 388E indicates that user interface element 316 has a match score from 99% to 100%.

FIGS. 4 and 5 together illustrate the process by which the system may receive user input that re-orders one or more of the user interface elements, thereby providing new data to the trained machine learning model. This new data may be instantiated as a new training data set that is used to re-train the model so that it emphasizes (e.g., recalculates weights) attributes differently in light of the set of requirements.

Turning first to FIG. 4, the GUI 400 includes user interface elements 504-580. The GUI 400 and the user interface elements 504-580 are analogous to those described above in the context of FIG. 3 and the GUI 300.

As schematically illustrated by a dashed arrow in FIG. 4, interface element 552 is repositioned, in response to received user input, from its existing position within the GUI 400 to a new position within row 404 and column 408. FIG. 5 illustrates a GUI 500 in which the interface element 552 is in its new position. As described above, this new position generates a new ranking (a higher match score than interface element 508) for the entity corresponding to the interface element 552. The system then applies the new ranking to generate new training data or update existing training data, which may be used to re-train the machine learning model.

The user interface elements 576 and 580 are shaded to indicate an evaluation status of the corresponding entities. Namely that an evaluation of the entity corresponding to interface element 576 is underway (gray) and that the entity corresponding to interface element 580 has been rejected. The remaining interface elements 504-572 are shaded to indicate that no additional evaluation has been performed (e.g., no direct contact with the corresponding entities).

FIG. 6A provides a schematic illustration of the re-ranking analysis initiated by repositioning interface element 552 in the new position shown in FIG. 5. FIG. 6A includes a depiction 602 of a first set of requirements and corresponding weights, a depiction 604 of attributes and attribute values for the entity corresponding to the interface element 508, and a depiction 608 of attributes and attribute values for the entity corresponding to the interface element 552.

As shown in depiction 602, the highest attribute weights are applied to attributes of “highest degree completed” and a “highest responsibility level.” These two attributes both have a weight of 0.6, which increases their contribution to the match score relative to other attributes with lower weights. A “number of years of work experience” has the least emphasis with a weight of 0.2. A “number of years of advanced education” is between the extremes, with a weight of 0.4. Based on the requirements illustrated in depiction 602, the system generates a match score based on attribute values in the depiction 604 in which the entity corresponding to interface element 508 has a higher match score than the entity corresponding to interface element 552. This first match score scenario is illustrated in FIG. 4.

The system interprets the relocation of the interface element 552 as data for re-training the ML model. In one example, the system implements this retraining as changing weights associated with various attributes, thereby changing the proportional contribution of different attribute values to an overall match score. These changes are illustrated in FIG. 6B as depiction 610.

Turning in detail to FIG. 6B, presented for purposes of illustration and convenience of explanation, the system increases weights associated with years of work experience (from 0.2 to 0.6) and a highest responsibility level (from 0.6 to 0.7) in response to the repositioning of the interface element 552 as shown in FIG. 5. Similarly, the system reduces weights of the highest degree completed (from 0.6 to 0.2). In some examples, an attribute value for a most heavily weighted attribute for an entity (e.g., highest degree completed in depiction 604) may be displayed in a GUI (e.g., as a pop-up window upon mousing over an icon in GUI 400 or GUI 500).

In some examples, the system may apply the re-trained model that uses these revised attribute weights to generate an updated ranked array of the attributes. Similarly, in some examples, a new ranked array of the various entities may be generated based on the new training.

5. Computer Networks and Cloud Networks

In one or more embodiments, a computer network provides connectivity among a set of nodes. The nodes may be local to and/or remote from each other. The nodes are connected by a set of links. Examples of links include a coaxial cable, an unshielded twisted cable, a copper cable, an optical fiber, and a virtual link.

A subset of nodes implements the computer network. Examples of such nodes include a switch, a router, a firewall, and a network address translator (NAT). Another subset of nodes uses the computer network. Such nodes (also referred to as “hosts”) may execute a client process and/or a server process. A client process makes a request for a computing service (such as, execution of a particular application, and/or storage of a particular amount of data). A server process responds by executing the requested service and/or returning corresponding data.

A computer network may be a physical network, including physical nodes connected by physical links. A physical node is any digital device. A physical node may be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node may be a generic machine that is configured to execute various virtual machines and/or applications performing respective functions. A physical link is a physical medium connecting two or more physical nodes. Examples of links include a coaxial cable, an unshielded twisted cable, a copper cable, and an optical fiber.

A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (such as, a physical network). Each node in an overlay network corresponds to a respective node in the underlying network. Hence, each node in an overlay network is associated with both an overlay address (to address to the overlay node) and an underlay address (to address the underlay node that implements the overlay node). An overlay node may be a digital device and/or a software process (such as, a virtual machine, an application instance, or a thread) A link that connects overlay nodes is implemented as a tunnel through the underlying network. The overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunneling is performed through encapsulation and decapsulation.

In an embodiment, a client may be local to and/or remote from a computer network. The client may access the computer network over other computer networks, such as a private network or the Internet. The client may communicate requests to the computer network using a communications protocol, such as Hypertext Transfer Protocol (HTTP). The requests are communicated through an interface, such as a client interface (such as a web browser), a program interface, or an application programming interface (API).

In an embodiment, a computer network provides connectivity between clients and network resources. Network resources include hardware and/or software configured to execute server processes. Examples of network resources include a processor, a data storage, a virtual machine, a container, and/or a software application. Network resources are shared amongst multiple clients. Clients request computing services from a computer network independently of each other. Network resources are dynamically assigned to the requests and/or clients on an on-demand basis. Network resources assigned to each request and/or client may be scaled up or down based on, for example, (a) the computing services requested by a particular client, (b) the aggregated computing services requested by a particular tenant, and/or (c) the aggregated computing services requested of the computer network. Such a computer network may be referred to as a “cloud network.”

In an embodiment, a service provider provides a cloud network to one or more end users. Various service models may be implemented by the cloud network, including but not limited to Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS). In SaaS, a service provider provides end users the capability to use the service provider's applications, which are executing on the network resources. In PaaS, the service provider provides end users the capability to deploy custom applications onto the network resources. The custom applications may be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider provides end users the capability to provision processing, storage, networks, and other fundamental computing resources provided by the network resources. Any arbitrary applications, including an operating system, may be deployed on the network resources.

In an embodiment, various deployment models may be implemented by a computer network, including but not limited to a private cloud, a public cloud, and a hybrid cloud. In a private cloud, network resources are provisioned for exclusive use by a particular group of one or more entities (the term “entity” as used in this section refers to a corporation, organization, person, or other entity). The network resources may be local to and/or remote from the premises of the particular group of entities. In a public cloud, cloud resources are provisioned for multiple entities that are independent from each other (also referred to as “tenants” or “customers”). The computer network and the network resources thereof are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a “multi-tenant computer network.” Several tenants may use a same particular network resource at different times and/or at the same time. The network resources may be local to and/or remote from the premises of the tenants. In a hybrid cloud, a computer network comprises a private cloud and a public cloud. An interface between the private cloud and the public cloud allows for data and application portability. Data stored at the private cloud and data stored at the public cloud may be exchanged through the interface. Applications implemented at the private cloud and applications implemented at the public cloud may have dependencies on each other. A call from an application at the private cloud to an application at the public cloud (and vice versa) may be executed through the interface.

In an embodiment, tenants of a multi-tenant computer network are independent of each other. For example, a business or operation of one tenant may be separate from a business or operation of another tenant. Different tenants may demand different network requirements for the computer network. Examples of network requirements include processing speed, amount of data storage, security requirements, performance requirements, throughput requirements, latency requirements, resiliency requirements, Quality of Service (QoS) requirements, tenant isolation, and/or consistency. The same computer network may need to implement different network requirements demanded by different tenants.

In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that the applications and/or data of different tenants are not shared with each other. Various tenant isolation approaches may be used.

In an embodiment, each tenant is associated with a tenant ID. Each network resource of the multi-tenant computer network is tagged with a tenant ID. A tenant is permitted access to a particular network resource only if the tenant and the particular network resources are associated with a same tenant ID.

In an embodiment, each tenant is associated with a tenant ID. Each application, implemented by the computer network, is tagged with a tenant ID. Additionally or alternatively, each data structure and/or dataset, stored by the computer network, is tagged with a tenant ID. A tenant is permitted access to a particular application, data structure, and/or dataset only if the tenant and the particular application, data structure, and/or dataset are associated with a same tenant ID.

As an example, each database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only a tenant associated with the corresponding tenant ID may access data of a particular database. As another example, each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only a tenant associated with the corresponding tenant ID may access data of a particular entry. However, the database may be shared by multiple tenants.

In an embodiment, a subscription list indicates which tenants have authorization to access which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is permitted access to a particular application only if the tenant ID of the tenant is included in the subscription list corresponding to the particular application.