404 is configured to retrieve one or more webpages from the Internet.

Browser/app 404 is configured to receive an input associated with a webpage. In some embodiments, the input is a query associated with a webpage. In some embodiments, the input is freeform text associated with a webpage.

Code associated with SDK client 406 is included in browser/app 404. SDK client 406 is configured to capture content associated with a webpage, process the content associated with the webpage into a specific format, and provide the processed content to cloud service 412. SDK client 406 includes functionality to interact with the annotated version of the web elements (e.g., the query response). SDK client 406 provides API(s) that enable actions, such as client, input, etc., to be performed. SDK client 406 is configured to provide error handling. An instruction step associated with a web automation solution may have an error handler. SDK client 406 is configured to cache a corresponding response for an instruction step for investigation and logging. In the event of an instruction execution failure not caused by web page changes, SDK client 406 is configured to continue and retry a script from a failed step without having to rerun prior steps. This ensures the scripting environment won't execute the same command or perform the same action repeatedly, especially for transaction-related tasks.

SDK client 406 is configured to determine, for a particular web element, a corresponding “role,” a corresponding “name,” and a corresponding “html_tag.” The “role” is a parameter that describes the role of the particular web element in an accessibility tree. The “name” is a parameter that represents the name of the web element as specified in the original webpage accessibility tree. The “html_tag” is a parameter that denotes the original html tag of the web element.

SDK client 406 is configured to request cloud service 412 to generate a response by providing to cloud service 412, via connection 410, the processed webpage content and the received input. Connection 410 may be a wired or wireless connection. Connection 410 may be the Internet, an intranet, a wireless area network, a personal area network, a wireless local area network, a virtual private network, etc.

Cloud service 412 includes inference pattern engine 414. For an initial iteration, associated with the webpage, inference pattern engine 414 utilizes the processed webpage content and the received query to generate a prompt for LLM 422. In some embodiments, LLM 422 is part of cloud service 412. In some embodiments, LLM 422 is a separate entity from cloud service 412. The notations for each element included in the processed webpage content help LLM 422 to determine the purpose of the elements. LLM 422 is trained to understand the semantics of web content. The prompt, the query, and the processed webpage content are provided to LLM 422 via connection 420. Connection 420 may be a wired or wireless connection. Connection 420 may be the Internet, an intranet, a wireless area network, a personal area network, a wireless local area network, a virtual private network, etc.

In response, LLM 422 generates a response. The LLM response is an annotated representation of web elements as specified in the query. The LLM response maps a variable included in the query to a corresponding webpage element included in the processed webpage content. This LLM response is designed to be user-friendly and easy to understand, in contrast to traditional HTML. It enhances the accessibility of web pages, allowing users to interact with the specified web elements as described in the LLM response. In addition to providing, for a particular web element, a corresponding “role,” a corresponding “name,” and a corresponding “html_tag,” the LLM response also includes a corresponding “identifier” for the particular web element. The identifier denotes a specified identifier for a given web element. Instead of using the specified identifier for a particular web element, a developer may utilize a variable included in the input that corresponds to the particular web element to generate the script to automate a task associated with the webpage. LLM 412 provides the LLM response to cloud service 412. Cloud service 412 is configured to provide the LLM response to client SDK 406. Client SDK 406 includes application programming interface(s) (APIs) that enables object-oriented programming interfaces to be generated based on the LLM response or inference pattern engine response. The APIs provide various functionality to interact with the web elements. The APIs are supported by one or more programming languages, such as Python, JavaScript, etc. Users associated with a client device may utilize the APIs to create web automation solutions for a wide range of everyday applications.

Inference pattern engine 414 derives a pattern from the LLM response to map the one or more variables included in the input to one or more corresponding elements associated with the webpage. In some embodiments, inference pattern engine 414 derives the pattern before providing the LLM response to client device 402. In some embodiments, inference pattern engine 414 provides the LLM response to client device 412 while deriving the pattern. In some embodiments, inference pattern engine 414 derives the pattern after providing the LLM response to client device 402.

In some embodiments, to derive the pattern, inference pattern engine 414 determines a plurality of beacon nodes associated with a tree data structure. The webpage has an associated tree data structure (e.g., document object model (DOM) tree). The tree data structure associated with a webpage may be a dynamic tree data structure. That is, each time the webpage is accessed, the corresponding tree data structure is different. However, a beacon node in a tree data structure is unique because it has a set of attributes (i.e., a fingerprint) that only maps to one element in the tree data structure, regardless of the tree data structure version. Examples of beacon nodes include a node that corresponds to a search box, a node that corresponds to a filter element on a left side of a webpage or application, a node that corresponds to a sponsored product element on a webpage or application.

Inference pattern engine 414 utilizes the paths associated with the plurality of beacon nodes (e.g., three beacon nodes) to map the one or more variables included in the input to one or more corresponding elements associated with the processed webpage content. A variable included in the input has a corresponding path in the tree data structure. Inference pattern engine 414 determines the corresponding paths from the plurality of beacon nodes to a target node corresponding to the variable included in the input (e.g., triangulation) and stores the determined paths in inference patterns store 432. For example, a variable included in the input may correspond to a login button (e.g., FIG. 1B). The corresponding paths from a plurality of beacons nodes to a node in the tree data structure corresponding to the login button (as indicated by the mapping generated by the LLM) is determined by inference pattern engine 414 and stored in inference patterns store 432 via connection 430. Connection 430 may be a wired or wireless connection. Connection 430 may be the Internet, an intranet, a wireless area network, a personal area network, a wireless local area network, a virtual private network, etc. In some embodiments, inference patterns store 432 is included in a storage device that is local to or remote from cloud service 412. The query response preserves the mapping of response nodes to their corresponding HTML elements via XPaths, DOM attributes and other distinctive patterns for identifying HTML elements within a webpage. Storing the inference patterns in the inference patterns store enables the cloud service to generate the query response for the same query and similar webpage without prompting LLM 422 to generate the same query response on CPU instances. This reduces latency and GPU costs associated with utilizing LLM 422 to generate the input response.

For one or more subsequent iterations (similar requests to the initial request), inference pattern engine 414 obtains a derived pattern stored in inference pattern store 432 and utilizes the stored derived pattern to generate an inference pattern engine response. Based on the stored derived pattern, inference pattern engine 414 maps the one or more variables included in the input to one or more corresponding elements associated with the webpage. Inference pattern engine 414 identifies the plurality of beacon nodes in the new version of the tree data structure. Inference pattern engine 414 identifies the plurality of beacon nodes in the new version of the tree data structure. Inference pattern engine 414 utilizes the corresponding stored paths from the plurality of beacon nodes to the one or more target nodes in the previous version of the tree data structure to predict the current paths from the plurality of beacon nodes in the new version of the tree data structure to the one or more target nodes in the new version of the tree data structure.

Inference pattern engine 414 provides metadata to cloud service 412. Examples of metadata include a post-processed HTML element tree, the query, the type of data that is to be extracted from the LLM, and/or configuration flags (e.g., which model to use for the query). Cloud service 412 generates a prompt based on the metadata, the input and the processed webpage content and provides the prompt, the input, and the processed webpage content to LLM 422. LLM 422 generates an LLM response based on the provided prompt, input, and processed webpage content. The LLM response maps one or more variables included in the input to one or more corresponding elements associated with the webpage. Inference pattern engine 414 receives the LLM response from LLM 422 and compares the LLM response to the inference pattern engine response. A correctness of the prediction is determined. In some embodiments, the prediction is correct, that is, inference pattern engine 414 correctly mapped all of the one or more variables included in the input to nodes in the new version of the tree data structure. The derived pattern from the previous iteration is maintained and an indication of the determined correctness is stored in inference patterns store 432.

In some embodiments, the prediction is partially correct, that is, inference pattern engine 414 correctly mapped some of the one or more variables included in the input to nodes in the new version of the tree data structure. In some embodiments, the prediction is greater than or equal to a confidence threshold (e.g., 95% accurate). In such embodiments, the derived pattern from the previous iteration is maintained and an indication of the determined correctness is stored in inference patterns store 432. In some embodiments, the prediction is less than a confidence threshold. In such embodiments, a new pattern is derived and stored in inference patterns store 432. An indication of the determined correctness may be stored with the new derived pattern.

In some embodiments, the prediction is incorrect, that is, inference pattern engine 414 incorrectly mapped all of the one or more variables included in the input to nodes in the new version of the tree data structure. In such embodiments, a new pattern is derived and stored in inference patterns store 432. An indication of the determined correctness may be stored with the new derived pattern.

After a plurality of iterations, inference pattern engine 414 is trained based on the plurality of LLM responses and can correctly map the one or more variables included in the input to one or more nodes in a new version of a tree data structure. Such a mapping has a confidence score that is greater than or equal to a confidence threshold. Instead of utilizing LLM 422 to generate the LLM response, inference pattern engine 414 obtains the derived pattern from a previous iteration stored in inference patterns store 432 and utilizes the obtained pattern to generate an inference pattern engine response that maps the one or more variables included in the input to one or more nodes in a new version of a tree data structure. Cloud service 412 provides the inference pattern engine response to client device 402 via browser/app 404.

As a result, the generated output is less likely to contain hallucinations since LLM 422 is not responsible for generating the response. Furthermore, the response time for client device 422 to receive an answer is significantly reduced (e.g., from seconds to milliseconds) because LLM 422 does not need to be prompted.

FIG. 4B is a block diagram illustrating a system to generate an adaptable script to automate a task associated with an application in accordance with some embodiments. In the example shown, system 450 includes a mobile device 452, a cloud service 412, a LLM 422, and an inference patterns store 432. Mobile device 452 may be a smart phone, a tablet, a handheld gaming device, a virtual reality headset, or any other portable computing device. In some embodiments, mobile device 452 is a client device, such as client device 402. Mobile device 452 includes one or more applications 454. Mobile device 452 is configured to receive an input from a user associated with mobile device 452. In some embodiments, the input is a query associated with an application. In some embodiments, the input is freeform text associated with an application.

The one or more applications 454, when executed by mobile device 452, have an associated UI that is viewable by a user associated with mobile device 452. The UI associated with the one or more applications have UI content, such as UI layout information and screen content, that is not easily accessible by the user associated with mobile device 452.

UI content retrieval service 456 is installed on mobile device 452 to enable the user associated with mobile device 452 to access the UI content associated with the one or more applications 454. UI content retrieval service 456 is configured to extract UI content from a UI associated with the one or more applications 454. In some embodiments, UI content retrieval service 456 is configured to extract UI content associated with an application that is running in the foreground of a display of mobile device 452. In some embodiments, UI content retrieval service 456 is configured to extract UI content associated with an application that is running in a background of the display of mobile device 452. UI content may include a UI layout, screen content, a screenshot, etc. In some embodiments, UI content retrieval service 456 is located on a separate device that communicates (wired or wirelessly) with client device 452. The wired connection may be a USB cable, lightning cable, or other type of mobile device cable. The wireless connection may be a Bluetooth connection, a Wi-Fi connection, an Airdrop connection, or other type of wireless connection.

Runtime agent 462 is configured to obtain the extracted UI content from UI content retrieval service 456 and process the obtained UI content into a consumable format (e.g., Javascript Object Notation (JSON), Extensible Markup Language (XML), screenshot, etc.). Runtime Agent 462 is configured to package the processed UI content with a user input and provide the packaged information as a request to cloud service 412. Runtime agent 462 is also configured to facilitate further communication with mobile device 452 (e.g., interacting with UI elements for automation purposes).

In some embodiments, runtime agent 462 is located on a device separate from mobile device 452, such as a client device. In some embodiments, runtime agent 462 is also installed on mobile device 452. In some embodiments, runtime agent 462 is installed on mobile device 452 as an application separate from UI content retrieval service 456. It is possible that in some embodiments, runtime agent 462 is installed on mobile device 452 in a same application as UI content retrieval service 456. However, it is desired to deploy UI content retrieval service 456 and runtime agent 462 across a plurality of devices in a uniform manner to reduce the amount of time and resources associated with debugging an error in UI content retrieval service 456 and/or runtime agent 462. For example, a standalone version of UI content retrieval service 456 and a version of UI content retrieval service 456 packaged with runtime agent 462 may be deployed. However, in the event there is a bug with UI content retrieval service 456, more time and resources are needed to debug both versions of UI content retrieval service 456 when compared to debugging either the standalone version of UI content retrieval service 456 or the version of UI content retrieval service 456 packaged with runtime agent 462.

In response to receiving the input and the packaged information, cloud service 412 includes inference pattern engine 414. For an initial iteration associated with application 454, cloud service 412 utilizes the packaged information to generate a prompt for LLM 422. In some embodiments, LLM 422 is part of cloud service 412. In some embodiments, LLM 422 is a separate entity from cloud service 412.

The notations for each element included in the processed content help LLM 422 to determine the purpose of the elements. LLM 422 is trained to understand the semantics of UI content. The prompt, the input, and the processed UI content are provided to LLM 422 via connection 420. Connection 420 may be a wired or wireless connection. Connection 420 may be the Internet, an intranet, a wireless area network, a personal area network, a wireless local area network, a virtual private network, etc.

In response, LLM 422 generates a response and provide the LLM response to inference pattern engine 414. The LLM response is an annotated representation of application elements as specified in the input. The LLM response maps a variable included in the input to a corresponding UI element. The LLM response enhances the accessibility of application UIs, allowing users to interact with the specified elements as described in the LLM response. In addition to providing, for a particular element, a corresponding “role,” a corresponding “name,” and a corresponding “html_tag,” the LLM response also includes a corresponding “identifier” and a corresponding “bounds” for the particular UI element. The identifier denotes a specified identifier for a given element. Instead of using the specified identifier for a particular UI element, a developer may utilize a variable included in the input that corresponds to the particular UI element to generate the script to automate a task associated with the application. The “bounds” value indicates a position or location of the particular element on a UI associated with the application.

Inference pattern engine 414 derives a pattern from the LLM response to map the one or more variables included in the input to one or more corresponding elements associated with application 454. In some embodiments, inference pattern engine 414 derives the pattern before providing the LLM response to runtime agent 462. In some embodiments, inference pattern engine 414 provides the LLM response to runtime agent 462 while deriving the pattern. In some embodiments, inference pattern engine 414 derives the pattern after providing the LLM response to runtime agent 462.

In some embodiments, to derive the pattern, inference pattern engine 414 determines a plurality of beacon nodes associated with a tree data structure. The application UI has an associated tree data structure. The tree data structure associated with the UI may be a dynamic tree data structure. That is, each time the application is accessed, the corresponding tree data structure is different. However, a beacon node in a tree data structure is unique because it has a set of attributes (i.e., a fingerprint) that only maps to one element in the tree data structure, regardless of the tree data structure version. Examples of beacon nodes include a node that corresponds to a search box, a node that corresponds to a filter element on a left side of a webpage or application, a node that corresponds to a sponsored product element on a webpage or application.

Inference pattern engine 414 utilizes the paths associated with the plurality of beacon nodes (e.g., three beacon nodes) to map the one or more variables included in the input to one or more corresponding elements associated with the processed application content. A variable included in the input has a corresponding path in the tree data structure. Inference pattern engine 414 determines the corresponding paths from the plurality of beacon nodes to a node corresponding to the variable included in the input (e.g., triangulation) and stores the determined paths in inference patterns store 432. For example, a variable included in the input may correspond to a login button (e.g., FIG. 1B). The corresponding paths from a plurality of beacons nodes to a node in the tree data structure corresponding to the login button (as indicated by the mapping generated by the LLM) is determined by inference pattern engine 414 and stored in inference patterns store 432 via connection 430. Connection 430 may be a wired or wireless connection. Connection 430 may be the Internet, an intranet, a wireless area network, a personal area network, a wireless local area network, a virtual private network, etc. In some embodiments, inference patterns store 432 is included in a storage device that is local to or remote from cloud service 412.

Cloud service 412 is configured to provide the LLM response to runtime agent 462, which uses the information in the LLM response to locate certain UI elements on a screen associated with an application 454 running on mobile device 452 and to interact with them to perform automation actions. Given the response from cloud service 412, runtime agent 462 is configured to facilitate interaction with UI elements by sending corresponding commands to mobile device 452 through a wired or wireless connection.

For one or more subsequent iterations (similar requests to the initial request), inference pattern engine 414 obtains a derived pattern stored in in inference pattern store 432 and utilizes the stored derived pattern to generate an inference pattern engine response. Based on the stored derived pattern, inference pattern engine 414 maps the one or more variables included in the input to one or more corresponding elements associated with the application UI. Inference pattern engine 414 identifies the plurality of beacon nodes in the new version of the tree data structure. Inference pattern engine 414 identifies the plurality of beacon nodes in the new version of the tree data structure. Inference pattern engine 414 utilizes the corresponding stored paths from the plurality of beacon nodes to the one or more target nodes in the previous version of the tree data structure to predict the current paths from the plurality of beacon nodes in the new version of the tree data structure to the one or more target nodes in the new version of the tree data structure.

Inference pattern engine 414 generates metadata. Examples of metadata include a post-processed UI element tree, the query, the type of data that is to be extracted from the LLM, and/or configuration flags (e.g., which model to use for the query). Cloud service 412 generates a prompt based on the metadata, the input, and the processed application content and provides the prompt, the input, and the processed application content to LLM 422. LLM 422 generates a response based on the provided prompt, input, and processed application content. The LLM response maps one or more variables included in the input to one or more corresponding elements associated with the application UI. Inference pattern engine 414 receives the LLM response from LLM 422 and compares the LLM response to the inference pattern engine response. A correctness of the prediction is determined. In some embodiments, the prediction is correct, that is, inference pattern engine 414 correctly mapped all of the one or more variables included in the input to nodes in the new version of the tree data structure. The derived pattern from the previous iteration is maintained and an indication of the determined correctness is stored in inference patterns store 432.

In some embodiments, the prediction is partially correct, that is, inference pattern engine 414 correctly mapped some of the one or more variables included in the input to nodes in the new version of the tree data structure. In some embodiments, the prediction is greater than or equal to a confidence threshold (e.g., 95% accurate). In such embodiments, the derived pattern from the previous iteration is maintained and an indication of the determined correctness is stored in inference patterns store 432. In some embodiments, the prediction is less than a confidence threshold. In such embodiments, a new pattern is derived and stored in inference patterns store 432. An indication of the determined correctness may be stored with the new derived pattern.

In some embodiments, the prediction is incorrect, that is, inference pattern engine 414 incorrectly mapped all of the one or more variables included in the input to nodes in the new version of the tree data structure. In such embodiments, a new pattern is derived and stored in inference patterns store 432. An indication of the determined correctness may be stored with the new derived pattern.

After a plurality of iterations, inference pattern engine 414 is trained based on the plurality of LLM responses and can correctly map the one or more variables included in the input to one or more nodes in a new version of a tree data structure. Such a mapping has a confidence score that is greater than or equal to a confidence threshold. Instead of utilizing LLM 422 to generate the LLM response, inference pattern engine 414 obtains the derived pattern from a previous iteration stored in inference patterns store 432 and utilizes the obtained pattern to generate an inference pattern engine response that maps the one or more variables included in the input to one or more nodes in a new version of a tree data structure. Inference pattern engine 414 provides the inference pattern engine response to runtime agent 462, which uses the information in the inference pattern engine response to locate certain UI elements on a screen associated with an application 454 running on mobile device 452 and to interact with them to perform automation actions.

As a result, the generated output is less likely to contain hallucinations since LLM 422 is not responsible for generating the response. Furthermore, the response time for client device 422 to receive an answer is significantly reduced (e.g., from seconds to milliseconds) because LLM 422 does not need to be prompted.

FIG. 5A is a flow diagram illustrating a process to generate an adaptable script to automate a task associated with a webpage in accordance with some embodiments. In the example shown, process 500 may be implemented by a client SDK, such as client SDK 406.

At 502, an input is received. In some embodiments, the input is a query. The query is a structured request, formulated in natural language, for specific web elements from a webpage. The query serves as a representation to extract precise information from the webpage. The query is structured in a manner that signifies a relationship between a component and the webpage. The query is comprised of one or more variables that correspond to one or more specific web elements associated with a webpage. The query is designed to be versatile across different types of websites.

In some embodiments, the input is a freeform input.

At 504, webpage content is processed. The processed webpage is a human-friendly representation of the HMTL associated with the webpage, with notations for each element. For a particular web element, the processed webpage content indicates a “role,” a “name,” and an “html_tag.” The “role” is a parameter that describes the role of the particular web element in an accessibility tree. The “name” is a parameter that represents the name of the web element as specified in the original webpage accessibility tree. The “html_tag” is a parameter that denotes the original html tag of the web element.

At 506, the input and the processed webpage content are provided to a cloud service.

At 508, a response is received from the cloud service. In some embodiments, the response is an LLM response. In some embodiments, the response is an inference pattern engine response. The response is a structured representation of specified web element nodes. The response maps a variable included in the input to a corresponding webpage element included in the processed webpage content.

At 510, an automated task is generated utilizing the response. Code associated with the automated task is generated utilizing the variables included in the input. For example, an automated task may include booking a flight on a travel website, purchasing a product on an e-commerce website, scheduling an appointment at a medical facility, etc. The variables “login_btn,” “search_box,” and “search_btn” from FIG. 1A may be utilized instead of “tf_22,” “APjFqB,” and “tf_194,” respectively.

In the event an identifier associated with a web element changes due to an update in the web page, an automated task may not function properly because a variable included in the script is not correctly mapped to the correct web element. Steps 504-508 may be repeated to enable the LLM to determine a new mapping between the updated web element identifier and the variable included in the query. For example, the response may map the variable “login_btn” to a web element having an identifier of “identifier_1.” The web page may be updated such that the web element having the identifier of “identifier_1” now has an identifier of “identifier_2.” Process 500 may be repeated to enable the LLM to update the mapping such that the variable “login_btn” is mapped to the web element having the identifier of “identifier 2.”

In some embodiments, steps 504-508 are periodically performed (e.g., daily, weekly, monthly, etc.). In some embodiments, steps 504-508 are performed in response to a user command. In some embodiments, steps 504-508 are performed as a background process.

Instead of having a developer debug the script line-by-line to determine which web element identifier has changed, implementing the LLM significantly reduces the time and resources needed to debug a nonfunctional script or to generate a new script. The LLM can effortlessly map the one or more variables included in the query to the one or more web elements included in the processed webpage content since the LLM is trained to understand the semantics of web content.

FIG. 5B is a flow diagram illustrating a process to generate an adaptable script to automate a task associated with an application in accordance with some embodiments. In the example shown, process 550 may be implemented by a runtime agent, such as runtime agent 462.

At 552, an input is received. In some embodiments, the input is a query. The query is a structured request, formulated in natural language, for specific web elements from an application. The query serves as a representation to extract precise information from the application. The query is structured in a manner that signifies a relationship between a component and the application. The query is comprised of one or more variables that correspond to one or more specific UI elements associated with the application. The query is designed to be versatile across different types of applications.

In some embodiments, the input is a freeform input.

At 554, UI content associated with an application running on a mobile device is obtained. The application has an associated UI that is viewable by a user associated with the mobile device. The UI has associated content that may not be easily accessible by the user associated with the mobile device. A UI content retrieval service is installed on the mobile device to obtain the UI content associated with the application. The UI content may include a UI layout, screen content, a screenshot, etc. The UI content retrieval service provides the obtained screen content to a runtime agent.

At 556, the obtained UI content is processed into a consumable format. The runtime agent processes the obtained UI content into a consumable format, such as JSON, XML, a screenshot, etc. In some embodiments, the consumable format is pre-defined. In some embodiments, the obtained screen content is processed into a consumable format based on a type of task that is to be automated.

At 558, the input and the processed screen content is provided to a cloud service.

At 560, a response is received from the cloud service. In some embodiments, the response is an LLM response. In some embodiments, the response is an inference pattern engine response. The response is a structured representation of specified UI element nodes. The response maps a variable included in the input to a corresponding UI element included in the processed user interface content.

At 562, a script for an automated task is generated utilizing the response. Code associated with the automated task is generated utilizing the variables included in the input. For example, an automated task may include finding the cheapest ride between multiple ride-sharing apps, accepting requests on a social media platform from users that meet certain criteria, purchasing an item from an e-commerce platform when it is below a certain price, etc.). The variables “Login,” “Tap to search,” and “Search” from FIG. 2B may be utilized instead of “32,” “34,” and “35,” respectively.

FIG. 6 is a flow diagram illustrating a process to provide a response in accordance with some embodiments. In the example shown, process 600 may be implemented by a cloud service, such as cloud service 412. Parts of process 600 are performed by an inference pattern engine included in the cloud service, such as inference pattern engine 414.

At 602, an input, and processed webpage or application content are received.

At 604, it is determined whether a confidence threshold has been reached for the webpage or application. A confidence threshold has been reached when the inference pattern engine is capable of mapping correctly greater than or equal to a confidence threshold one or more variables in the input to one or more corresponding elements in the processed webpage or application content.

In response to a determination that a confidence threshold has not been reached for the webpage or application, process 600 proceeds to 606. In response to a determination that the confidence threshold has been reached for the webpage or application, process 600 proceeds to 628.

At 606, it is determined whether a derived pattern associated with the webpage or application is stored. In response to a determination that a derived pattern associated with the webpage or application is not stored (e.g., this is the initial request for a mapping associated with the webpage or application), process 600 proceeds to 608. In response to a determination that a derived pattern associated with the webpage or application is stored (e.g., this is a subsequent request for a mapping associated with the webpage or application), process 600 proceeds to 616.

At 608, a prompt based on the input and processed webpage content or application is generated. The prompt, the input, and the processed webpage content or application are provided to the LLM. An example of a generated prompt is:

- You are an expert in understanding the structure of the web page. You will be given a simplified Web Page accessibility tree (created following Aria spec) and a GraphQL-like query that is supposed to query various web page elements. Provide a hypothetical response to such a query in a GraphQL-like response format. Return only response in the following format:
  - {
    - {response}
- }

At 610, a response is received from the LLM. The LLM response is a structured representation of specified element nodes. The LLM response maps a variable included in the input to a corresponding element included in the processed webpage content or processed UI content. For each element, the query response may indicate a “role,” a “name,” an “id,” and/or an “html_tag.”