In one embodiment, the virtual agent on a worker device 110 performs a masking and anonymization process on the set of features analyzed for each event to mask, anonymize, or scramble any sensitive or confidential information (e.g., social security number, private information) obtained for the worker device 110. The virtual agent provides the images of the video streams 204 and the set of events and features extracted for the events to the process mining system 130. In one instance, the images of the video stream may also be considered a feature for an event.

The process mining system 130 generates visual logs 208 from images of the video streams 204. In one embodiment, the process mining system 130 groups images by similarity into activity groups. Images in one activity group may have similar pixel data, and thus, may be images that have similar content. The process mining system 130 generates visual logs 208 by arranging activity groups of images in chronological order. The activity groups may also be sorted by worker device or by worker session. In the example shown in FIG. 2, a visual log 208A is generated from images obtained from worker device 110A. Similar visual logs can be generated from images obtained from worker devices 110B, 110C, 110D, and so on. In one embodiment, the process mining system 130 identifies activity groups based on the captured events by the virtual agent. In such an embodiment, an activity group may also refer to a logical group of events captured by the virtual agent that define a specific functionality or objective.

The process mining system 130 extracts text data from the images of the visual logs 208. The text data may include one or more text fields identified on an image, along with the spatial location of the text field in the image. Among other things, the text data may include the name of one or more applications running on a worker device 110 or data values that were present in these applications. The process mining system 130 determines an activity label for each activity group that indicates one or more activities performed on a worker device 110 when the images were captured. The activity label may refer to the name of an application on the worker device 110 or the type of task being processed on the worker device 110. For example, the activity label may be a name of an e-mail application, a word processing application, or an invoice processing application that was detected in the image. The activity label may be inferred from the extracted text data or pixel data of the image. In another embodiment, the process mining system 130 determines an activity label for each activity group that indicates one or more activities performed on a worker device 110 based on a logical group of events and features for those events (and/or additional metadata on the events) that were captured by the virtual agent as described in detail above. The activity group may also be associated with one or more images at or around the time (e.g., one minute before or after, two minutes before or after) the events for the activity group occurred.

In one embodiment, the process mining system 130 identifies and assigns a case identifier to the images obtained from the worker devices 110. The case identifier is a data value that can be used to identify a particular case. The case identifier may be referred to by the organization with a label consistent with naming conventions of the organization. For example, each instance of a mortgage application process may be unique to the particular transaction between the lender and an applicant for a house, and the lender may assign a case identifier value of “APP-0123456” that is referred to by the lender organization as an “Application ID.” For each activity group, the process mining system 130 determines a case identifier specific to a case being processed in the activity group. The case identifier may be inferred from the text data. In another embodiment, the process mining system 130 identifies case identifiers at an application level and across applications. The process mining system 130 may automatically identify images or screenshots obtained from worker devices 110 where the identified case identifier is manifested and extracts the case identifier against the events captured on an image.

The process mining system 130 generates event logs 212 from the visual logs. The event logs 212 transform information associated with the visual logs 208 to a structured form of text. In one embodiment, the format of the event logs 212 may be identical or be similar to the format of existing event logs, such that the event log could be applied to existing process mining techniques that use conventional event logs. This way, the process mining system 130 can be applied to existing process mining tools, while taking advantage of the information captured in the visual logs 204 that was previously not available for process mining. In one instance, the event logs 212 are generated for each case identifier. Specifically, the process mining system 130 may identify activity groups of images assigned to a particular case identifier, and record the timestamp of an activity group, the activity label for the activity group, and the worker account associated with the activity group in the event log 212.

The process mining system 130 performs process mining using the event logs 212. In the example shown in FIG. 2, the process mining system 130 generates a process map indicating the sequence in which workers access certain applications for a particular operation process. As shown in FIG. 2, for a total number of 5,000 cases, workers have either accessed a sequence of {App1, App2, App3, App4, App6} or a sequence of {App1, App2, App3, App5, App6} to complete the required tasks. In particular, 4,000 cases have been processed using App5, and 1,000 cases have been processed using App4. Responsive to determining that the use of App5 is a bottleneck in efficiency, the process mining system 130 may identify workers who have used App5 to process their cases, and re-train these workers to proceed with App4 instead to improve the overall efficiency of the operation process. However, this is one example, and it is appreciated that in other embodiments, the process maps can be generated using different perspectives. As an example, while application transition is one perspective illustrated in FIG. 2, other perspectives may include generating process maps to indicate flow of different tasks and activities, hand-offs between resources, and the like.

The worker device 110 is a computing device such as a smartphone with an operating system such as ANDROID® or APPLE® IOS®, a tablet computer, a laptop computer, a desktop computer, or any other type of network-enabled device. A typical worker device 110 includes the hardware and software needed to connect to the network 122 (e.g., via WiFi and/or 4G or other wireless telecommunication standards).

The worker device 110 allows a worker to perform tasks related to operation processes to the extent the worker is authorized to do so within the organization. The worker device 110 may include an operating system and various applications that run on the operating system that enable the workers to fulfill their tasks. For example, for a business organization, the worker device 110 may include browser applications that allows a worker in the accounting department to interact with the Internet, an invoice processing application that is used to process invoices, and the like. As another example, for a lender organization, the worker device 110 may include a mortgage processing application that mortgage officers and underwriters can use to process mortgage applications from customers.

A worker may each be associated with a unique virtual account that can be used to log into a worker device 110. In one instance, a worker may be associated with multiple virtual agents, where a virtual agent is on each worker device 110 the worker logs onto. The events captured from all the virtual agents will be stitched together based on the worker ID of the worker, and therefore, this allows the process mining system 130 to obtain a holistic view of the actions and activities performed by the worker irrespective of the worker devices 110 the worker logs onto. Thus, on one worker device 110, multiple workers can login and multiple virtual agents each associated with a different worker may capture events related to the session. In such an instance, the worker device 110 may generate and maintain a dedicated session for the account of the worker such that the worker can process tasks on the worker device 110. Thus, activities performed on the worker device 110 may be traced back to the corresponding worker through the login history of the virtual account. In another instance, the worker device 110 may generate and maintain a public session, in which multiple workers can process tasks on the worker device 110 without logging in through a virtual account. In yet another instance, it is not required to trace each activity back to the worker, and the worker name or worker account may be anonymized or replaced by a device name for the worker device 110.

The worker device 110 includes a software agent or virtual agent deployed by the process mining system 130 that captures video streams of a screen of the worker device 110 while the worker a processes one or more tasks related to an operation process on the worker device 110 as well as captured events from, for example, the operating system, applications, or interface devices as described in conjunction with FIG. 2 above. The software agent provides the video streams to the process mining system 130. Each image received by the agent management module 310 may be associated with a time stamp indicating the time the image was taken on a respective worker device 110. In one embodiment, the software agent captures images whenever the worker interacts with the mouse or keyboard (or any other input device), such as mouse clicks or keyboard presses. For example, the software agent may capture image every 5 seconds regardless of what happens on the worker device 110 as long as the worker device 110 is operating a virtual session. In another embodiment, the software agent captures an image responsive to detecting an event on the worker device 110. The event can include actions initiated by the worker, such as pressing the “tab” or “space” key on the keyboard, clicking the mouse, and the like. The software agent may be configured to append the event that triggered the capture (e.g., mouse-click, pressing “enter” key) to the image as metadata, and provide images appended with such metadata to the process mining system 130.

While the system 100 illustrated in FIG. 1 is described in conjunction with an embodiment in which the worker device 110 is a computing device with a software agent, it is also appreciated that in other embodiments, different configurations can be applied to obtain images of workers processing tasks related to an operation process. For example, in one embodiment, workers of an organization may process tasks offline on documents, and a software agent within or connected to an external camera may capture images of the documents while the worker is processing tasks related to the operation process. The images captured by the camera may be provided to the process mining system 130 as video streams.

The network 122 provides a communication infrastructure between the worker devices 110 and the process mining system 130. The network 122 is typically the Internet, but may be any network, including but not limited to a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile wired or wireless network, a private network, or a virtual private network.

Process Mining System

FIG. 3 is a high-level block diagram illustrating a detailed view of the process mining system 130, in accordance with an embodiment. The process mining system 130 is comprised of modules including an agent management module 310, a visual log generator 315, an activity analysis module 320, an event log generator 325, a process mining module 330, and a case identifier module 335. The process mining system 130 also includes a visual logs datastore 350, an image analysis data store 355, and an event logs data store 360. Some embodiments of the process mining system 130 have different modules than those described here. Similarly, the functions can be distributed among the modules in a different manner than is described here.

The agent management module 310 develops and maintains a software agent program that capture images from worker devices 110. Specifically, responsive to receiving a request, the agent management module 310 may distribute the software agent program or virtual agent program to worker devices 110 of the organization, such that the software agent can be downloaded and installed on each worker device 110. Alternatively, an individual associated with the process mining system 130 may install the software agent directly on each worker device 110 through, for example, a CD. Once installed, the software agent is configured to capture video streams of screens of worker devices 110 and provide the video streams to the agent management module 310. In one instance, the software agent obtains a video stream in the form of a continuous video that includes a plurality of video frame images, each taken at a predetermined interval. In another instance, the software agent obtains a video stream that includes a plurality of images each taken responsive to a key event such as the worker pressing the “enter” key or clicking on the mouse connected to the respective worker device 110. In another instance, the virtual agent is configured to collect information and data about activity occurring on the worker device 110 via applications or frameworks (e.g., accessibility frameworks) integrated with the virtual agent. The virtual agent is also configured to capture and record events (e.g., mouse clicks, keyboard entries) occurring on the worker device 110, extract features for each of the events, and/or mask and anonymize the features before the features are provided to the process mining system 130.

Moreover, the software agent may be configured to selectively obtain images according to a predetermined criterion provided by the organization or the process mining system 130. For example, the software agent may be configured to detect which applications are running on the worker device 110 and obtain images only when applications of interest to the organization are running on the worker device 110.

The agent management module 310 may develop multiple versions of the software agent based on the configuration of the worker devices 110 of the organization making the request. Specifically, different organizations may have worker devices 110 with different hardware configurations and operating systems. The agent management module 310 may develop versions of the software agent program for different worker device 110 configurations. Moreover, the agent management module 310 may continuously update the software agent by providing the worker devices 110 with software update patches or bug fixes.

The visual log generator 315 generates visual logs from images of the video streams received by the agent management module 310. The visual log generator 315 groups images by similarity into activity groups. In one embodiment, the images are grouped together based on the similarity between pixel data. For example, the visual log generator 315 may determine clusters of images that have less than a threshold difference in pixel data as one activity group. As another example, the visual log generator 315 may first generate an embedding for each image using a machine-learned model and determine clusters of images that have less than a threshold difference in the embeddings as one activity group. The visual log generator 315 arranges the activity group of images in chronological order to generate the visual logs.

FIG. 4 illustrates an example visual log 408 generated from a video stream of images from a worker device 110, in accordance with an embodiment. The visual log 408 illustrated in FIG. 4 includes images 432, 436, 440 that were obtained from a video stream from a particular worker session WA. While FIG. 4 illustrates three example images, in reality, there may be many more images obtained from the video stream for a worker session. Image 432 has a timestamp of 1/23/2020 12:34 pm, image 436 has a timestamp of 1/23/2020 12:36 pm, and image 440 has a timestamp of 1/23/2020 12:58 pm.

As shown in FIG. 4, images 432 and 436 display the same invoice processing application with similar pixel data, except that the worker has filled in additional data fields in image 436 relative to image 432. Thus, responsive to determining that the difference in pixel data of images 432 and 436 are less than a predetermined threshold, the visual log generator 315 may group images 432 and 436 as one activity group 450. Also shown in FIG. 4, image 440 displays a different travel invoice viewing application with significantly different pixel data than images 432 and 436. Thus, the visual log generator 315 may assign a new activity group 454 to image 440 alone.

In another embodiment, the visual log generator 315 may group images together responsive to determination that one or more events captured by the virtual agent can be grouped together as an activity group. Specifically, the visual log generator 315 can perform one or more activity discovery processes to identify different activities occurring for each of the worker devices 110. In one instance, the activity discovery process is performed using predictive analytics or machine-learned models. The activity discovery process can also receive feedback from human operators to correct any discrepancies or errors in the activity predictions, and incorporate the feedback into the activity discovery process. The images associated with the activity group may be images that were captured immediately after the events occurred or around the time the events occurred on the worker device 110.

In one embodiment, the visual log generator 315 upon detecting applications that are not related to the business or organization of the worker (e.g., screenshots related to worker's personal social networking account), may mask the entire image or selective regions covering the unrelated application, or may also optionally delete those images. The visual log generator 315 may receive indications from other modules of the process mining system 130 that a certain image is a whitelisted image and may discard this image from the visual logs. The list of whitelisted applications or websites may differ depending on the organization and depending on the operation process.

The activity analysis module 320 extracts text data from the images that can be used to infer activity labels and data values contained in the images. In one embodiment, the activity analysis module 320 extracts text data by applying a model to identify one or more text fields in the image and their relative locations in the image. In one instance, the image analysis module 332 applies optical character recognition (OCR) methods to identify the text data. In another instance, the activity analysis module 320 may manage an application programming interface, such as the Microsoft Windows Automation API, to communicate with the worker devices 110 and obtain the textual data.

The text fields can include information that can be used to infer the name of an application or an online website application. For example, a text field may include the name of a native application on the worker device 110. As another example, a text field may include the URL address of a website displayed in a browser application of the worker device 110. The text fields can also include one or more data values for the application being displayed in an image. The data values may be values for one or more data fields of an application. For example, a text field may include the name of the recipient an invoice is to be paid to. As another example, a text field may include the total amount of an invoice. The data values may be associated with respective labels that the application uses to refer to the data values.

FIG. 5A illustrates text data extracted from the example image 440 shown in FIG. 4, in accordance with an embodiment. In particular, image 440 shows an application for viewing travel-related invoices. As shown in FIG. 5A, the activity analysis module 320 performs OCR to detect text data in the image 440. The identified text data is indicated by dotted boxes in the image 440. Among other things, the text data includes a text field with the name of the application “ViewTravelInvoices.” The text data also includes a text with a label for the recipient of the invoice “TO,” and a corresponding text field including the data value for the recipient “Mr. X.” The text data also includes a text field with a label for the description of the travel invoice “Travel Details,” and a corresponding text field including the data values “Destination: Washington D.C., Hotel: Marriott D.C. 3 Nights.”

FIG. 5B illustrates another example of text data extracted from the example image 436 shown in FIG. 4, in accordance with an embodiment. In particular, image 436 shows an application internal to an organization for processing invoices. As shown in FIG. 5B, the text data includes a text field with the name of the application “ProcessYourInvoice.” The text data includes text fields with labels “Vendor,” “Invoice No.,” “Due Date,” “Discount” and corresponding text fields with respective data values “Paper Inc.,” “INV019485,” “2/3/2020,” and “NO.” With respect to the description of the items, the text data includes text fields with labels “Item,” “Description,” “Unit Cost,” “Quantity,” “Total,” and corresponding text fields with respective data values “Pens,” “Ballpoint-Black,” “$1.99,” and “$1,990.”

The activity analysis module 320 also determines an activity label for each activity group in the visual logs that indicates activities performed on the worker devices 110 when the images were captured. In one embodiment, the activity label is the name of the application on the worker device 110, and the activity analysis module 320 may determine the activity label from the extracted text data. For example, the activity analysis module 320 may search the extracted text fields and identify a text field that matches an application in a list of applications. The identified text field can be assigned as the activity for the image. For example, the activity label for the image 440 shown in FIG. 5A may be “ViewTravelInvoices,” and the activity label for the image 436 shown in FIG. 5B may be “ProcessYourInvoice.”

However, embodiments are not limited hereto, and in other embodiments, the activity label may be any other characterization of the images that indicates the activities performed in the images when the images were captured at varying levels of granularity. For example, the activity label may be the type of task being performed in the image. In such an example, the activity label for the image 440 shown in FIG. 5A may be “Viewing Travel Invoices,” and the activity label for the image 436 shown in FIG. 5B may be “Enter Data for Invoice.” As another example, the activity label for an image may be “Enter Registration Details.”

In another embodiment, the activity analysis module 320 determines an activity label for an image by identifying a similarity between the image and one or more images that have known activity labels. The known activity labels for the identified images that have a similarity above a threshold with the image are assigned as the activity label for the image. For example, the activity analysis module 320 may perform clustering or other unsupervised machine-learning techniques to identify a cluster of one or more images that have a similarity above a threshold with the image, and assign the known activity label for the cluster of images as the activity label for the image. The similarity may be determined as a metric that increases as a difference between the pixel data or a transformation of the pixel data between two images decreases, and vice versa.

The activity analysis module 320 stores the activity labels and data values in association with the images in the image analysis data store 355. For example, in the images shown in FIGS. 5B, the stored text data for each image includes the activity label for the image and data values of the image. In particular, the data values are formatted in brackets, in which the first element of the bracket is the label for the data value, and the second element is the data value. However, embodiments are not limited hereto, and the image analysis module 320 may store the extracted text data in any available format compatible with the process mining system 130.

In one embodiment, the activity analysis module 320 data may merge extracted text data from images in an activity group, such that a single set of text data is stored for images of the activity group. The activity analysis module 320 may extract text data from the image with the latest timestamp in the activity group and associate the text data for the image as the text data for the activity group, since the image with the latest timestamp is likely to be the most comprehensive within the activity group. For example, returning to the example illustrated in FIG. 4, activity group 450 may be associated with extracted text data from image 436, since image 436 has the latest timestamp in the group and thus may be assigned a single activity label. In another embodiment, the activity analysis module 320 may store extracted text data with respect to an individual image basis.

In another embodiment, the activity analysis module 320 may determine activity groups (which are logical groupings of captured events that device a specific functionality of objective) and the activity labels for those activity groups from the events captured by the virtual agents on one or more worker devices 110 and the set of features for the events associated with the events, even without the data extracted from the images of the visual log. The activity analysis module 320 identifies the ordering or sequence in which the events occurred (e.g., via timestamp collected for the events) to form the activity groups and labels. As an example, a first event may be captured when a worker enters the first name of a customer in an invoice processing application, a second event subsequent the first event may be captured when the worker enters the last name of the customer. The activity analysis module 320 may group the first event and the second event together as an activity group and infer an activity label “enter customer name John Smith” based on the features extracted for these events that include detailed information on what occurred during the worker session. In such an embodiment, the activity analysis module 320 can infer different activity groups and labels for those activity groups from the events captured by the virtual agents and the set of extracted features (e.g., application name, widget name, browser information) for those events as described above.

In one instance, the set of features extracted for events may or may not include image information from the visual logs, and/or text data extracted from the visual logs. In some embodiments, both image and non-image features may be available for the activity analysis module 320 to process. If the non-image features are not rich enough to infer the activity groups and/or labels, the activity analysis module 320 may obtain additional information from the screenshot images as features to infer the activity groups and/or labels. For example, the activity analysis module 320 may initially attempt to determine an activity group including one or more captured events based on the set of features extracted for the events. If the non-image features are not rich enough, the activity analysis module 320 may obtain the text data (e.g., via OCR) from the screenshots as described in conjunction with FIGS. 5A-5C and use the extracted data as image features to infer the activity groups and labels. As another example, the activity analysis module 320 may further determine the activity groups and labels based on both a combination of image and non-image features for the captured events.

The case identifier module 325 identifies and assigns a case identifier to the images obtained from worker devices 110. The case identifier is a data value that can be used to identify a particular instance of an operation process. The case identifier may vary across different operation processes and organizations, even if the instances are for the same entity or transaction. For example, for the same individual in an accident, the individual may be assigned a patient identifier value at a hospital, while the insurance claim for the individual may be assigned a claim identifier value at the insurance company. The assigned case identifiers may also be stored in the image analysis data store 355 in association with images in the visual logs 350.

The case identifier module 325 may determine the case identifier from the extracted text data. The case identifier may be referred to by the organization with a particular label, or in some cases, may be identified by a particular format. The case identifier module 325 may identify a case identifier for an image from the extracted text data once the label or format for the case identifier is known. Returning to the example in FIG. 5B, responsive to determining that the label for the case identifier for invoice processing operations is “Invoice No.,” the case identifier module 325 searches the stored text data for the image 436, and identifies the data value “INV019485” as the case identifier for the image. As another example, responsive to determining that the case identifier has a format of starting with “INV,” the case identifier module 325 searches the stored text data to identify a data value that has the specified format.

In some cases, images may be captured that do not include the case identifier in the screen. In one embodiment, the case identifier module 325 may determine case identifiers for such images by determining similarity between data values on these images to data values on images that have already been assigned a case identifier. For example, the case identifier module 325 may compare the data values for a subset of labels in the unassigned image to those in an assigned image, and determine the similarity as the number or proportion of data values in the assigned image that match those in the unassigned image. The similarity may be determined among images obtained from the same worker session, images obtained from the same worker, or images obtained across different workers and worker devices 110. The case identifier module 325 may identify an assigned image having above a threshold similarity with the unassigned image and assign the case identifier for that image to the unassigned image.

FIG. 5C illustrates an example of assigning a case identifier to an example image 438, in accordance with an embodiment. In particular, the example image 438 shown in FIG. 5C may be obtained after the worker for the example image 436 in FIG. 5B scrolls to the bottom portion of the application, and thus, may be associated with the same case identifier. The case identifier module 325 compares data values for the extracted labels in the image 438 to those in the previous image 436 that already has been assigned a case identifier of “INV019485.” The similarity between these two images may be significantly high since the data values in the image 438 for a subset of labels “Due Date,” “Discount,” “Item,” “Description,” “Unit Cost,” “Quantity,” “Total” match those of the image 436. Based on the determined similarities, the case identifier module 325 may assign the same case identifier “INV019485” to the image 438.

By assigning case identifiers to images or activity groups in the visual log, the process mining system 130 can identify when a worker has finished working on one case and started processing a different case in the operation process. Moreover, in many instances, handover of tasks for one particular case may occur from one worker to another. In such an instance, images from workers affiliated with the case can be collated to provide a more comprehensive view of the process flow of the case. For example, for a given case of an operation process, the process mining system 130 can identify the case identifier associated with the case. The process mining system 130 can identify images in the visual log assigned to the case identifier across different workers or different worker devices 110. The order in which the case was handled across workers or worker devices can be determined by, for example, analyzing the chronological order of the images based on time stamps. For example, a first subset of images may show worker A processing a particular case at a first time. A second subset of images may show worker B processing the particular case at some second time subsequent the first time. Based on this chronological order of images assigned to the particular case, the process mining system 130 may determine that a handover of this case occurred from worker A to worker B at the second time. In this manner, the process mining system 130 can determine whether a particular case was handled by different workers, and when handovers of the case occurred in the process.

FIG. 6 illustrates an example handover of tasks for an operation process, in accordance with an embodiment. As shown in FIG. 6, the process mining system 130 obtains images 436, 442, 444, 446, 448 in chronological order from worker session WA. Images 436, 442 are assigned case identifiers of “INV019485,” while images 444, 446, 448 are assigned case identifiers of “INV018393.” Also shown in FIG. 6, the process mining system 130 obtains images 450, 452, 454, 456, 458 in chronological order from worker session WB. Images 450, 452, 454 are assigned case identifiers of “INV016392,” while images 456, 458 are assigned the same case identifier of “INV018393” with images 436, 442 from worker session WA. Thus, the process mining system 130 is able to identify that a handover of tasks has occurred from worker session WA to worker session WB, and these images can be collated to provide an overall process flow of the case.

In one embodiment, the case identifier module 325 may obtain the label or format for the case identifier from individuals affiliated with the organization, such as individuals in managerial positions. In some embodiments, the case identifier module 325 it may be difficult to obtain the label or format for the case identifier directly from the organization. In such an instance, the case identifier module 325 may obtain records from the organization that contain information about previously processed instances of an operation process and determine the label for the case identifier from the records. The records may store one or more labels and data values corresponding to these labels for the previously processed instances.

FIG. 7 illustrates a record 760 of an organization including labels and data values for previously processed instances of a mortgage application process, in accordance with an embodiment. As shown in FIG. 7, the first row of the record contains a set of labels including “Name,” “Application No.,” “Processor,” and “Loan Amount.” Each row thereafter contains data values for each label for a particular instance of the operation process. For example, the second row corresponds to an instance for a particular mortgage applicant, and contains data values “John S.,” “APAP03827,” “Philipp N.,” and “$637,000” that respectively correspond to the set of labels. While the label “Application No.” and its corresponding data values are the case identifiers for the operation process, this information may not be known to the case identifier module 325 beforehand for various reasons.

In one embodiment, the case identifier module 325 identifies the label or format for the case identifier by computing the variance for a data value. The variance for a data value indicates the frequency in which different data values appear in the record. The case identifier module