134 (e.g., including some or all of the item verification data and other item data that pertains to the selected item 150) to the mobile device 102 and/or the shopping cart 104, from the item data store 120.

In some implementations, retrieving and providing item verification data can be based at least in part on shopping cart specifications of a shopping cart that is paired with a mobile device that requested the data. For example, the item data request can include, along with the obtained item identifier of the selected item 150, an identifier of the shopping cart 104 that is paired with the mobile device 102, and/or an identifier of the device 102. Upon receiving the identifier of the shopping cart 104 and/or the identifier of the device 102, for example, the server system 106 can access the cart data store 110 and retrieve (and optionally, cache for later use) shopping cart specifications that correspond to the shopping cart 104. As another option, the mobile device 102 can provide the server system 106 with the shopping cart specifications that correspond to the paired shopping cart 104 as part of the item data request. After receiving the shopping cart specifications for the shopping cart 104 (e.g., including data that indicates each type of sensor, including location sensor(s) and other sensors that may exist on the cart), for example, the server system 106 can retrieve and provide item verification data that corresponds to the selected item 150 and to the types of item verification sensors that exist on the cart 104. By retrieving and providing verification data that pertains to a particular cart, for example, an amount of data transmitted over the network(s) can be reduced, along with data transmission times.

In some implementations, retrieving and providing item verification data (and optionally, other item data) for an item can be performed without the item having been scanned. For example, the mobile device 102 can be associated with a user account that has a historical purchase history maintained by the server system 106. When the mobile device 102 and the shopping cart 104 are paired, for example, the server system 106 can retrieve and provide item verification data (and optionally, other item data) that pertains to a subset of the items 150 that have been frequently purchased by the user, based on the historical purchase history associated with the user's account. As another example, as the mobile device 102 and the shopping cart 104 move throughout the retail environment 108, location information of the device and/or cart (e.g., based on wireless signal strength data, Global Positioning System (GPS) data, overhead light-based data, or other suitable data) can be provided to the server system 106. In response to receiving the device and/or cart location information, for example, the server system 106 can retrieve and provide item verification data (and optionally, other information) that pertains to a subset of the items 150 that are in proximity to the device and/or cart (e.g., based on item location information maintained in the item data store 120). As another example, the server system 106 can provide a compact data structure that includes appropriate item verification data (and optionally other information) that pertains to all of the items 150 that are available in the retail environment 108 to the mobile device 102 and/or shopping cart 104 when the device and cart are paired.

By using the server system 106 to retrieve and provide item verification data (and optionally, other item data) as the data is used (and/or, in anticipation of the data being used), data updates can be centrally managed. For example, the retail environment 108 may include tens of thousands of different types of items 150, and the relevant verification data and other product information for such items can change over time. When changes occur, for example, the item data store 120 and/or the cart data store 110 can be updated, and the mobile device 102 and/or the shopping cart 104 can receive current data for particular items upon request. Thus, the data storage requirements of both the mobile device 102 and the shopping cart 104 can be reduced, along with an amount of data transmitted over the network(s).

Once the item data 134 has been retrieved and provided by the server system 106, for example, the mobile device 102 and/or the shopping cart 104 can receive the data 134, and can use the data 134 to verify whether the item 150 scanned by the mobile device 102 is actually added to the cart 104, according to location sensors of the device 102 and/or cart, and according to other item verification sensors of the cart 104. In general, item verification data of a particular type can be used by item verification sensors of a corresponding type to verify an item. For example, location verification data (e.g., data that specifies one or more locations of the item according to a planogram and/or heat map) can be used to verify the item, in view of a location of the mobile device 102 and/or the shopping cart 104 when the item scan occurred and/or when the item was placed in the cart 104. As another example, infrared (IR) verification data (e.g., data that defines a size/shape of an item, as detected by IR sensors when the item enters the cart 104) can be used to verify the item 150 according to the shopping cart's IR sensors. Example techniques for verifying an item according to IR sensor data are described in U.S. Application No. 63/276,386, which is herein incorporated by reference. As another example, sound verification data (e.g., data that defines a sound that the item produces when placed in the cart 104, as detected by sound sensors) can be used to verify the item 150 according to the cart's sound sensors (e.g., one or more microphones). Example techniques for verifying an item according to sound sensor data are described in U.S. Application No. 63/276,462, which is herein incorporated by reference. As another example, weight verification data (e.g., data that defines a weight bump that the item generates when placed in the cart 104, as detected by weight sensors) can be used to verify the item 150 according to the cart's weight sensors (e.g., one or more scales). Example techniques for verifying an item according to weight sensor data are described in U.S. Application No. 63/276,474, which is herein incorporated by reference. Other types of item verification data and corresponding sensors are possible, and the various sensor types can be used singly or in combination with other sensors (e.g., in combination with item location data) to verify scanned items. Item verification can include determining whether item verification data for a scanned item matches data collected by the corresponding sensors, as described in further detail in examples below.

As described in examples throughout this document, item location data can be used to verify an item. For example, the item data 134 can include item location data 124 (e.g., including one or more item locations 126 according to a planogram and/or a heat map of past item scans by other mobile device users) for the item 150. As items are scanned by multiple different users in the retail environment 108, potentially through the use of multiple different applications (e.g., customers using a shopping application, employees using an inventory application, etc.), location data for each item scan can be determined (e.g., based on wireless signal strength data, Global Positioning System (GPS) data, or other suitable location data provided through use of a scanning device) and can be stored (e.g., by the item data store 120) for use in aggregating and providing the item location data 124. In general, location data for an item that has been collected within a certain period of time (e.g., the past day, the past week, the past month, or another suitable period of time) can be used for aggregating and providing item location data. If an item becomes available at a new location, for example, the new location can eventually be recognized as a valid location for the item, as the item is scanned at the new location by multiple different users over time. Item verification can include determining whether a location of a current scan of an item matches an expected item location 126 for the item according to its item location data 124 (e.g., the current scan location is within a threshold distance of five feet, 10 feet, 20 feet, or another suitable threshold distance of the expected item location). When the location of the current item scan matches the expected item location 126 according to its item location data 124, for example, the scanned item can be verified with a higher level of confidence, whereas the scanned item may be unverified (or verified with a lower level of confidence) when the location of the current item scan does not match the expected item location. By considering the location of a current item scan relative to an expected item location according to its item location data when verifying an item, for example, deceptive practices (e.g., switching a high-cost item with a low-cost item having a similar physical profile) can be potentially be detected and prevented.

In some implementations, once an item has been successfully or unsuccessfully verified, a user can be notified through output generated by the mobile device 102 and/or the shopping cart 104. In the present example, the mobile device 102 (shown here as device 102b) can add the selected item (e.g., one of items 150) to a virtual shopping cart maintained by the device 102 and/or servers 106, which can be presented to the user at interface 160. As another example, the shopping cart 104 can provide an indication (e.g., visual, sonic, and/or tactile) of successful and/or unsuccessful item verification. Additional user notifications and interfaces are described in further detail in examples below.

In some implementations, a user may not be notified of an item being successfully or unsuccessfully verified. For example, item verification results can be maintained in association with a virtual shopping cart (e.g., with a scanned item being added to the cart, along with its verification status), without providing the item verification results to the user. By maintaining item verification results in the background in association with the user's virtual shopping cart, for example, a relatively uninterrupted shopping experience can be facilitated, while tracking the verification statuses of items that have been added to the cart. The verification statuses of each item in the virtual shopping cart can be aggregated, possibly with one or more other factors (e.g., a user's purchase history, a user's reputation score, etc.) to determine an overall verification status of the shopping cart 104. A high user reputation score, for example, can indicate a high level of virtual cart accuracy for the user for previous shopping trips (e.g., a virtual cart being shown to accurately represent the contents of a physical cart during checkout), whereas low reputation scores can indicate a low level of virtual cart accuracy for the user for previous shopping trips (e.g., a virtual cart being shown to not accurately represent the contents of a physical cart during checkout). The overall verification status of the shopping cart, for example, can be used as a factor in directing the user to an appropriate station during a checkout process.

In some implementations, item verification results for an item in a virtual shopping cart can be adjusted after the item has been added to the virtual shopping cart, based on item location data that has been collected from item scans performed by other users. For example, when item 150 is initially scanned, the item location data 124 may not include item location 126 as an expected location of the item. If the item 150 is scanned at an unexpected location, for example, the scanned item may be initially associated with a low level of confidence. However, during the customer's shopping trip, additional scans of the item 150 may be performed over time by other shoppers, in proximity to item location 126. In the present example, item location 126 can eventually become recognized as an expected location for item 150 (e.g., when a threshold number of scans have occurred at the location), and thus the item verification result for item 150 in the customer's virtual shopping cart can be adjusted, such that the scanned item is associated with a high level of confidence. Item verification results can be periodically adjusted (e.g., once a minute, once per five minutes, or at another suitable time interval) and/or in response to the customer indicating (e.g., through the shopping application running on the mobile device 102) that the customer intends to end a current shopping trip.

During stage (C), a shopping cart may potentially be authorized for automated checkout. For example, the system 100 can perform operation 136, in which a determination of whether the user of the mobile device 102 (shown here as device 102c) and the shopping cart 104 (shown here as cart 104c) is authorized to proceed to an automated checkout station 170, or is to proceed to a manual checkout station 172. The automated checkout station 170, for example, can include RFID scanners, weight sensors, and/or high resolution cameras that can be used to determine whether the contents of the shopping cart 104, as tallied by the mobile device 102, correspond to what is being detected by the station 170. For example, the station 170 can measure the weight of the cart 104 and its contents, and determine whether the measured weight is within a threshold of the expected weight of the cart 170 and its contents, according to the mobile device tally. As another example, the station can use its RFID scanners to verify items with RFID tags that automated device sensors (e.g., IR scanners) are unable to verify with confidence. As another example, the station can use its cameras to capture one or more images of the cart 104 and its contents. If the cart 104 is validated across one or more of these signals, the customer may be automatically charged for the tallied contents and permitted to simply depart the retail environment 108. The manual checkout station 172, for example, can be a Point of Sale (POS) terminal at which items are removed from the shopping cart 104, an employee of the retail environment 108 individually scans the items to tally a total price, and the user provides a payment before departing the environment 108.

In general, determining whether a shopping cart is authorized for automated checkout can include determining a cart accuracy score for a virtual shopping cart, from verification results associated with items that have been placed in the shopping cart. Determining the cart accuracy score used for cart authorization, for example, can be based on a quantity of items in the cart (e.g., an absolute quantity, a percentage quantity, etc.) that have been verified, and/or an aggregated confidence score (e.g., averaged and/or weighted) for items in the cart, as described in further examples below. In some implementations, one or more additional authorization factors (e.g., a user's purchase history, a user's reputation score, etc.) can be considered when determining whether a shopping cart is authorized for automated checkout. For example, if a user is a regular customer of a store, and/or is a user of one or more additional services provided by the store (e.g., credit services, loyalty services, etc.), and/or has a high reputation score, a lower cart accuracy score may qualify the user for an automated checkout—whereas if the user is not a regular customer, and/or is not a user of additional services, and/or has a low reputation score, a higher cart accuracy may qualify the user for the automated checkout. In the present example, the shopping cart 104 is authorized for automated checkout (e.g., the virtual cart's accuracy score meets a threshold value), and the user is notified through output generated by the mobile device 102, which can be presented to the user at interface 162. At stage (D₁), for example, the user can proceed with the mobile device 102 and the shopping cart 104 to the automated checkout station 170 to perform an automated checkout operation 138. However, if the shopping cart 104 were not to be authorized for automated checkout (e.g., the virtual cart's accuracy score does not meet a threshold value), the user would be notified through output generated by the mobile device 102, and at stage (D₂), for example, the user could proceed with the mobile device 102 and the shopping cart 104 to the manual checkout station 172 to perform a manual checkout operation 140.

In some implementations, an additional item check may be performed after an automated checkout operation. For example, after the automated checkout operation 138 is performed on the shopping cart 104 (at stage (D₁)), the user can be presented with further instructions (e.g., through output presented by the mobile device 102, through an indicator on the shopping cart 104, and/or through output presented by the automated checkout station 170) for completing the shopping trip. If the automated checkout station 170 confirms that the user's virtual shopping cart accurately represents the contents of the physical shopping cart 104, for example, the user's transaction can be finalized and the user can be permitted to depart the retail environment 108. If the automated checkout station 170 is unable to confirm that the user's virtual shopping cart accurately represents the contents of the physical shopping cart 104 within a threshold level of certainty, for example, an employee of the retail environment 108 can be directed to perform an additional item verification (e.g., by scanning one or more random items, high value items, and/or specifically unverified items in the shopping cart 104). If the shopping cart 104 passes the additional item verification, for example, the user's transaction can be finalized and the user can be permitted to depart the retail environment 108. If, however, the shopping cart 104 does not pass the additional item verification, for example, the user can be directed to the manual checkout station 172 to perform the manual checkout operation 140. Similarly, if the automated checkout station 170 determines that the user's virtual shopping cart does not accurately represent the contents of the physical shopping cart 104 (e.g., one or more items were detected by the automated checkout station 170 as being in the shopping cart 104 that are not in the user's virtual shopping cart), the user can be directed to the manual checkout station 172 to perform the manual checkout operation 140.

In some implementations, upon completion of a shopping trip, a user's transaction may be automatically finalized and the user may be permitted to depart without first proceeding to an automated checkout station or a manual checkout station. For example, if a virtual cart's accuracy score meets a threshold value and one or more additional authorization factors pertain to the user (e.g., the user's reputation score meets a threshold value and/or the user subscribes to a loyalty program and/or uses credit services of the store), the user can be provided with a notification (e.g., on the mobile device 102 and/or the shopping cart 104) to confirm the transaction. After the user's transaction for the shopping trip is finalized, for example, the user can be permitted to simply depart the store. Example techniques for determining whether a shopping cart is authorized for automated checkout and verifying items by an automated checkout system are described in U.S. Application No. 63/276,471, which is herein incorporated by reference.

FIG. 2 depicts an example system 200 for verifying items placed in physical shopping carts. The system 200, for example, can include mobile device 102 (shown in FIG. 1), shopping cart 104 (also shown in FIG. 1), and server system 106 (also shown in FIG. 1), in communication over network(s) 250. The network(s) 250, for example, can include one or more communication networks, including a LAN (local area network), a WAN (wide area network), and/or the Internet. In addition to or instead of communicating over network(s) 250, in some implementations, devices, carts, and/or servers may communicate directly using an appropriate communications protocol (WiFi, BLUETOOTH low energy (BLE), near-field communication (NFC), etc.). For example, the mobile device 102 can perform a wireless pairing operation (e.g., WiFi Direct pairing, BLUETOOTH low energy (BLE) pairing, near-field communication (NFC) pairing) with the a cart computing system 210 of the shopping cart 104 to establish an association with the cart 104.

As shown in FIG. 2, the cart 104 can be a smart cart. In some implementations, the cart 104 can be a preexisting shopping cart in the retail environment 108 that has been modified/updated to include one or more components used for performing the item verification techniques described herein. Since the components used for item verification may be low cost, modifying existing shopping carts with the components can be simple and inexpensive relative to producing new carts.

The shopping cart 104 can include optional mounting fixture(s) 230 (e.g., on or near a cart handle), optional indicator(s) 240a-b (e.g., on or near the cart handle and/or around a top perimeter of the cart), the cart computing system 210, motion detection device(s) 220, and location determination device(s) 260. The mounting fixture(s) 230 can have a housing or other configuration that can removably receive the mobile device 102 (e.g., smart phone, digital assistant, tablet, etc.) of a user. The housing or other configuration can be sized to fit mobile devices of varying sizes. The mounting fixture(s) 230 can also be located at one or more other regions of the cart 104 to facilitate ease of access, attachment, and removal of the mobile device 102 from the cart 104.

The cart computing system 210, as described further in reference to FIG. 3, can include processor(s), memory, a power source, and a communication interface. In addition, the cart computing system 210 can be in communication (e.g., wired, wireless) with the mobile device 102, and can include and/or be in communication with the motion detection device(s) 220, the location determination device(s) 260, and/or the indicator(s) 240a-b. In some implementations, in addition to providing power to the cart computing system 210, the power source of the cart computing system 210 can provide power to one or more additional components of the shopping cart 104, including but not limited to the motion detection device(s) 220, the location determination device(s) 260, and/or the indicator(s) 240a-b. In some implementations, the power source can also provide power to the mobile device 102 (e.g., to charge the device 102) while the device 102 is mounted at the mounting fixture(s) 230.

The motion detection device(s) 220, for example, can include one or more sensors along the top perimeter of the shopping cart 104 (e.g., directed toward an interior of the cart), and can be configured to detect motion indicative of an item (e.g., item 150) being placed in or removed from the cart 104. In some implementations, the motion detection device(s) 220 of the shopping cart 104 and/or the cart computing system 220 can include one or more optical sensors (e.g., infrared (IR) sensors) that are sufficiently spaced to detect and track vertical and rotational motion of the item 150 as it passes through a horizontal plane defined by the top perimeter of the cart 104. The optical sensors, for example, can be configured to translate movement and velocity of items entering or leaving the cart 104, in a z-dimension.

The location determination device(s) 260, for example, can include one or more devices that are affixed to the shopping cart 104 and/or the cart computing system 210. To determine a position of the cart 104 within the retail environment 108, for example, the location determination device(s) 260 can be configured to collect and analyze various sorts of data in various implementations, such as wireless signal strength data (e.g., in environments in which wireless access points are distributed throughout the environment), overhead light data (e.g., in environments in which light-based positioning systems are in place), Global Positioning System (GPS) data, or other suitable data. As another example, the location determination device(s) 260 can be configured to collect data for determining a cart location (e.g., wireless signal strength data from multiple different access points in the environment 108, light data from overhead lights in the environment 108, and/or GPS data), and can provide the data to the cart computing system 210, which can determine the location of the cart 104 within the retail environment 108, based on the received data. In some implementations, a location of a shopping cart in a retail environment can be determined in response to a location determination request. For example, when an item is scanned by the mobile device 102, the device 102 can transmit a request to the cart computing system 210 to determine a location of the shopping cart 104 at a time of the scan. As another example, when the motion detection device(s) 220 detects that an item has entered or exited the cart, the cart computing system 210 can determine a location of the shopping cart 104 at a time of the entering/exiting. In some implementations, a location of a shopping cart in a retail environment can be continually determined. For example, as the shopping cart 104 moves through the retail environment 108, the location determination device(s) 260 can continually collect data to determine the cart's location, and the cart computing system 210 can determine (or can provide data for determining) the cart's path through the environment 108.

The indicator(s) 240a-b, for example, can include one or more output devices (e.g., indicator lights, speakers, tactile devices, etc.) on or near the handle of the shopping cart 104 and/or around a top perimeter of the cart (e.g., a ring of upward-facing indicator lights). In general, the indicator(s) 240a-b can be configured to convey verification status information that pertains to an addition of an item to the cart 104, and/or an overall state of the cart. In some implementations, the indicator(s) 240a-b can be configured to provide verification status information to a user of the shopping cart 104 instead of, or in addition to, verification status information provided by the mobile device 102. For example, the indicator(s) 240a-b can include a first indicator that indicates a successful item verification (e.g., a green light, a chime tone, etc.), and can include a second indicator that indicates an unsuccessful item verification (e.g., a red light, a buzzer tone, etc.). In some implementations, the indicator(s) 240a-b can be configured to provide information that indicates an overall state of the shopping cart 104. For example, the indicator(s) 240a-b can include different indicators (e.g., different colored lights, different light intensities, and/or different flashing light patterns) that indicate whether the cart 104 is active, is inactive, is malfunctioning, is waiting for an item to be added, has detected that an item has been added (e.g., either with or without an indication of whether the item has been successfully or unsuccessfully verified), and/or other suitable cart states. As another example, the indicator(s) 240a-b can include different indicators for directing a user during a checkout process. For example, the indicator(s) 240a-b can include a first indicator (e.g., a green light) that indicates that a transaction is complete and that a customer may leave the store, a second indicator (e.g., a yellow light) that indicates that a user is to proceed to a station where a partial check of the cart contents is performed (e.g., by an employee), and a third indicator (e.g., a red light) that indicates that a user is to proceed to a manual checkout station where the entire cart contents are determined and the transaction is completed. In some implementations, the indicator(s) 240a-b can be an LED light bar and/or an LED light strip. One or more other light configurations can be used for the indicator(s) 240a-b.

FIG. 3 depicts an example system 300 for verifying items placed in physical shopping carts. The system 300, for example, can include cart computing system 210 (also shown in FIG. 2), mobile device 102 (also shown in FIG. 1 and FIG. 2), and server system 106 (also shown in FIG. 1 and FIG. 2), in communication over one or more network(s) 250. The cart computing system 210, for example, can be part of a shopping cart, such as the shopping cart 104 (shown in FIG. 1 and FIG. 2). In some implementations, the cart computing system 210 can include one or more components that are separate from a shopping cart and/or in communication with one or more components of the shopping cart (e.g., a cloud-based system and/or service). In such a scenario, the cart computing system 210 can, for example, perform operations for multiple shopping carts. The cart computing system 210 can include processor(s) 310, memory 312, a power source 314, a communication interface 316, motion detection device(s) 220 (also shown in FIG. 2), location determination device(s) 260 (also shown in FIG. 2), and optionally, additional item verification hardware 320. The processor(s) 310 can be configured to execute one or more of the processes/techniques described herein.

The cart computing system 210 can be configured to activate the location determination device(s) 260 (and optionally, the additional item verification hardware 320) in response to identification of the item by the mobile device 102 (e.g., through a scan of the item, a captured image of the item, etc.) and/or when an item is detected as entering the shopping cart 104. For example, the motion detection device(s) 220 can provide a notification of when the item enters the shopping cart to the cart computing system 210, which can then activate the location determination device(s) 260 and optionally, one or more sensors 322a-c of the item verification hardware 320. The motion detection device(s) 220, for example, can include one or more sensors attached to the shopping cart 104, to collect information about an item as it passes through a top horizontal plane of the cart and enters the cart. For example, the motion detection device(s) 220 can include IR emitters, IR sensors, and IR optical flow sensors. The IR emitters and IR sensors can be alternating in a single line along a top perimeter of the shopping cart 104, for example, while the IR optical flow sensors can be interspersed along the single line at the top perimeter of the shopping cart 104 at predetermined distances/intervals. The IR emitters can be activated to generate light across the top horizontal plane of the shopping cart 104 as the item passes through the plane and into the cart. The generated light can be detected by the IR sensors as IR data, which can be used to determine when an item enters and/or exits the shopping cart 104, for example, and/or to generate a shadow map of the item.

In some implementations, data from the motion detection device(s) 220 can be used to determine a velocity of an item as it enters a shopping cart. As the item moves through the top horizontal plane and into the shopping cart 104, for example, the IR optical flow sensors can detect rotational and/or motion data of the item, which can be used to determine velocity of the item as it moves into the cart. Because of variable distances between the IR optical flow sensors and the item, a relative translation rate or other data received from the IR optical flow sensors may not directly translate into velocity of the item. In some implementations, trigonometric functions can be used to determine relative item and IR optical flow sensor positioning over time. For example, a curve can be generated that indicates a position of the item over time as the item passes through the top horizontal plane of the shopping cart 104. The curve, for example, can also be a map of time values corresponding to a constant change in item position, for example. A velocity of the item can be identified using the curve, for example.

The optional item verification hardware 320, for example, can include one or more sensors (e.g., sensors 322a-c) that can be used to detect an item that is placed in (or removed from) the shopping cart 104, to measure various physical properties of the detected item, and to provide item data based on the measurements. For example, each of the sensor(s) 322a-c can be a different type of sensor that measures a different physical property, such as one or more IR sensors (e.g., sensors of the motion detection device(s) 220 and/or similar sensors) for measuring a size/shape of an item, one or more sound sensors (e.g., microphones) for measuring a sound produced by the item, one or more weight sensors (e.g., scales) for measuring a weight bump produced by the item, one or more cameras for visually recognizing the item, or other suitable sensors. The item verification hardware 320, for example, can include a single type of sensor, or can include a combination of multiple different sensor types. In some implementations, data from the motion detection device(s) 220 can be used to perform item verification. For example, sensors of the motion detection device(s) 220 can be sensors of the item verification hardware 320 (e.g., any of the sensors 322a-c). For example, data from the motion detection device(s) 220 (e.g., IR sensor data) can be used to reconstruct an item in multidimensional (e.g., 3D) space to determine a size/shape of the item.

The mobile device 102, for example, can be a smart phone, digital assistant, tablet, or other sort of mobile computing device. Functionality described herein can be presented in a mobile application or other software suite presented at the mobile device 102. The mobile device 102 can be used to scan items that a customer desires to purchase, and can be used by the customer to apply offers, promotions, and/or coupons to their transaction. A virtual shopping cart can be accessible at the mobile device 102 and can include various items that the customer scans with the mobile device 102 and puts in their shopping cart 104. In some implementations, the virtual shopping cart can also include virtual products that the customer selects from an online shopping application presented at the mobile device 102.

The mobile device 102 can include processor(s) 330, memory 332, a power source 334, a communication interface 336, input device(s) 340, output device(s) 342, and location determination device(s) 344. The processor(s) 330 can be configured to execute one or more of the processes/techniques described herein. The input device(s) 340 can include one or more of a touch screen display, keyboard, mouse, microphone, and/or imaging sensors, and can receive user input. For example, a user can scan an identifier on packaging of an item using a camera (e.g., one of the input device(s) 340) of the mobile device 102. As another example, the user can use a Radio Frequency Identification (RFID) scanner (e.g., another of the input device(s) 340) of the mobile device 102 to scan an RFID tag of the item. The user can also provide input indicating an item quantity on a touch screen display of the mobile device 102, for example. The output device(s) 342 can include display screens, speakers, and/or other types of devices for delivering content/information to a user. For example, the output device(s) 342 can present information in graphical user interface (GUI) displays at the mobile device 102. The output device(s) 342, for example, can be an LCD touch screen that can present an application or software suite that allows a user to scan items, add items to their virtual shopping cart, access account information, review transactions, and/or complete transactions. The output device(s) 342 can also provide notifications, messages, and/or alerts to the user indicating whether an item is verified. The output device(s) 342 can provide one or more other forms of output to the user, as described herein.

Similar to the location determination device(s) 260, for example, the location determination device(s) 344 can be configured to collect data for determining a location of the mobile device 102 in the retail environment 108 (e.g., wireless signal strength data from multiple different access points in the environment, light data from overhead lights in the environment, and/or GPS data), and the device 102 can determine its location based on the collected data. In some implementations, a location of a mobile device in a retail environment can be determined in response to a location determination request. For example, when the motion detection device(s) 220 detects that an item has entered or exited the cart, the cart computing system 210 can transmit a request to the mobile device 102 to determine a location of the device 102 at a time of the entering/exiting. As another example, when an item is scanned by the mobile device 102, the device 102 can determine its location at a time of the scan. In some implementations, a location of a mobile device in a retail environment can be continually determined. For example, as the mobile device 102 moves through the retail environment 108, the location determination device(s) 344 can continually collect data to determine the mobile device's location, and the mobile device can determine (or can provide data for determining) the device's path through the environment 108.

The server system 106 can include one or more computing servers (e.g., application servers, cloud servers, data servers, etc.). For example, the server system 106 can communicate with, and can store, update, and retrieve data in the cart data store 110 and the item data store 120. The cart data store 110, for example, can maintain information associated with various shopping carts in the retail environment 108 (e.g., cart identifiers, current pairings with mobile devices, cart specifications, etc.). The item data store 120, for example, can maintain information associated with various items in the retail environment 108 (e.g., item identifiers, verification data associated with the items, item prices, item descriptions, item weights, item images, item locations, available item quantities, etc.).

FIGS. 4A-B comprise a flow diagram of an example technique 400 for verifying items placed in physical shopping carts. The example technique 400, for example, includes coordinated operations of a mobile device (e.g., mobile device 102, also shown in FIGS. 1-3) and a cart computing system (e.g., cart computing system 210, also shown in FIG. 2 and FIG. 3). In the present example, some operations of the example technique 400 are described with reference to an item verification engine 402, which can be executed by the mobile device 102 or the cart computing system 210. Optionally, the item verification engine 402 may be executed by the server system 106, with the mobile device 102 forwarding sensor data from the mobile device and/or the cart computing system 210 to the server system 106, and with the server system directly obtaining item verification data from the item data store 120 and not providing the item verification data to the mobile device 102. In general, the example technique 400 is not a linear process, in that operations may be triggered in response to item scans performed by the mobile device 102 and/or by sensor readings by the cart computing system 210. Also, since item verification data may be retrieved by the server system 106 and provided to the mobile device 102 and/or cart computing system 210 in response to an item scan, for use in verifying an item placed in the shopping cart 104, the item verification data may or may not be available to the verification engine 402 before location data (and optionally, other sensor data) is available. For example, a user may scan an item using the mobile device 102, then place the item in the shopping cart 104 before or after the verification data for the item happens to arrive from the server system 106. To provide a smoother shopping experience for the user, for example, various data caching techniques and timer activation techniques can be employed to handle data transmission variances, such that the user is not inconvenienced by a strictly linear process (e.g., a process in which an item is to be placed in a cart only when item verification data is available).

In a regular scenario, the mobile device 102 is used to perform an item scan at 404. Referring to FIG. 1, for example, a user of the mobile device 102 can select an item 150 from the storage unit 152, and can scan the item 150 with the device 102. Optionally, the device 102 can provide a user interface control through which a user can specify a quantity of items to be added to the shopping cart 104. After performing the item scan (and optionally receiving an item quantity specified by the user), for example, the mobile device 102 can provide a scan notification (at 406) to the item verification engine 402 and to the cart computing system 210. For implementations in which a mobile device location is used for verifying an item, for example, the scan notification (406) can include a location of the mobile device 102 when the item scan occurred. At 408, the item verification engine 402 (e.g., executed by the cart computing system 210 or the mobile device 102) can receive the scan notification. In response to receiving the scan notification, for example, at 410, the item verification engine 402 optionally activates a verification timer (e.g., 5 seconds, 10 seconds, 20 seconds, or another suitable period of time) during which the engine 402 waits for item verification data, the location data, and optional sensor data to be received. Further, in response to receiving the scan notification from the mobile device 102, for example, at 412 the cart computing system can optionally activate its various other sensors (e.g., the item verification hardware 320, shown in FIG. 3), and can activate a sensor timer (e.g., 5 seconds, 10 seconds, 20 seconds, or another suitable period of time) during which the cart computing system 210 waits for sensor data to be received, as a result of the item 150 being placed in the shopping cart 104. For implementations in which the cart computing system 210 executes the item verification engine 402, for example, the verification timer and the sensor timer can be the same timer. For implementations in which the mobile device location is used for verifying the item and other sensor data is not used for verifying the item, for example, the sensor timer may not be used. For implementations in which the shopping cart location is used for verifying the item, for example, the cart computing system 210 can determine a current location of the shopping cart 104 in response to receiving the scan notification.

According to the regular scenario described above, verification/sensor timers are activated in response to an item scan by a mobile device. However, in an alternate scenario, a user may place an item in a shopping cart without scanning the item, or after performing an unsuccessful scan. To account for such scenarios, for example, the cart computing system 210 can detect an item entering a shopping cart at 422. In response to the cart computing system 210 detecting the item, for example, the cart computing system 210 can optionally activate its various sensors and its sensor timer at 412, and the item verification engine 402 can activate its verification timer at 410. In the present example, to detect the item 150 entering the shopping cart 104, cart computing system 210 can receive data from motion detection device(s) 220 (shown in FIG. 2 and FIG. 3). With both the verification timer and the sensor timer being activated (or a single combined timer in the case of the cart computing system 210 executing the item verification engine), for example, potential problems or uncommon scenarios can be appropriately handled.

By activating a shopping cart's item verification hardware (e.g., including one or more types of sensors) in response to an item scan notification provided by a mobile device and/or when an item is detected as entering the cart, battery power consumed by the item verification hardware can be conserved. For example, some sensors may consume a little power (or no power) while in an inactive mode, and a greater amount of power while actively performing a sensing operation. Further, by activating one or more sensors immediately prior to a time when the sensor(s) are likely to be used and by activating a limited sensor timer, for example, data collected by the sensors can be limited to a time period corresponding to the sensor. Thus, a relatively small amount of sensor data may be cached at any time while potentially waiting for item verification data to be available.

Returning now to the regular scenario, for example, while the verification timer and the sensor timer (or the single combined timer) are activated, the mobile device 102 can obtain item verification data at 414. For example, the mobile device 102 can transmit, to the server system 106, an item data request that includes the item identifier of the item 150, determined from the scan of the item 150. In response to receiving the item data request, for example, the server system 106 can access the item data store 120 (shown in FIG. 1), can retrieve item verification data that includes location data for the item (e.g., one or more known locations of the item 150 in the retail environment 108 according to a planogram and/or heat map), can optionally retrieve other item verification data (e.g., verification data that pertains to optional sensors of the shopping cart 104) that corresponds to the item identifier in the item data request, and can provide the item verification data (and optionally, other item data, such as item price, description, weight, image, etc.) to the mobile device 102. As another example, the mobile device 102 can access cached item verification data (and optionally, other item data) that was received from the server system 106 before the item 150 was scanned. At 416, the mobile device 102 can provide the item verification data received from the server system 106 to the item verification engine 402, and the item verification engine 402 can receive the item verification data at 418. Optionally, at 420, the mobile device 102 can prompt the user to place the item 150 in the cart 104. For example, the mobile device 102 can provide a visual and/or audible message to the user through a mobile application running on the device 102.

While the item verification hardware 320 of the cart computing system 210 is activated and the sensor timer is running, for example, the user can place item 150 in the shopping cart 104. At 424, the motion detection device(s) 220 (shown in FIG. 3), and optionally, one or more sensors of the item verification hardware 320 can sense the item 150 being placed in the shopping cart 104 by the user. In response to the item 150 being sensed, for example, the cart computing system 210 can provide a notification to the item verification 402 that the item was sensed (at 428). For implementations in which a shopping cart location is used for verifying an item, for example, the item sensed notification (428) can include a location of the shopping cart 104 when the item was sensed by the cart and/or when the cart computing system 210 received the scan notification. Optionally, the notification that the item was sensed can include additional sensor data associated with the item 150, and the cart computing system 210 can deactivate the sensors of the item verification hardware 320 (at 432). At 430, for example, the item verification engine 402 can receive the notification that the item was sensed (with optional cart location data and/or other sensor data from the cart computing system 210). For implementations in which a mobile device location is used for verifying an item, for example, a request to determine a device location can optionally be provided by the item verification engine 402 to the mobile device 102, in response to receiving the notification that the item was sensed. After receiving the notification that the item was sensed, for example, the item verification engine 402 may proceed to verify the item 150 (as shown in FIG. 4B).

In an alternate scenario, the sensor timer may expire (at 426), before an item is sensed. For example, the user may scan an item and not place the item in the shopping cart 104. As another example, the item may be detected as beginning to enter the shopping cart 104 (e.g., the motion detection device(s) 220 may be triggered), but the user may not actually place the item in the cart 104. As another example, one or more sensor devices 322a-c of the item verification hardware 320 may malfunction. In such scenarios, for example, the cart computing system 210 can provide a notification (at 434) to the item verification engine 402 that an item was not sensed, and can deactivate the sensors of the item verification hardware 320 (at 432). At 436, for example, the item verification engine 402 can receive the notification that the item was not sensed, and can provide the notification to the mobile device 102. At 438, for example, the mobile device 102 can optionally present an appropriate message to the user based on the notification and/or a current scenario. For example, if an item was detected as entering the shopping cart 104 and the item was not scanned and not sensed, instructions can be provided to the user to scan the item and place the item in the cart. As another example, if the item was scanned and the item was not sensed, a message can be provided to the user to place the item in the shopping cart 104.

Referring now to FIG. 4B, for example, the example technique 400 for verifying items placed in physical shopping carts can continue. In the regular scenario, the item verification engine 402 has received item verification data (at 418), and has received a notification that an item has been sensed (at 430). In the present example, the item verification engine 402 can determine (at 452) that the item has been sensed and that item verification data has been received, can proceed to perform verification of the item (at 456), and can verify that a count of items as indicated by the motion detection device(s) 220 and optional other sensor data corresponds to a quantity of items specified as being placed in the cart by a user, to determine verification results (e.g., a successful verification of the item 150 based on the item's verification data and the received item location data and optional sensor data, or an unsuccessful verification). (Item verification is described in further detail below with reference to FIG. 5.) At 458, for example, the item verification engine 402 can provide verification results to the mobile device 102, and optionally, to the cart computing system 210. At 460, for example, the mobile device 102 can receive the verification results and can optionally present the verification results to the user. For example, the mobile device 102 can present the verification results through a mobile application running on the device 102. At 462, the mobile device 102 can update a virtual cart status. For example, the mobile device 102 can add data representing a scanned and verified item (e.g., an item identifier, an item price, an item description, an item weight, an item image, an item quantity, an item verification confidence value, and/or other appropriate item data) to a virtual cart maintained by a mobile application running on the device 102. As another example, the mobile device 102 can add data representing a scanned and unverified item to the virtual shopping cart (e.g., along with data that indicates that the item is unverified, such as the item's verification confidence value being under a threshold value for the item). At 464, for example, the cart computing system 210 can optionally present verification results to the user. For example, cart computing system 210 can cause the indicator(s) 240a-b to provide information indicative of the verification results (e.g., green light for successful item verification, red light for unsuccessful verification, or another suitable visual, audible, or tactile indicator).

In an alternate scenario, the verification timer may expire (at 454), before item verification data is received. For example, the user may scan an item and place the item in the shopping cart 104, however the verification engine 402 does not receive item verification data for the scanned item before the verification timer expires. In such a scenario, the item verification engine 402, for example, can provide a notification (at 466) that the item is not verified to the mobile device 102. At 468, for example, the mobile device 102 can optionally present an appropriate message to the user based on the notification and/or current scenario. For example, the mobile device 102 can present, through a mobile application running on the mobile device 102, a message that the system was unable to verify the item and/or that potential network issues exist. At 462, the mobile device 102 can update a virtual cart status. For example, the mobile device 102 can add data representing the scanned item to a virtual cart maintained by a mobile application running on the device 102.

Although the present scenarios are primarily directed to the verification of items being added to a physical shopping cart and the updating of a corresponding virtual shopping cart, in some implementations, users may have the option of indicating items to be removed from the physical shopping cart. For example, a user can select an item from a virtual cart presented by a mobile application running on the mobile device 102, and indicate through the application that the item is to be removed from the physical cart. The motion detection device(s) 220 and/or other cart sensors can provide sensor data that results from removal of the item from the physical cart, for example, and the item verification engine 402 can verify whether the item has been actually been removed. After an item has been removed from the physical cart, for example, the item can also be removed from the virtual cart. If an item removal is not successfully verified, for example, a virtual cart's overall accuracy score may be decreased, whereas a successfully verified item removal may have a neutral impact on the accuracy score (other than removing the item's individual verification confidence value from an overall accuracy score for the cart).

FIG. 5 depicts an example process flow 500 for determining a verification status of an item, and presenting a notification to a user based on the verification status. Some operations of the example process flow 500 (e.g., optionally adjusting physical sensor data based on motion data, comparing location sensor data and optional physical sensor data with item verification data, and determining whether an item is verified) may be performed by the item verification engine 402 (shown in FIGS. 4A-B), which may be executed by the mobile device 102 or the cart computing system 210 (or optionally, the server system 106). Some operations of the example process flow 500 (e.g., providing notifications/prompts to users, tracking unverified items, managing a virtual cart, and managing an automated checkout option) may be performed by the mobile device 102, or a combination of the mobile device 102 and the server system 106 (e.g., with a cloud-based mobile application running on the mobile device 102).

At 502, for each type of physical sensor that may be included on the shopping cart 104 (e.g., image sensors, sound sensors, weight sensors), item verification data may optionally be adjusted, based on motion data from the motion detection device(s) 202. For example, after receiving item verification data for a scanned item, and receiving sensor data from one or more sensors 322a-c (shown in FIG. 3), the verification engine 402 can use the motion data to adjust the sensor data from each of the one or more sensors 322a-c. The motion data, for example, can indicate a velocity of the item 150 as it enters the shopping cart 104, whereas the item verification data can be associated with a baseline velocity (e.g., an average velocity of the item 150 being added to carts when pre-generating the item verification data for the item). If the velocity of the item 150 being added to the shopping cart 104 is greater than or less than the baseline velocity associated with its item verification data, for example, the verification engine 402 can proportionally adjust the item verification data for the item 150 and the sensor type, according to the increase or decrease in velocity. For item verification data for a sound sensor, for example, the item verification engine 142 can increase the amplitude of a sound sample in the item verification data when the item's velocity is greater than the baseline velocity, or can decrease the amplitude of the sound sample when the item's velocity is less than the baseline velocity. As another example, for item verification for a weight sensor, the item verification engine 142 can increase an expected weight bump in the item verification data when the item's velocity is greater than the baseline velocity, or can decrease the expected weight bump when the item's velocity is less than the baseline velocity. Alternately, the item verification engine 142 can adjust the sensor data for each type of sensor, based on the motion data—for example, decreasing values of the sensor data when item's velocity is greater than the baseline velocity, or increasing values of the sensor data when the item's velocity is less than the baseline velocity.

At 504, for a location sensor and for each optional physical sensor, the received (and optionally adjusted) sensor data can be compared with the received (and optionally adjusted) item verification data for that sensor. For example, the item verification engine 402 can compare data from one or more location sensors (e.g., location determination device(s) 260 of the cart computing system 210 and/or location determination device(s) 344 of the mobile device 102, shown in FIG. 3) with the location verification data for the item 150 (e.g., comparing a determined location in the retail environment 108 at which the item 150 was scanned by the mobile device 102 and/or was added to the shopping cart 104, with known location(s) of the item 150 in the retail environment 108 according to a planogram and/or heat map). As another example, the item verification engine 402 can compare the IR sensor data associated with item 150 being placed in the shopping cart 104, with the IR verification data for the item (e.g., comparing a sensed shadow profile with an expected shadow profile for the item). As another example, the item verification engine 402 can compare the sound sensor data associated with item 150 being placed in the shopping cart 104, with the sound verification data for the item (e.g., comparing a sound sample captured by a cart's microphone(s) with an expected sound sample for the item). As another example, the item verification engine 404 can compare the weight sensor data associated with item 150 being placed in the shopping cart 104, with the weight verification data for the item (e.g., comparing a weight bump detected by a cart's scale(s) with an expected weight bump for the item). The item verification engine 402, for example, may compare sensor data with item verification data for a single sensor type, or may compare data for two or more sensor types and combine the comparison results.

In some implementations, location data can be determined for a mobile device and for a shopping cart, in response to an item detection event (e.g., an item being scanned by the mobile device, or the item being sensed by the shopping cart). For example, data from location sensor(s) of the mobile device 102 and data from location sensor(s) of the cart computing system 210 can be provided to the item verification engine 402 (shown in FIGS. 4A-B) in response to the item 150 being scanned by the mobile device 102, in response to the item 150 being sensed by the cart computing system 210, or in response to both events. In some implementations, location data can be determined for a mobile device in response to an item scanned event, and location data can be determined for a shopping cart in response to an item sensed event. For example, when item 150 is scanned by the mobile device 102, data from locations sensor(s) of the mobile device can be provided to the item verification engine 402. When item 150 is later sensed by the cart computing system 210 when it is added to the shopping cart 104, for example, data from location sensor(s) of the shopping cart can be provided to the item verification engine 402.

At 506, the item verification engine 402 can determine whether there is a threshold match, for the location sensor(s) and optionally, for each other type of sensor, between the received (and optionally adjusted) sensor data and the received (and optionally adjusted) item verification data for that sensor. For example, the item verification engine 402 can determine whether a mobile device location at which a scan of the item 150 was performed and/or a shopping cart location at which the item entered the cart is within a threshold distance of the item's determined location according to its item location data. As another example, the item verification engine 402 can determine whether a sensed shadow profile for the item 150 matches the expected shadow profile for the item within a threshold margin (e.g., a sufficiently similar size and shape). As another example, the item verification engine 402 can determine whether a sound sample captured by a cart's microphone(s) matches the expected sound sample for the item within a threshold margin (e.g., a sufficiently similar wave form). As another example, the item verification engine 402 can determine whether a weight bump detected by a cart's scale(s) matches the expected weight bump for the item within a threshold margin (e.g., a sensed weight bump value within a threshold range of the expected weight bump). The item verification engine 402, example, may determine whether a threshold match exists for the data from the location sensor(s), or may determine whether threshold matches exist for the location data and optional data from one or more additional sensor types, and combine the determination results. For example, when data from location sensor(s) and data from other sensor types are used to verify an item, the verification results as determined by each individual verification technique can be aggregated, to generate a composite verification result. In some implementations, a verification result can include a confidence score (e.g., a percentage value) that indicates a likelihood of an item that has been scanned being the same item that has been placed in the cart. A high verification confidence score, for example, can indicate that that a scanned item was likely placed in the cart, whereas a low verification confidence score can indicate that an item that is different from the scanned item may have been placed in the cart. In addition, the item verification engine 402, for example, can determine whether a count of items as indicated by the data from the motion detection device(s) 220 and other sensor data corresponds to a quantity of items specified by a user as being placed in the cart (e.g., by performing multiple matches of sensor data and item verification data).

Referring now to FIG. 6A, an example map 600 is shown that represents item planogram locations and aggregated item location data over time. The map 600, for example, can represent the retail environment 108, including various fixtures 602a-n on which items can be placed, and from which shoppers can select the items during shopping trips. As multiple different shoppers use their mobile devices to scan an item (e.g., item 150) and place the item in their shopping carts, for example, location data from the mobile devices and/or shopping carts can be aggregated to generate a heat map for the item. Item location data can be maintained and aggregated over a predetermined period of time (e.g., thirty days, sixty days, one hundred and eighty days, or another suitable time period), and can be collected through the use of multiple different applications (e.g., customers using a shopping application, employees using an inventory application, etc.).

In the present example, location 610 is a planogram location of item 150 in the retail environment 108 (e.g., a specified location of the item according to a store plan), and thus is an expected location for mobile devices to scan the item and/or for shopping carts to sense an addition of the item. In some implementations, aggregated location data for an item can be used to identify one or more non-planogram locations for the item, which can be designated as expected locations for mobile device scans for the item and/or for shopping cart additions of the item. For example, when a sufficient number of detection events (e.g., tens or hundreds of item scans and/or cart additions) occur for the item in a non-planogram location, the location can be designated as an expected location for the item. In the present example, location 620 is a non-planogram location that is designated as an expected location of item 150 in the retail environment, based on aggregated location data (e.g., a heat map) for the item. In some implementations, detection events for an item may occur through devices used by a threshold number of different users (e.g., five, ten, or another suitable number), and/or through the use of at least two different application types, before a non-planogram location is designated as an expected location for the item. By considering the sources of detection events when determining whether a non-planogram location is to be considered as an expected location, for example, potentially fraudulent application use can be mitigated.

In general, a location-based verification confidence score for an item can be inversely proportional to a distance from its expected location within a retail environment. For example, if a scan and/or cart addition of item 150 occurs within area 612 or 622 (e.g., five feet, ten feet, or another suitable distance from respective expected item locations 610, 620), a relatively high location-based verification confidence score can be associated with the item. As another example, if a scan and/or cart addition of item 150 occurs outside of area 612 but within area 614, or outside of area 622 but within area 624 (e.g., within a range of five to ten feet, ten feet to twenty feet, or another suitable range of an expected item location), a moderate location-based verification confidence score can be associated with the item. As another example, if a scan and/or cart addition of item 150 occurs outside of area 614 or 624 (e.g., further than ten feet, twenty feet, or another suitable distance from an expected item location), a relatively low location-based verification confidence score can be associated with the item. For example, if a detection event occurs for item 150 in locations 630a-n, the verification confidence score for the item can be a relatively low value.

Referring now to FIG. 6B, an example of determined item locations and corresponding item confidence scores is shown. In the present example, various different customers can scan an item (e.g., a box of Frosty Flakes) using their respective mobile devices and can add the item to their respective shopping carts. Location data from a mobile device and/or shopping cart that results from the item being scanned by the mobile device and/or sensed by the cart can be compared to an expected location 650 of the item in the store. Based at least in part on a distance between the location data and the expected location, for example, a location-based verification confidence score can be determined for the item. In the present example, various different customers (e.g., Customer 1, Customer 2, Customer 3, Customer 4, and Customer 5) have each scanned the item (e.g., a box of Frosty Flakes selected from a store fixture) and added the item to their respective shopping cart. Customer 1, for example, scanned and/or added the item at location 660a (e.g., 0.5 meters from expected location 650), and a verification confidence score for the item can be 95%. Customer 2, for example, scanned and/or added the item at location 660b (e.g., 0.4 meters from expected location 650), and a verification confidence score for the item can be 94%. Customer 3, for example, scanned and/or added the item at location 660c (e.g., 54.5 meters from expected location 650), and a verification confidence score for the item can be 1%. Customer 4, for example, scanned and/or added the item at location 660d (e.g., 0.4 meters from expected location 650), and a verification confidence score for the item can be 96%. Customer 5, for example, scanned and/or added the item at location 660e (e.g., 10 meters from expected location 650), and a verification confidence score for the item can be 60%. In general, the distance between the location data and the expected location can be a significant contribution to an overall verification confidence score for an item, which can be adjusted based on sensor data from one or more other sensors, or other factors.

In some implementations, a proximity between a mobile device and a shopping cart can be used as a factor for adjusting a verification confidence score for an item. For example, in response to an item detection event having occurred (e.g., item 150 being scanned by mobile device 102, or item 150 being sensed by shopping cart 104), the location of the mobile device 102 and the location of the shopping cart 104 at the time of the event can be determined and compared. As another example, the location of the mobile device 102 can be determined when the item 150 is scanned by the mobile device, the location of the shopping cart 104 can be determined when the item 150 is sensed by the shopping cart, and the location of the mobile device 102 at the time of the scan can be compared to the location of the shopping cart 104 at the time of the item being sensed. In general, greater distances between a mobile device and a shopping cart during a shopping trip may indicate that item scans are potentially unreliable (e.g., the cart is unattended, item switching may have occurred, etc.). If the distance between the location of the mobile device 102 and the shopping cart 104 is under a threshold value (e.g., five feet, ten feet, or another suitable value), the verification confidence score for the item can remain unadjusted, whereas if the distance between the locations meets or exceeds the threshold value, for example, the verification confidence score for the item can be negatively adjusted in proportion to the distance.

In addition to or instead of the aforementioned techniques for optionally adjusting item verification data, comparing sensor data with item verification data, and determining whether there is a threshold match (e.g., as described with respect to boxes 502, 504, and 506), in some implementations, determining a match between sensor data (e.g., from physical sensors or location determination devices) and item verification data can include various machine learning techniques. For example, the received location data from the location determination device(s) 260 of the cart computing system 210 and/or from the location determination device(s) 344 of the mobile device 102, and optionally, the received motion data from the motion detection device(s) 220, and the received sensor data from one or more sensors 322a-c can be provided by the verification engine 402 to one or more pre-trained machine learning algorithms (e.g., classification algorithms) that can generate a verification confidence score for the item 150. For example, the location of an item scan and/or the location of an item being added to a cart, and the item's location verification data can be provided to a location classification algorithm to generate a location verification confidence score. Optionally, the motion data and the received sensor data for each sensor type can be individually provided to a corresponding algorithm for the sensor type to generate a verification confidence score according to the sensor type. For example, the motion data and the received sensor data from an IR sensor can be provided to an IR sensor classification algorithm to generate an IR verification confidence score, the motion data and the received sensor data from a sound sensor can be provided to a sound sensor classification algorithm to generate a sound verification confidence score, and/or the motion data and the received sensor data from a weight sensor can be provided to a weight sensor classification algorithm to generate a weight verification confidence score. In some implementations, the various verification confidence scores can be aggregated to determine an overall verification confidence score for an item. In the present example, the location verification confidence score, the IR verification confidence score, the sound verification confidence score, and the weight verification confidence score can be aggregated (e.g., weighted, averaged, etc.) to generate an overall verification confidence score for the item 150. In some implementations, the location data, the motion data, and the received sensor data for each sensor type can be provided to a composite algorithm that has been pre-trained based on data from multiple different sensor types. For example, the location of the item scan and/or the location of the item being added to the cart, the motion data, and the received sensor data from the IR sensor, the sound sensor, and the weight sensor can be provided to a composite classification algorithm to generate an overall verification confidence score for the item 150.

If the item verification engine 402 determines that there is a threshold match between the sensor data and the item verification data for the item 150, for example, the engine 402 can provide a notification to the mobile device 102 that the item is verified (at 508). The notification, for example, can include a binary verification result (e.g., verified or unverified), and/or a verification confidence score (e.g., a confidence percentage). In response to the notification, for example, the mobile device 102 can optionally notify the user that the item 150 is verified (at 510). For example, the mobile device 102 can provide a visual and/or audible message to the user through a mobile application running on the device 102. At 512, for example, the mobile device 102 and/or the server system 106 can add the item 150 to the user's virtual shopping cart. For example, the mobile device 102 and/or the server system 106 can maintain a tally of items that have been scanned by the user (and optionally, also verified by the system 100) during a current shopping trip. At 514, for example, the mobile device 102 can notify the user that an automated checkout option is available to the user, for when the user chooses to end the current shopping trip.

Referring to FIG. 7A, an example interface 700 is shown that can notify a user of items in a virtual shopping cart, checkout options available to the user, and/or verification statuses of items added to a physical shopping cart. For example, the mobile device 102 can present the interface 700 through a mobile application, which can be executed by the device 102 and/or the server system 106 (e.g., as a cloud-based application). In the present example, the interface 700 includes a list of items 710 that have been scanned by the user (e.g., using the mobile device 102), have been verified by the system 100, and added to the user's virtual shopping cart. The list of items 710 can include, for each item in the list, various sorts of data associated with the item, such as a description, price, image, quantity, weight, and/or other item data. In some implementations, some of the item data may be presented to the user (e.g., through the interface 700), whereas other item data may be maintained in the background. For example, the computing device 102 and/or the server system 106 can maintain an expected weight of each item in the list of items 710, for use in verifying the shopping cart 104 during an automated checkout process (e.g., weighing the cart 104 and its contents by the automated checkout station 170, and comparing an expected weight to an actual weight). In the present example, the interface 700 can present a total price of items in the virtual shopping cart, and can present a notification 720 (e.g., a message) to the user that the user may proceed to an automated checkout station when finished with the shopping trip.

Referring to FIG. 8A, example interfaces 800, 820 are shown that can notify users of items in a virtual shopping cart and available checkout options. For example, the mobile device 102 can present the interfaces 800, 820 through a mobile application, which can be executed by the device 102 and/or the server system 106 (e.g., as a cloud-based application). In the present example, the interface 800 presents information (e.g., description, price, image, quantity, weight, etc.) associated with items that have been scanned by the user (e.g., using the mobile device 102), have been verified by the system 100 (e.g., with each item having been assigned a corresponding verification confidence score), and added to the user's virtual shopping cart. The interface 820, for example, facilitates completing a transaction for the user's shopping trip, including presenting an order summary that includes a subtotal, possible discounts, estimated tax, and a total charge for the shopping trip, along with information related to a selected payment method. Each of the interfaces 800, 820 in the present example can present a notification (e.g., a “Pay & Go” message) to inform the user that the current shopping trip qualifies for an automated checkout option.

Referring again to FIG. 5, if the item verification engine 402 determines that there is not a threshold match between the sensor data and the item verification data for the item 150, for example, the engine 402 can provide a notification to the mobile device 102 that the item is not verified (at 520). In response to the notification, for example, the mobile device 102 can optionally notify the user that the item 150 is not verified (at 522), or that a count of items as indicated by the sensor data does not correspond to an item quantity specified by the user. For example, the mobile device 102 can provide a visual and/or audible message to the user through a mobile application running on the device 102.

Optionally, the mobile device 102 can track instances of unverified items, and/or provide an option for the user to retry verifying an item (or to correct a specified item quantity). For example, at 524, the mobile device 102 and/or server system 106 can track an instance of an unverified item (e.g., by adding data representing the item 150 to a list of unverified items). At 526, for example, the mobile device 102 and/or server system 106 can determine whether the unverified items exceeds a threshold. The threshold, for example, can be an absolute threshold (e.g., a number of items in a list of unverified items), a price threshold (e.g., a dollar value of items in the list of unverified items), a percentage threshold (e.g., a percentage of items in the list of unverified items relative to a number of scanned items), or another sort of threshold. If the unverified items do not exceed the threshold, for example, the user can optionally be given the option to retry verification of the item 150 (at 528). If the option to retry item verification is available (and/or if the user chooses the option), the user can be prompted to rescan the item 150 and/or correct the item quantity (at 530), and the item verification process can be restarted (at 532). For example, the mobile device 102 can prompt the user to rescan the item 150 through a mobile application running on the device 102. If the option to retry item verification is unavailable (and/or if the user does not choose the option), or if the unverified items exceeds the threshold (at 526), the automated checkout option can be removed (at 534). For example, the mobile device 102 and/or the server system 106 can maintain data that indicates the user's virtual shopping cart includes one or more unverified items during a current shopping trip. At 536, for example, the mobile device 102 can notify the user that a manual checkout option is available to the user, for when the user chooses to end the current shopping trip.

Referring to FIG. 7B, an example interface 750 is shown that can notify a user of items in a virtual shopping cart, verification statuses of the items, and/or checkout options available to the user. For example, the mobile device 102 can present the interface 750 through a mobile application, which can be executed by the device 102 and/or the server system 106 (e.g., as a cloud-based application). In the present example, the interface 750 includes a list of items 760 that have been scanned by the user (e.g., using the mobile device 102), however not all of the items have been verified by the system 100. Unverified items, for example, can be marked with a visual indication 762 (e.g., a graphical indicator, a note, a color, a font, etc.) that indicates that the items have not been verified. As another example, unverified items can be maintained in a separate list from verified items. As another example, information about whether an item has been verified or unverified can be maintained in the background and not presented to a user. In the present example, the interface 750 can present a notification 770 (e.g., a message) that the user may proceed to a manual checkout station when finished with the shopping trip.

Referring to FIG. 8B, example interfaces 840, 860 are shown that can notify users of items in a virtual shopping cart and available checkout options. For example, the mobile device 102 can present the interfaces 840, 860 through a mobile application, which can be executed by the device 102 and/or the server system 106 (e.g., as a cloud-based application). In the present example, the interface 840 presents information (e.g., description, price, image, quantity, weight, etc.) associated with items that have been scanned by the user (e.g., using the mobile device 102), that have been verified by the system 100 (e.g., with each item having been assigned a corresponding verification confidence score), and added to the user's virtual shopping cart. In the present example, verification confidence information for each item is not directly surfaced to the user, in that some items may have a relatively high verification confidence score, whereas other items may have a relatively low verification confidence score. An overall cart accuracy score (e.g., based on aggregating the individual item verification confidence scores, and optionally based on other factors, such as a user's purchase history, a user's reputation score, and other suitable factors) can be maintained in the background, for example, can be continually updated as the user adds and/or removes items from the physical shopping cart, and can be used to provide appropriate instructions to the user for finalizing a transaction for the current shopping trip. The interface 860, for example, facilitates completing a transaction for the user's shopping trip, including presenting an order summary total charge for the shopping trip, along with information related to a selected payment method. Each of the interfaces 840, 860 in the present example, can present a notification (e.g., a “Proceed to store checkout” message) to inform the user that the current shopping trip does not qualify for an automated checkout option, and that the user is to proceed to a manual checkout station instead.

FIG. 9 shows an example of a computing device 900 and an example of a mobile computing device that can be used to implement the techniques described here. The computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connecting to the memory 904 and multiple high-speed expansion ports 910, and a low-speed interface 912 connecting to a low-speed expansion port 914 and the storage device 906. Each of the processor 902, the memory 904, the storage device 906, the high-speed interface 908, the high-speed expansion ports 910, and the low-speed interface 912, are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 902 can process instructions for execution within the computing device 900, including instructions stored in the memory 904 or on the storage device 906 to display graphical information for a GUI on an external input/output device, such as a display 916 coupled to the high-speed interface 908. In other implementations, multiple processors and/or multiple buses can be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices can be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory 904 stores information within the computing device 900. For example, the memory 904 is a volatile memory unit or units. For example, the memory 904 is a non-volatile memory unit or units. The memory 904 can also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device 906 is capable of providing mass storage for the computing device 900. For example, the storage device 906 can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. The computer program product can also be tangibly embodied in a computer- or machine-readable medium, such as the memory 904, the storage device 906, or memory on the processor 902.

The high-speed interface 908 manages bandwidth-intensive operations for the computing device 900, while the low-speed interface 912 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. For example, the high-speed interface 908 is coupled to the memory 904, the display 916 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 910, which can accept various expansion cards (not shown). In the implementation, the low-speed interface 912 is coupled to the storage device 906 and the low-speed expansion port 914. The low-speed expansion port 914, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) can be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device 900 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server 920, or multiple times in a group of such servers. In addition, it can be implemented in a personal computer such as a laptop computer 922. It can also be implemented as part of a rack server system 924. Alternatively, components from the computing device 900 can be combined with other components in a mobile device, such as a mobile computing device 950. Each of such devices can contain one or more of the computing device 900 and the mobile computing device 950, and an entire system can be made up of multiple computing devices communicating with each other.

The mobile computing device 950 includes a processor 952, a memory 964, an input/output device such as a display 954, a communication interface 966, and a transceiver 968, among other components. The mobile computing device 950 can also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor