may be a high-performance device, such as a desktop or laptop, or a resource-limited device, such as a mobile phone. A high-performance device may have a dedicated GPU and a high-capacity or constant power source. A resource-limited device, on the other hand, may not have a GPU and may have limited battery capacity. As such, the algorithms that could be practically used by a virtual reality system 100 depends on the capabilities of its one or more computing systems 110.

In embodiments where the one or more computing systems 110 is a high-performance device, an embodiment of the passthrough feature may be designed as follows. Through the external-facing cameras 105A-B of the VR display device 135, a sequence of images of the surrounding physical environment may be captured. The information captured by the cameras 105A-B, however, may be misaligned with what the user's eyes may capture since the cameras could not spatially coincide with the user's eyes (e.g., the cameras may be located some distance away from the user's eyes and, consequently, have different viewpoints). As such, simply displaying what the cameras captured to the user may not be an accurate representation of what the user should perceive.

Instead of simply displaying what was captured, the passthrough feature may re-project information captured by the external-facing cameras 105A-B to the user. Each pair of simultaneously captured stereo images may be used to estimate the depths of observed features. As explained above, to measure depth using triangulation, the one or more computing systems 110 may find correspondences between the stereo images. For example, the one or more computing systems 110 may determine which two pixels in the pair of stereo images correspond to the same observed feature. A high-performance one or more computing systems 110 may solve the correspondence problem using its GPU and optical flow techniques, which are optimized for such tasks. The correspondence information may then be used to compute depths using triangulation techniques. Based on the computed depths of the observed features, the one or more computing systems 110 could determine where those features are located within a 3D space (since the one or more computing systems 110 also knows where the cameras are in that 3D space). The result may be represented by a dense 3D point cloud, with each point corresponding to an observed feature. The dense point cloud may then be used to generate 3D models of objects in the environment. When the system renders a scene for display, the system could perform visibility tests from the perspectives of the user's eyes. For example, the system may cast rays into the 3D space from a viewpoint that corresponds to each eye of the user. In this manner, the rendered scene that is displayed to the user may be computed from the perspective of the user's eyes, rather than from the perspective of the external-facing cameras 105A-B.

The process described above, however, may not be feasible for a resource-limited computing unit (e.g., a mobile phone may be the main computational unit for the VR display device 135). For example, unlike systems with powerful computational resources and ample energy sources, a mobile phone cannot rely on GPUs and computationally-expensive algorithms (e.g., optical flow) to perform depth measurements and generate an accurate 3D model of the environment. Thus, to provide passthrough on resource-limited devices, an optimized process is needed.

In particular embodiments, the computing device may be configured to dynamically determine, at runtime, whether it is capable of or able to generate depth measurements using (1) the GPU and optical flow or (2) the optimized technique using video encoder and motion vectors, as described in further detail below. For example, if the device has a GPU and sufficient power budget (e.g., it is plugged into a power source, has a full battery, etc.), it may perform depth measurements using its GPU and optical flow. However, if the device does not have a GPU or has a stringent power budget, then it may opt for the optimized method for computing depths.

FIG. 3 illustrates components of the VR system 100. There are real-world objects 145 and real-world environments 150. The real-world objects can include items (such as keyboards, cups, controllers, etc.) and furniture (such as desks, chairs, couches, etc.). The real-world environment 150 may correspond to a user's office, living room, kitchen, etc. FIGS. 4-5 illustrate example environment maps 170. With reference to FIG. 4, the environment map 170 comprises the one or more real-world objects 145 in the real-world environment 150. The environment map 170 may be provided to VR applications, alongside behavior logic 171 (e.g., which may include real-world object identification, MR object identification, safe traversal and interaction information, etc.) and developer guidelines (e.g., best practices, how to use and how not to use the real-world and MR objects, etc.). With reference to FIG. 5, a number of different VR applications 173 may utilize the environment map 170 to generate a MR experience for the user. For example, a VR media streaming application may utilize a real-world chair as an MR couch for the user to sit on while watching a movie with other VR users (who may be represented as VR avatars sitting on the same MR couch to create a co-presence experience). As another example, a VR board game application may utilize a real-world table as an MR play area for board game, for the user to sit at so the user can virtually play a board game against other VR users (who may be represented as VR avatars around the MR play area to create a co-presence experience).

In particular embodiments, the VR system 100 may receive an instruction to transition to a first VR environment 160 of a first VR application 173. The user may select a VR application 173, which may instruct the VR system 100 to initiate a VR experience by transitioning to a VR environment. As an example and not by way of limitation, the user may select a VR media streaming application, and the VR system will receive an instruction to transition to a VR movie theater environment. Although this disclosure describes the VR system 100 receiving instructions in a particular manner, this disclosure contemplates the VR system 100 receiving instructions in any suitable manner.

FIG. 6 illustrates example MR surfaces 156 corresponding to real-world objects 145. A create user interface (UI) 157 may appear on the VR display device 135 for a user to mark a surface of a real-world object 145 to be rendered as a MR surface 156. The user may adjust the dimensions of the MR surface 156, e.g., by adjusting parameters on the create UI 157 or by selecting and dragging the edges of the MR surface 156 to a desired dimension (e.g., via an edit user interface 158). The dimensions of the MR surface 156 may be determined by the VR system 100 semi-automatically (e.g., by dragging one or more edges of the MR surface 156) or automatically. The user may then view the MR surface 156 corresponding to the real-world object 145, while being aware of or alerted to the presence of other real-world objects 145 (e.g., a cat on the real-world couch) by displaying a passthrough view of the real-world object 145 or displaying a pass-through view of the object (e.g., the cat on the real-world couch) on the MR object 155. Thus, the user may safely and seamlessly traverse the VR environment 160 without breaking the VR immersion. A technical advantage of the embodiments may include allowing the user to safely navigate within VR environments by integrating real-world objects as suitable MR objects, such that the user does not need to clear the real-world environment of furniture or other obstacles.

FIG. 7 illustrates example MR objects 155 and their related MR surfaces 156. The MR surface 156 may correspond to a two-dimensional (2D) surface of the three-dimensional (3D) MR object 155. The MR surface 156 of a real-world object 145 may correspond to a planar surface (e.g., the MR surface 156 of the real-world object 145 may correspond to a top planar surface of the real-world object, such as a surface of a real-world desk). The MR object 155 may encompass the real-world object 145. That is, the MR surface 156 of the MR object 155 may correspond to a bounding box (e.g., two or more MR surfaces 156 of the MR object 155 may form a bounding box enclosing the real-world object 145). Alternatively, the MR surface 156 of the MR object 155 may conform to the dimensions of the real-world object (e.g., the two or more MR surface 156 of the MR object 155 may correspond to two or more real-world surfaces of the real-world object such that the MR object 155 matches the shape of the real-world object 145).

FIGS. 8-9 illustrates example customization features for MR objects 155. With reference to FIG. 8, the VR system 100 may populate a bounding box 164 with one or more components. For example, the MR object 155 corresponding to a seat and having a bounding box 164 may be populated with MR cushions, planks, tiles, etc. Additionally, the VR system 100 may allow a user to customize the MR object 155 with various configurations. For example, a MR couch may be conformed to include components such as armrests, backrests, or attachment points (e.g., attachment points 161). Additionally, one or more planar projections 162 may be used to determine the dimensions of the bounding box 164. With reference to FIG. 9, the VR system 100 may incorporate items within a shell 174 (e.g., a shell enclosing the VR environment 160), and leverage real-world objects 145 as MR objects 155 within the shell 174. Additionally, the VR system may incorporate a virtual user interface (UI) 190. The virtual UI 190 may be incorporated into MR objects (e.g., the virtual UI 190 may appear as a desktop on top of a MR desk). The VR system 100 may identify MR objects (e.g., MR surfaces 156) the user may act on, and confirm with the user that the MR object is usable. For example, the VR system 100 may confirm with the user that the MR surface 156 of an MR desk is usable as a desktop. Additionally, the VR system 100 may automatically or semi-automatically identify a real-world object 145 as potential MR objects 155. For example, the VR display device 135 may scan the real-world environment 150 and identify real-world objects 145 that can be leveraged as MR objects 155.

FIG. 10 illustrates a capture of a real-world object 145. The VR system 100 may be used to capture—either manually, semi-automatically, or automatically—the real-world object 145 to be leveraged as a MR object 155. As an example and not by way of limitation, the user may manually select a real-world object to be rendered as a MR object 155, and manually assign a MR functionality to the MR object 155 (e.g., the user may select a real-world couch to be rendered as a MR couch having the MR functionality of a seat, or the user may select the real-world couch to be rendered as a MR pile of rocks providing the functionality of a place to hide behind). As another example and not by way of limitation, the VR system may use an object recognition algorithm to recognize the object type of the real-world object, and ask the user to confirm whether to add the real-world object as an MR object and apply a MR functionality to the MR object 155. As yet another example and not by way of limitation, the user VR system 100 may automatically identify a real-world object 145 using an object recognition algorithm and automatically add it as a MR object 155 in the VR environment (i.e., without requiring user confirmation), and assign an appropriate MR functionality to the MR object 155.

FIG. 11 illustrates properties of MR objects 155. One property of the MR object 155 is a movement 176 of the MR object 155. That is, as the real-world object 145 moves, the location of the corresponding MR object 155 within the VR environment 160 may be updated. Another property of the MR object 155 is the function of the MR object 155. That is, the VR system 100 may automatically classify the real-world object 145 and/or the MR object 155 based on its function. As an example and not by way of limitation, a real-world desk may be classified as a MR desk, and a real-world couch may be classified as a MR couch. As another example and not by way of limitation, the real-world desk may be classified as a MR pillar for a user to hide behind, and the real-world couch may be classified as a MR obstacle to avoid. The real-world desk may also be classified as MR platform for the user to stand on, and the real-world couch may also be classified as a MR pillar (e.g., for a user to hide behind). Additionally, the VR system 100 may adjust the dimensions of the MR object 155. That is, the MR couch may be adjusted using the one or more adjustments 177 to extend the length, width, and/or height of the MR couch. As an example and not by way of limitation, a MR desk may be “extended” to turn into a long MR conference table. Additionally, the VR system may adjust a style 178 of the MR object 155. For example, a pattern, color, theme, or other form of styling may be applied to the MR object 155. The style may be based on the VR environment 160 and/or determined by policies or rules of the VR application.

FIG. 12A illustrates customizing the appearance of an MR object 155 based on the VR environment 160. That is, the appearance and/or style of the MR object 155 may be customized to blend in with or correspond to the VR environment 160. The following are examples of an MR overlay of MR objects 155 on real-world objects 145. As an example and not by way of limitation, referencing the left-side example of FIG. 12A, in a “dilapidated” VR environment, a real-world couch may be rendered as a tattered MR couch. As another example and not by way of limitation, referencing the center example of FIG. 12A, in a “luxury resort” VR environment, a real-world couch may be rendered as a luxurious MR couch. As yet another example and not by way of limitation, referencing the right-side example of FIG. 12A, in a “outdoor” VR environment, a real-world couch may be rendered as a MR log bench.

FIG. 12B illustrates customizing the functionality of an MR object 155 based on the VR environment 160. That is, the MR functionality of the MR object 155 may be customized to a functionality that is applied by the VR application and appropriate to the VR environment 160. As an example and not by way of limitation, referencing the left-side example of FIG. 12B, in a “strategy gaming” VR environment, a real-world table may be rendered as a MR boardgame, functioning as a virtual interactive boardgame with VR game pieces the user can interact with and move on the MR surface 156. As another example and not by way of limitation, referencing the center example of FIG. 12B, in a “museum” VR environment, a real-world table may be rendered as a MR marble pillar holding a MR painting, functioning as a virtual obstacle the user has to walk around. As yet another example and not by way of limitation, referencing the right-side example of FIG. 12B, in a “camping” VR environment, a real-world table may be rendered as a MR stone tower, allowing the user to step on the real-world table to be transported to the top of the MR stone tower to get an elevated view of the landscape below in the VR environment.

FIG. 13 illustrates a user interface for classifying MR objects 155. A technical challenge may include determining what type of MR object 155 a real-world object 145 should be rendered as by the VR system 100. The solution presented by the embodiments disclosed herein to address this challenge may be to utilize a user interface to help a user classify the MR functionality of MR objects 155. In particular embodiments, a user interface for classifying MR objects 155 may be presented to a user when they select a real-world object 145 to be rendered as a MR object 155, to classify the type of object the MR object 155 should be rendered as. As an example and not by way of limitation, the user interface may present options for a user to classify an MR object as a couch, a desk, a tabletop, a whiteboard, or a generic room item. FIGS. 14-16 illustrates enclosures 179 for VR environments 160. With reference to FIG. 14, the enclosure 179 can ensure the space size of the VR environment is suitable according to the MR object 155. For example, the enclosure 179 around a MR desk may be larger to assume the user will sit in a chair next to the MR desk. Thus, the enclosure 179 may be expanded to allow the user to comfortably sit within the VR environment 160. As another example, the enclosure 179 around a MR couch may be limited to just the MR couch, as the user may not need more space beyond the immediate vicinity around the MR couch to comfortably remain within the VR environment 160. With reference to FIG. 15, the enclosure 179 may separate the VR environment 160 and MR objects 155. For example, the VR environment 160 may cover the space between MR objects 155, and extend to the MR surfaces 156 of the MR objects 155. The enclosure 179 may include the MR objects 155 entirely, and not just the usable MR surfaces 156. With reference to FIG. 16, regardless of whether the enclosure 179 encloses the MR objects 155, the entire room of the real-world environment 150 may be accepted for the boundary of the VR environment 160. Thus, the user 102 may freely transition between multiple spaces within the enclosure 179 of the VR environment 160, allowing the user 102 to utilize multiple MR surfaces 156.

In particular embodiments, the VR system 100 may access attribute information of one or more real-world objects 145 in the real-world environment 150 proximate to the VR display device 135. Attribute information may comprise one or more spatial dimensions of the one or more real-world objects 145. As an example and not by way of limitation, attribute information of a real-world desk may comprise the length and width of the surface of the desk, and attribute information of a real-world bookshelf may comprise the length, width, and height of the real-world bookshelf. More information on determining dimensions of real-world object for use in mixed-reality applications may be found in U.S. patent application Ser. No. 17/204,699, filed 17 Mar. 2021, which is incorporated by reference. Attribute information may comprise one or more volumetric shapes of the one or more real-world objects 145. As an example and not by way of limitation, attribute information of a real-world couch may comprise the shape of the real-world couch, including the seat, armrests, and backrest of the real-world couch. Attribute information may comprise one or more real-world functionalities of the one or more real-world objects. As an example and not by way of limitation, attribute information of a real-world couch may include the functionality of the real-world couch (e.g., as a seat), and attribute information of a real-world desk may include the functionality of the real-world desk (e.g., a surface to work on). This attribute information may be determined using computer vision algorithms (e.g., object recognition algorithms, map depth and spatial recognition algorithms) or user-provided information. Although this disclosure describes the VR system 100 accessing particular attribute information of particular objects in a particular manner, this disclosure contemplates the accessing any suitable attribute information of any suitable objects in any suitable manner.

In particular embodiments, the VR system 100 may determine, for each of the one or more real-world objects 145, one or more first MR functionalities applicable to the real-world object 145 based on its attribute information. The MR functionality applicable to the real-world object 145 may correspond to one or more real-world functionalities of the real-world object 145. As an example and not by way of limitation, if a real-world couch has the real-world functionality of providing a place to sit, the MR functionality will correspondingly provide a place to sit. On the other hand, the MR functionality applicable to the real-world object 145 may not correspond to one or more real-world functionalities of the real-world object 145 (see FIG. 12B). As an example and not by way of limitation, a real-world couch may be rendered as a MR couch in one VR environment (e.g., in a VR media streaming application), and as a MR pile of rocks in another VR environment (e.g., in a VR western video game environment). As another example and not by way of limitation, if a real-world table has the real-world functionality of providing a usable surface, the MR functionality may provide an alternative functionality, e.g., a blockade or obstacle to avoid. As another example and not by way of limitation, a real-world wall may be rendered as a MR wall in one VR environment, and then rendered as a MR whiteboard in another VR environment. Although this disclosure describes the VR system 100 determining one or more functionalities in a particular manner, this disclosure contemplates the VR system 100 determining one or more functionalities in any suitable manner.

In particular embodiments, the VR system 100 may render, for one or more displays of the VR display device 135, the first VR environment 160 with one or more MR objects 155 corresponding to the one or more real-world objects 145. That is, while the user 102 is immersed within the VR environment 160, the user 102 may be able to utilize the MR objects 155 within the VR environment 160. That is, the user 102 may use the real-world object 145 as a MR object 155 in the VR environment 160 with no loss of immersion. Each MR object 155 may comprise one of the first MR functionalities applicable to the corresponding real-world object 145. As an example and not by way of limitation, A MR couch may serve the functionality of providing the user a place to sit while immersed in a VR media streaming application, similar to the real-world couch. The MR object 155 may comprise one or more MR functionalities that does not correspond to the real-world object 145. As another example and not by way of limitation, a MR table may be used as an obstacle (e.g., a pile of rocks) to hide behind while the user is in a VR gaming application. Although this disclosure describes the VR system 100 rendering the VR environment 160 in a particular manner, this disclosure contemplates the VR system 100 rendering the VR environment 160 in any suitable manner.

In particular embodiments, the VR system 100 may identify one or more real-world objects 145 in the real-world environment 150 proximate to the VR display device 135. That is, the VR system 100 may detect whether there are real-world objects 145 close or proximate to the user 102, and notify or present these real-world objects 145 to the user 102. The VR system 100 may identify a planar surface of one of the real-world objects 145 in the real-world environment 150, and apply an MR functionality to the planar surface of the one of the real-world objects 145. As an example and not by way of limitation, if the user 102 is proximate to a real-world desk (as in FIGS. 7 and 9-10), the VR system 100 may identify the top surface of the real-world desk, and apply the MR functionality of providing a working surface to the planar surface of the real-world desk. The VR system 100 may generate a bounding box containing one of the real-world objects 145 in the real-world environment 150, and apply an MR functionality to the bounding box containing the real-world object 145. As an example and not by way of limitation, if the user 102 is next to a real-world couch (as in FIGS. 7 and 9-10), the VR system 100 may identify the real-world couch, generate a bounding box 164 that encloses the real-world couch, and apply the MR functionality of providing a barrier or obstacle to hide behind while the user is in a VR gaming application. The VR system 100 may generate a three-dimensional (3D) mesh of one of the real-world objects 145 in the real-world environment 150, and apply an MR functionality to the 3D mesh of the real-world object 145. As an example and not by way of limitation, if the user 102 is next to a real-world couch (as in FIGS. 7 and 9-10), the VR system 100 may identify the real-world couch, generate a mesh that matches the volume and shape the real-world couch, and apply the MR functionality of providing the user a place to sit, such that the corresponding MR couch has seats, armrests, and a backrest that match and correspond to the real-world couch. Although this disclosure describes identifying real-world objects 145 in a particular manner, this disclosure contemplates identifying real-world objects 145 in any suitable manner.

In particular embodiments, the VR system 100 may render, for the one or more displays of the VR display device 135, one or more suggestions for applying one or more MR functionalities to the one or more of the real-world objects 145. The VR system 100 may receive an indication of a user selection corresponding to one of the suggestions for applying the MR functionalities to the real-world object 145, and apply the MR functionalities to the real-world objects 145 responsive to the user selection. That is, the VR system 100 may suggest MR functionalities based on the real-world object (e.g., the VR system 100 may suggest the MR functionality of providing a work surface for a real-world desk, or the MR functionality of providing a place to sit for the real-world couch). The VR system 100 may suggest MR functionalities based on the available functionalities within the VR application that are suitable for the particular real-world object 145 (e.g., in a first-person shooter VR gaming application, the VR system 100 may suggest the MR functionality of a barricade to hide behind). Although this disclosure describes suggesting MR functionalities in a particular manner, this disclosure contemplates suggesting MR functionalities in any suitable manner.

FIG. 17 illustrates a process for safely traversing the real-world environment to switch between MR objects 155. As the user 102 switches from one MR object 155 to another, or selects to switch from one MR object 155 to another, the VR system 100 may request a confirmation that the user intends the switch. For example, if the user 102 is at a MR desk and wants to switch to a MR couch, the VR system 100 may present a passthrough view of the real-world environment 150 to ensure the user 102 can safely move about the real-world environment to switch from using the MR desk to the MR couch. Thus, the user 102 may avoid any objects, pets, or other obstacles that may impede their safe passage from one MR object to another MR object.

FIG. 18 illustrates a sample VR environment 160 with multiple MR objects 155. The multiple MR objects 155 can be added or omitted to the VR environment 160 based on the needs of the VR environment 160. For example, the VR environment 160 may contain a MR desk, a MR table, a MR couch, and a MR whiteboard. In a workspace VR environment, the user may only need the MR desk and the MR whiteboard. Thus, the VR system 100 may omit the MR couch and the MR table from the VR environment 160. For user safety, the VR system 100 may alert the user (e.g., via a boundary warning) if the user approaches an omitted object. As an example and not by way of limitation, the VR system 100 may show the user the omitted object (e.g., via a passthrough view of the real-world environment, or as a boundary warning alert through the user's personal UI). More information on spatial awareness and virtual boundaries may be found in U.S. patent application Ser. No. 17/139,434, filed 31 Dec. 2020, which is incorporated herein by reference.

FIG. 19 illustrates a guided traversal for the user 102 in the VR environment 160. If the user 102 wishes to switch from one MR object 155 to another, or alternatively, if the VR system 100 recommends the user 102 to switch from one MR object to another, the VR system 100 may provide a guide 181 to allow the user 102 to safely traverse the VR environment 160. For example, if the user 102 is at a MR desk, the VR system 100 may recommend the user to switch to the MR whiteboard. The VR system 100 may then provide the guide 181 indicating to the user 102 a route to traverse to safely reach the MR whiteboard. Alternatively, the VR system 100 may present a passthrough view of the real-world environment to help the user safely navigate to the new space.

In particular embodiments, the VR system 100 may access floorplan information of the real-world environment 150, and render, for the one or more displays of the VR display device 135, the first VR environment 160. The floorplan information may correspond to one or more areas of the user's room, or to one or more rooms in the user's home or real-world environment 150. In particular embodiments, the floorplan information may be gathered by one or more cameras of the VR display device 135. As an example and not by way of limitation, the VR system 100 may use an algorithm (e.g., object recognition algorithm, map depth and spatial recognition algorithm) to identify floors, walls, objects, and any other boundaries of the real-world environment 150. In particular embodiments, the floorplan information may be determined by user-provided information. As an example and not by way of limitation, the user may manually draw boundaries defining the floors, walls, objects, and any other boundaries of the real-world environment 150. This process can occur as the user defines the boundary for a VR application (e.g., during a setup period for the VR system 100, where the user may input boundaries for the enclosure 179). The VR environment 160 may then be rendered based on the one or more areas or one or more rooms of the user's real-world environment 150. The first VR environment 160 may correspond to the floorplan information.

In particular embodiments, the VR system 100 may receive an indication the VR display device 135 has moved to a new area of the real-world environment 150 corresponding to the floorplan information. The new area of the real-world environment comprising one or more new real-world objects 145. That is, the user 102 may have moved to a new room in their real-world environment 150 (e.g., the user 102 has moved from the living room to the office), and the new room has new real-world objects 145 for the VR system 100 to identify. The VR system 100 may access attribute information of one or more new real-world objects 145 in the new real-world environment 150 proximate to the VR display device 135. As an example and not by way of limitation, if the first room was a living room containing a real-world couch and a real-world coffee table, and the new room was an office containing a real-world desk, the VR system may access attribute information (such as dimensions and functionality) of the real-world desk. The VR system 100 may render, for the one or more displays of the VR display device 135, a second VR environment 160 with one or more new MR objects 155 corresponding to the one or more new real-world objects 145. Each new MR object 155 may comprise one or more determined new MR functionalities applicable to the corresponding new real-world object 145. As an example and not by way of limitation, if the user has moved from their living room (containing the real-world couch and real-world coffee table) and into their office (containing the real-world desk), the real-world desk may be rendered as an MR desk the user can sit at, or an obstacle for the user to avoid. Although this disclosure describes floorplan information of real-world environments 145 in a particular manner, this disclosure contemplates floorplan information of real-world environments 145 in any suitable manner.

In particular embodiments, the VR system 100 may receive an instruction to transition to a second VR environment 160 of the first VR application 173. As an example and not by way of limitation, if the user is in a VR gaming environment of a VR gaming application, and the VR gaming application transitions into a new gaming level or area, the VR environment may change from a first VR gaming environment of the VR gaming application to a second VR gaming environment of the VR gaming application. The VR system 100 may determine, for each of the one or more real world objects 145, one or more second MR functionalities based on the second VR environment 160 of the first VR application 173 and applicable to the real-world object 145 based on its attribute information. The VR system 100 may render, for one or more displays of the VR display device 135, the second VR environment 160 with one or more MR objects 155 corresponding to the one or more real-world objects 145, where each MR object 155 may comprise one of the second MR functionalities applicable to the corresponding real-world object 145. As an example and not by way of limitation, for a real-world environment 150 comprising a real-world couch and a VR gaming application, in the first VR environment 160 the real-world couch may be rendered as a MR car seat in a virtual car for the user to sit on while the user is “driving” in the VR gaming application, and the armrest of the real-world couch may be rendered as a virtual dashboard for the virtual car. In a second VR environment 160, the real-world couch may be rendered as a MR barricade to hide behind. A technical advantage of the embodiments may include allowing the user to utilize a real-world object in a number of different ways according to changing VR environments. Although this disclosure describes transitioning between VR environments in a particular manner, this disclosure contemplates transitioning between VR environments in any suitable manner.

In particular embodiments, the VR system 100 may receive an instruction to transition to a second VR environment 160 of a second VR application 173. As an example and not by way of limitation, the user may switch from a VR gaming application (having the first VR environment 160) to a VR media streaming application (having a second VR environment 160). The VR system 100 may determine, for each of the one or more real world objects 145, one or more second MR functionalities based on the second VR environment 160 of the second VR application 173 and applicable to the real-world object 145 based on its attribute information. The VR system 100 may render, for one or more displays of the VR display device 135, the second VR environment 160 with one or more MR objects 155 corresponding to the one or more real-world objects 145, where each MR object 155 may comprise one of the second MR functionalities applicable to the corresponding real-world object 145. As an example and not by way of limitation, for a real-world environment 150 comprising a real-world couch, if the user switches from a VR gaming application (the first VR application) to a VR media streaming application (the second VR application), the real-world couch (which may have been rendered as an obstacle for the user to hide behind while in the VR gaming application) may be re-rendered as MR theater seating for the VR media streaming application. A technical advantage of the embodiments may include allowing the user to utilize a real-world object 145 in a number of different ways as different MR objects 155 with different functionalities according to changing VR environments 160. That is, users may leverage the MR object 155 across different VR applications 173 without needing to re-identify the real-world object 145, or re-create the MR object 155, because the VR system 100 retains and shares the attribute information. Although this disclosure describes transitioning between VR environments in a particular manner, this disclosure contemplates transitioning between VR environments in any suitable manner.

FIG. 20 illustrates a VR environment 160 breach warning. To maintain immersion of the VR environment 160, the boundary of the enclosure 179 is not present. However, if the user 102 is about to breach the enclosure, or actually breaches the enclosure 179, the VR system 100 may alert the user 102. The alert may be a pop-up warning (e.g., an exclamation point, a stop sign, a guardian wall texture, etc.). The alert may also appear as a transition to a passthrough view of the real-world environment 150, or as a virtual wall showing the boundary of the enclosure 179. The alert may appear as a notification (e.g., textual notification on the VR display device 135 or on via a virtual UI).

FIG. 21 illustrates a MR object 155 proximity notice. MR objects 155 may be highlighted, outlined, or made apparent to the user 102 based on proximity of the user 102 to the MR object 155. As an example and not by way of limitation, if the user 102 is standing next to the MR couch, the seat of the MR couch may be highlighted for the user 102. If the user 102 is not proximate to the MR object 155, the MR object 155 is not made apparent to the user 102. As an example and not by way of limitation, if the user 102 is standing far away from a MR desk, the MR desk may not be highlighted for use by the user 102. However, if the user 102 moves toward the MR desk, then an outline may appear indicating the user 102 may use the MR desk. Thus, the MR object 155 is not made apparent until it becomes relevant.

FIG. 22 illustrates a passthrough view of the MR object 155. The VR system 100 may present a passthrough view 192 of the MR surface 156 of the MR object 155 to show the corresponding surface of the real-world object 145. This may be done via a passthrough portal. For example, if the user wishes to view what items are on the real-world desk corresponding to the MR desk, the user may use the passthrough portal to view the real-world desk. The user can thus safely interact with the MR object 155 by ensuring what is present on the corresponding real-world object 145. FIG. 23 illustrates a trigger 193 for presenting identified MR objects 155. The user can select the trigger 193 to present identified MR objects 155, MR surfaces 156, or any other identified or defined boundaries. The trigger 193 may be, for example, a recenter button of the VR display device 135.

FIG. 24 illustrates the virtual user interface (UI) 190 interacting with the MR object 155. When the user is proximate to or at the MR object 155, the virtual UI 190 may adapt its form according to the MR object 155. For example, if the user 102 is at a MR desk, the virtual UI 190, which ordinarily appears as a floating virtual object proximate to the user 102, may adapt its form to turn into a multi-monitor display. This may be useful when the user 102 is in a VR workspace environment, and the virtual UI 190 mimics the appearance of a multi-monitor work display. As another example, the virtual UI 190 may appear as a widescreen television when the user 102 is at a MR couch, thus mimicking the appearance of a living room configuration. More information on user interfaces for interacting with mixed-reality and virtual-reality applications may be found in U.S. patent application Ser. No. 17/174,037, filed 11 Feb. 2021, which is incorporated herein by reference.

FIG. 25 illustrates a MR co-presence configuration. The MR object 155 may be configured to enable a comfortable co-presence experience for the user 102. As an example and not by way of limitation, the user 102 may be seated at the MR couch, and across from the MR couch in the VR environment 160 is an additional user avatar 103 (corresponding to an additional user), where the additional user avatar 103 is rendered on a VR couch in the VR environment 160. As yet another example and not by way of limitation, the additional user avatar 103 (corresponding to the additional user) may be seated at the MR couch next to the user 102 (e.g., the user 102 and the additional user avatar 103 are seated next to each other on the same MR couch). As another example, in a workplace VR environment, the user 102 may be seated at a MR conference table across from the additional user avatar 103.

One technical challenge may include integrating VR representations of other users into a first user's VR experience, where adding a floating or randomly placed avatar representing these other users may break the VR immersion experience for the first user. The solution presented by the embodiments disclosed herein to address this challenge may be to place avatars corresponding to the other users on or around MR objects 155 to improve the co-presence experience of the first user and the other users. In particular embodiments, the VR system 100 may render, for the one or more displays of the VR display device 135, an avatar corresponding to a second user (e.g., the additional user avatar 103) within the first VR environment 160. The placement of the avatar within the first VR environment 160 at the MR object 155 may correspond to the determined MR functionality. That is, rather than having a floating avatar within the VR environment 160, the avatar of the second user may be purposefully placed within the VR environment 160. As an example and not by way of limitation, in a co-presence situation, the user 102 and the additional user avatar 103 may be placed side by side on a couch while the user 102 is in a VR media streaming application watching a movie. As another example and not by way of limitation, the user 102 and the additional user avatar 103 may be placed at opposite ends of a table while in a VR tennis game application. Although this disclosure describes rendering a second user within the VR environment 160 in a particular manner, this disclosure contemplates a second user within the VR environment 160 in any suitable manner.

FIG. 26 illustrates features of a MR workplace configuration. Features of the MR workplace may include turning a vertical surface (such as a wall) into a whiteboard or a virtual monitor display. For example, a wall may be rendered as a MR whiteboard for the user 102 to work on. As another example, the virtual user interface (UI) 190 may be adapted into a whiteboard or monitor display on the MR surface 156 of a wall. Another feature of the MR workplace may be using the VR system 100 to automatically detect planar surfaces (such as walls or desks) to allow the user to quickly add or “pin” content 195 to the planar MR surface 156. Yet another feature of the MR workplace may be allowing the user to view a passthrough view 192 of the MR surface 156. For example, if the user wishes to work on a MR desk that corresponds to a real-world desk, the user may view a passthrough view 192 of the real-world desk to ensure the user can safely work on the MR desk by ensuring the real-world desk is free of any objects or hazards. Another feature of the MR workplace may be allowing the user 102 to use a tuner 194 to toggle between the VR workplace environment 160 and the real-world environment 150.

FIG. 27 illustrates features for MR experience developers (e.g., VR application developers who may determine MR functionalities for MR objects 155 in VR applications). Features that may be relevant to VR application developers creating MR experiences include a “tabletop” function, which can be used to permit access for users 102 and/or additional user avatars 103 to MR objects 155 and/or MR surfaces 156. Thus, key MR objects 155 and/or MR surfaces 156 may be integrated into the VR environment more seamlessly. For example, a MR experience developer may assign a common MR object, such as a MR desk, to have a standard function within the MR experience. Another feature for MR experience developers includes coordinating two or more MR objects 155 and/or MR surfaces 156, thus allowing a VR application to direct users between the two or more MR objects 155 and/or MR surfaces 156. Another feature for MR experience developers includes differentiating between MR objects 155 and VR objects 165. The platform may signify which objects are MR objects 155, and which objects are VR objects 165. Thus, future users may be able to interact with the MR objects 155. Another feature for MR experience developers may include allowing for adjustment of settings 196 related to the MR experience. The settings 196 can be used to give users control over their MR experience, e.g., by allowing the user to leverage real-world objects as different types of pre-selected MR objects. A developer may determine, based on the attribute information associated with the MR object 155, what the MR object 155 will be in the VR environment 160. For example, a real-world bench may be turned into a MR obstacle for the user to walk around, or a MR object for the user to hide behind.

FIG. 28 illustrates an example method 2800 for leveraging real-world objects as MR objects in VR environments. The method may begin at step 2810, where one or more computing systems may receive an instruction to transition to a first VR environment of a first VR application. At step 2820, the one or more computing systems may access attribute information of one or more real-world objects in a real-world environment proximate to a VR display device. At step 2830, the one or more computing systems may determine, for each of the one or more real-world objects, one or more first MR functionalities applicable to the real-world object based on its attribute information. At step 2840, the one or more computing systems may render, for one or more displays of the VR display device, the first VR environment with one or more MR objects corresponding to the one or more real-world objects. Each MR object may comprise one of the first MR functionalities applicable to the corresponding real-world object. Particular embodiments may repeat one or more steps of the method of FIG. 28, where appropriate. Although this disclosure describes and illustrates particular steps of the method of FIG. 28 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of FIG. 28 occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for leveraging real-world objects as MR objects in VR environments including the particular steps of the method of FIG. 28, this disclosure contemplates any suitable method for leveraging real-world objects as MR objects in VR environments including any suitable steps, which may include all, some, or none of the steps of the method of FIG. 28, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method of FIG. 28, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of FIG. 28. Finally, although this disclosure describes leveraging real-world objects as MR objects in VR environments, this disclosure contemplates leveraging real-world objects as MR objects in any suitable environments. As an example, this disclosure contemplates leveraging real-world objects as MR objects in augmented reality (AR) environments, where the AR system may use the attribute information of a real-world object to determine a suitable MR functionality to apply to the real-world object in the AR environment.

FIG. 29 illustrates an example network environment 2900 associated with a social-networking system. Network environment 2900 includes a client system 2930, a social-networking system 2960, and a third-party system 2970 connected to each other by a network 2910. Although FIG. 29 illustrates a particular arrangement of client system 2930, social-networking system 2960, third-party system 2970, and network 2910, this disclosure contemplates any suitable arrangement of client system 2930, social-networking system 2960, third-party system 2970, and network 2910. As an example and not by way of limitation, two or more of client system 2930, social-networking system 2960, and third-party system 2970 may be connected to each other directly, bypassing network 2910. As another example, two or more of client system 2930, social-networking system 2960, and third-party system 2970 may be physically or logically co-located with each other in whole or in part. Moreover, although FIG. 29 illustrates a particular number of client systems 2930, social-networking systems 2960, third-party systems 2970, and networks 2910, this disclosure contemplates any suitable number of client systems 2930, social-networking systems 2960, third-party systems 2970, and networks 2910. As an example and not by way of limitation, network environment 2900 may include multiple client system 2930, social-networking systems 2960, third-party systems 2970, and networks 2910.

This disclosure contemplates any suitable network 2910. As an example and not by way of limitation, one or more portions of network 2910 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 2910 may include one or more networks 2910.

Links 2950 may connect client system 2930, social-networking system 2960, and third-party system 2970 to communication network 2910 or to each other. This disclosure contemplates any suitable links 2950. In particular embodiments, one or more links 2950 include one or more wireline (such as for example Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless (such as for example Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical (such as for example Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In particular embodiments, one or more links 2950 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular technology-based network, a satellite communications technology-based network, another link 2950, or a combination of two or more such links 2950. Links 2950 need not necessarily be the same throughout network environment 2900. One or more first links 2950 may differ in one or more respects from one or more second links 2950.

In particular embodiments, client system 2930 may be an electronic device including hardware, software, or embedded logic components or a combination of two or more such components and capable of carrying out the appropriate functionalities implemented or supported by client system 2930. As an example and not by way of limitation, a client system 2930 may include a computer system such as a desktop computer, notebook or laptop computer, netbook, a tablet computer, e-book reader, GPS device, camera, personal digital assistant (PDA), handheld electronic device, cellular telephone, smartphone, augmented/virtual reality device, other suitable electronic device, or any suitable combination thereof. This disclosure contemplates any suitable client systems 2930. A client system 2930 may enable a network user at client system 2930 to access network 2910. A client system 2930 may enable its user to communicate with other users at other client systems 2930.

In particular embodiments, client system 2930 may include a web browser 2932, and may have one or more add-ons, plug-ins, or other extensions. A user at client system 2930 may enter a Uniform Resource Locator (URL) or other address directing the web browser 2932 to a particular server (such as server 2962, or a server associated with a third-party system 2970), and the web browser 2932 may generate a Hyper Text Transfer Protocol (HTTP) request and communicate the HTTP request to server. The server may accept the HTTP request and communicate to client system 2930 one or more Hyper Text Markup Language (HTML) files responsive to the HTTP request. Client system 2930 may render a webpage based on the HTML files from the server for presentation to the user. This disclosure contemplates any suitable webpage files. As an example and not by way of limitation, webpages may render from HTML files, Extensible Hyper Text Markup Language (XHTML) files, or Extensible Markup Language (XML) files, according to particular needs. Such pages may also execute scripts, combinations of markup language and scripts, and the like. Herein, reference to a webpage encompasses one or more corresponding webpage files (which a browser may use to render the webpage) and vice versa, where appropriate.

In particular embodiments, social-networking system 2960 may be a network-addressable computing system that can host an online social network. Social-networking system 2960 may generate, store, receive, and send social-networking data, such as, for example, user-profile data, concept-profile data, social-graph information, or other suitable data related to the online social network. Social-networking system 2960 may be accessed by the other components of network environment 2900 either directly or via network 2910. As an example and not by way of limitation, client system 2930 may access social-networking system 2960 using a web browser 2932, or a native application associated with social-networking system 2960 (e.g., a mobile social-networking application, a messaging application, another suitable application, or any combination thereof) either directly or via network 2910. In particular embodiments, social-networking system 2960 may include one or more servers 2962. Each server 2962 may be a unitary server or a distributed server spanning multiple computers or multiple datacenters. Servers 2962 may be of various types, such as, for example and without limitation, web server, news server, mail server, message server, advertising server, file server, application server, exchange server, database server, proxy server, another server suitable for performing functions or processes described herein, or any combination thereof. In particular embodiments, each server 2962 may include hardware, software, or embedded logic components or a combination of two or more such components for carrying out the appropriate functionalities implemented or supported by server 2962. In particular embodiments, social-networking system 2960 may include one or more data stores 2964. Data stores 2964 may be used to store various types of information. In particular embodiments, the information stored in data stores 2964 may be organized according to specific data structures. In particular embodiments, each data store 2964 may be a relational, columnar, correlation, or other suitable database. Although this disclosure describes or illustrates particular types of databases, this disclosure contemplates any suitable types of databases. Particular embodiments may provide interfaces that enable a client system 2930, a social-networking system 2960, or a third-party system 2970 to manage, retrieve, modify, add, or delete, the information stored in data store 2964.

In particular embodiments, social-networking system 2960 may store one or more social graphs in one or more data stores 2964. In particular embodiments, a social graph may include multiple nodes—which may include multiple user nodes (each corresponding to a particular user) or multiple concept nodes (each corresponding to a particular concept)—and multiple edges connecting the nodes. Social-networking system 2960 may provide users of the online social network the ability to communicate and interact with other users. In particular embodiments, users may join the online social network via social-networking system 2960 and then add connections (e.g., relationships) to a number of other users of social-networking system 2960 to whom they want to be connected. Herein, the term “friend” may refer to any other user of social-networking system 2960 with whom a user has formed a connection, association, or relationship via social-networking system 2960.

In particular embodiments, social-networking system 2960 may provide users with the ability to take actions on various types of items or objects, supported by social-networking system 2960. As an example and not by way of limitation, the items and objects may include groups or social networks to which users of social-networking system 2960 may belong, events or calendar entries in which a user might be interested, computer-based applications that a user may use, transactions that allow users to buy or sell items via the service, interactions with advertisements that a user may perform, or other suitable items or objects. A user may interact with anything that is capable of being represented in social-networking system 2960 or by an external system of third-party system 2970, which is separate from social-networking system 2960 and coupled to social-networking system 2960 via a network 2910.

In particular embodiments, social-networking system 2960 may be capable of linking a variety of entities. As an example and not by way of limitation, social-networking system 2960 may enable users to interact with each other as well as receive content from third-party systems 2970 or other entities, or to allow users to interact with these entities through an application programming interfaces (API) or other communication channels.

In particular embodiments, a third-party system 2970 may include one or more types of servers, one or more data stores, one or more interfaces, including but not limited to APIs, one or more web services, one or more content sources, one or more networks, or any other suitable components, e.g., that servers may communicate with. A third-party system 2970 may be operated by a different entity from an entity operating social-networking system 2960. In particular embodiments, however, social-networking system 2960 and third-party systems 2970 may operate in conjunction with each other to provide social-networking services to users of social-networking system 2960 or third-party systems 2970. In this sense, social-networking system 2960 may provide a platform, or backbone, which other systems, such as third-party systems 2970, may use to provide social-networking services and functionality to users across the Internet.

In particular embodiments, a third-party system 2970 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects, which may be communicated to a client system 2930. As an example and not by way of limitation, content objects may include information regarding things or activities of interest to the user, such as, for example, movie show times, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example and not by way of limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

In particular embodiments, social-networking system 2960 also includes user-generated content objects, which may enhance a user's interactions with social-networking system 2960. User-generated content may include anything a user can add, upload, send, or “post” to social-networking system 2960. As an example and not by way of limitation, a user communicates posts to social-networking system 2960 from a client system 2930. Posts may include data such as status updates or other textual data, location information, photos, videos, links, music or other similar data or media. Content may also be added to social-networking system 2960 by a third-party through a “communication channel,” such as a newsfeed or stream.

In particular embodiments, social-networking system 2960 may include a variety of servers, sub-systems, programs, modules, logs, and data stores. In particular embodiments, social-networking system 2960 may include one or more of the following: a web server, action logger, API-request server, relevance-and-ranking engine, content-object classifier, notification controller, action log, third-party-content-object-exposure log, inference module, authorization/privacy server, search module, advertisement-targeting module, user-interface module, user-profile store, connection store, third-party content store, or location store. Social-networking system 2960 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management-and-network-operations consoles, other suitable components, or any suitable combination thereof. In particular embodiments, social-networking system 2960 may include one or more user-profile stores for storing user profiles. A user profile may include, for example, biographic information, demographic information, behavioral information, social information, or other types of descriptive information, such as work experience, educational history, hobbies or preferences, interests, affinities, or location. Interest information may include interests related to one or more categories. Categories may be general or specific. As an example and not by way of limitation, if a user “likes” an article about a brand of shoes the category may be the brand, or the general category of “shoes” or “clothing.” A connection store may be used for storing connection information about users. The connection information may indicate users who have similar or common work experience, group memberships, hobbies, educational history, or are in any way related or share common attributes. The connection information may also include user-defined connections between different users and content (both internal and external). A web server may be used for linking social-networking system 2960 to one or more client systems 2930 or one or more third-party system 2970 via network 2910. The web server may include a mail server or other messaging functionality for receiving and routing messages between social-networking system 2960 and one or more client systems 2930. An API-request server may allow a third-party system 2970 to access information from social-networking system 2960 by calling one or more APIs. An action logger may be used to receive communications from a web server about a user's actions on or off social-networking system 2960. In conjunction with the action log, a third-party-content-object log may be maintained of user exposures to third-party-content objects. A notification controller may provide information regarding content objects to a client system 2930. Information may be pushed to a client system 2930 as notifications, or information may be pulled from client system 2930 responsive to a request received from client system 2930. Authorization servers may be used to enforce one or more privacy settings of the users of social-networking system 2960. A privacy setting of a user determines how particular information associated with a user can be shared. The authorization server may allow users to opt in to or opt out of having their actions logged by social-networking system 2960 or shared with other systems (e.g., third-party system 2970), such as, for example, by setting appropriate privacy settings. Third-party-content-object stores may be used to store content objects received from third parties, such as a third-party system 2970. Location stores may be used for storing location information received from client systems 2930 associated with users. Advertisement-pricing modules may combine social information, the current time, location information, or other suitable information to provide relevant advertisements, in the form of notifications, to a user.

Systems and Methods

FIG. 30 illustrates an example computer system 3000. In particular embodiments, one or more computer systems 3000 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 3000 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 3000 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 3000. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems 3000. This disclosure contemplates computer system 3000 taking any suitable physical form. As example and not by way of limitation, computer system 3000 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these. Where appropriate, computer system