256, and an accelerometer 258. In some embodiments, aerial system 204 can include an altitude and heading reference system (AHRS). Using these devices, the orientation, heading, and height of the aerial vehicle (and of camera 250) can be determined. This information, when used with a GPS location for the aerial vehicle, can be used to infer the location of one or more points in an image taken from the aerial vehicle as described in further detail below.

Aerial system 204 can also include an onboard computing system 274 that can work in conjunction with the other components of aerial system 204. It should be understood that while the path planning module 270 is presented as part of the ground system 202, it can alternatively be incorporated directly in the onboard computing system 274 of aerial system 204. Furthermore, in some embodiments, the path planning module 270 can reside in both the ground system 202 and the onboard computing system 274. The two modules may be configured to communicate with one another and share information to allow for the adjustment of the flight path in real-time (e.g., during the flight mission).

In one example, the onboard computing system 274 further includes an image capture and processing system 260, also referred to simply as processing system 260. Processing system 260 may be used to store, process, and transmit image information. Additionally, in some cases, processing system 260 can receive GPS or other coordinate information about one or more target locations. To facilitate these tasks, image capture and processing system 260 may include one or more processors 262 as well as memory 264. Memory 264 can store instructions for programs that facilitate storing, processing, and transmitting image information.

Both ground system 202 and aerial system 204 can include communication systems. For example, ground system 202 can include a ground communication system 220 and aerial system 204 can include a UAV communication system 272. These communication systems enable information to be transmitted between ground system 202 and aerial system 204 via a network 280. Thus, the type of communication components used in each communication system can be selected according to the type of network used. In some cases, a cellular network could be used so that each communication system includes a cellular radio or other component that enables cellular communication. Using a cellular network may enable information to be exchanged while aerial system 204 is in the air where Wi-Fi or other networks might be unavailable. In other cases, network 280 could comprise any kind of local area network and/or wide area network. In some cases, network 280 may be a Wi-Fi network. Alternatively, ground system 202 and aerial system 204 could be connected by wires, for example, when aerial system 204 is on the ground and near ground system 202.

In different embodiments, one or more components of aerial system 204 could be disposed within a single computing device. Examples of computing devices that could be used include, but are not limited to: laptop computers, tablet computers, smartphones or other computing devices.

With reference now to FIGS. 3A and 3B, some examples of location factors 230 (see FIG. 2) are provided. In different embodiments, the path planning module can be in communication with external or internal knowledge repositories about a wide range of external environmental conditions, such as environmental conditions database 232. The environmental conditions database 232 can provide data regarding real-time or forecasted conditions that may affect high-resolution image capture and flight performance at the target location. Each of the data points described with reference to FIGS. 3A and 3B can be considered by the path planning algorithm to help determine the optimal path for the UAV to travel that takes into account the unique or specific characteristics of the target location.

While in some embodiments the information obtained by the path planning module is based on access to databases describing historical patterns of environmental conditions for a particular target location and/or predicted environmental conditions for the target location, in other embodiments, the information can be supplemented by sources that offer real-time data for the target location. For example, in some embodiments, data can be collected during manned or other unmanned aviation fly-bys over the target location to build a better localized and instantaneous report of near-current flight conditions rather than only on what is forecast. This can include raw weather data collected by some type of sensor suite including surface and airborne observations, radar, lightning, satellite imagery, and profilers.

In some cases, real-time weather information can be obtained via Flight Information Services (FIS), which is a mechanism for disseminating meteorological (MET) and aeronautical information (AI) to displays in the cockpit in order to enhance pilot situational awareness, provide decision support tools, and improve safety. FIS augments traditional pilot voice communication with Flight Service Stations (FSSs), ATC facilities, or Airline Operations Control Centers (AOCCs). FIS can provide textual and graphical information that can be used by the module to fine-tune the recommended flight path closer to the expected time of departure. Specifically, an FAA-sponsored system referred to as FIS-B may be accessed to receive and display a suite of broadcast weather and aeronautical information products. In other cases, non-FAA FIS Systems can be used to obtain FIS data over both the aeronautical spectrum and on other frequencies using a variety of data link protocols.

In some other examples, reported weather conditions applicable to travel over the target location can be obtained from one or more available sources, such as but not limited to METARs/SPECIs, PIREPs, RAREPs. In one embodiment, the system can monitor the ADS-B flight tracks (position, heading, speed, altitude) for planes flying over or near the target location and/or over a portion of the route to be taken based on the generated flight path to determine the environmental conditions. Such data sources offer real-time, highly accurate information about environmental conditions in the target location. One or more of these and other data sources can provide information that can cause the path planning module to make localized changes in the flight path based on selected parameters. The data can be collected immediately before (up to two or three hours) the selected flight path (“mission”) is sent to the UAV, as well as in real-time while the UAV is on its mission, in order to continually optimize the remainder of the mission for best photogrammetry. Thus, the path planning module can continue to modify or adjust the flight path not only prior to the mission, but during the flight itself.

As shown in FIG. 3A, some of these conditions may include weather 302, lighting conditions 304, and/or thermals 306. Each of these aspects can affect the performance of the UAV and detract from the success of its mission. In a first example, weather 304 can strongly impact lighting and flying conditions. Data for real-time or predicted weather conditions can be obtained by the path planning module and decisions regarding the path generated as well as the time at which the flight should occur can be based at least in part on such data. Weather conditions can encompass a variety of different factors. As a general matter, image noise decreases with decreasing temperatures. Noise is a random variation of image density, and the camera(s) will have a built-in image noise reduction system. The presence of noise gives an image an uneven grainy look, occluding details in the image. In cold weather, there is less noise as well as less water vapor, resulting in less haze.

Generally, extremely hot and cold temperatures represent adverse conditions for flying a drone. Both can cause adverse reactions for different constituents of your UAV. As a result, reduced flight performance and/or more rapid drain of the UAV battery may occur. In environments that are too hot, the motors will be made to work harder. In too cold environments, the efficiency of a battery decreases, in some cases to the point that the UAV battery drops below the critical voltage. Thus, in some embodiments, in response to receiving data describing extremes of temperature in the target location for the selected time and/or day, the path planning algorithm can be configured to plan shorter flights, or recommend an alternative time. Furthermore, temperature can affect visibility; for example, cold air may be preferable as it often prevents the build-up of haze.

Similarly, it is typically unadvisable for UAVs to fly when it is raining or snowing. The path planning algorithm may incorporate data regarding expected precipitation intensity and precipitation probability. Precipitation probability refers to probability of precipitation at the given time and location by considering whether the target area generally receives measurable precipitation and the confidence that precipitation will happen in any place in the forecasted area. In addition, overcast days can cause the lighting in the target area to take on a blue tinge. Cameras with an auto white balance feature can be fooled by this and overcompensate. The proposed path can include instructions for the camera to disable the white balance feature if such conditions are forecast. Similarly, the brightness of snow can affect exposure and color meters for the camera, leading to images that are underexposed, and white colors to appear gray or blue. Again, the path planning algorithm can include instructions in such cases to adjust the exposure meter and/or color meter of the camera.

As another non-limiting example, the quantity of water vapor which is present in the air can affect the flight performance of the UAV. If the humidity is very close to 1%, the UAV will become damp or wet. Monitoring of this weather metric may be useful if the operator recognizes that the UAV will be flying on multiple occasions in damp, humid conditions, which will eventually cause damage to the onboard equipment. In addition, UAVs typically do not handle moisture caused by fog very well and can obstruct views of the target structures. Thus, if fog is forecast for the target location, the system may recommend delaying the flight until the fog has passed.

In a second example, lighting 304 can strongly impact visibility, clarity, and resolution of the captured images. Data for real-time or predicted lighting conditions can be obtained by the path planning module and decisions regarding the path generated as well as the time at which the flight should occur can be based at least in part on such data. Expected lighting conditions can be based on a variety of different factors, such as atmospheric haze, heat shimmer, and seasonal variations (see FIG. 5). In general, the amount of moisture, dust and smoke particles visible in the air (haze) is a major factor affecting visibility. A low haze layer can have a detrimental effect on image quality. In some embodiments, the path planning algorithm will seek to output a path that instructs the UAV to fly on a day when there is minimal haze, and visibility remains clear for several miles. In other embodiments, the plan planning algorithm can suggest that UAV equipped with a camera featuring an ultraviolet (UV) filter that reduces atmospheric reflections or a polarizing filter, should be used for this particular mission to minimize the effect of haze. A polarizer will reduce reflections dependent upon your angle to the sun, and also tends to darken blue skies and whiten clouds. In another embodiment, the path planning algorithm can generate a proposed path that directs the UAV to fly at a lower altitude (where there is less atmosphere), and therefore less haze.

Furthermore, the presence of clouds can affect visibility. Overcast, cloudy conditions above the target area will tend to make an aerial scene look flat by diffusing the light. High, thin clouds can have little or great effect, depending on how much of it there is. In some cases, overall diffusion caused by an overcast sky can be of benefit for shooting structures on their normally-shaded side or when you wish to reduce overall contrast and have no harsh shadows. However, broken clouds can create unusual shadow patterns on the ground, which generally detract from the aerial images. Thus, in some embodiments, the path planning algorithm can determine, based on data indicating the target location will be associated with a great deal of cloud cover (e.g., around or above 30% or 40% of the sky is covered by clouds), that the mission should not be performed until another day.

In addition, if the sun is particularly bright at certain times of day, rising heat waves may cause a shimmering effect in the air, particularly near ground level above surfaces that absorb heat (like a road surface) or reflect light (like a shiny metal roof). Excessive shimmer can distort the view, and is more noticeable when using a long lens at fairly low altitude. In some embodiments, the path planning algorithm can be configured to delay the flight of the UAV to the target location until the heat waves are expected to decrease.

In a third example, thermals and wind 306 can strongly impact flying conditions (see FIG. 4 below). Data for real-time or predicted thermal conditions can be obtained by the path planning module and decisions regarding the path generated as well as the time at which the flight should occur can be based at least in part on such data. High winds and thermals (rising currents of warm air) can affect an aircraft's maneuverability, and potentially lead to bumpy image capture sessions. Furthermore, high wind speed makes it difficult for a drone to maintain its position, resulting in complicated maneuvering or shorter flight times. Wind can also push the UAV off course, forcing the motors to work harder in order to remain stable and on course. In response to data indicating high winds, the path planning algorithm can recommend postponing the flight until wind speed has slowed. However, a lack of wind can also create problems, leading to a build-up of haze, particularly on warm days over metropolitan areas. In such cases, the path planning algorithm can generate a proposed path instructing the UAV to fly at lower altitude to offset the effect of haze.

Other factors can also influence the path that will be selected by the path planning algorithm. In some embodiments, the path planning module can access information from structural and geographical characteristics repository 234, as shown in FIG. 3B. This information can be provided or made available by the insurance company which may keep records of structural details for its insured properties. In one example, the path planning module may make reference to a policy database of the requesting insurer or party to identify and retrieve information about the insured structures or structures that are otherwise relevant to the end-user and located within the target region. For example, upon learning that a disaster has occurred, an insurance company (or other party) may prepare a list of structures in the disaster area that are insured and thus require damage assessments to be performed.

In different embodiments, such data may include target property data 322, primary structure dimensions 324, primary structure features of interest 326, secondary structures associated with property 328, land attributes 330, no-fly zones 332, and/or known viewing obstacles or obstructions 334. Target property data 322 refers to location information. As a general matter, each structure may be associated with location information. As used herein, the term “location information” refers to any kind of information that can be find a geographic location for an object. Location information may include latitude and longitude information. Location information could also comprise a street address. It may be appreciated that location information provided in one format (for example, a street address for a structure) could be converted into another format (for example, a latitude and longitude position). When the location information is specific to a structure (such as a house, office building, or any other structure) the term “structure location” may be used. By obtaining a list of structure locations, the system can capture images of areas that include the structure locations. For example, an airplane may fly over the disaster area and photograph areas according to the provided list of structure locations. In addition, the structure may be associated with a point address or a locator code or name created for common addresses that contain a street number and street name. This locator role uses feature classes with polygon or point geometry as the primary reference data. Thus, each feature in the primary reference data corresponds to a single address.

Primary structure dimensions 324 refer to geo-spatial details for the target destination, including parcel boundaries and a centroid for the structure and any additional secondary structures associated with the structure, as well as other physical properties of the structure that may influence the path selected for image capture, such as the perimeter shape and size of the structure as observed from various angles and altitudes. In addition, primary structure features of interest 326 refer to physical features of the structure that may be tagged as being of particular value or significance to the objectives of the mission. In such cases, the proposed path generated by the path planning algorithm can include instructions for the UAV to position itself at a location adjacent to or with an otherwise unobstructed view of the tagged (“called-out”) feature of the structure. For example, in damage assessments, some features that may be of particular interest can include but are not limited to windows and their arrangement and location on the structure, pools, sunrooms, solar panels, decks, shutters, and other exterior facets of the structure such as balconies, sidings, gutters, roof tiles, external stairways, paved areas adjacent to the structure, rails, posts, pillars, etc.

Secondary structures associated with property 328 refer to additional structures that are separate from the primary structure but nevertheless should be included during image collection. For example, pools, sheds, MIL house, arbors, garden structures, patios, gazebos, or any other out-building or structure can be identified and its dimensions and location information relative to the primary structure can be applied to alter the path selected and ensure images of all pertinent structures are clearly captured.

In addition, land attributes 330 refer to unusual or otherwise significant physical features of the land on which the structure is disposed, including but not limited to vegetation, trees, hills, etc. Such information can affect the altitude that can be flown by the UAV, and also represent additional targets that can be highlighted by the end-user for image capture. No-fly zones 332 generally refer to locations or portions of a target location that have been designated as regions where coverage is simply unnecessary or locations where the UAVs are not allowed to fly. Finally, known viewing obstacles 334 refer to aspects of the property that can affect whether the UAV can obtain clear, unobstructed images of the structure from particular angles or heights. Some possible obstacles can include awnings, walls or gates, shrubbery near to the structure, or other physical features that can block the view of the camera and/or cause a part of the structure to be in shadow from a particular viewpoint.

As noted earlier, the time of year (i.e., seasons) can affect the conditions in which the image is being taken. Thus, depending on the time of year, the path planning module can adapt the proposed path to optimize for the presence of photo-friendly light and wind conditions. Referring now to FIG. 4, an example of the effects of seasonality is depicted. The illustration presents a building 402 at a target location 400 and some possible variations in the location of the sun and day length, such as but not limited to sunrise, sunset, shadow length, sun angle, solar eclipse, sun position, sun phase, sun height, etc. Thus, lighting conditions can change dramatically through the course of the year. In FIG. 4, it can be observed that the movement of the sun between sunrises 410 and sunsets 420 move across the sky along different paths during the summer, winter, and spring/autumn seasons. Similarly, thermals 430 can move in one direction during winter and another during summer. Seasonality can thus represent an important factor in path planning. The path planning algorithm can be configured to automatically factor the seasonal variations in sun position and movement and thermal activity for the desired flight date when determining the optimal flight path.

Thus, as may be understood from FIGS. 2, 3A, 3B, and 4, where the path planning module has access to knowledge about the flight workspace, including details about the target area and any NFZs along the way, the path can be computed in an offline phase before its execution. Such data can facilitate the performance of the algorithm by shaping the path to accommodate the factors applicable to the target area. In some embodiments, portions of the target area may be covered with distinct resolutions through flights at different altitudes, according to the importance of each feature or subregion or based on the preferences of the end-user. The path planning module can generate multiple paths to evaluate their respective attributes such as efficiency, resolution, lighting, preferences, etc. and also take into account the properties of the UAV such as speed, size, start time, and so on. In some embodiments, the algorithm relies on the concept of 4D trajectory-based operations, i.e., a sequence of waypoints that consists of 3D coordinates and associated timestamps that a UAV passes. In one embodiment, during in the in-flight phase, as a redundancy mechanism, when the UAV enters the airspace, its trajectories can be adapted due to emergent events, such as bad weather or emergency operations, using in-flight CDR methods or back-up or alternate paths that can be provided to the onboard computing system. For example, avoiding obstacles with waypoint guidance can be accomplished by creating diversion points off the planned path and outside of the obstacle by recomputing segments or the entire set of waypoints.

In order to provide some context for the improvements offered by the proposed systems, FIG. 5 presents an example of a conventional image capture mission for the reader's review. In FIG. 5, a schematic view of an aerial vehicle 500 performing a survey of a region 502 following a disaster is shown. As used herein, the term “aerial vehicle” refers to any kind of plane, helicopter, drone, or other flying vehicles. In this exemplary embodiment, aerial vehicle 500, also referred to simply as vehicle 500, is a plane operated by a pilot. However, in other embodiments, vehicle 500 could be remotely operated. Vehicle 500 includes at least one camera 510 for capturing images. In the example shown in FIG. 5, camera 510 is used to capture an image of a target area 520. Vehicle 500 may also include additional systems to facilitate capturing, processing, and transmitting image information about one or more areas.

As shown in FIG. 5, the vehicle 500 flies directly over target area 520, which includes four houses, three of which (530, 532, 534) represent insured properties. The flight path is relatively straight, corresponding to a direct route over the structures and allowing for image capture from directly above the homes at different times (e.g., T1, T2, T3, etc.) along different points in the path. Thus, the images collected will be limited in what features they reveal. Even for images collected by other camera orientations, the path, being a generic flyover that may be used across multiple locations, does not accommodate specific aspects of the structures that may need to be more clearly photographed, or adjust for obstacles that may obstruct the view of the structures.

As noted earlier, the path planning algorithm can receive information about the structure dimensions and other structural characteristics. It can be appreciated that the shape of the area of interest can be critical to coverage path planning. In FIG. 6A, one example of a footprint for a target area 610 is shown in which the path planning algorithm has generated a proposed path 620 that includes multiple waypoints 622. A footprint generally refers to the area on a target site that is used by a building structure and is defined by the perimeter of the building, or the horizontal projected area of the built structure from the roof down to ground level.

In different embodiments, the flight path can include instructions for operation of the camera or UAV maneuvering that should occur when the UAV reaches or approaches a particular waypoint. Depending on the features of the structure, the coverage path shape can vary from the shape of the target area 610, as shown in FIG. 6A, to accommodate the factors described above and ensure optimal image collection of the salient structural features. While the algorithm may sometimes generate rectangular areas or simplify the coverage area shape to a rectangle, in other cases the path over the target area may encompass more complex shapes such as concave and convex polygons representing irregular areas. It should also be appreciated that the optimal path may not include a closed shape, but rather segments extending from one waypoint to the next (see FIG. 7). Indeed, a target area can include different shapes and complexities. Generally, simple areas of interest, such as rectangular and convex polygons, do not require any discretization method or decomposition technique, and can typically be explored by back-and-forth and spiral flight patterns. The back-and-forth pattern usually defines the sweep direction based on the major axis and uses 90-degrees angles for turning maneuvers. In polygonal cases, the turning angle may vary both back-and-forth and spiral. In general, these patterns require low computational time to find coverage paths and are relatively easy to perform by the UAV. In larger and more complex areas of interest, an exact cellular decomposition may be applied to split the scenario into subregions that are explored by a series or sequence of pre-determined paths. The resulting sub-regions can be covered by different path directions to obtain an optimal coverage. In such areas, there is the possibility to explore four back-and-forth alternatives, varying the direction and the orientation, aiming at minimizing the distance among the subareas.

It can be appreciated that the path output by the algorithm will also instruct the UAV the height or altitudes at which the images should be captured. In FIG. 6B, a house 650 is shown for reference as a drone 660 moves up and down (vertically) between two points at A and B represented by a first distance 662. This difference in height can impact the angle at which the target structural feature is viewed and so affect the image that is captured. Depending on the feature that is being targeted (e.g., a roof corner 652), the altitude can significantly affect how the feature appears in the captured image. Similarly, the horizontal distance between the UAV and the structure will also be considered. Variations in the horizontal distance (i.e., moving horizontally closer to or further away from the structure) can also impact the angle at which the target structural feature is viewed and so affect the image that is captured, for example a second distance 690 between UAV at position “B” and UAV at position “D”. Depending on the feature that is being targeted (e.g., the roof corner 652), the distance from a structure can significantly affect how the feature appears in the captured image. Thus, both spatial attributes can be modulated by the algorithm to generate a path that offers the highest resolution and/or quality images for the target structure and its individual components or features of interest.

FIGS. 7-9 present two examples of pre-generated optimized paths that may be generated by some embodiments of the path planning algorithm and implemented by a UAV. In FIG. 7, a first vehicle 702 is shown as it performs its flight along a first optimized route (“first route”) 704 over a first target location 700 that includes a first house 710, a second house 720, a third house 730, and a fourth house 740. The houses have been damaged by a recent disaster such as a tornado or hurricane. This type of collection or survey can be performed across large areas of land to support the automated processing and generation of damage assessments for multiple structures in an affected region. In many cases, homes and other buildings in a city may have insurance policies that cover loss following a disaster such as a tornado.

In contrast to the relatively simple and nonspecific path traditionally traversed by drones during aerial photography sessions over multiple structural targets (see FIG. 5), the first route 704 has been fine-tuned and adapted, based on environmental and structural characteristics for first target location 700, to maximize the opportunity of the UAV to capture the most salient images as well as images of highest quality and resolution reflecting all features of interest.

The first vehicle 702 includes at least one camera. In one example, the camera can be configured to rotate or reorient its lens to capture images at different angles. In other examples, the vehicle itself can rotate to reorient the camera as needed. In FIG. 7, the directionality of the camera viewfinder is represented by a plurality of arrows extending from the first vehicle's base at each waypoint. In some embodiments, it can be of great importance that the imagery capture occurs at varying angles in order to, for example, collect sufficient image data to generate a 3D model. In some embodiments, the imagery can be obtained using both vertical imagery (nadir, 90 degrees) and oblique (45 degrees) imagery techniques. In general, vertical imagery offers an approximately straight-down aerial view of properties and locations, providing keen insight into rooftops and property surroundings, and at-a-glance situational awareness of large-scale catastrophe scenarios such as city-wide flooding. In contrast, oblique imagery provides an approximately 45° perspective of properties and locations from all four cardinal directions, allowing viewers to see and measure not only the top of objects but the sides as well (e.g., the external walls and windows of a building). In some embodiments, the pre-programmed route can include instructions for adjusting the orientation of the camera upon approaching or reaching specific waypoints. The two types of image techniques can be used to collect multiple images of the same structure.