250 would generate six ticket records in ticketing database 255. Ticket management system 250 would submit a request to reservation management system 240 to update the reservation record stored in reservation database 245 to include six ticket numbers that identify each ticket record generated. That is, in this example, a single reservation record stored in reservation database 245 would include ticket numbers identifying six ticket records stored in ticketing database 255.

FIG. 3 is a block diagram of an example revenue management system (“RMS”) 300 (e.g., RMS 140 of FIG. 1) that is suitable for implementing aspects of the present invention. RMS 300 is generally configured to optimize one or more objective metrics by executing automated, data-driven decisions concerning travel services inventory using an active learning framework to iteratively train demand models. In an exemplary implementation, RMS 300 includes: training set compiler 310, demand model trainer 320, forecasting service 330, optimization service 340, auditing service 350, and reinforcement learning service 360. In an embodiment, RMS 300 is hosted by computing resources provided by reservation environment 200 of FIG. 2. Alternatively, in some implementations, one or more of the modules as shown as part of the RMS 300 (e.g., training set compiler 310, demand model trainer 320, forecasting service 330, optimization service 340, auditing service 350, and reinforcement learning service 360) may be implemented as separate components, servers, computers, etc. For example, the, auditing service 350 and/or the reinforcement learning service 360 may be systems that are interpedently controlled outside of the RMS 300.

Training set compiler 310 is configured to populate, compile, or build training sets of booking data by interacting with reservation management system 240. Demand model trainer 320 is configured to train one or more demand models using training sets obtained from training set compiler 310. In an embodiment, demand model trainer 320 is implemented using a machine learning algorithm. Forecasting service 330 is configured to generate estimated booking data for active travel services using demand models trained by demand model trainer 320.

Optimization service 340 is configured to select one or more inventory control attributes that maximize one or more objective metrics of reservation environment 200 using estimated booking data generated by forecasting service 330. Optimization service 340 is further configured to interact with inventory management system 230 to include the selected one or more inventory control attributes.

Auditing service 350 is configured to detect variances between estimated booking data generated by forecasting service 330 and transient booking data stored in reservation database 245 corresponding to active travel services departing on a future date (“active travel services”). Auditing service 350 is further configured to activate reinforcement learning service 360 in response to detecting a variance between estimated booking data and transient booking data that exceeds a calculated threshold. In an embodiment, auditing service 350 includes a demand shock detection process. In an embodiment, an auditing service 350 generates a confidence cone by aggregating a set of probabilities computed for multiple flights with each probability estimating a likelihood that a given flight among the multiple flights will receive a given number of bookings by a given day-to-departure (“DTD”) based on a demand forecast obtained using the demand model trainer 320.

When activated, reinforcement learning service 360 is configured to increase price disparity within an updated training set by increasing a diversity of observations of price-demand pairs. To that end, reinforcement learning service 360 intercepts the one or more inventory control attributes selected by optimization service 340 and processes those attributes according to an exploration-exploitation strategy. The exploration-exploitation strategy includes an operating mode of reinforcement learning service 360 transitioning between an exploration mode and an exploitation mode. In an embodiment, reinforcement learning service 360 has a given probability (e.g., 5%) of transitioning to the exploration mode; otherwise, an operating mode of reinforcement learning service 360 is the exploitation mode. In an embodiment, reinforcement learning service 360 computes the optimal probability of transitioning to the exploration mode to maximize the net value of the gain of information minus the cost of exploration.

The exploitation mode of reinforcement learning service 360 defines an operating mode in which reinforcement learning service 360 forwards the one or more inventory control attributes selected by optimization service 340 to inventory management system 230. The exploration mode of reinforcement learning service 360 defines an operating mode in which reinforcement learning service 360 replaces the one or more inventory control attributes selected by optimization service 340 with alternative inventory control attributes. For example, the selected inventory control attribute may define pricing information for a given transportation service as € 250. In this example when in the exploration mode, reinforcement learning service 360 may replace the € 250 pricing information defined by the selected inventory control attribute with € 300 pricing information to create an alternative inventory control attribute. Upon replacing the inventory control attributes selected by optimization service 340, reinforcement learning service 360 forwards the alternative inventory control attribute(s) to inventory management system 230.

FIG. 4 illustrates a conceptual, high-level overview of using booking data in demand forecasting and resource optimization. As discussed above, demand forecasting techniques involving models trained by machine learning and/or predictive analytic methodologies may be implemented in various contexts, including travel reservation systems like reservation environment 200. Booking data stored in reservation database 245 facilitates various aspects of demand forecasting and resource optimization for reservation environment 200. As illustrated in FIG. 4, a line 410 associated with a current departure date partitions historical or archived booking data 420 corresponding to previously departed travel services (“inactive travel services”) from transient booking data 430 corresponding to travel services departing on future dates (“active travel services”) and estimated booking data 440 corresponding to forecasting data for active travel services (e.g., generated by forecasting service 330).

A demand model trainer (e.g., demand model trainer 320 of FIG. 3) can apply machine learning and/or predictive analytic methodologies (e.g., regression algorithms) to historical booking data 420 to train demand models. Demand models trained using historical booking data may then be used to select one or more inventory control attributes for active travel services associated with transient booking data 430 that maximize one or more objective metrics of the revenue management system 300. Such inventory control attributes may include attributes configured to: adjust availabilities among multiple fare classes for specific active travel services, cancel an active travel service to reduce supply, and the like. Examples of objective metrics include: utilization rate per segment, utilization rate per flight, revenue per segment, revenue per flight, revenue per passenger-mile, and the like.

FIG. 5 illustrates an example of a demand model 510 that could be trained using historical booking data 420. As illustrated in FIG. 5, demand model 510 estimates demand as a function of price. One skilled in the art will appreciate that a function characterizing demand model 510 includes a number of parameters (e.g., price elasticity, product attractiveness, competitor pricing, competitor switching sensitivity, and the like) that estimate various underlying market factors that impact demand. Such parameters define a shape of demand model 510, and thus impact a measure of how well demand model 510 fits a set of observations provided by historical booking data 420. Misestimating parameters of demand model 510 may thereby negatively impact an ability of demand model 510 to forecast future demand.

FIG. 6A and FIG. 6B illustrate data sparsity issues that may impact demand model training. For example, in some instances, historical booking data may lack sufficient data points (or observations) to train a model that accurately forecasts future demand. A comparison between FIGS. 6A and 6B illustrates how such data sparsity issues may impact demand model training. FIG. 6A illustrates an example graph 600 that includes a horizontal axis 602 corresponding to price and a vertical axis 604 corresponding to demand. The historical data presented in graph 600 includes observations of price-demand pairs largely centered about a mid-point of horizontal axis 602. As seen in FIG. 6A, machine learning and/or predictive analytic methodologies may train a wide range of demand models from such sparse historical data. In some instances, demand models trained on such sparse historical data may conflict with generally recognized principles. For example, demand model 610 estimates that increasing price of a resource will increase demand for that resource.

FIG. 6B illustrates an example graph 650 that includes a horizontal axis 652 corresponding to price and a vertical axis 654 corresponding to demand. Unlike the historical data presented in graph 600, the historical data presented in graph 650 includes observations of price-demand pairs that are both increased in number and more distributed across horizontal axis 652. That is, the observations of price-demand pairs presented in graph 650 are more diverse than those presented in graph 600. As seen in FIG. 6B, machine learning and/or predictive analytic methodologies may still train a range of demand models from such historical data. However, a comparison between FIGS. 6A and 6B illustrates that the range of demand models trained from the historical data presented in graph 650 is narrower than the range of demand models trained from the historical data presented in graph 600. This comparison suggests that a positive correlation exists between the diversity of training data provided to machine learning and/or predictive analytic methodologies and the precision of models obtained from such methodologies.

FIG. 7 is a flowchart illustrating an example method 700 of implementing a reinforcement learning framework for iteratively training a demand model, in accordance with an embodiment of the invention. In an embodiment, method 700 is implemented by revenue management system 300 of FIG. 3.

At step 701, estimated booking data is generated for an initial time with a demand model trained using a training set of historical booking data. In an embodiment, generating the estimated booking data includes computing a probability that a travel service or flight will receive a given number of bookings by a given day-to-departure (“DTD”) based on a demand forecast obtained using the demand model.

At step 703, detecting a variance between the estimated booking data and transient booking data observed at the initial time that exceeds a defined threshold. In an embodiment, detecting the variance includes an outlier detection process identifying observations within the transient booking data that diverge from the estimated booking data. In an embodiment, the outlier detection process generates a confidence cone by aggregating a set of probabilities computed for multiple flights with each probability estimating a likelihood that a given flight among the multiple flights will receive a given number of bookings by a given DTD based on a demand forecast obtained using the demand model. In an embodiment, the necessary threshold is computed to detect a demand shock of a given magnitude at a given statistical accuracy within a desired detection time.

At step 705, activating a reinforcement learning service in response to detecting the variance. In an embodiment, the reinforcement learning service is configured to transition between an exploitation mode and an exploration mode. In an embodiment, the transition between the exploitation and exploration mode is determined by computing the entropy of the training data set to determine an uncertainty of the demand model estimated from the training data set. In an embodiment, the reinforcement learning service includes an Epsilon-Greedy algorithm. In an embodiment, the reinforcement learning service determines which fare classes to explore when the exploration mode is activated.

In an exemplary embodiment, step 705 may include estimating a cost of exploring at a given fare class by computing the difference in expected revenue between offering the exploratory fare class and offering the fare class that would have been selected by the exploitation mode, and selecting a fare class to explore that maximizes the expected improvement in demand model accuracy as a result of increased price diversity in the updated training set while not exceeding a maximum exploration cost threshold.

Additionally, or alternatively, in an embodiment, the reinforcement learning service includes an offer threshold process configured to effectuate the exploration mode by modifying offer frequencies of multiple fare classes based a respective number of offers for each fare class identified within the historical booking data. In an embodiment, modifying the offer frequencies of the multiple fare classes includes reducing an offer frequency of a given fare class with a highest number of offers among the multiple fare classes. Additionally, or alternatively, in an embodiment, modifying the offer frequencies of the multiple fare classes includes increasing an offer frequency of a given fare class with less than a pre-defined minimum number of offers.

At step 707, creating an updated training set including enhanced booking data observed at a subsequent time after activating the reinforcement learning service. For example, enhanced booking data may include more diverse price-demand observations than if the reinforcement learning service was not activated, due to price exploration conducted by the reinforcement learning service. For example, the increased price diversity in the updated training set may allow for more accurate demand model estimation, as seen in FIG. 6B.

At step 709, updating a parameter of the demand model by training the demand model using the updated training set. In an embodiment, the parameter includes: a price elasticity parameter, a product attractiveness parameter, a demand volume parameter, or a temporal adjustment parameter. Additionally, in some implementations, the parameters may further include a competitor pricing parameter, a competitor switching sensitivity parameter, a substitute resource pricing parameter, a complement resource pricing parameter, or a seasonal adjustment parameter.

In an embodiment, method 700 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In an embodiment, method 700 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

FIG. 8 illustrates a graphical view comparing active booking data with estimated booking data generated using a demand model, in accordance with an embodiment of the invention. In particular, FIG. 8 presents graph 800 that includes a horizontal axis 802 corresponding to days to departure and a vertical axis 804 corresponding to current bookings. In general, graph 800 illustrates a comparison between estimated booking data generated by forecasting service 330 and transient booking data stored in reservation database 245 corresponding to active travel services. Each observation or data point (e.g., observations 822, 824, and/or 826) in graph 800 represents a particular travel service (e.g., flight) that is scheduled to depart on a future date.

As seen in graph 800, each travel service typically has a minimal number of bookings (e.g. zero bookings) when initially released by inventory management system 230 for booking on or about 350 days to departure (or other various departure days, depending on the airline). Over time the number of bookings associated with each active travel service increases as available spaces on a given travel service are booked and transitioned from unreserved travel services inventory to reserved travel services inventory. Spaces in unreserved travel services inventory generally remain unbooked when inventory management system 230 closes booking for each travel service on or about zero days to departure. By tracking a number of bookings for each travel service from a time that inventory management system 230 initially releases a given travel service for booking to a time that inventory management system 230 closes booking, the observations in graph 800 collectively represent transient booking data (e.g., transient booking data 430 of FIG. 4).

In FIG. 8, that transient booking data (e.g., transient booking data 430) is overlaid by the estimated booking data (e.g., estimated booking data 440) generated by forecasting service 330, which is represented by confidence cone 810. As illustrated in FIG. 8, a substantial number of observations in the transient booking data (including observations 822, 824, and 826) exist external to an upper confidence bound 812 and a lower confidence bound 814 of confidence cone 810. That is, a substantial number of observations diverge from confidence cone 810. If that substantial number of observations exceeds a defined threshold for activating reinforcement learning service 360, one or more parameters of the demand model associated with confidence cone 810 may be updated at a subsequent time by retraining the demand model using an updated training set including enhanced booking data with increased price diversity, as discussed above. In an embodiment, the updated training set is generated and the demand parameters are updated whether or not the exploration mode in reinforcement learning service 360 is activated. For example, the only difference is whether or not the prices used in the generation of the updated training set involve price experimentation or not. Upon retaining the demand model using the updated training set, confidence cone 910 may be generated.

FIG. 9 illustrates a graphical view comparing the estimated booking data from FIG. 8 with estimated booking data generated after updating one or more parameters of the demand model from an updated training set. In particular, FIG. 9 presents graph 900 that includes a horizontal axis 902 corresponding to days to departure and a vertical axis 904 corresponding to current bookings. As illustrated in FIG. 9, some observations in the transient booking data (e.g., 822 and 826) remain external to an upper confidence bound 912 and a lower confidence bound 914 of confidence cone 910. However, a comparison between FIGS. 8 and 9 illustrates that fewer observations in the transient booking data diverge from confidence cone 910 than diverged from confidence cone 810. For example, observation 824 exists internal to upper confidence bound 912 and lower confidence bound 914 of confidence cone 910. As such, the one or more parameters that were updated by retraining the demand model using the updated training set positively impacted the accuracy of demand predictions generated using the retrained demand model.

FIG. 10 is an example graph 1000 illustrating a graphical view comparing respective mean absolute percentage error of revenue management systems implementing passive and active learning frameworks over time, in accordance with an embodiment of the invention. For example, graph 1000 illustrates a mean absolute percentage error on the vertical axis over time (e.g., departure date) on the horizontal axis. In particular, graph 1000 illustrates a comparison for percentage variances between a forecast error in an estimated parameter for RMS that uses passive learning (e.g., passive learning results 1010), and a forecast error in an estimated parameter for an RMS that uses active learning (e.g., active learning results 1020). Thus, as illustrated, the active learning results 1020 generally have a lower forecasted error than the passive learning results 1010. In particular, the peaks 1022, 1024 for the active learning results 1020 have a forecast error that is reduced in magnitude than peaks 1012, 1014 for the passive learning results 1010.

FIG. 11 is an example graph 1100 illustrating a graphical view comparing respective average cumulative revenue results of revenue management systems implementing passive and active learning frameworks over time, in accordance with an embodiment of the invention. For example, graph 1100 illustrates a percentage difference in cumulative average revenue on the vertical axis over time (e.g., departure date) on the horizontal axis. In particular, graph 1100 illustrates a comparison for percentage differences between a revenue performance metric for a hypothetical psychic RMS with perfect knowledge of customer behavior (e.g., hypothetical optimal results 1110), and a revenue performance metric for an RMS with active learning (e.g., active learning results 1120) with respect to a RMS baseline with passive learning (e.g., passive learning results