120), and user activity and behavior data (stored, e.g., in the user activity and behavior database 122). The search engine 116 may collect, parse, and/or store data in an index or other similar structure to facilitate the identification and retrieval of information in response to received queries for information. This may include, but is not limited to, forward search indexes, inverted indexes, N-gram indexes, and so on.

As described above, example embodiments may be utilized for ranking and/or selection of job listings. These job listings may be posted by job posters (entities that perform the posting, such as businesses) and stored in job listing database 124.

FIG. 2 is a block diagram illustrating application server module 114 of FIG. 1 in more detail, in accordance with an example embodiment. While in many embodiments the application server module 114 will contain many subcomponents used to perform various different actions within the social networking system 110, in FIG. 2 only those components that are relevant to the present disclosure are depicted.

A content impression component 200 may receive, at runtime, one or more job listings and determine which users of the online network to present the job listings to as “impressions”. An impression is a single display of the job listing in a graphical user interface. There may be numerous ways these impressions may be presented and numerous channels on which these impressions may be presented. For example, the impressions may be presented in an email to a user, in a feed of the online network, or as results of a job search. Some of these impressions may be for job listings where the corresponding entity that posted the job listing has agreed to pay for the impression. Such impressions are called “sponsored impressions”. It should be noted that there is a distinction between the corresponding entity having agreed to pay for the impression and the corresponding entity actually paying for the impression. It is possible that the entity may have agreed to pay for an impression but, at the time the impression is made, the daily budget established by the entity has been used up, and thus it becomes possible for the sponsored impression to be displayed without an actual charge being applied to the entity's account. Alternatively, job posters may pay for clicks instead of impressions.

More particularly, companies may establish a daily budget they want to spend and a cost per impression based on a predicted number of impressions that they believe will occur each day. The actual display of the job listings, however, is based on a variety of factors that may vary based on the individual sets of users potentially served the job listing on a particular day.

All of these determinations may be considered by a sponsored content selector 202, which may consider information about the current user (i.e., the user for whom the piece(s) of content are being considered for display), the channel in which the display may occur, and the content itself (such as how closely features of the content match attributes of the user, or what the user is searching for (e.g., a search query submitted by the user)) to identify an ordered list of sponsored pieces of content available for display to the user in the channel. In this context, “channel” refers to a type of user interface or portion of a user interface on which content may be displayed. For example, a social networking service may provide multiple user interfaces or portions of user interfaces where job listings may be displayed to a user, including a job search channel (where users perform explicit searches for job listings, such as by specifying a job search query), a jobs-you-may-be-interested-in channel (where job listings are presented to a user who has not performed an explicit search, based on information the social networking service knows about the user), an email channel (where job listings are presented in emails rather than as part of a web-based user interface), and a feed channel (where job listings are presented as part of a content feed), where other types of feed content, such as news articles and updates about the user's connections, are intermingled.

In an example embodiment, the sponsored content selector 202 implements a model that determines the ordered list of sponsored pieces of content, possibly from the features described above. In some example embodiments, this model may be a machine learned model that is trained by a machine learning algorithm. The training may include using training data, such as labeled pieces of content, to learn one or more weights assigned to each of one or more features, such as user features, content features, and channel features. In an example embodiment, the machine learning algorithm may be selected from among many different potential supervised or unsupervised machine learning algorithms. Examples of supervised learning algorithms include artificial neural networks, Bayesian networks, instance-based learning, support vector machines, random forests, linear classifiers, quadratic classifiers, k-nearest neighbor, decision trees, and hidden Markov models. Examples of unsupervised learning algorithms include expectation-maximization algorithms, vector quantization, and information bottleneck method. In an example embodiment, the model implemented by the sponsored content selector 202 bases its selection on a combination of the relevance of the sponsored pieces of content to the user and/or the search performed by the user and how efficiently displaying each of the sponsored pieces of content will utilize the budgets set for the sponsored pieces of content by the job posters.

Regardless of how it is determined, the sponsored content selector 202 outputs an ordered list of sponsored pieces of content.

Separately, an organic content selector 204 identifies an ordered list of organic pieces of content available for display to the user in the channel. This may be performed using, for example, information about the current user, the channel in which the display may occur, and the content itself (such as how closely features of the content match attributes of the user, or what the user is searching for (e.g., a search query submitted by the user). The organic pieces of content may be pieces of content posted by job posters who have not elected to sponsor the pieces of content.

In an example embodiment, the organic content selector 204 implements a model that determines the ordered list of organic pieces of content, possibly from the features described above. In some example embodiments, this model may be a machine learned model that is trained by a machine learning algorithm. The training may include using training data, such as labeled pieces of content, to learn one or more weights assigned to each of one or more features, such as user features, content features, and channel features. In an example embodiment, the machine learning algorithm may be selected from among many different potential supervised or unsupervised machine learning algorithms. Examples of supervised learning algorithms include artificial neural networks, Bayesian networks, instance-based learning, support vector machines, random forests, linear classifiers, quadratic classifiers, k-nearest neighbor, decision trees, and hidden Markov models. Examples of unsupervised learning algorithms include expectation-maximization algorithms, vector quantization, and information bottleneck method. In an example embodiment, the model implemented by the organic content selector 204 bases its selection on the relevance of the organic pieces of content to the user and/or the search performed by the user.

The ordered list of sponsored pieces of content and the ordered list of organic pieces of content are then passed to a blending component 206. The blending component 206 determines an ordering in which to present a combination of sponsored pieces of content and organic pieces of content. This may be performed in a number of different ways. In one example embodiments, there are fixed slots in a list of results that are assigned to either sponsored or organic results. For example, if a graphical user interface is designed to output ten results at a time, a fixed slot assignment may indicate that the first two results on every results page are organic results, the next two are sponsored results, the next three are organic results, and so on. In an alternative embodiment, the slot assignments are dynamically adjusted. In a further example embodiment, the blending component 206 may remove any duplicate pieces of content using a duplicate remover. Unlike pure advertisements, sponsored pieces of content only vary from organic pieces of content by virtue of their respective poster being willing to pay for an impression. Thus, it is possible for a piece of content, such as a job listing, to be both sponsored and organic. This situation can arise, for example, if a company is willing to pay a social networking service to have an impression of the job posting be displayed in results on the social network service, but where the company has also posted the same job posting on a third-party web site which is crawled by the social networking service to obtain job listings which are unpaid. In an example embodiment, if the same piece of content exists in both the ordered list of sponsored pieces of content and the ordered list of organic pieces of content, blending component 206 removes that piece of content from the ordered list of organic pieces of content. In another example embodiment, if the same piece of content exists in both the ordered list of sponsored pieces of content and the ordered list of organic pieces of content, the lower ranked piece of content is removed.

The runtime output of the blending component 206 is an ordered list of results, likely with a mixture of sponsored content and organic content, which can be sent to a graphical user interface for display to a user.

The budgets sent to the sponsored content selector 202 may be sent via a budgeting GUI 208. The budgeting GUI 208 provides a place for content posters to enter their budget for sponsored placement of the content. As will be described in more detail later, in an example embodiment the budgeting GUI 208 may also provide a place where the content posters are able to see forecasts for performance of their content (based on a performance-based metric), and use this information to set the respective budgets for the pieces of content. In one example embodiment, a slider may be provided where a content poster 212 can increase or decrease the amount of the budget for a piece of content, and see a dynamically updated forecast of performance based on the budget indicated by the current position of the slider.

The forecast of performance for a particular budget may be obtained from the content performance forecasting model 210. The content performance forecasting model 210 uses high-budget and low-budget versions of the same pieces of content, on different randomly selected subgroups of users, and based on those results the content performance forecasting model 210 is trained to accurately predict content performance without bias. This production of training data includes promoting the first piece of content to users in the first subgroup using the budget value assigned to the first subgroup, which includes assigning a bid for each of one or more impressions of the first piece of content based on the budget value assigned to the first subgroup. This bid is then used by a sponsored content selector 202 to determine how high to rank the first piece of content among other sponsored pieces of content to display to users in the first subgroup, an impression being a single display of the first piece of content to a single user. A similar technique is used to promote the first piece of content to users in the second subgroup.

FIG. 3 is a diagram illustrating the content performance forecasting model 210 in accordance with an example embodiment. A budget assigner 300 may obtain content and associated budget. In other words, each piece of content has its own budget. These pieces of content may be ones that are selected for use in training the content performance forecasting model 210. For each of these pieces of content, the budget corresponding to the piece of content may then be split asymmetrically in two (i.e., in unequal amounts). The higher budget portion may be called “high budget” and the lower budget portion may be called “low budget”, although this is not indicative of any requirement that the respective budgets be any particular levels. Any split is permitted, and thus a 51-49 split (where one portion represents 51% of the original budget and another portion represents 49% of the original budget) would still result in 51% portion being called “high budget” and the 49% portion being called “low budget”. Likewise a 98-2 split is also possible. Nevertheless, a typical range may be somewhere between 50-50 and 70-30.

In an example embodiment, the split is based on a fixed amount selected by an administrator, such as 60-40 split. This may be selected based on empirical observation. In another example embodiment, however, the split may be dynamically determined using a rules-based algorithm that applies rules to one or more metrics to determine a split, or learned via a machine-learning algorithm.

The asymmetric budget split fraction may be defined as a, namely the split needed to achieve a desired level of statistical precision in the incrementality estimate. The incrementality estimate comes from comparing performance of high and low budget job versions. Hence, the variation from a binary treatment BudgetHigh_k∈{0, 1} may be used to determine the level of a needed to reach a desired level of statistical precision. In the following, an analysis for this binary treatment relating it to the estimate of incrementality is conducted.

For the binary treatment BudgetHigh_kconsider the following regression where k denotes a job posting version:

Applies^k=cons+β×BudgetHigh_k+ε_k (1a)

The coefficient β is related to incrementality δ (defined as δ=d Number of Applies/d Budget) through the equation

β=δ·(2α−1)·Budget, (2a)

where Budget is a job listing's daily budget and α is the fraction of a job listing's budget assigned to the high-budget job listing version as defined above.

Derivation: denote by ƒ( ) a job listing version's value delivery function, that maps budget into expected number of applies. Then β=ƒ(α·Budget)−ƒ((1−α)·Budget)=(α−(1−α))·δ·Budget+((1−α))·Budget)−ƒ((1−α)·Budget)=(2α−1)·δ·Budget, where the second equality assumes linearity in the neighborhood [α·Budget, (1−α)·Budget] and uses the definition of incrementality δ.

Suppose one wants the 95% confidence interval of the incrementality estimate to be a fraction φ of the level of δ(i.e., [(1−φ)·δ, (1+φ)·δ]). As β and δ are linearly related, a power calculation can be performed for β and it can be related to incrementality δ through equation (2a).

For the necessary precision in terms of p (defined as the randomization fraction, typically being 0.5), one can calculate the minimum detectable effect size (MDE) for β as if one attempted to reject the null-hypothesis of β=0 with a threshold for the t-statistic of 1.96/φ. (This is because

1
.
9

⁢
6

=

β
-

(

1
-
ϕ

)

⁢
β

SE
⁡
(

β
^

)

=

ϕ
⁢
β

SE
⁡
(

β
^

)

=

ϕ
⁢
t

⇒
t

=

1
.
9

⁢
6

ϕ

.

)

Hence:

(

3
⁢
a

)

⁢

MDE
⁡
(
β
)

=

(

1.96
ϕ

+
0.8

)

·

σ
2

N
⁢

p
⁡
(

1
-
p

)

,

where all parameters are defined as in the table below.

Using (β)=M(δ)×(2α−1)×Budget, one arrives at

α
=

(

1
+

(

1.96
ϕ

+
0.8

)

·

σ

N
⁢

p
⁡
(

1
-
p

)

·

1

M
⁢
D
⁢

E
⁡
(
δ
)

×
B
⁢
u
⁢
d
⁢
g
⁢
e
⁢
t

)

/

2
.

Component
Info

φ
Desired width of the 95% confidence interval estimate

σ
Standard deviation of applies (or any other performance

metric) per budget split market (one submarket), typcioally

winsorized at the 95th percentile

Np(1 − p)
N: number of active jobs with a budget. p = 0.5 (equal

randomization)

MDE(δ)
Estimate of incrementality to detect

Budget
Average budget among all active jobs with a budget

α
Resulting level of the asymmetric budget split fraction

Since each piece of content has its budget split into two unequal portions, the result is that there are two versions of each piece of content: a “high budget” version and a “low budget” version. Each of these versions is then randomly assigned to a different subgroup of users. More particularly, the asymmetric budget assigner 300 obtains a list of users from a user database 302. This list of users may be ones selected to be used to train the content performance forecasting model 210, although in some embodiments all available users in the online network may be used for this purpose and thus the list of users is a list of all available users. Regardless, the asymmetric budget assigner 300 randomly assigns the users in the list of users into either a first subgroup or a second subgroup. These subgroups may be the same size (or at least substantially the same size; if the list of users has an odd number of users in it, then it would be impossible for the subgroups to be exactly the same size).

The high-budget version of a piece of content is then randomly assigned to either the first or second subgroup, with the low-budget version assigned to the subgroup that the high-budget version was not. The content impression component 200 then serves the version of the piece of content to users in the subgroup to which that version is assigned. Thus, for example, if the high-budget version is assigned to the first subgroup and the low-budget version is assigned to the second subgroup, then the high budget is used to place the piece of content as sponsored pieces of content served to users in the first subgroup, and the low budget is used to place the piece of content as sponsored pieces of content served to users in the second subgroup.

The results from both subgroups are then received from the content impression component 200 and may be designated as training data 304. The training data 304 may then be fed as input to a machine learning algorithm 306 to train a content performance model 308. The content performance model 308 is trained to output a forecast of performance according to a performance-based metric of some sort. In one example embodiment, the performance-based metric is applications to a job associated with a job posting (where the job posting is the piece content). In another example embodiment, the performance-based metric is clicks on the piece of content.

The training may include using the training data 304, but may also use features extracted from other data, such user features, content features, and channel features, or features obtained from other models, such as from an auction entry model 310 and/or engagement model 312. In an example embodiment, the machine learning algorithm 306 is a linear regression algorithm. In another example embodiment, the machine learning algorithm 306 may be selected from among many different potential supervised or unsupervised machine learning algorithms. Examples of supervised learning algorithms include artificial neural networks, Bayesian networks, instance-based learning, support vector machines, random forests, linear classifiers, quadratic classifiers, k-nearest neighbor, decision trees, and hidden Markov models. Examples of unsupervised learning algorithms include expectation-maximization algorithms, vector quantization, and information bottleneck method.

FIG. 4 is a screen capture illustrating a first screen 400 of a budgeting GUI 208 in accordance with an example embodiment. On this first screen 400, a recommended budget 402 is displayed along with a button 404 that the user may click on to edit the recommended budget to a custom budget. Also displayed is an estimated amount of applicants/month 406 based on the recommended budget 402. This estimated amount of applicants/month 406 is obtained by the budgeting GUI 208 from the content performance forecasting model 210, which reflects a forecasted value based on the budget (in this screen, the recommended budget 402).

FIG. 5 is a screen capture illustrating a second screen 500 of the budgeting GUI 208 in accordance with an example embodiment. Here, the user has selected button 404 from FIG. 4, which brings up a drop down button 502 where the user can select a time period for which the budget will cover (e.g., daily, weekly, monthly). Also displayed is an input box 504 where the user can type in an exact budget number, but a slider bar 506 is also provided that the user can select to lower and raise the budget.

FIG. 6 is a screen capture illustrating a third screen 600 of the budgeting GUI 208 in accordance with an example embodiment. Here, the user has moved the slider bar 506 down to reduce the budget, which has caused a corresponding change in the estimated amount of applicants/month 406. This estimated amount is based on the forecast from the content performance forecasting model 210, using the budget provided by the user based on the position of the slider bar 506.

FIG. 7 is a screen capture illustrating a fourth screen 700 of the budgeting GUI 208 in accordance with an example embodiment. Here, the user has moved the slider bar 506 up to increase the budget, which has caused a corresponding change in the estimated amount of applicants/month 406. This estimated amount is based on the forecast from the content performance forecasting model 210, using the budget provided by the user based on the position of the slider bar 506.

In other example embodiments, the incrementality estimates may be used in other ways other than as part of a forecasting model. For example, budget-aware forecasts may be built based on segment-level estimates of incrementality, each segment being a different category of content. Here, first, heterogenous incrementality estimates are obtained at the segment level (i.e., as a function of job listing features). These incrementality estimates are derived by comparing the difference in the performance metric between high and low budget job versions relative to the difference in budget between the high and low budget job versions. Second, these segment-level estimates can be directly used for budget-aware forecasts.

In another example, budget-aware forecasts may be built based on job posting-level simulations calibrated through market-level incrementality estimates, as derived as explained above. Here, an auction replay for an individual job posting may be performed. An auction replay is a tool for simulation of content placement and display, where auctions are used to determine how to place content, such as where content is placed more prominently if the content provider bids higher for display of the content than other content being displayed. Auction replay uses historical data of such auctions and simulates the outcome of these past auctions if a piece of content had a different budget level. This allows simulation of performance of a piece of content (as measured by, for example, number of impressions, clicks, applies, etc.) at different budget levels. In sum, auction replay is a simulation tool that outputs data on performance at different hypothetical budget levels for each job listing.

One technical issue with auction replay is that it can be biased. More particularly, the simulated mapping between budget and performance may not correspond to the actual mapping between budget and performance, due to the issues mentioned earlier. This may be addressed by calibrating the auction replay through incrementality estimates obtained from causal estimation based on data obtained through asymmetric budget split (to ensure that simulated performance of job postings in the simulation corresponds to performance of job postings estimated based on data obtained through asymmetric budget split).

FIG. 8 is a flow diagram illustrating a method 800 for training a machine-learned model in accordance with an example embodiment. At operation 802, a first piece of content and a first budget for the first piece of the content are obtained. In an example embodiment, the first budget is obtained from a first content poster, but in other example embodiments the first budget may be obtained from other sources, or created by the online service operating the method 800. The first budget is indicative of a first value the first content poster is willing to pay over a particular time period to promote the first piece of content. At operation 804, the first budget is split by value into a high budget value and a low budget value, the high budget value being different and higher than the low budget value. At operation 806, each user in a first group of users is randomly assigned into a first subgroup or a second subgroup. At operation 808, the high budget value is randomly assigned to either the first subgroup or the second subgroup and the low budget value is assigned to the subgroup to which the high budget value was not assigned.

At operation 810, the first piece of content is promoted to users in the first subgroup using the budget value assigned to the first subgroup. At operation 812, the first piece of content is promoted to users in the second subgroup using the budget value assigned to the second subgroup. At operation 814, results produced by promoting the first piece of content to users in the first subgroup and results produced by promoting the first piece of content to users in the second subgroup are designated as training data. Designating something as training data may be accomplished in a variety of ways, from having a field in a record indicating the data is training data, to simply using the training data as input to a machine learning algorithm used to train a model. It should be noted that while the above steps describe performing the asymmetric budget split for a single piece of content (the first piece of content), in an example embodiment, these steps are repeated for multiple pieces of content, to create a more complete set of training data. Indeed, in some example embodiments, the asymmetric budget split operations are performed on a large number of pieces of content, possibly all pieces of content that have budgets. At operation 816, a first machine-learned model is trained using a machine learning algorithm by feeding the training data into the machine learning algorithm. This training includes training the first machine-learned model to forecast a value for a performance-based metric for a given piece of content with a given budget. It should be noted that while this figure depicts a single training operation, in an example embodiment, the machine-learned model may be trained multiple times. This allows the model to be retrained when, for example, new training data is created or user feedback about the performance of the machine-learned model is received.

At operation 818, a graphical user interface is caused to be rendered. The graphical user interface may include a first screen where a user can select a second piece of content and a second screen where a user can adjust a budget for the second piece of content. At operation 820, upon receiving user input adjusting the budget for the second piece of content, information about the second piece of content and the adjusted budget for the second piece of content are sent to the first machine-learned model to receive a forecast of a performance-based metric, the graphical user interface further displaying the forecast for the adjusted budget. The second screen includes a slider that the user can slide in a first direction or a second direction, wherein sliding the slider in the first direction adjusts the budget downwards and wherein sliding the slider in the second direction adjusts the budget upwards.

FIG. 9 is a block diagram 900 illustrating a software architecture 902, which can be installed on any one or more of the devices described above. FIG. 9 is merely a non-limiting example of a software architecture, and it will be appreciated that many other architectures can be implemented to facilitate the functionality described herein. In various embodiments, the software architecture 902 is implemented by hardware such as a machine 1000 of FIG. 10 that includes processors 1010, memory 1030, and input/output (I/O) components 1050. In this example architecture, the software architecture 902 can be conceptualized as a stack of layers where each layer may provide a particular functionality. For example, the software architecture 902 includes layers such as an operating system 904, libraries 906, frameworks 908, and applications 910. Operationally, the applications 910 invoke API calls 912 through the software stack and receive messages 914 in response to the API calls 912, consistent with some embodiments.

In various implementations, the operating system 904 manages hardware resources and provides common services. The operating system 904 includes, for example, a kernel 920, services 922, and drivers 924. The kernel 920 acts as an abstraction layer between the hardware and the other software layers, consistent with some embodiments. For example, the kernel 920 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 922 can provide other common services for the other software layers. The drivers 924 are responsible for controlling or interfacing with the underlying hardware, according to some embodiments. For instance, the drivers 924 can include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth.

In some embodiments, the libraries 906 provide a low-level common infrastructure utilized by the applications 910. The libraries 906 can include system libraries 930 (e.g., C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries 906 can include API libraries 932 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render in two dimensions (2D) and three dimensions (3D) in a graphic context on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. The libraries 906 can also include a wide variety of other libraries 934 to provide many other APIs to the applications 910.

The frameworks 908 provide a high-level common infrastructure that can be utilized by the applications 910, according to some embodiments. For example, the frameworks 908 provide various graphical user interface functions, high-level resource management, high-level location services, and so forth. The frameworks 908 can provide a broad spectrum of other APIs that can be utilized by the applications 910, some of which may be specific to a particular operating system 904 or platform.

In an example embodiment, the applications 910 include a home application 950, a contacts application 952, a browser application 954, a book reader application 956, a location application 958, a media application 960, a messaging application 962, a game application 964, and a broad assortment of other applications, such as a third-party application 966. According to some embodiments, the applications 910 are programs that execute functions defined in the programs. Various programming languages can be employed to create one or more of the applications 910, structured in a variety of manners, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a specific example, the third-party application 966 (e.g., an application developed using the ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or another mobile operating system. In this example, the third-party application