How Yelp built a back testing engine for safer, smarter ad budget allocation

Topic was presented at Munich Datageeks - April 2026

Abstract

This talk presents a backtesting (simulation) engine built at Yelp to evaluate changes to its ad budget allocation system before committing to expensive, slow A/B tests. Yelp funds itself through advertisers who set monthly budgets that the platform translates into daily budgets, impressions, clicks and leads, with a feedback loop where each day's allocation decisions affect the rest of the month. Because Yelp's campaign volume (hundreds of thousands, not millions) and month-long budget cycles make A/B testing costly and slow, the team built a system to cheaply test many algorithm variants against historical data first. The talk walks through the engine's architecture: a YAML configuration defining the search space, the SigOpt optimizer proposing candidate parameters, git submodules linking directly into production code so simulations exactly match what will ship, historical data pulled from Redshift, and machine learning models that predict "what-if" outcomes under different budgets. Results and metrics are logged to MLflow for stakeholder review before promising candidates move to real A/B testing.

About the speaker

The speaker, Samuel, is an applied scientist on Yelp's Recommendation Intelligence team. He previously worked on Yelp's Ad Budget Allocation team, where the project described in this talk originated. Before joining Yelp, he worked as a data scientist across multiple industries for five years, and moved to Munich a couple of years ago to join Yelp. His academic background is in statistics. He is also a long-time blogger: he wrote for Towards Data Science for many years, and since that publication left Medium, he now writes for a different publication on Medium.

Transcript summary

Yelp's business model and advertising

Yelp is a platform connecting people with local businesses, popular mainly in North America (the US and Canada), with around 74 million people visiting its app or website each month and annual net revenue of roughly $1.5 billion. Like many other platforms, Yelp makes money through targeted advertising: advertisers set a budget, typically a monthly one (e.g. $300 for a small business), which Yelp uses to sell impressions and clicks to its users. When a user searches for something like "burgers" or "electronics," some of the results shown are sponsored, and this is what advertisers pay for.

The campaign journey and the feedback-loop problem

Although advertisers set a monthly budget, Yelp's system actually works by deciding a daily budget each day. Based on that daily budget, an ad receives impressions over the course of the day, which can convert to clicks and then to leads. At the end of the day, the recorded impressions, clicks, and leads are used to compute the daily billing for that campaign. The speaker's team was responsible for two parts of this process: setting the daily budget and where to allocate it, and computing the billing at the end of the day.

What makes this complicated is a feedback loop: the allocation decisions made on one day affect what happens on the following days, and so on for the rest of the month. This feedback loop is presented as the central reason the team needed a more sophisticated simulation engine rather than relying on simple estimation methods.

Why A/B testing alone isn't enough

The standard process for improving an algorithm is to design it, implement it in code, and then validate it in production with an A/B test — splitting campaigns randomly so half run on the existing algorithm and half on the new one, then promoting the new algorithm if it outperforms the status quo. While this is the gold standard, it poses two problems at Yelp's scale:

1. Campaign volume is in the hundreds of thousands, not millions, which is too small to comfortably rely on the statistical properties that make A/B testing efficient.
2. Because budgets are monthly, a change needs to be observed for at least a month to judge its effect, making each test slow.

Together, these constraints make A/B testing expensive at Yelp: by the time the team is ready to run one, they want to already be reasonably confident the new algorithm is good, since they can't afford to test many ideas this way.

From notebook to backtesting engine

This need for confidence before A/B testing is what motivated backtesting: a new layer where, once a code change is implemented, it can be tested against historical data. Backtesting is highly scalable and inexpensive, allowing the team to evaluate hundreds of alternative configurations before deciding what is worth promoting to a real A/B test.

The speaker clarifies a naming choice: he calls this "backtesting" rather than "simulation," since technically the algorithm is being tested against the past. He prefers this term specifically to set expectations with internal stakeholders — the results show what performance would have been achieved last month, not a promise of what will happen next month. In practice, he treats backtesting and simulation as near-synonyms in this context.

Before this architecture existed, the whole process ran inside a Jupyter notebook. Since the production system contains substantial business logic spread across repositories, testing a new idea meant copying that logic into the notebook — a process the speaker describes as unmanageable, and one that could never guarantee the notebook exactly matched what was actually running in production.

Backtesting engine architecture

The engine is composed of several components, each covered in turn:

• Configuration file. A YAML dictionary defines the variables entering the simulation: a date interval (e.g. December 1–31, 2025), an experiment name for tracking, and the search space for the parameters to be backtested. The team uses the optimization platform SigOpt to search this space, which requires a metric to optimize — in the example given, average cost per lead, since advertisers benefit most from spending as little as possible per lead. The configuration also sets a maximum number of runs (trials), and defines whatever parameters are relevant to the test — for example, the name of the allocation algorithm (status quo vs. a new "algorithm X"), where algorithm X additionally takes a parameter called alpha, a real number between -10 and 10.
• The SigOpt optimizer. On its first iteration, SigOpt picks parameter values at random from the defined search space — for example, the status quo algorithm with alpha = 3.53. Note that alpha is not actually used when the algorithm is status quo, but SigOpt has no awareness of that and treats it as any other candidate.
• Production repositories via git submodules. This is described as the most important part of the architecture. Rather than reimplementing or copying production business logic into the backtesting tool (as the old notebook-based approach required), the team uses git submodules to reference the actual production repositories directly, without copying any code. To test a new algorithm, the team opens a new branch on the production repository, develops the change there, and can immediately backtest it through the engine. If the simulation results are good enough to promote, no additional work is needed — the already-tested branch is simply pushed to production. This guarantees that what was simulated is exactly what ships, avoiding the bugs and drift that come from maintaining a separate copy of production logic.
• Historical data. Because backtesting relies on the past, the team reads historical campaign data (e.g. last month's campaigns) from Redshift. Some of this data represents characteristics unaffected by the simulation, such as how long a campaign has been running.
• Machine learning "what-if" models. Historical data alone isn't sufficient, because a campaign's actual budget on a given day (say $10) will differ under the algorithm being tested (say $13), and the team has no direct record of how many impressions, clicks, and leads would have resulted from that different budget. To answer this "what-if" question, machine learning models are trained on historical data to predict impressions, clicks, and leads under a different daily budget. These models are monitored on an ongoing basis to ensure their predictions remain reliable.

Validating the backtesting engine against reality

Before this engine existed, the baseline way to estimate the effect of a hypothetical algorithm change was rough manual ("napkin") math — a fairly low bar to beat. After building the engine, the team tested it over time: when a simulated change was later rolled out via a real A/B test, they could compare the simulation's predictions to the actually observed results. This comparison showed the simulations to be quite reliable, and validating the tool against real outcomes in this way helped build trust with other stakeholders in the results the engine produced.

Running the optimization loop and reviewing results

With the candidate parameters, the production repository (via git submodules), historical data, and the what-if prediction models all in place, the engine produces a full simulation: what would have happened last month if the candidate algorithm and parameters had been used. Multiple metrics are tracked, including average cost per click, average cost per lead, and margin (the difference between advertiser revenue and Yelp's cost). While the illustrative example optimizes for minimizing average cost per lead, the actual target used in practice is typically more complex, combining several metrics.

The overall process is iterative: after the first run produces a candidate and its resulting metrics, those metrics are passed back to the SigOpt optimizer, which proposes a new candidate; this repeats until the configured number of runs is reached (25 in the example, producing 25 candidates and 25 sets of metrics). The optimizer is agnostic to why any given metric moves the way it does — it only uses the metric values to try to minimize the target — but because of this feedback, each successive candidate is expected, on average, to be somewhat better than the last.

All candidates and their metrics are logged to MLflow, which makes it easy to share results with other stakeholders through intuitive visualizations. From there, the team can identify the most promising candidates and decide what to move forward into real-world A/B testing in production.