Until the mid-1980s, much of drug discovery ran at a singular test-at-a-time speed. A chemist made a handful of compounds, handed them to a biologist, waited for test results, then studied those and only then decided what to make next.
Then the industry changed. Automation and technology changed testing.
Pfizer’s early screening system went from about 800 compounds a week in 1986 to 7,200 by 1989.1 Combinatorial chemistry changed the process such that the number of drug-like molecules a chemist could produce in a year rose roughly 800-fold through the 1980s and ’90s.2
It should have been a golden age. Instead, the number of new drugs approved per billion dollars of R&D kept halving about every nine years, a trend now called Eroom’s law (Moore’s law spelled backward). Drug candidates were more likely to fail in clinical trials than they had been in the 1970s.3
Eroom’s law
Half the output. Same research spending.
Historically, new drug approvals per R&D dollar halved roughly every nine years.
Drug approvals per inflation-adjusted R&D dollar
Illustrative index · starting level = 100
More noodles were sticking than ever. A smaller share of them were cooked.
AI is now handing every company a catapult. Whether that leads to a golden age or to another Eroom’s law depends on the wall.
Token Market Fit
The catapult runs on tokens. Tokens are the small units of language that AI models process and generate, the basic unit of AI computation.
In business, what we are seeing now are extraordinary changes in where tokens are being used. Fal calls some of this “token market fit.” Talia Goldberg, a partner at Bessemer Venture Partners, spoke on the tech show TBPN about industries that are having this token market fit opportunity, which included coding, video production, and possibly high-frequency trading.4
It makes sense that these industries are seeing token market fit. These are tasks where outcomes (maybe besides video production) are quick to check. The majority of tokens being spent are basically shots on goal. These tokens are being used to make attempts. A coding agent writes a fix, runs the tests, and retries. A video team generates many takes to keep one after A/B testing. A trading firm will test thousands of strategy variants against the same history.
A big prize and a quick check. That’s where more throws pay. But those three checks differ dramatically in how reliably they tell the truth.
Fat Tails
If outcomes follow a bell curve, extra throws help, but slowly. The best of 10 attempts typically lands about 1.5 standard deviations above average. The best of 1,000 lands about 3.2. A hundred times the effort gets you roughly twice as far above average.
If outcomes are fat-tailed, where one result is worth hundreds of ordinary ones, the math flips. In an illustrative Pareto model, the same hundredfold increase in attempts makes the best result about 20 times larger.5
Typical best of 10: 1.5 standard deviations above average.
Fat tails are more common than people or businesses may realize. In one study of drugs launched in the early 1990s, the top tenth accounted for more than half of post-launch returns, and only about a third earned back the average cost of developing a drug.6 Box-office revenue is so skewed that a single blockbuster can move the average. Even small product tweaks behave this way. At Microsoft’s Bing, the top 2% of tested ideas produced about three-quarters of the gains.7
That’s why the token market fit list isn’t random. A video that breaks out, a trading strategy that works, an algorithm that gets faster every time it runs. In each of these, the best attempt is worth far more than the average one. The same goes for a bug fix, where most attempts fail the tests and the one that passes is the only one that counts.
Finance readers may recognize this as option math. Each attempt is a cheap option on the tail, and options are worth the most when outcomes are volatile. Where there’s no tail, there’s nothing to buy. A thousand attempts at processing an invoice gets you the same invoice.
It also means pharma’s bet on the catapult was rational. With a prize that skewed, more throws should have paid off. They didn’t, because a big prize is only half the equation. You also need to know which noodle won.
Think about an advertiser who can generate 1,000 pieces of ad content. If ads behave like Bing’s tweaks, a handful of them will carry the campaign. However, a thousand variants aren’t a thousand independent throws. A study of AI-generated research ideas suggested that from 4,000 generated ideas, only about 200 were distinct.8 The throws can lack distinction from each other. And the advertiser only has the traffic to test 20.
The Wall
Think of the wall as the screener. Plenty of noodles stuck, but sticking doesn’t mean cooked.
A catapult doesn’t change the screener. Throwing more things at the wall doesn’t make any single thing better. It just gives you more throws. A better wall makes every extra throw worth more than before.
There was a study conducted by Scannell & Bosley where they explained that over the 60 years before publishing, massive technological and scientific advancements should have made drug discovery cheaper and more efficient. Instead, inflation-adjusted R&D costs per approved drug skyrocketed. They argued that the success of drug discovery is overwhelmingly sensitive to the predictive validity of experimental models (i.e., how well a lab assay or disease model predicts clinical outcomes in humans) rather than high-throughput scale.
The practical question is how often a noodle that sticks turns out to be a cooked noodle.
One key finding of their research was that, in their model, an absolute change of 0.1 in the correlation coefficient (predictive validity) between a drug screening model’s output and human clinical outcomes can have a massive impact on drug discovery that vastly outweighs massive changes in high-throughput efficiency. These changes can offset a 10-fold or even 100-fold change in brute-force operational efficiency.2
The same gap shows up in AI. In a 2024 study called Large Language Monkeys, researchers had an AI model (Llama-3-8B) attempt math problems. Going from 100 samples to 10,000 samples raised “a right answer exists” from 82.9% to 98.4%, while “we picked it,” by majority vote or a reward model, only moved from 40.5% to 41.4%. The right answer was almost always in the pile. The system was barely more likely to pick it.9
An AI judge is not automatically a better wall. In the 2024 research-idea study, the best tested LLM judge agreed with human reviewers 53% of the time. A coin flip is 50%, and the human reviewers only agreed with each other 56% of the time.8 The walls that work are built on outcomes rather than opinions: tests that run, conversions, trial results.
For the advertiser, something has to pick which 20 get tested, and that pick is a wall too.
Prize and Check
04 / The decision
A big prize. A check you can trust.
Choose a task to see what the next dollar should buy.
One working fix can be worth far more than a pile of failures. A strong test suite gives fast feedback. Keep a held-out check for regressions; passing incomplete tests is still an imperfect wall.
Conceptual positions, not measured scores. “Prize” means how much the winning attempt is worth relative to an ordinary one.
A bug fix with tests has a big prize and a check you can mostly trust, so buy throws. A backtested trading strategy has a big prize and a fast check that can lie, so more throws mostly buy flukes (try about 45 unrelated variants on five years of data and you should expect one to show a Sharpe of 1.0 by luck).10 An ad creative has a big prize, but the check is slow and someone else owns it. Invoices have no tail, so automate once and stop.
Ask for N
With independent, normally distributed measurement error, the best of 10 equally good options looks better than it is by about 1.5 standard deviations of that error. So any result that reaches a decision-maker should come with how many attempts produced it.5
Who Owns the Verdict
What is interesting is that Meta is becoming a full kitchen, in the sense that they are supplying users with the catapult. Meta reported in 2024 that, in a single month, more than 1 million advertisers made more than 15 million ads with its generative AI tools. Advertisers are generating content from Meta products and are helping them screen for the best content. Meta also rebuilt its ad retrieval system, Andromeda, to handle the growth in creative. Meta has the wall and the catapult.11
Go back to the advertiser with 1,000 pieces of ad content and the traffic to test 20. That traffic belongs to Meta, and so do the results.
If Meta has the catapult and the wall, and competitors have access to similar generators, another variant is not much of an edge. The advantage moves to whatever outcome data only you have, such as first-party conversions, a test suite, or trial results. The choice is to build your own wall or accept the platform’s terms.
The Same Wall
AI has now come back to pharma. An early 2024 analysis found that AI-discovered drugs passed Phase I, mostly a safety check, 80–90% of the time. They passed Phase II, the test of whether the drug works, about 40% of the time, comparable to historical industry averages. The sample is early and small. So far, the catapult is new but the wall is the same.12
What to Do Monday
Many AI budgets are catapult budgets. Scannell & Bosley’s model gives executives a reason to question that: a 0.1 gain in screen validity can offset a 10x to 100x gain in throughput.2 The question for executives is to determine what share of AI spend goes to generating versus judging (or making a better screen).
If your business had ten times the candidates tomorrow, could you test them all, tell the real winners from the lucky ones, and act on them? For the advertiser with the traffic to test 20, ten times the ad content changes nothing.
The next dollar should go to whichever answer is no. AI can help build the wall as well as the catapult.
Notes
- Pereira and Williams describe the early Pfizer screening system: 800 compounds a week in 1986, rising to 7,200 by 1989. This is a history of one company, not a measure of the whole industry. Pereira & Williams, Origin and evolution of high throughput screening (2007). ↩
- Scannell and Bosley’s decision model finds that, when useful candidates are rare, a 0.1 absolute increase in predictive correlation can offset a 10-fold or even 100-fold increase in screening efficiency. This is a model result under stated assumptions, not a universal conversion rate. The noodle counts in the animations are illustrative, not study data. Scannell & Bosley, When Quality Beats Quantity (2016). ↩
- The original Eroom’s law analysis examined the six decades from 1950 to approximately 2010. It found roughly a nine-year halving in approvals per inflation-adjusted R&D dollar. Scannell et al., Diagnosing the decline in pharmaceutical R&D efficiency (2012). ↩
- In her TBPN interview (about 49:40–50:59), Talia Goldberg describes token market fit as productive consumption of roughly $10,000 in tokens per end user per month. She credits fal with the phrase and names coding, video/media and possibly high-frequency trading. TBPN interview with Talia Goldberg, token market fit discussion. ↩
- These examples assume independent attempts. For standard normal draws, the expected maximum is about 1.54 at N=10 and 3.24 at N=1,000. For a Pareto tail with index 1.5, a typical maximum scales approximately as N^(1/1.5), giving about 21.5 times the scale from 100 times as many attempts. The interactive illustration uses a fixed sample of independent standard-normal or Pareto(1.5) draws. Its typical-best calculation is the median maximum; Pareto multiples are relative to 10 attempts. Heights are compressed for display. ↩
- For 118 drugs introduced in 1990–1994, the authors estimated that the top tenth generated over half the present value of post-launch returns, and only about one-third exceeded average R&D costs. Grabowski, Vernon & DiMasi, Returns on Research and Development for 1990s New Drug Introductions (2002). ↩
- Azevedo and colleagues’ analysis of Bing experiments estimates that the top 2% of ideas produced 74.8% of historical gains. The figure concerns the studied platform and its quality metric, not all advertising campaigns. Azevedo et al., A/B Testing with Fat Tails (2020). ↩
- In this 2024 research-idea experiment, 4,000 generated ideas per topic yielded about 200 non-duplicates under the study’s similarity filter. The best tested AI evaluator agreed with human comparisons 53.3% of the time; human reviewers agreed with one another 56.1%. Si, Yang & Hashimoto, Can LLMs Generate Novel Research Ideas? (2024). ↩
- On MATH with Llama-3-8B-Instruct, increasing samples from 100 to 10,000 raised coverage from 82.9% to 98.44%. Over the same range, the largest improvement from the tested majority-vote and reward-model selectors was 40.50% to 41.41%. Coverage means a correct answer exists among the samples; it does not mean the system can identify it. Brown et al., Large Language Monkeys (2024). ↩
- The backtest example assumes independent strategy variants and five years of daily observations. Bailey and López de Prado illustrate how trying about 45 variants can produce an apparently impressive Sharpe ratio without skill. Real strategies can be dependent and financial returns need not match the simple model. Bailey & López de Prado, Backtest Overfitting in Financial Markets. ↩
- Meta reported in December 2024 that more than one million advertisers had created more than 15 million ads with its generative tools in a month. Its Andromeda post describes an ad retrieval system designed to handle growing creative volume. Meta Engineering, Meta Andromeda (December 2, 2024). ↩
- This early 2024 analysis of AI-native biotech pipelines reported Phase I success of 80–90% and Phase II success around 40%. The Phase II sample was small, and the authors described its rate as comparable to historical industry averages. It is early evidence, not proof that AI cannot improve efficacy. Jayatunga et al., How successful are AI-discovered drugs in clinical trials? (2024). ↩
