AI Pricing Benchmarks

Case leaderboard

Ranking across all complete benchmark cases with shared tie points

Rank	Model	Case points	Point share	Mean loss across cases	Median loss across cases	Compared cases

Mean revenue loss across cases

All complete benchmark cases together

Median revenue loss across cases

Same cases as the mean chart; the median is less sensitive to a few large failures

Near-optimal leaderboard

Most cases with relative revenue loss at or below 5%

Rank	Model	Near-optimal cases	Share	Compared cases

Severe-loss leaderboard

Fewest cases with relative revenue loss at or above 25%

Rank	Model	Severe-loss cases	Share	Compared cases

Under-pricing leaderboard

Fewest cases priced more than 5% below the optimum

Rank	Model	Under-price cases	Share	Compared cases

Similar-pricing leaderboard

Most cases within +/-5% of the optimal price

Rank	Model	Similar-price cases	Share	Compared cases

Over-pricing leaderboard

Fewest cases priced more than 5% above the optimum

Rank	Model	Over-price cases	Share	Compared cases

Static across-method performance

Filtered static benchmark cases; regression included when matching output is available

Rank	Model	Mean loss	Median loss	Near-optimal	Severe-loss	Case points	Compared cases

Static mean revenue loss across methods

Shared static cases for the selected methods

Static regression worst-fit instances

Filtered static instances ranked by absolute revenue gap

Instance	Status	Demand model	Parameter set	Sigma	History	MC	R²	\|Price gap\|	\|Revenue gap\|

Static regression best-fit instances

Filtered static instances ranked by smallest absolute revenue gap

Instance	Status	Demand model	Parameter set	Sigma	History	MC	R²	\|Price gap\|	\|Revenue gap\|

Static regression closest P-hat to P-star

Filtered static instances ranked by smallest absolute price gap

Rank	Instance	Demand model	Parameter set	Sigma	History	MC	P_hat	P_star	Price gap

Static regression closest revenue-hat to revenue-star

Filtered static instances ranked by smallest absolute revenue gap

Rank	Instance	Demand model	Parameter set	Sigma	History	MC	Revenue_hat	Revenue_star	Revenue gap

Mean relative revenue loss across five runs

Each complete shared case has equal weight

Mean absolute relative price gap across five runs

Mean |P − P*| / P*: lower is closer to optimal price

Basic model averages and benchmarks

Signed gaps show direction; absolute gaps measure distance

Model	Shared cases	Runs per case	Mean revenue loss	Median revenue loss	Mean signed price gap (P − P*)	Mean absolute price gap \|P − P*\|	Mean signed relative price gap	Mean absolute relative price gap

Per-run results and fixed benchmarks

Same complete cases; each fixed benchmark appears once

Model	Run	Shared cases	Mean revenue loss	Median revenue loss	Mean signed relative price gap	Mean absolute relative price gap

Three-model summary

Answer availability · mean · median

Model	Answered	Answer rate	Mean loss	Median loss	P75 loss	P90 loss	Win share

Distribution of relative revenue loss

Single plot across the selected models

Mean relative revenue loss

Complete matched cases

Median relative revenue loss

Complete matched cases

Answer availability

Matched inputs

Head-to-head wins

Lowest loss wins

Under / similar / over pricing

Complete matched cases · Similar means within +/-5% of the optimal price

Grouped comparison: Demand model

Mean loss by matched subgroup

Performance by method

Includes heuristic benchmark; lower severe-loss and higher near-optimal are better

Method	N	Mean	Median	Near-optimal	Severe-loss

Mean comparisons

Paired t-test on matched differences

Comparison	N	Mean diff	T p-value	Bootstrap 95% CI

Mean-loss bars

Bars use complete matched cases and include the heuristic benchmark

Near-optimal rate

Relative revenue loss ≤ 5%

Severe-loss rate

Relative revenue loss ≥ 25%

Mean loss by demand distribution

One shared scale across all distributions; four methods per panel

Under / similar / over pricing

Complete matched cases only

Direction detail

Price: P_hat vs P_star. Revenue: R_hat vs R_star. Complete matched cases only.

Method	N	Under (P_hat < 0.95 P_star)	Similar (0.95-1.05 P_star)	Over (P_hat > 1.05 P_star)	Near-optimal (R_hat ≥ 0.95 R_star)	Severe loss (R_hat ≤ 0.75 R_star)

Quartile comparison

Q1 · median · Q3, winner highlighted by column

Model	Q1	Median	Q3

Density of relative revenue loss

Kernel density over 0-15% relative revenue loss

CDF of relative revenue loss

Higher curve means more cases at or below each loss level

Density of paired differences

First listed model versus the others; displayed from -10% to +10%; negative favors the first model

Paired difference diagnostics

Difference quartiles and win share

Comparison	N	Mean diff	Median diff	P25 diff	P75 diff	T p-value	Bootstrap mean 95% CI

Expert role: matched mean loss

No and Yes shown side by side for each LLM

Overall pairwise t tests

Current across-model filters plus selected models

Comparison	N	Mean diff	T p-value	Bootstrap 95% CI

Pairwise t tests by levelOpen to calculate detailed split-level results Mean differences are first model minus second model

Level	Comparison	N	Mean diff	T p-value	Bootstrap 95% CI

Are the five runs' means equal?

Joint test of H0: mean loss in run 1 = run 2 = run 3 = run 4 = run 5, for each basic model

Are the five runs' medians, upper quartiles, maxima and minima equal?

Joint permutation tests across the five runs, for each basic model

Advanced models: are the runs' means equal?

Joint test of H0: mean loss in run 1 = run 2, for each advanced model

Advanced models: are the runs' medians, upper quartiles, maxima and minima equal?

Joint permutation tests across the advanced-model runs

Basic model run comparisons

Repeated-measures ANOVA (two runs) of relative revenue loss on the same input cases

Comparison	N	Mean diff	F	df	p	Bootstrap 95% CI

Run comparisons by levelOpen to calculate detailed split-level results Uses the split variable and included levels selected above

Level	Comparison	N	Mean diff	F	df	p	Bootstrap 95% CI

Overall matched level t tests

Selected models compared across matched level pairs

Comparison	N	Mean diff	T p-value	Bootstrap 95% CI

Matched level t tests by split level

Mean differences are first level minus second level within the same model

Level	Comparison	N	Mean diff	T p-value	Bootstrap 95% CI

True demand and recommended prices

Ground-truth curve with the price history, the optimal price and model estimates

Revenue generated at each price

Ground-truth revenue curve with the same recommended-price markers

Price history shown to the models

Observed prices and demands for the selected benchmark case

Price estimates

Mean recommended price per method. "Loss at the mean price" scores the averaged price as one recommendation; "Mean loss across answers" averages the losses of the individual answers. They differ when a model's answers vary.

Method	Answers	Mean recommended price	Difference from P*	Revenue at the mean price	Loss at the mean price	Mean loss across answers

All shared cases

Each model cell shows its recommended price and relative revenue loss. Expand a case to see its matching inputs.

Actual demand, least-squares fit, Regression Heuristic fit and the AI's stated fit

Demand curves over the price history shown to the models

Demand lines and the prices they imply

Selected instance, model, run and prompt variant

Source	Intercept	Slope	MSE on the price history	Price implied by the line	Recommended price	Relative revenue loss

All stated fits of this model on this instance

Every run and prompt variant whose reasoning states a linear demand equation

Run	Prompt variant	Stated demand (quoted)	Intercept	Slope	Recommended price	Relative revenue loss

How often do the models state a linear demand fit?

Share of answers with an explicit linear equation, and how its slope compares with the regression slope

Model	Answers	Static	Seasonal	Duopoly	All	Median stated slope / regression slope	Stated slope within 10% of regression

Basic models, full grid

Omnibus repeated-measures ANOVA of H0: all methods have the same mean loss, for each run

Basic and advanced models, shared cases

The same omnibus test on the cases every model was run on

Noise effect relative to noise 0.1

Mean loss at noise 0.3 minus mean loss at noise 0.1, with the old and with the new histories

Method	Instances	Noise 0.1 mean	Old 0.3 mean	Old effect	p	New 0.3 mean	New effect	p	Change in effect	p

Old versus new noise 0.3 histories

Paired t test by instance of H0: mean loss is the same on the old and the new noise 0.3 histories

Method	Instances	Old 0.3 mean	New 0.3 mean	New − old	t	p	Old 0.3 median	New 0.3 median

Mean loss by segment

Each LLM, the best-observed price and the regression heuristic, with the clear best and worst LLM in each segment

LLM comparisons by segment

Mean loss and paired differences between LLMs (percentage points); highlighted p-values are below the selected level

LLMs versus the heuristics by segment

LLM mean loss minus heuristic mean loss (percentage points); positive values mean the LLM loses more