Methodology

Intelligence: three publisher inputs, equal weight

The MM Relative Intelligence Index v4.4 combines LiveBench and ARC-AGI-2 capability results with LMArena Text human preference. Preference can reflect style and presentation as well as correctness, so this editorial mix is not a pure capability test. It does not measure every form of intelligence, robotics performance, or suitability for your particular task.

LiveBench overall

Broad performance on language, reasoning, mathematics, and coding tasks. Publisher method.

ARC-AGI-2 verified

Abstract reasoning on unfamiliar visual puzzles. Publisher method.

LMArena Text overall

Blind human preference between text responses. Publisher results.

We put each result on the same 0–100 reference scale, then average the available components equally. With all three present, each contributes one third. Equal weighting is our editorial choice: no publisher gets an extra vote because it publishes more tests. The signals can still be correlated.

A score of 80 means a relative position on this scale. It does not mean 80% accuracy, an 80% chance of success, or twice the capability of a score of 40. Specialist benchmarks remain available separately.

Missing results stay unknown

Three sources: complete. One or two: provisional. None: unranked. We use every available eligible core result. We never choose a model’s best sources, borrow a predecessor’s score, or fill an empty result with zero or an estimate.

A provisional score averages only its observed components. Its bar carries an asterisk; its hover text and screen-reader label give the missing sources and a possible full-core range. This shows how far the three-source average could move if each unknown component were anywhere from 0 to 100. A single-source score reflects that benchmark alone, not a comprehensive intelligence measure.

Example: relative scores of 80, 60, and 70 give a complete score of 70. These numbers are illustrative.

With only one observed relative score of 80, the provisional score is 80*, but its possible full-core range is 26.7–93.3. Both absent benchmark results remain unknown.

The range measures missing coverage, not statistical confidence. It holds the observed results fixed and does not capture test noise. A complete row’s coverage range collapses to its score; that does not make the underlying measurements certain.

We sort by the observed score, so provisional positions are tentative. Selective reporting can still raise an average; no arithmetic can reveal an unpublished result. On comparison pages, a higher Intelligence or Efficiency value receives a better-value highlight only when its entire coverage range exceeds the other model’s range. Overlap means the missing evidence could reverse the order.

Cost: a defined estimate, not your bill

We multiply a model’s published LiveBench output tokens per case by its lowest listed USD output-token rate across currently available provider endpoints. We retain the selected configuration, provider, source, and observation date.

Output cost = output tokens per case × USD price per million tokens ÷ 1,000,000

For example, 10,000 output tokens at $2 per million cost an estimated $0.02. Input tokens, caching, fixed fees, and provider execution differences are excluded. The priced route may differ from the benchmark’s route. Missing token accounting or price leaves the model unranked; the estimate is not invoiced spend or cost per successful task.

Efficiency: an explicit 50/50 tradeoff

MM Intelligence-Cost Efficiency v1.9 averages Intelligence and affordability with equal weight. It includes only models with an eligible Intelligence score and a positive output-cost estimate. One- or two-source provisional coverage and its range carry through from Intelligence.

Affordability uses a logarithmic scale within that eligible group: the cheapest model gets 100 and the most expensive gets 0. Equal cost ratios have equal spacing. If the cheapest costs $0.01 and the most expensive $1, a $0.10 model gets 50. If all costs are identical, every model gets a neutral 50.

Efficiency = (Intelligence + affordability) ÷ 2

This is a preference score, not intelligence per dollar or a universal winner. A cheaper model can compensate for a lower Intelligence score. Prices and group membership can change affordability even when a model’s own price is unchanged, so compare Efficiency scores only within the same published snapshot.

The rules behind the numbers

Eligible models

Reviewed current product-line generations that are active or preview with at least one available provider endpoint. Earlier generations stay on benchmark and model pages; a new generation without the required benchmark inputs remains unranked rather than borrowing its predecessor’s score.

Primary evidence

Headline Intelligence uses original benchmark-publisher artifacts. Developer marketing claims do not enter it. We preserve model identity, configuration, version, sources, and dates; fixture data is labeled.

Run selection

For each model and benchmark, we select the eligible run with the largest published output-token spend, then the latest observation on a tie. Where token accounting is absent, we use the latest eligible run. This is a disclosed high-budget comparison, not equal-compute testing or a claim that more tokens always improve results.

Ties

Equal published Intelligence or Efficiency scores at four decimal places share a rank: 1, 1, 3. Equal Cost estimates share a rank too. Display order within a tie does not imply a winner; small nonzero differences do not establish statistical significance.

Versioned references

Each benchmark uses the worst and best scores of the same fixed 20-model reference group, including historical members. Version 4.0 adds LMArena Text, pins a newer ARC-AGI-2 artifact, and removes one reference model without a Text result; v4.1 adds the exact Mistral Large 4 high-effort LiveBench row without changing the reference group. Version 4.2 pins a newer LiveBench artifact; its new Muse Wonder row remains quarantined pending complete entity publication. v3.3 remains bound to its two-source rule and 21-model group. Scores across versions are not comparable. New entrants do not rescale the group. Results outside its range are capped at 0 or 100, so the scale can saturate. Changing inputs, reference group, weights, or selection rules requires a new index version.

Freshness

The index reads its exact pinned benchmark editions, even when a publisher’s rolling leaderboard advances. A missing reference result or invalid reference range makes the index unavailable. A recent price observation does not make an older benchmark current.

Other evidence

The supporting API coverage-point aggregates reward breadth of published evidence and are not our Intelligence score. We publish no speed ranking without comparable Model Markets measurements of route, region, workload, tier, and time window.