Methodology

Evidence classification, observation scope, freshness, storage, and independent measurement rules.

Evidence classes

Official fact

A model developer, cloud, exchange, or benchmark owner states a first-party fact in an official API or document.

Provider-reported claim

An inference provider or aggregator reports its price, availability, capability, uptime, latency, or throughput.

Third-party benchmark

An external benchmark project publishes a versioned result under its own methodology.

Independent measurement

Model Markets executes a standardized probe and preserves request, environment, result, and evidence.

Collection pipeline

Collect primary evidence

Official APIs and documents are fetched by independent source adapters. Raw payloads are content-addressed for reproducibility when S3 archival is configured.

Normalize durable identities

Mutable aliases resolve to immutable model versions. Provider, endpoint, region, workload, service tier, and units remain explicit.

Validate before publication

Empty, malformed, or incomplete results fail closed. Collector health is updated, while the last-known-good current snapshot remains intact.

Separate storage by question

Redis answers what is true now, Postgres holds relational identities, ClickHouse records what happened over time, and S3 preserves original evidence.

Measure under disclosure

Independent probes record their AWS region, workload, parameters, returned model version, cost, latency, throughput, errors, and validity. One region is never described as global.

Publish with provenance

Every material value exposes source classification, URL or probe ID, timestamps, method, version, units, and raw evidence reference.

Comparison-page selection

Comparison pages are generated for new flagship models, meaningful subvariants, and manually selected high-intent peers. Candidates must share a comparable workload, pass a source-completeness threshold, and use one canonical unordered pair. Reverse-order, query-parameter, and low-evidence duplicates are not indexed.

Index limits

Automatic pairing is capped at 12 peers per model and 50,000 total indexed comparisons. A pair is removed from the sitemap when its evidence falls below the publication threshold. This keeps the comparison surface useful without creating a full model-by-model Cartesian product.

Telemetry percentiles

ClickHouse aggregate views compute p50, p75, p90, p95, and p99 for time to first token, output tokens per second, and end-to-end latency. Success, timeout, rate-limit, and provider-error rates remain separate.

What is not measured yet

No paid inference probes were run for this delivery. No source snapshot is relabeled as independent telemetry, and no single-region result is generalized globally.