From 2026-08-06T20:17:02.706Z to 2026-08-07T00:07:44.737Z, the run requested deepseek-v4-flash and deepseek-v4-pro and recorded returned model IDs deepseek-v4-flash and deepseek-v4-pro. All 344 planned attempts produced terminal records. In the 288-case U.S. primary matrix, Flash returned 71/144 strict exact answers (49.3%) and Pro 81/144 (56.3%). Primary end-to-end latency was 12.75 s at p50 and 162.29 s at p95 (n=288). The dated all-role cost estimate was a cache-miss upper bound of USD 39.913229 with complete coverage (344/344 rows). Synthetic strict-JSON tasks are not general model quality.
This independent benchmark tested exact retrieval and synthesis at provider-counted prompts from 32K to approximately 950K tokens. It used deterministic synthetic English records, one paid attempt per case, streaming Chat Completions, non-thinking mode, temperature 0, JSON Output, and a strict family-specific grader. The versioned dataset is archived on Zenodo under DOI 10.5281/zenodo.21838863, available through the Hugging Face Dataset Viewer, and preserved in the frozen GitHub v1.0.0 release.
DeepSeek 1M Context Benchmark Results
| Item | Frozen design or observed release |
|---|---|
| Protocol | deepseek-v4-long-context-retrieval-v1.1.0 |
| Models | deepseek-v4-flash, deepseek-v4-pro |
| Task families | Single record, multi-hop join, latest version, event ordering |
| Prompt tiers | 32K, 128K, 512K, approximately 950K |
| Primary accuracy | 288 cases from us-east-1 |
| Excluded pilot | 20 calls: 10 per network vantage |
| India validation | 36 matched single-record calls from ap-south-1 |
| Attempts | One per case; no automatic retry |
| Run window | 2026-08-06T20:17:02.706Z to 2026-08-07T00:07:44.737Z |
What DeepSeek officially documents
DeepSeek’s official Models & Pricing documentation was the frozen source for the tested model IDs, the documented 1M context capacity, the documented 384K maximum output, and dated rates. A context capacity is an input-and-output budget, not proof that a model will retrieve or synthesize every requested detail at every position. This study targeted four returned prompt-token bands: 24K–36K, 110K–140K, 450K–550K, and 850K–980K. The final tier is described as approximately 950K, not one million input tokens. Independently of the model’s documented 384K maximum, the study’s request contract capped each generated answer at 256 tokens; it did not test long-form maximum output.
The current specification and planning details belong in the DeepSeek V4 context and output limits guide. For current model status and positioning, see the DeepSeek V4 models guide. This page owns the independent measured benchmark.
Original test methodology
The corpus generator created deterministic synthetic records with objective answers and no external-knowledge requirement. Flash and Pro received the same unsalted base fixture for each matched primary condition. A deterministic role-and-region prefix separated pilot, primary, U.S., and India request prefixes while keeping the two models paired within a role and vantage.
Operational preflight disclosure
Before the paid U.S. pilot began, the first Step Functions execution stopped during the S3 result-existence preflight because the execution role lacked the required bucket-list permission. It stopped before any provider POST, so it generated no provider request or provider cost. The role was corrected with least-privilege access, and the same frozen run ID and version were launched under a clearly labeled infrastructure-only retry execution. None of the 20 paid pilot cases were retried, and the event is outside every model-result denominator.
- Exact single-record retrieval: find one keyed record and return six exact fields.
- Two-record multi-hop join: find a source, follow its link, and return the linked record’s fields.
- Latest-version conflict resolution: select the highest numeric version despite corpus order.
- Scattered event ordering: find four events and return their codes and checksums in sequence order.
Every response had to be one exact JSON object with the frozen keys, values, arrays, and types. No repair prompt or manual correction was allowed. Provider-reported prompt tokens determined tier validity.
Overall V4 Flash vs V4 Pro accuracy
| Model | Exact | Rate | Wilson 95% CI | Valid JSON | Field accuracy | E2E p50 | E2E p95 | Latency n | Cost coverage | Dated cost value |
|---|---|---|---|---|---|---|---|---|---|---|
| deepseek-v4-flash | 71/144 | 49.3% | 41.3%–57.4% | 144/144 | 70.5% | 7.12 s | 84.89 s | 144 | 144/144 | USD 8.177713 |
| deepseek-v4-pro | 81/144 | 56.3% | 48.1%–64.1% | 144/144 | 73.7% | 15.07 s | 168.28 s | 144 | 144/144 | USD 25.411423 |
The strict primary denominator remained 288 even when a request or output failed. There were 136 non-exact primary terminal records; their stages remain in the sanitized attempt dataset and the failure table.
Accuracy by prompt tier
| Model | Tier | Exact | Rate | Wilson 95% CI | E2E p50 | E2E p95 | n |
|---|---|---|---|---|---|---|---|
| deepseek-v4-flash | 128k | 22/36 | 61.1% | 44.9%–75.2% | 5.99 s | 7.54 s | 36 |
| deepseek-v4-flash | 32k | 24/36 | 66.7% | 50.3%–79.8% | 1.93 s | 2.62 s | 36 |
| deepseek-v4-flash | 512k | 15/36 | 41.7% | 27.1%–57.8% | 12.39 s | 34.48 s | 36 |
| deepseek-v4-flash | 950k | 10/36 | 27.8% | 15.8%–44.0% | 82.41 s | 91.21 s | 36 |
| deepseek-v4-pro | 128k | 22/36 | 61.1% | 44.9%–75.2% | 12.69 s | 21.51 s | 36 |
| deepseek-v4-pro | 32k | 26/36 | 72.2% | 56.0%–84.2% | 3.89 s | 5.59 s | 36 |
| deepseek-v4-pro | 512k | 17/36 | 47.2% | 32.0%–63.0% | 67.35 s | 124.01 s | 36 |
| deepseek-v4-pro | 950k | 16/36 | 44.4% | 29.5%–60.4% | 158.50 s | 186.19 s | 36 |
Accuracy by task family
| Model | Family | Exact | Rate | Field accuracy |
|---|---|---|---|---|
| deepseek-v4-flash | event_ordering | 14/36 | 38.9% | 69.4% |
| deepseek-v4-flash | latest_version | 32/36 | 88.9% | 94.4% |
| deepseek-v4-flash | multi_hop_join | 0/36 | 0.0% | 31.0% |
| deepseek-v4-flash | single_record | 25/36 | 69.4% | 89.4% |
| deepseek-v4-pro | event_ordering | 24/36 | 66.7% | 80.6% |
| deepseek-v4-pro | latest_version | 30/36 | 83.3% | 93.7% |
| deepseek-v4-pro | multi_hop_join | 0/36 | 0.0% | 31.7% |
| deepseek-v4-pro | single_record | 27/36 | 75.0% | 95.8% |
Position sensitivity
| Model | Position | Exact | Rate |
|---|---|---|---|
| deepseek-v4-flash | beginning | 28/48 | 58.3% |
| deepseek-v4-flash | end | 23/48 | 47.9% |
| deepseek-v4-flash | middle | 20/48 | 41.7% |
| deepseek-v4-pro | beginning | 31/48 | 64.6% |
| deepseek-v4-pro | end | 27/48 | 56.3% |
| deepseek-v4-pro | middle | 23/48 | 47.9% |
Paired Flash-versus-Pro outcomes
Across 144 matched fixtures, both models passed 62, Flash alone passed 9, Pro alone passed 19, and both failed 54. The paired exact-pass delta, Flash minus Pro, was -6.9%. This is a task-specific paired comparison, not a general-purpose model ranking.
JSON and field diagnostics
Primary outputs produced 288/288 parseable JSON objects and 288/288 exact key sets. At field level, 1194/1656 fields matched exactly (72.1%). These diagnostics never override strict exact match.
Latency, tokens, cache accounting, and dated cost
The U.S. primary end-to-end latency was 12.75 s at p50 and 162.29 s at p95 across n=288 transport-complete streams. The all-role known-usage cost estimate was a cache-miss upper bound of USD 39.913229 with complete coverage (344/344 rows). These figures use the dated 2026-08-06 price snapshot and are not a current price quote.
For current implementation guidance, see the DeepSeek API guide, context-caching guide, and current pricing page.
Failure and exclusion accounting
All terminal states stay visible. The public failure file distinguishes transport, prompt-tier calibration, JSON parsing, exact-key, and strict-value stages. There was no automatic retry and no failed row was silently removed from its assigned denominator.
Provider-counted prompt lengths
Matched India network-vantage validation
The India slice contains 36 matched single-record pairs at 32K and approximately 950K. Across available paired end-to-end timings, the India-minus-U.S. delta had p50 -490 ms and p95 133.93 s with n=36. This labels AWS client network vantages, not Indian users, model-hosting location, or recurring reliability. Ongoing availability belongs in the regional DeepSeek reliability report.
Reproducibility and downloads
- Versioned Zenodo archive — DOI 10.5281/zenodo.21838863
- Hugging Face Dataset and interactive viewer
- Primary 288-case CSV
- India 36-pair validation CSV
- All 344 sanitized terminal rows
- Canonical JSON summary
- Versioned release, protocol, calibration, charts, QA, and checksums
The public rows are reconstructed through an explicit allowlist. They exclude credentials, authorization headers, raw prompts, raw response bodies, private S3 object keys, local paths, and account data. The validator export contains aggregate field counts rather than per-field expected and observed values, so the release does not fabricate a field-level dataset.
How to cite this benchmark
For measurements from release v1.0.0, cite the immutable version DOI: 10.5281/zenodo.21838863. This identifies the exact archived dataset used by this article.
Chat Deep AI. (2026). DeepSeek 1M Context Benchmark: Retrieval Accuracy, Latency, and Cost (Version 1.0.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.21838863
The concept DOI resolves to the latest Zenodo version. Use the version DOI above when reproducibility depends on this exact release. A machine-readable citation is available in CITATION.cff; Hugging Face provides queryable dataset configurations, while the GitHub v1.0.0 tag remains the immutable source-and-evidence release.
Limitations
- Primary accuracy uses exactly 288 us-east-1 primary_accuracy rows.
- The 20 pilot_excluded rows are preserved but excluded from every primary accuracy denominator.
- The 36 ap-south-1 rows are a matched client network-vantage check, not a sample of Indian users or provider hosting.
- Synthetic English records and strict JSON grading do not measure general model quality.
- Latency belongs to this workload and narrow run window; it is not recurring reliability evidence.
- Prices are dated and can change.
- Cost totals with incomplete usage coverage are labeled known-usage partial sums and are never presented as full-run upper bounds.
- Token and cache totals are reported as complete only when every row contains the full usage tuple; otherwise the release exposes known-usage totals with observed and missing row counts.
This work belongs to the DeepSeek research hub and supports the site’s independent DeepSeek AI guide. For model internals rather than measurements, use the DeepSeek V4 architecture explainer. To evaluate a production application on its own data and release gates, use the DeepSeek evaluation framework.
Sources and related research
- DeepSeek Models & Pricing — current model specifications and the dated price source.
- Create Chat Completion — streaming, model, finish-reason, fingerprint, and usage fields.
- DeepSeek Context Caching — cache behavior and returned hit/miss accounting.
- Official DeepSeek V4 Pro model card and technical report — vendor architecture and vendor-reported benchmarks.
- Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows — separate independent research with a different protocol and corpus.
Frequently asked questions
Does DeepSeek support a one-million-token context window?
DeepSeek documented a one-million-token context capacity for the tested V4 model IDs on the protocol freeze date. This study tested provider-counted prompt tiers from 32K to approximately 950K; it did not call 950K input tokens one million input tokens.
What does the DeepSeek 1M context benchmark test?
It tests exact single-record retrieval, a two-record join, latest-version conflict resolution, and scattered event ordering across four provider-counted prompt tiers and three target positions. It does not test general knowledge, writing quality, agents, or web search.
How many prompts were tested?
The execution plan produced 344 terminal rows: 288 U.S. primary cases, 20 excluded pilot calls, and 36 matched India network-vantage calls. Only the 288 primary cases enter the accuracy denominator.
Why test 32K, 128K, 512K, and approximately 950K tokens?
The four tiers show behavior from a substantial prompt to near the documented capacity. The approximately 950K band stays below the documented one-million-token context capacity; separately, the study capped each generated answer at 256 tokens. That study cap is not the model’s documented maximum output, and 950K is not described as one million input tokens.
Does target position affect DeepSeek retrieval accuracy?
Across both models, beginning targets returned 59/96 strict exact answers, middle targets 43/96, and end targets 50/96. Position semantics differ by family, so the detailed table is the correct interpretation.
Which is better for long context, V4 Flash or V4 Pro?
Pro recorded the higher task-specific strict exact rate: 81/144 versus 71/144 for Flash. This narrow synthetic strict-JSON benchmark cannot establish a universal winner; paired outcomes, latency, cost, and the task family still matter.
How much does a near-1M-token DeepSeek API request cost?
For the approximately 950K primary tier, Flash had cost coverage for 36/36 rows and Pro for 36/36. The observed-row means were USD 0.133020 for Flash and USD 0.413326 for Pro. A full-tier cache-miss upper bound exists only where coverage is complete. These estimates use the 2026-08-06 official-price snapshot and are not current quotes.
Does the India subset measure DeepSeek’s server location or regional reliability?
No. It is a bounded 36-pair comparison from the AWS ap-south-1 client network vantage. It does not identify model-hosting location, represent Indian users, or establish recurring regional reliability.