48 live model-case results. Two independently executed frozen tracks. No composite score. This evidence-led audit tests DeepSeek’s Responses API across Flash and Pro, including streaming, JSON Schema, function tools, server-side Web Search, ignored fields, stateless behavior, unsupported input, and errors.
Download the 48-record public evidence package and read its verification limits. Evidence access was updated September 4, 2026; the benchmark remains the August 19 snapshot.
Scope and current contract — verified September 23, 2026: the 48 results below were executed on August 19 against the text-only
deepseek-v4-flashanddeepseek-v4-pro, before Vision Exp launched. RSP-23 remains a valid historical observation for those two text models and must not be generalized todeepseek-flash(DeepSeek-V4.1-Flash), which now accepts images; the olddeepseek-v4-flashname also routes there. The current image-capable model,deepseek-flash(DeepSeek-V4.1-Flash), processes Responsesinput_imageparts supplied by public URL, Base64 data URL, or an image Files APIfile_id. This audit did not test Vision quality.
The completed core run produced 42 selected results under Protocol 1.10.0. A separate completed optional run produced six selected Web Search results under Protocol 1.11.1. Combining only those selected records yields 48 unique model-case results: 34 PASS and 14 FAIL, with no selected BLOCKED or NOT_RUN result.
A “result” here means one model tested once against one frozen case. PASS and FAIL describe that case’s exact assertions; they are not percentages, ratings, model rankings, or proof of full OpenAI Responses API parity.
Browse the DeepSeek research hub for the site’s other dated reports, benchmarks, and evidence notes.
Results at a glance
| Scope | Live results | PASS | FAIL | RECOVERED | BLOCKED | NOT_RUN |
|---|---|---|---|---|---|---|
| Combined | 48 | 34 | 14 | 0 | 0 | 0 |
deepseek-v4-flash | 24 | 17 | 7 | 0 | 0 | 0 |
deepseek-v4-pro | 24 | 17 | 7 | 0 | 0 | 0 |
Flash and Pro each recorded 17 PASS and seven FAIL results from 24 live samples. Equal counts do not make the models equivalent and are not a ranking: the matrix records whether each response met its own frozen assertions.
The Web Search result: tool calls appeared, the final message did not
RSP-17 through RSP-19 tested the documented web_search tool path, its dated alias, and compatibility with documented ignored context/location fields on both models. In all six single live samples, the completed response contained web_search_call output items, showing that the server-side tool path executed. None contained the final message item required by the frozen case, so all six results are FAIL.
This is deliberately a response-shape finding. Because the required final message was absent, the audit makes no claim about the truth, source quality, completeness, or usefulness of search results. Six one-shot cases also cannot establish full API parity or a reliability rate.
For request examples and field-by-field implementation guidance beyond this result set, use the DeepSeek API Web Search guide.
Exact FAIL taxonomy
| Assertion-specific taxonomy | Count out of 14 FAIL results | Scope |
|---|---|---|
web_search_missing_final_message_after_tool_execution | 6 | RSP-17–19 on both models: tool-call output present, required final message absent. |
wrong_response_media_type | 6 | RSP-16, RSP-20, and RSP-24 on both models: exact target-specific negative response observed, but not as application/json. |
json_schema_instance_validation_failed | 1 | Flash RSP-10: HTTP 200 JSON content did not validate as the requested schema instance. |
visible_text_exact_mismatch_trailing_period | 1 | Pro RSP-21: exact marker gained one terminal period; other frozen fixed response-field assertions passed. |
The 14-result denominator is the observed FAIL set, not a severity score. The six media-type findings concern the response envelope for negative cases. Flash RSP-10 concerns returned structured content, and Pro RSP-21 concerns one punctuation character. None supports a broad incompatibility claim.
For endpoint-specific request examples and application-side validation, use the DeepSeek JSON Output guide.
DeepSeek V4 Flash: 24 live results
| Case | Surface | Frozen test | Status | Observed assertion outcome | Evidence SHA-256 |
|---|---|---|---|---|---|
RSP-01 | Baseline | Plain string input and exact visible marker | PASS | All frozen assertions passed in this one live sample. | 5c66061349fdcdcd… |
RSP-02 | Baseline | Instructions parameter is applied | PASS | All frozen assertions passed in this one live sample. | f24f241652d62aac… |
RSP-03 | Baseline | Developer-role input item is accepted | PASS | All frozen assertions passed in this one live sample. | a21a66dcaddd1a18… |
RSP-04 | Baseline | Default thinking response exposes documented reasoning shape | PASS | All frozen assertions passed in this one live sample. | b8e172eb489f82a0… |
RSP-05 | Streaming | Semantic SSE terminal event and no DONE sentinel | PASS | All frozen assertions passed in this one live sample. | e0392e695282d1a8… |
RSP-06 | Streaming | SSE sequence numbers strictly increase | PASS | All frozen assertions passed in this one live sample. | 27cbae14127f3ab2… |
RSP-07 | Streaming | Output deltas reconstruct terminal visible text | PASS | All frozen assertions passed in this one live sample. | b717e64121337b8c… |
RSP-08 | Streaming | Output cap produces an incomplete terminal event | PASS | All frozen assertions passed in this one live sample. | c2f0f9df24567f60… |
RSP-09 | Structured output | JSON object mode returns parseable JSON | PASS | All frozen assertions passed in this one live sample. | 04f927d94058e2a1… |
RSP-10 | Structured output | Simple JSON Schema is enforced | FAIL | HTTP 200 JSON; visible content did not validate against the requested JSON Schema. | 0b5ecedee3565ac0… |
RSP-11 | Structured output | Nested JSON Schema with enum and required fields | PASS | All frozen assertions passed in this one live sample. | b4465ede76dba2a5… |
RSP-12 | Structured output | JSON Schema without required name is rejected | PASS | All frozen assertions passed in this one live sample. | bf5d4b7cf7690a2d… |
RSP-13 | Function tools | Forced function call emits schema-valid arguments | PASS | All frozen assertions passed in this one live sample. | 882ecb9b00f8683f… |
RSP-14 | Function tools | Function output round trip with complete stateless history | PASS | All frozen assertions passed in this one live sample. | 1bded8f28153b566… |
RSP-15 | Function tools | Tool choice none suppresses a declared function | PASS | All frozen assertions passed in this one live sample. | cf45d8045e4cb9c3… |
RSP-16 | Function tools | Invalid function name is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | c62e51db87c4f24c… |
RSP-17 | Web Search | Forced current web search tool path | FAIL | 11 web_search_call items observed; required final message item absent. | 06da44becd79a555… |
RSP-18 | Web Search | Forced web_search_2025_08_26 alias | FAIL | 11 web_search_call items observed; required final message item absent. | 95126eeaeae8e5fc… |
RSP-19 | Web Search | Ignored web search context and location fields remain compatible | FAIL | 11 web_search_call items observed; required final message item absent. | f5d0a8b5c28bf273… |
RSP-20 | Web Search | Named web search choice without declared tool is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | feda919477a6eac4… |
RSP-21 | Ignored fields | Unsupported top-level fields are silently ignored | PASS | All frozen assertions passed in this one live sample. | 50a0516192b99f6f… |
RSP-22 | Stateless | Previous response ID is not used for server state | PASS | All frozen assertions passed in this one live sample. | 673e306adfdaabd5… |
RSP-23 | Unsupported input | Image input compatibility uses placeholder rather than vision | PASS | All frozen assertions passed in this one live sample. | c58888dc966c46b4… |
RSP-24 | Errors | Missing both input and instructions is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | bd92fc83056ecce4… |
DeepSeek V4 Pro: 24 live results
| Case | Surface | Frozen test | Status | Observed assertion outcome | Evidence SHA-256 |
|---|---|---|---|---|---|
RSP-01 | Baseline | Plain string input and exact visible marker | PASS | All frozen assertions passed in this one live sample. | c846d1d525a66aa1… |
RSP-02 | Baseline | Instructions parameter is applied | PASS | All frozen assertions passed in this one live sample. | ed78252968a6389e… |
RSP-03 | Baseline | Developer-role input item is accepted | PASS | All frozen assertions passed in this one live sample. | f88b5716ea1469b3… |
RSP-04 | Baseline | Default thinking response exposes documented reasoning shape | PASS | All frozen assertions passed in this one live sample. | 646aaa764387582d… |
RSP-05 | Streaming | Semantic SSE terminal event and no DONE sentinel | PASS | All frozen assertions passed in this one live sample. | 050b602adb6e5224… |
RSP-06 | Streaming | SSE sequence numbers strictly increase | PASS | All frozen assertions passed in this one live sample. | 316071c525b9f64a… |
RSP-07 | Streaming | Output deltas reconstruct terminal visible text | PASS | All frozen assertions passed in this one live sample. | a7bf93cae7c0924a… |
RSP-08 | Streaming | Output cap produces an incomplete terminal event | PASS | All frozen assertions passed in this one live sample. | 90defa2fddd853a7… |
RSP-09 | Structured output | JSON object mode returns parseable JSON | PASS | All frozen assertions passed in this one live sample. | 85d1186bb2479e5d… |
RSP-10 | Structured output | Simple JSON Schema is enforced | PASS | All frozen assertions passed in this one live sample. | af3dd044a7258c12… |
RSP-11 | Structured output | Nested JSON Schema with enum and required fields | PASS | All frozen assertions passed in this one live sample. | c6b9b4a303f3f2c1… |
RSP-12 | Structured output | JSON Schema without required name is rejected | PASS | All frozen assertions passed in this one live sample. | 7d78d5a2fbb562af… |
RSP-13 | Function tools | Forced function call emits schema-valid arguments | PASS | All frozen assertions passed in this one live sample. | a0fe45b99d619097… |
RSP-14 | Function tools | Function output round trip with complete stateless history | PASS | All frozen assertions passed in this one live sample. | 00ddb164ccc237a0… |
RSP-15 | Function tools | Tool choice none suppresses a declared function | PASS | All frozen assertions passed in this one live sample. | 2514db2cefcdc9f5… |
RSP-16 | Function tools | Invalid function name is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | 6b94f3b1df5ca875… |
RSP-17 | Web Search | Forced current web search tool path | FAIL | 11 web_search_call items observed; required final message item absent. | a4e37c8a36037b3e… |
RSP-18 | Web Search | Forced web_search_2025_08_26 alias | FAIL | 11 web_search_call items observed; required final message item absent. | 7e692de1a2f1fb33… |
RSP-19 | Web Search | Ignored web search context and location fields remain compatible | FAIL | 11 web_search_call items observed; required final message item absent. | 6be7d1c566075170… |
RSP-20 | Web Search | Named web search choice without declared tool is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | 9478d511590b60ad… |
RSP-21 | Ignored fields | Unsupported top-level fields are silently ignored | FAIL | Request accepted; exact marker gained one trailing period while the other fixed response-field assertions passed. | 5c777a4a0fe58220… |
RSP-22 | Stateless | Previous response ID is not used for server state | PASS | All frozen assertions passed in this one live sample. | 8c4ba8c89af3f67c… |
RSP-23 | Unsupported input | Image input compatibility uses placeholder rather than vision | PASS | All frozen assertions passed in this one live sample. | 945b51fd2d7eb9d7… |
RSP-24 | Errors | Missing both input and instructions is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | a8361a7fe6d78c36… |
How the audit was run
- Each protocol snapshot froze its cases, expected statuses, assertions, runner, fixture, cost accounting, and source ledger before its corresponding execution.
- The core track selected 21 cases per model and excluded RSP-17–19. The later optional track selected only RSP-17–19 per model.
- Every public result is re-derived from a selected manifest row plus its SHA-256-matched evidence file. Unknown failing assertion arrays stop derivation instead of receiving an invented label.
- The combined matrix requires 48 unique model-case keys, 24 per model, with no selected BLOCKED or NOT_RUN result.
For the endpoint contract and request fields referenced in this matrix, see the DeepSeek API documentation.
For reusable mock, regression, SSE, and opt-in smoke-test patterns, use the DeepSeek API testing guide.
Provider-reported usage and the price estimate
| Scope | Input tokens | Cached input | Output tokens | Reasoning tokens | Total tokens | Peak-price estimate |
|---|---|---|---|---|---|---|
deepseek-v4-flash | 154,794 | 130,304 | 2,031 | 18 | 156,825 | $0.07079028 |
deepseek-v4-pro | 126,887 | 105,472 | 1,978 | 26 | 128,865 | $0.17532372 |
| Combined | 281,681 | 235,776 | 4,009 | 44 | 285,690 | $0.246114 |
The token counts are provider-reported usage from both tracks. Server-side Web Search can add internal model work, so its aggregate input usage should not be read as local prompt length. The dollar amount applies the frozen peak token rates to reported usage only. It is not an invoice, account debit, provider-enforced cap, or measured total spend. Numeric account-balance observations are omitted because attribution to this audit was not proven; error responses without usage stay “not reported,” never zero.
For current token prices and peak/off-peak windows, see the DeepSeek pricing page.
Official sources refreshed during execution
- DeepSeek Responses API guide
- Create Response API reference
- DeepSeek models and pricing
- DeepSeek Claude Code integration and Web Search cost note
Both runs refreshed these official source contracts independently. The public derivative records HTTP status, media type, content hash, and frozen-byte or required-marker proof without including private request, response, credential, or account material.
For official release dates and model changes, see DeepSeek’s official change log.
Download the public evidence and check the results
Evidence access update — September 4, 2026: we located the retained records for both August 19 runs and rechecked their manifest and tree hashes, all 48 selected evidence records, result counts, reported usage and the sanitized public derivative. The original results and test date have not changed; this was an offline verification, not a new API benchmark or an external certification.
- Download the public evidence package (ZIP, 13.6 KiB) — the unchanged
public-results.json, a README, the publisher’s retained-record verification note, an offline aggregate verifier and file checksums. - View the package file checksums (TXT).
What is public: all 48 sanitized model-case records, source run IDs, full evidence hashes, case-specific outcomes, failure categories and provider-reported usage. Eight records do not report usage; they are not treated as zero. After extracting the ZIP, readers with Node.js can run node verify-public-results.mjs to check the JSON identity and recalculate the published aggregates without an API key or network request.
What is not public: raw request and response records, private manifests, account data and credentials. This download is not a complete raw-run reproduction kit. The included verifier checks the public derivative, not the withheld source bytes. Hash agreement establishes file identity; it does not independently witness the historical execution or validate every test-design choice.
Historical-source limit: the core Protocol 1.10.0 fixture, runner and protocol source bytes were not recovered during this review. Their hashes are retained in the original run records; the later Protocol 1.11.1 files are not substitutes. The original JSON also preserves its old preparation-state marker; the new README explains the September 4 publication without rewriting those historical bytes.
ZIP SHA-256: c44aa4b4abed49e0d79f4a4e6fb7bbc8e3c1da2c97888ed9b77485c1e74cd37b
public-results.json SHA-256: b84a59b5c73c8a62ba7a029e543b5fdf01bb53c24e54e3fe33ab0ec279bbd9d9
Provenance: two independent completed runs
| Run | Protocol | Track | Selected results | UTC execution window | Files | Manifest SHA-256 | Canonical tree SHA-256 |
|---|---|---|---|---|---|---|---|
RSP1100-CORE-20260819T190014Z | 1.10.0 | core | 42 | 2026-08-19T19:01:15.831Z to 2026-08-19T19:03:29.862Z | 46 | b297b7e1005c3905b03ee01840145d832568fbc51ee36fa76bdc1139a6ccc4dc | 026b7257085220df7cf48cdf34c4aefaf8e70ad09e8b1b94d0b0f2ebc109a44a |
RSP1111-OPTIONAL-20260819T212207Z | 1.11.1 | optional_web_search | 6 | 2026-08-19T21:22:07.459Z to 2026-08-19T21:24:17.467Z | 10 | f27b622e7025454476b7f928d55d9a76305667c53d08fc9fe5b0ff7d114f003c | 7df2bcd2a7fee2178299c921d76523a4c94e67e8e7a193fc36e13b2865367cb4 |
The core and optional tracks are not presented as one continuous run. The table identifies the retained source records and historical fixture, runner and protocol hashes; it does not list the contents of the public ZIP. The unchanged sanitized public-results.json combines only the 42 selected core rows and six selected optional rows, retaining the source run ID and full evidence SHA-256 for each. See the download section for what readers can verify publicly and which source material remains private.
Limits on interpretation
- One sample per model-case cannot estimate reliability, variance, or a general failure rate.
- PASS is case-specific and does not establish full API parity. FAIL is assertion-specific and does not establish broad incompatibility.
- The Web Search cases grade the frozen response shape, not search truth, source quality, completeness, or usefulness.
- RSP-23 is an August 19 placeholder-compatibility result for the text-only Flash and Pro models; it is not a Vision Exp test and does not measure image understanding.
- The ignored-field cases grade observable acceptance and fixed response fields; they do not prove internal non-use.
- Provider documentation, models, pricing, and behavior can change after the 2026-08-19 UTC execution windows.
Frequently asked questions
What does this DeepSeek Responses API audit test?
It reports 48 live model-case results executed on August 19 across the text-only deepseek-v4-flash and deepseek-v4-pro: 21 core cases per model in one completed run and three server-side Web Search cases per model in a second completed run. The surfaces include baseline responses, streaming, structured output, function tools, Web Search, ignored fields, stateless behavior, unsupported input, and errors. Vision Exp launched later and was not tested, so these totals and outcomes must not be read as evidence about its image handling or quality.
What were the Flash and Pro results?
Flash recorded 17 PASS and 7 FAIL results. Pro independently recorded 17 PASS and 7 FAIL results. Each model has 24 live results; these are assertion outcomes, not an overall score, percentage, rating, winner, or ranking.
What happened in the six Web Search tests?
In every single live sample, the completed response contained server-side web_search_call output items but no final message item, so the frozen response-shape assertion failed. Tool execution was observed; the audit did not grade search-result truth, source quality, completeness, full API parity, or reliability across repeated runs.
Does a passing case mean full OpenAI Responses API compatibility?
No. PASS means one received response met that model-case’s frozen assertions. The cases cover different behavior surfaces, and one sample per model-case cannot establish full parity, reliability, variance, or a general failure rate.
What produced the 14 FAIL results?
Six were response media-type mismatches on exact negative cases, six were Web Search responses with tool-call output but no final message item, one Flash result failed requested JSON Schema content conformance after HTTP 200, and one Pro result differed only by a trailing period in an exact-text marker. Each label is assertion-specific.
How should the usage-derived amount be interpreted?
The combined $0.246114 figure applies frozen peak token rates to provider-reported usage from both runs. It is an estimate, not an invoice, account debit, provider-enforced cap, or measured total spend. Numeric account-balance observations are omitted because attribution to this audit was not proven.
Where can I download and check the 48 results?
Download the public evidence ZIP from the evidence section above. It contains the unchanged sanitized public-results.json, all 48 result records, a README, file checksums and an offline verifier for counts, failure categories and reported usage. Raw request and response records, private manifests and account data are not included, so this is not a complete raw-run reproduction kit. Chat-Deep.ai rechecked the retained source records on September 4, 2026; the original test date remains August 19, 2026.