48 live model-case results. Two independently executed frozen tracks. No composite score. This evidence-led audit tests DeepSeek’s Responses API across Flash and Pro, including streaming, JSON Schema, function tools, server-side Web Search, ignored fields, stateless behavior, unsupported input, and errors.
The completed core run produced 42 selected results under Protocol 1.10.0. A separate completed optional run produced six selected Web Search results under Protocol 1.11.1. Combining only those selected records yields 48 unique model-case results: 34 PASS and 14 FAIL, with no selected BLOCKED or NOT_RUN result.
A “result” here means one model tested once against one frozen case. PASS and FAIL describe that case’s exact assertions; they are not percentages, ratings, model rankings, or proof of full OpenAI Responses API parity.
Browse the DeepSeek research hub for the site’s other dated reports, benchmarks, and evidence notes.
Results at a glance
| Scope | Live results | PASS | FAIL | RECOVERED | BLOCKED | NOT_RUN |
|---|---|---|---|---|---|---|
| Combined | 48 | 34 | 14 | 0 | 0 | 0 |
| deepseek-v4-flash | 24 | 17 | 7 | 0 | 0 | 0 |
| deepseek-v4-pro | 24 | 17 | 7 | 0 | 0 | 0 |
Flash and Pro each recorded 17 PASS and seven FAIL results from 24 live samples. Equal counts do not make the models equivalent and are not a ranking: the matrix records whether each response met its own frozen assertions.
The Web Search result: tool calls appeared, the final message did not
RSP-17 through RSP-19 tested the documented web_search tool path, its dated alias, and compatibility with documented ignored context/location fields on both models. In all six single live samples, the completed response contained web_search_call output items, showing that the server-side tool path executed. None contained the final message item required by the frozen case, so all six results are FAIL.
This is deliberately a response-shape finding. Because the required final message was absent, the audit makes no claim about the truth, source quality, completeness, or usefulness of search results. Six one-shot cases also cannot establish full API parity or a reliability rate.
For request examples and field-by-field implementation guidance beyond this result set, use the DeepSeek API Web Search guide.
Exact FAIL taxonomy
| Assertion-specific taxonomy | Count out of 14 FAIL results | Scope |
|---|---|---|
web_search_missing_final_message_after_tool_execution | 6 | RSP-17–19 on both models: tool-call output present, required final message absent. |
wrong_response_media_type | 6 | RSP-16, RSP-20, and RSP-24 on both models: exact target-specific negative response observed, but not as application/json. |
json_schema_instance_validation_failed | 1 | Flash RSP-10: HTTP 200 JSON content did not validate as the requested schema instance. |
visible_text_exact_mismatch_trailing_period | 1 | Pro RSP-21: exact marker gained one terminal period; other frozen fixed response-field assertions passed. |
The 14-result denominator is the observed FAIL set, not a severity score. The six media-type findings concern the response envelope for negative cases. Flash RSP-10 concerns returned structured content, and Pro RSP-21 concerns one punctuation character. None supports a broad incompatibility claim.
For endpoint-specific request examples and application-side validation, use the DeepSeek JSON Output guide.
DeepSeek V4 Flash: 24 live results
| Case | Surface | Frozen test | Status | Observed assertion outcome | Evidence SHA-256 |
|---|---|---|---|---|---|
RSP-01 | Baseline | Plain string input and exact visible marker | PASS | All frozen assertions passed in this one live sample. | 5c66061349fdcdcd… |
RSP-02 | Baseline | Instructions parameter is applied | PASS | All frozen assertions passed in this one live sample. | f24f241652d62aac… |
RSP-03 | Baseline | Developer-role input item is accepted | PASS | All frozen assertions passed in this one live sample. | a21a66dcaddd1a18… |
RSP-04 | Baseline | Default thinking response exposes documented reasoning shape | PASS | All frozen assertions passed in this one live sample. | b8e172eb489f82a0… |
RSP-05 | Streaming | Semantic SSE terminal event and no DONE sentinel | PASS | All frozen assertions passed in this one live sample. | e0392e695282d1a8… |
RSP-06 | Streaming | SSE sequence numbers strictly increase | PASS | All frozen assertions passed in this one live sample. | 27cbae14127f3ab2… |
RSP-07 | Streaming | Output deltas reconstruct terminal visible text | PASS | All frozen assertions passed in this one live sample. | b717e64121337b8c… |
RSP-08 | Streaming | Output cap produces an incomplete terminal event | PASS | All frozen assertions passed in this one live sample. | c2f0f9df24567f60… |
RSP-09 | Structured output | JSON object mode returns parseable JSON | PASS | All frozen assertions passed in this one live sample. | 04f927d94058e2a1… |
RSP-10 | Structured output | Simple JSON Schema is enforced | FAIL | HTTP 200 JSON; visible content did not validate against the requested JSON Schema. | 0b5ecedee3565ac0… |
RSP-11 | Structured output | Nested JSON Schema with enum and required fields | PASS | All frozen assertions passed in this one live sample. | b4465ede76dba2a5… |
RSP-12 | Structured output | JSON Schema without required name is rejected | PASS | All frozen assertions passed in this one live sample. | bf5d4b7cf7690a2d… |
RSP-13 | Function tools | Forced function call emits schema-valid arguments | PASS | All frozen assertions passed in this one live sample. | 882ecb9b00f8683f… |
RSP-14 | Function tools | Function output round trip with complete stateless history | PASS | All frozen assertions passed in this one live sample. | 1bded8f28153b566… |
RSP-15 | Function tools | Tool choice none suppresses a declared function | PASS | All frozen assertions passed in this one live sample. | cf45d8045e4cb9c3… |
RSP-16 | Function tools | Invalid function name is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | c62e51db87c4f24c… |
RSP-17 | Web Search | Forced current web search tool path | FAIL | 11 web_search_call items observed; required final message item absent. | 06da44becd79a555… |
RSP-18 | Web Search | Forced web_search_2025_08_26 alias | FAIL | 11 web_search_call items observed; required final message item absent. | 95126eeaeae8e5fc… |
RSP-19 | Web Search | Ignored web search context and location fields remain compatible | FAIL | 11 web_search_call items observed; required final message item absent. | f5d0a8b5c28bf273… |
RSP-20 | Web Search | Named web search choice without declared tool is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | feda919477a6eac4… |
RSP-21 | Ignored fields | Unsupported top-level fields are silently ignored | PASS | All frozen assertions passed in this one live sample. | 50a0516192b99f6f… |
RSP-22 | Stateless | Previous response ID is not used for server state | PASS | All frozen assertions passed in this one live sample. | 673e306adfdaabd5… |
RSP-23 | Unsupported input | Image input compatibility uses placeholder rather than vision | PASS | All frozen assertions passed in this one live sample. | c58888dc966c46b4… |
RSP-24 | Errors | Missing both input and instructions is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | bd92fc83056ecce4… |
DeepSeek V4 Pro: 24 live results
| Case | Surface | Frozen test | Status | Observed assertion outcome | Evidence SHA-256 |
|---|---|---|---|---|---|
RSP-01 | Baseline | Plain string input and exact visible marker | PASS | All frozen assertions passed in this one live sample. | c846d1d525a66aa1… |
RSP-02 | Baseline | Instructions parameter is applied | PASS | All frozen assertions passed in this one live sample. | ed78252968a6389e… |
RSP-03 | Baseline | Developer-role input item is accepted | PASS | All frozen assertions passed in this one live sample. | f88b5716ea1469b3… |
RSP-04 | Baseline | Default thinking response exposes documented reasoning shape | PASS | All frozen assertions passed in this one live sample. | 646aaa764387582d… |
RSP-05 | Streaming | Semantic SSE terminal event and no DONE sentinel | PASS | All frozen assertions passed in this one live sample. | 050b602adb6e5224… |
RSP-06 | Streaming | SSE sequence numbers strictly increase | PASS | All frozen assertions passed in this one live sample. | 316071c525b9f64a… |
RSP-07 | Streaming | Output deltas reconstruct terminal visible text | PASS | All frozen assertions passed in this one live sample. | a7bf93cae7c0924a… |
RSP-08 | Streaming | Output cap produces an incomplete terminal event | PASS | All frozen assertions passed in this one live sample. | 90defa2fddd853a7… |
RSP-09 | Structured output | JSON object mode returns parseable JSON | PASS | All frozen assertions passed in this one live sample. | 85d1186bb2479e5d… |
RSP-10 | Structured output | Simple JSON Schema is enforced | PASS | All frozen assertions passed in this one live sample. | af3dd044a7258c12… |
RSP-11 | Structured output | Nested JSON Schema with enum and required fields | PASS | All frozen assertions passed in this one live sample. | c6b9b4a303f3f2c1… |
RSP-12 | Structured output | JSON Schema without required name is rejected | PASS | All frozen assertions passed in this one live sample. | 7d78d5a2fbb562af… |
RSP-13 | Function tools | Forced function call emits schema-valid arguments | PASS | All frozen assertions passed in this one live sample. | a0fe45b99d619097… |
RSP-14 | Function tools | Function output round trip with complete stateless history | PASS | All frozen assertions passed in this one live sample. | 00ddb164ccc237a0… |
RSP-15 | Function tools | Tool choice none suppresses a declared function | PASS | All frozen assertions passed in this one live sample. | 2514db2cefcdc9f5… |
RSP-16 | Function tools | Invalid function name is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | 6b94f3b1df5ca875… |
RSP-17 | Web Search | Forced current web search tool path | FAIL | 11 web_search_call items observed; required final message item absent. | a4e37c8a36037b3e… |
RSP-18 | Web Search | Forced web_search_2025_08_26 alias | FAIL | 11 web_search_call items observed; required final message item absent. | 7e692de1a2f1fb33… |
RSP-19 | Web Search | Ignored web search context and location fields remain compatible | FAIL | 11 web_search_call items observed; required final message item absent. | 6be7d1c566075170… |
RSP-20 | Web Search | Named web search choice without declared tool is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | 9478d511590b60ad… |
RSP-21 | Ignored fields | Unsupported top-level fields are silently ignored | FAIL | Request accepted; exact marker gained one trailing period while the other fixed response-field assertions passed. | 5c777a4a0fe58220… |
RSP-22 | Stateless | Previous response ID is not used for server state | PASS | All frozen assertions passed in this one live sample. | 8c4ba8c89af3f67c… |
RSP-23 | Unsupported input | Image input compatibility uses placeholder rather than vision | PASS | All frozen assertions passed in this one live sample. | 945b51fd2d7eb9d7… |
RSP-24 | Errors | Missing both input and instructions is rejected | FAIL | Exact negative-case error observed; response media type was not application/json. | a8361a7fe6d78c36… |
How the audit was run
- Each protocol snapshot froze its cases, expected statuses, assertions, runner, fixture, cost accounting, and source ledger before its corresponding execution.
- The core track selected 21 cases per model and excluded RSP-17–19. The later optional track selected only RSP-17–19 per model.
- Every public result is re-derived from a selected manifest row plus its SHA-256-matched evidence file. Unknown failing assertion arrays stop derivation instead of receiving an invented label.
- The combined matrix requires 48 unique model-case keys, 24 per model, with no selected BLOCKED or NOT_RUN result.
For the endpoint contract and request fields referenced in this matrix, see the DeepSeek API documentation.
For reusable mock, regression, SSE, and opt-in smoke-test patterns, use the DeepSeek API testing guide.
Provider-reported usage and the price estimate
| Scope | Input tokens | Cached input | Output tokens | Reasoning tokens | Total tokens | Peak-price estimate |
|---|---|---|---|---|---|---|
| deepseek-v4-flash | 154,794 | 130,304 | 2,031 | 18 | 156,825 | $0.07079028 |
| deepseek-v4-pro | 126,887 | 105,472 | 1,978 | 26 | 128,865 | $0.17532372 |
| Combined | 281,681 | 235,776 | 4,009 | 44 | 285,690 | $0.246114 |
The token counts are provider-reported usage from both tracks. Server-side Web Search can add internal model work, so its aggregate input usage should not be read as local prompt length. The dollar amount applies the frozen peak token rates to reported usage only. It is not an invoice, account debit, provider-enforced cap, or measured total spend. Numeric account-balance observations are omitted because attribution to this audit was not proven; error responses without usage stay “not reported,” never zero.
For current token prices and peak/off-peak windows, see the DeepSeek pricing page.
Official sources refreshed during execution
- DeepSeek Responses API guide
- Create Response API reference
- DeepSeek models and pricing
- DeepSeek Claude Code integration and Web Search cost note
Both runs refreshed these official source contracts independently. The public derivative records HTTP status, media type, content hash, and frozen-byte or required-marker proof without including private request, response, credential, or account material.
For the dated release chronology behind this test window, see the DeepSeek API update timeline.
Reproducibility: two independent completed runs
| Run | Protocol | Track | Selected results | UTC execution window | Files | Manifest SHA-256 | Canonical tree SHA-256 |
|---|---|---|---|---|---|---|---|
RSP1100-CORE-20260819T190014Z | 1.10.0 | core | 42 | 2026-08-19T19:01:15.831Z to 2026-08-19T19:03:29.862Z | 46 | b297b7e1005c3905b03ee01840145d832568fbc51ee36fa76bdc1139a6ccc4dc | 026b7257085220df7cf48cdf34c4aefaf8e70ad09e8b1b94d0b0f2ebc109a44a |
RSP1111-OPTIONAL-20260819T212207Z | 1.11.1 | optional_web_search | 6 | 2026-08-19T21:22:07.459Z to 2026-08-19T21:24:17.467Z | 10 | f27b622e7025454476b7f928d55d9a76305667c53d08fc9fe5b0ff7d114f003c | 7df2bcd2a7fee2178299c921d76523a4c94e67e8e7a193fc36e13b2865367cb4 |
The core and optional tracks are not presented as one continuous run. Each has its own immutable directory, manifest, canonical tree, file count, protocol snapshot, fixture, and runner provenance. The sanitized public-results.json combines only the 42 selected core rows and six selected optional rows, retaining the source run ID and evidence SHA-256 for each.
Limits on interpretation
- One sample per model-case cannot estimate reliability, variance, or a general failure rate.
- PASS is case-specific and does not establish full API parity. FAIL is assertion-specific and does not establish broad incompatibility.
- The Web Search cases grade the frozen response shape, not search truth, source quality, completeness, or usefulness.
- RSP-23 tests documented placeholder compatibility, not image understanding.
- The ignored-field cases grade observable acceptance and fixed response fields; they do not prove internal non-use.
- Provider documentation, models, pricing, and behavior can change after the 2026-08-19 UTC execution windows.
Frequently asked questions
What does this DeepSeek Responses API audit test?
It reports 48 live model-case results across deepseek-v4-flash and deepseek-v4-pro: 21 core cases per model in one completed run and three server-side Web Search cases per model in a second completed run. The surfaces include baseline responses, streaming, structured output, function tools, Web Search, ignored fields, stateless behavior, unsupported input, and errors.
What were the Flash and Pro results?
Flash recorded 17 PASS and 7 FAIL results. Pro independently recorded 17 PASS and 7 FAIL results. Each model has 24 live results; these are assertion outcomes, not an overall score, percentage, rating, winner, or ranking.
What happened in the six Web Search tests?
In every single live sample, the completed response contained server-side web_search_call output items but no final message item, so the frozen response-shape assertion failed. Tool execution was observed; the audit did not grade search-result truth, source quality, completeness, full API parity, or reliability across repeated runs.
Does a passing case mean full OpenAI Responses API compatibility?
No. PASS means one received response met that model-case’s frozen assertions. The cases cover different behavior surfaces, and one sample per model-case cannot establish full parity, reliability, variance, or a general failure rate.
What produced the 14 FAIL results?
Six were response media-type mismatches on exact negative cases, six were Web Search responses with tool-call output but no final message item, one Flash result failed requested JSON Schema content conformance after HTTP 200, and one Pro result differed only by a trailing period in an exact-text marker. Each label is assertion-specific.
How should the usage-derived amount be interpreted?
The combined $0.246114 figure applies frozen peak token rates to provider-reported usage from both runs. It is an estimate, not an invoice, account debit, provider-enforced cap, or measured total spend. Numeric account-balance observations are omitted because attribution to this audit was not proven.
How can the 48 results be reproduced and checked?
Use the two independent immutable run records: RSP1100-CORE-20260819T190014Z for 42 core results and RSP1111-OPTIONAL-20260819T212207Z for six Web Search results. The public derivative records each run’s manifest hash, canonical tree hash, file count, frozen fixture, runner, and protocol provenance, plus the evidence hash for every selected model-case.