02 — The OpenAI-Compatible API as Integration Contract¶

Every serious inference engine speaks OpenAI-compatible. Every serious agent framework speaks OpenAI-compatible. This is not because the spec is beautiful — it isn’t — but because it is the de facto contract of the LLM ecosystem. If you own inference infrastructure and don’t own this spec at a details level, you will ship a subtly-broken endpoint and the agent team will hate you.

Why this doc exists (and why you specifically need it)¶

At Zoho your day job is agentic harnesses. Every one of those harnesses talks to LLMs over what is effectively OpenAI’s chat/completions API. When you become the person who serves the models too, you now own both sides of that contract. That’s a rare pair of shoes to be wearing and it’s where you make yourself indispensable: you understand the exact way that an SSE early-close breaks a tool-calling loop, or the exact way that a mis-typed logprobs field breaks a re-ranker.

The spec has ~5 pieces that matter for production. This doc walks all five, plus the failure modes.

1. Streaming via Server-Sent Events (SSE)¶

The wire format¶

POST /v1/chat/completions
Content-Type: application/json
{ "model": "zoho-llama-70b", "messages": [...], "stream": true }

Response: Content-Type: text/event-stream, with each event a JSON object prefixed by data: :

data: {"id":"cmpl-1","object":"chat.completion.chunk","created":1720044800,"model":"zoho-llama-70b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"cmpl-1","object":"chat.completion.chunk","created":1720044800,"model":"zoho-llama-70b","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"cmpl-1","object":"chat.completion.chunk","created":1720044800,"model":"zoho-llama-70b","choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]}

data: {"id":"cmpl-1","object":"chat.completion.chunk","created":1720044800,"model":"zoho-llama-70b","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Key invariants a compliant server MUST hold:

The first chunk carries the role: assistant in delta but usually empty content. This is the TTFT-observable boundary.
Subsequent chunks carry delta.content — raw text fragments, not whole strings.
The last chunk before [DONE] carries finish_reason (stop / length / tool_calls / content_filter) and an empty delta.
The literal string data: [DONE]\n\n terminates the stream. Not JSON. Many clients break on this.
Each SSE event is data: <json>\n\n (double newline separator). Miss the double newline and Envoy/nginx will buffer.

The failure modes you will hit¶

Proxy buffering. nginx, Envoy, Cloudflare all default to buffering. Configure proxy_buffering off in nginx; X-Accel-Buffering: no header. Or the client TTFT will be pinned to the end of the response, not the first token. Every enterprise on-prem deployment gets bitten by this once.
Half-closed connections on cancel. When the client cancels mid-stream, your engine keeps generating and burns GPU-seconds. vLLM and SGLang both handle this if you propagate cancellation properly through your Python async chain. Test this explicitly: cancel from the client, then check whether GPU utilization drops.
JSON fragmentation across chunks. Naively serializing tool call arguments causes them to appear in fragments across delta chunks. The client must accumulate. Your tests must include a tool-call streaming case.
Heartbeat. Long TTFT (>60s for reasoning models) can trigger LB idle timeouts. Send SSE comment lines : heartbeat\n\n every 15 seconds during long prefills.
UTF-8 mid-character splits. Tokenizer output can split a multi-byte character across two token boundaries. If you emit raw bytes, the client sees mojibake. Buffer partial UTF-8 sequences until they close. vLLM handles this; check your custom middleware doesn’t undo it.

What the OpenAI spec doesn’t mandate but production needs¶

Usage in stream. stream_options: {"include_usage": true} returns a final chunk with token counts. vLLM supports this; enable it or your billing/metrics get harder.
Server-emitted X-Request-Id — propagate for tracing (see 05_observability.md).

2. Tool calls (function calling)¶

This is the piece you care about most given your agentic work.

Request shape¶

{
  "model": "zoho-llama-70b",
  "messages": [...],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_crm_lead",
        "description": "Fetch a CRM lead by ID",
        "parameters": {
          "type": "object",
          "properties": {
            "lead_id": {"type": "string", "description": "CRM lead ID"}
          },
          "required": ["lead_id"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}

tool_choice values: "auto" / "none" / "required" / {"type":"function","function":{"name":"..."}}.

Response shape (non-streaming)¶

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_abc",
        "type": "function",
        "function": {
          "name": "get_crm_lead",
          "arguments": "{\"lead_id\":\"L-4291\"}"
        }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

Note: arguments is a JSON-encoded string, not a JSON object. This is a wart from the original OpenAI spec, but if you emit an object here your clients break.

Streaming tool calls¶

delta.tool_calls is an array where each element is a partial update, indexed. The arguments string fragments across chunks. Concrete example fragments:

delta: { tool_calls: [{ index: 0, id: "call_abc", type: "function", function: { name: "get_crm_lead", arguments: "" } }] }
delta: { tool_calls: [{ index: 0, function: { arguments: "{\"lead_i" } }] }
delta: { tool_calls: [{ index: 0, function: { arguments: "d\":\"L-4291\"}" } }] }

The client must reconstruct by concatenating arguments fragments by index.

How the engines actually produce this¶

They don’t magically know to call tools. Two mechanisms:

Model-native tool tokens. Llama-3.x, Qwen 2.5+, Mistral trained with tool-call templates that emit <|tool_call|>{...}</|tool_call|> (varies by model). vLLM’s tool-parser plugin (--tool-call-parser hermes / llama3_json / mistral / pythonic etc.) parses these tokens and re-shapes them into OpenAI-compatible tool_calls. You must select the right parser per model or tool-calling silently fails.
Constrained decoding. For models without native tool tokens, use structured output (§4) to force the model into a JSON schema. Slower but works universally.

The failure modes you will hit¶

Wrong tool-parser flag. vLLM defaults to no parser — without --tool-call-parser, tool calls appear as raw text in content. Silent, ugly, common.
Malformed arguments. The model emits {"lead_id": L-4291} (no quotes on the string). Guarded structured output (§4) fixes this at inference time; add a JSON-schema validator on your side as belt-and-braces.
Parallel tool calls. OpenAI supports multiple simultaneous tool calls in one assistant message. Not all model prompting formats do. Test explicitly.
Tool call ID reuse. Some clients expect id values unique per conversation, not per response. Generate a UUID.

3. `logprobs`¶

logprobs: true, top_logprobs: 5 in the request → each token in the response carries the top-5 log-probabilities.

"logprobs": {
  "content": [{
    "token": "Hello",
    "logprob": -0.1,
    "top_logprobs": [
      {"token": "Hello", "logprob": -0.1},
      {"token": "Hi", "logprob": -2.3},
      ...
    ]
  }, ...]
}

What logprobs are used for in production:

Confidence calibration. Re-rank retrieval by response confidence.
Cheap classifiers. Ask model “answer yes or no” and look at logprob(" Yes") - logprob(" No") — don’t waste tokens on chain-of-thought if you just need a probability.
Guardrail voting. Multiple models’ logprobs → vote weighted by confidence.
Evaluation. Reference-free eval via next-token likelihood.

vLLM support: logprobs and top_logprobs supported. Small overhead but nonzero (extra softmax broadcast). Don’t enable by default; expose as an API parameter.

4. Structured output (JSON schema / grammar)¶

The API surface¶

OpenAI’s canonical form (2024+):

{
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "crm_lead_summary",
      "strict": true,
      "schema": { "type": "object", "properties": { ... }, "required": [...] }
    }
  }
}

Also response_format: {"type": "json_object"} (weaker — just “be JSON, no schema”).

How it works under the hood¶

The engine compiles the JSON schema into an FSM/pushdown automaton.
At each decode step, the automaton produces the set of legal next tokens.
Logits for illegal tokens are masked to -inf.
The model can only sample tokens that keep the output valid.

The libraries you’ll actually see¶

XGrammar (used by SGLang and vLLM by default in 2026). Pushdown-automaton-based, near-zero overhead on the hot path. This is the current SOTA.
Outlines (regex → FSM). Still used, especially for simpler grammars.
llguidance (Guidance’s guided decoder). Solid.
lm-format-enforcer. Older, still fine.

vLLM: --guided-decoding-backend xgrammar (default from ~v0.6+). SGLang: XGrammar built in.

Failure modes¶

Schema compilation cost. Large schemas take milliseconds to compile per request. Cache compiled FSMs by schema hash — XGrammar does this; verify.
Model fights the schema. If your schema is very different from the model’s training distribution, quality collapses. The model tokens with any nonzero prob may all be illegal; you get gibberish. Iterate on the schema before blaming the model.
strict: true semantics. In OpenAI’s spec, strict: true means “schema is enforced.” In some engines this maps to guided decoding; in others it doesn’t. Verify per engine.

The Zoho-day-job payoff¶

Your agent tools have schemas. Right now if the model emits invalid JSON, your harness catches the error and retries — wasting tokens and latency. With structured output turned on, the model cannot emit invalid JSON. Retry rate → 0. Ship this to prod.

5. Everything else (briefly)¶

temperature, top_p, top_k, frequency_penalty, presence_penalty, min_p, repetition_penalty. Standard. Note: min_p is a vLLM/SGLang extension not in OpenAI’s original spec; it’s better than top_p for most workloads. Expose both.
max_tokens / max_completion_tokens. OpenAI moved to max_completion_tokens for reasoning models. Support both aliases.
n. Number of completions. Costs multiply. Rarely used in production; be sure to bill correctly.
seed. Deterministic sampling knob. Useful for eval reproducibility; not truly deterministic under continuous batching (different co-batched requests → different kernel launch orders → tiny FP nondeterminism). Document this.
user. Free-text user ID for abuse tracking. Log this into your traces (05_observability.md).
stop. Array of stop sequences. Model stops generating on match. Cheap and effective for prompt scaffolding.

Compatibility gotchas (what breaks even when everyone’s “OpenAI compatible”)¶

Tokenizer mismatch. OpenAI’s usage.prompt_tokens uses tiktoken. Your open-model server uses the model’s own tokenizer. Client code that pre-counts tokens for rate limiting will disagree. Document the tokenizer explicitly and provide a /v1/tokenize endpoint.
system_fingerprint and service_tier. OpenAI-native fields many clients ignore. Emit anything reasonable; empty string is fine.
Role names. OpenAI supports system / user / assistant / tool / function (legacy) / developer (reasoning models). Some open models don’t have a dedicated system-role template. Your server should map appropriately.
Extended tool schemas. OpenAI’s spec has grown — things like strict: true, image inputs in content arrays, audio. Coverage in open engines lags by 3–6 months. Test with the exact client libraries your agents use.
Reasoning model quirks. OpenAI’s o-series exposes hidden reasoning; open reasoning models (DeepSeek R1, QwQ) put reasoning in <think>...</think> tags. vLLM has --reasoning-parser (2026) to convert. Configure per model.

The integration-contract mindset¶

When you serve LLMs at Zoho, you are not just “running vLLM.” You are publishing a contract that dozens of internal teams will build on. Contract stability is worth more than 10% throughput improvements. Rules for staying trustworthy:

Version your endpoint. /v1/ today, /v2/ when you break something. Never break /v1/ silently.
Compatibility tests. Have a test suite that runs the top-3 client libraries (OpenAI Python, LangChain, LlamaIndex) against your server and asserts behavior. Run in CI on every vLLM upgrade.
Deprecation windows. If you must remove a field, announce it 60 days ahead with a header (Deprecation: ...) served on the field’s endpoint.
Explicit model names. --served-model-name zoho-llama-70b-v2.1 rather than the HF path. Clients pin against your name.

Reading list¶

OpenAI API reference — platform.openai.com/docs/api-reference/chat. Read every field.
vLLM OpenAI-compatible server docs — docs.vllm.ai/en/stable/serving/openai_compatible_server.html.
SGLang API reference.
XGrammar paper (MLSys 2025).
The Anthropic Messages API — not because you serve it, but because it’s what a good API looks like when re-designed 5 years later with hindsight.

Exit test for this doc¶

Write, from memory, the SSE wire format for a two-token response with a stop finish. No looking.
Explain what goes wrong when nginx is in front of your vLLM server and proxy_buffering is default.
Given a JSON schema for a CRM lead, describe how XGrammar turns it into a decode-time constraint.
Debug a report: “our LangChain agent gets 'NoneType' object has no attribute 'startswith' when talking to the internal Llama endpoint.” Name at least three plausible causes without seeing the code.

02 — The OpenAI-Compatible API as Integration Contract¶

Why this doc exists (and why you specifically need it)¶

1. Streaming via Server-Sent Events (SSE)¶

The wire format¶

The failure modes you will hit¶

What the OpenAI spec doesn’t mandate but production needs¶

2. Tool calls (function calling)¶

Request shape¶

Response shape (non-streaming)¶

Streaming tool calls¶

How the engines actually produce this¶

The failure modes you will hit¶

3. logprobs¶

4. Structured output (JSON schema / grammar)¶

The API surface¶

How it works under the hood¶

The libraries you’ll actually see¶

Failure modes¶

The Zoho-day-job payoff¶

5. Everything else (briefly)¶

Compatibility gotchas (what breaks even when everyone’s “OpenAI compatible”)¶

The integration-contract mindset¶

Reading list¶

Exit test for this doc¶

3. `logprobs`¶