The request succeeded. The response was valid JSON. The SDK stayed quiet. And yet the application collapsed.

This is the story of what happens when you treat an LLM provider swap like a configuration change instead of a structural gamble. You paste in a new base URL, swap the API key, and keep the request body identical because the docs promise an OpenAI-compatible endpoint. For a basic "hello world" prompt, it works. You celebrate. Then real traffic hits, and the seams split open.

The Illusion of Wire Compatibility

Compatibility at the HTTP layer is shallow. A 200 status code and a JSON body mean the server accepted your message. It does not mean the server thinks the same way as the previous one. OpenAI-compatible endpoints share a request shape, but they do not share a behavioral contract. Two providers can ingest identical payloads and return answers that diverge in subtle, destructive ways.

Your code makes assumptions. You assume message.content is a string because it always was before. You assume a tool call arrives with clean, parseable JSON. You assume finish_reason signals what you think it signals. These assumptions are invisible until they are fatal.

Consider the crash that started it all:

const text = response.choices[0].message.content.trim();

This line looks innocent. It worked for weeks. Then the new provider returned a tool call. In that moment, message.content was not an empty string. It was null. The actual payload lived inside message.tool_calls, but the parser had already moved on, calling .trim() on nothing. The API did not throw. The network layer did not complain. Your own parser killed the request.

Where Providers Quietly Diverge

The differences do not announce themselves in changelogs. They sit in the margins of the response object, waiting for edge cases.

Tool-call formatting. One provider sends tool arguments as a pre-validated JSON object. Another sends them as an escaped string inside a field. A third might split a long tool call across multiple streaming deltas, forcing you to buffer chunks before you can even see if the structure is valid. If your application expects a single parseable blob, it chokes.

Finish reasons. OpenAI uses specific strings like "stop", "length", "tool_calls", and "content_filter". A compatible provider might return "end_turn" or simply omit the field when the model hits the token ceiling. If your retry or fallback logic waits for "length" to detect truncation, it will sit idle while the user sees a half-finished answer.

Usage fields. Some providers strip token counts from streaming responses to shave milliseconds off latency. Others append usage only to the final chunk, or omit it entirely in non-streaming calls. If you charge customers per token and your accounting code expects usage.total_tokens to exist in every response object, your billing pipeline will silently record zeros.

Streaming behavior. Server-sent events are supposed to be standard, yet providers flush buffers at different frequencies. Event boundaries vary. One provider terminates a stream with a [DONE] signal. Another drops the connection cleanly with no sentinel at all. If your client blocks waiting for a specific closing marker, it hangs.

Errors and timeouts. A rate limit might arrive as a 429 with a retry-after header from one provider, and as a vague 502 from another. Some providers accept the request and then go quiet for two minutes before a network timeout. The OpenAI SDK will not magically normalize these into the exception types your logs expect.

Defensive Parsing for Unpredictable Shapes

The fix is not to trust the schema. The fix is to treat every response as a suspect.

Do not assume content is a string. Check it before you touch it.

const content = response.choices?.[0]?.message?.content;
const text = typeof content === "string" ? content.trim() : "";

Do not assume tool arguments are valid JSON. The model proposes an action. Your code must decide whether that proposal is safe enough to execute. Wrap every tool argument parse in a try-catch. If JSON.parse throws, treat the tool call as malformed garbage and route it to a failure handler. A hallucinated bracket or a missing quote should never bubble up as an unhandled exception.

If tool_calls exists but content is missing, your application should recognize a state transition. The user did not get a chat reply. The system got a work order. Those are two different paths, and your router should know the difference before it attempts string manipulation.

Behavioral Tests Before You Deploy

Pinging the endpoint with a "hi" message proves the network works. It proves nothing about your application.

Before you redirect production traffic, run a targeted behavioral test suite against the new provider:

  • Normal text response. Verify that content exists, is a string, and can be passed through your sanitization pipeline without casting errors.
  • Forced tool call. Set tool_choice to required. Confirm the provider honors it, and check whether content arrives as null, an empty string, or a missing key. Each of those states needs its own handler.
  • Malformed tool arguments. Inject scenarios where the model returns broken JSON inside tool arguments. Ensure your parser rejects them gracefully instead of crashing the worker.
  • Response near the token limit. Push the context window. Check the finish_reason. If the provider returns something unexpected when truncation happens, your summarization or retry logic must know how to react.

These are integration tests, not unit tests. They exercise the real relationship between your code and the provider's personality. Pass them before you call the migration done.

Build an Internal Contract

Provider differences should stop at your network boundary. Do not let them leak into business logic.

Create a normalization layer that consumes the raw SDK response and emits an object your application actually owns. Map provider-specific eccentricities into a stable internal format. If Provider A returns tool arguments as strings and Provider B returns objects, your mapper flattens both into your own ToolRequest structure. If usage is missing, your mapper either estimates it or flags the gap, but it never lets undefined seep into your cost-tracking modules.

If finish_reason is nonstandard, translate it into your own enum of terminal states: COMPLETE, TRUNCATED, TOOL_CALL, FILTERED. Your app should decide what to do based on these clean abstractions, not by sniffing raw strings from a third-party server.

This layer turns provider swaps from a game of whack-a-mole into a single-file change. You rewrite the mapper, run the behavioral tests, and move on. Your application remains untouched.

A Dependency Upgrade, Not a Config Tweak

Switching LLM providers is not like swapping CDN endpoints. It is closer to changing your database from PostgreSQL to MySQL. You would never assume the same connection string means identical query behavior. You would test locking semantics, migration paths, and indexing quirks. LLMs deserve the same respect. They are probabilistic systems masquerading as standard APIs, and their responses carry assumptions about formatting, truncation, and control flow that can shatter your application without raising a single network error.

The bug was never in the connection. It was in the assumption that compatibility means sameness. It does not. Validate the shape. Test the edges. Own the contract.

Source: The Bug Only Happened After I Switched LLM Providers

Community: GyaanSetu AI on Telegram