Запрос прошел успешно. Ответ был валидным JSON. SDK хранил молчание. И все же приложение рухнуло.

Это история о том, что происходит, когда замену провайдера LLM воспринимают как простое изменение конфигурации, а не как структурный риск. Вы вставляете новый базовый URL, меняете API-ключ и оставляете тело запроса прежним, потому что документация обещает OpenAI-совместимый эндпоинт. Для простого промпта «hello world» это работает. Вы празднуете успех. Но затем приходит реальный трафик, и швы расходятся.

Иллюзия совместимости на уровне протокола

Совместимость на уровне HTTP поверхностна. Код состояния 200 и JSON-тело означают лишь то, что сервер принял ваше сообщение. Это не значит, что сервер думает так же, как предыдущий. OpenAI-совместимые эндпоинты имеют схожую структуру запроса, но они не разделяют единый поведенческий контракт. Два провайдера могут принимать идентичные полезные нагрузки и возвращать ответы, которые различаются едва заметными, но разрушительными способами

Pinging the endpoint with a "hi" message proves the network works. It proves nothing about your application.

Before you redirect production traffic, run a targeted behavioral test suite against the new provider:

  • Normal text response. Verify that content exists, is a string, and can be passed through your sanitization pipeline without casting errors.
  • Forced tool call. Set tool_choice to required. Confirm the provider honors it, and check whether content arrives as null, an empty string, or a missing key. Each of those states needs its own handler.
  • Malformed tool arguments. Inject scenarios where the model returns broken JSON inside tool arguments. Ensure your parser rejects them gracefully instead of crashing the worker.
  • Response near the token limit. Push the context window. Check the finish_reason. If the provider returns something unexpected when truncation happens, your summarization or retry logic must know how to react.

These are integration tests, not unit tests. They exercise the real relationship between your code and the provider's personality. Pass them before you call the migration done.

Build an Internal Contract

Provider differences should stop at your network boundary. Do not let them leak into business logic.

Create a normalization layer that consumes the raw SDK response and emits an object your application actually owns. Map provider-specific eccentricities into a stable internal format. If Provider A returns tool arguments as strings and Provider B returns objects, your mapper flattens both into your own ToolRequest structure. If usage is missing, your mapper either estimates it or flags the gap, but it never lets undefined seep into your cost-tracking modules.

If finish_reason is nonstandard, translate it into your own enum of terminal states: COMPLETE, TRUNCATED, TOOL_CALL, FILTERED. Your app should decide what to do based on these clean abstractions, not by sniffing raw strings from a third-party server.

This layer turns provider swaps from a game of whack-a-mole into a single-file change. You rewrite the mapper, run the behavioral tests, and move on. Your application remains untouched.

A Dependency Upgrade, Not a Config Tweak

Switching LLM providers is not like swapping CDN endpoints. It is closer to changing your database from PostgreSQL to MySQL. You would never assume the same connection string means identical query behavior. You would test locking semantics, migration paths, and indexing quirks. LLMs deserve the same respect. They are probabilistic systems masquerading as standard APIs, and their responses carry assumptions about formatting, truncation, and control flow that can shatter your application without raising a single network error.

The bug was never in the connection. It was in the assumption that compatibility means sameness. It does not. Validate the shape. Test the edges. Own the contract.

Source: The Bug Only Happened After I Switched LLM Providers

Community: GyaanSetu AI on Telegram