The request succeeded. The response was valid JSON. The SDK stayed quiet. And yet the application collapsed.

This is the story of what happens when you treat an LLM provider swap like a configuration change instead of a structural gamble. You paste in a new base URL, swap the API key, and keep the request body identical because the docs promise an OpenAI-compatible endpoint. For a basic "hello world" prompt, it works. You celebrate. Then real traffic hits, and the seams split open.

The Illusion of Wire Compatibility

Compatibility at the HTTP layer is shallow. A 200 status code and a JSON body mean the server accepted your message. It does not mean the server thinks the same way as the previous one. OpenAI-compatible endpoints share a request shape, but they do not share a behavioral contract. Two providers can ingest identical payloads and return answers that diverge in subtle, destructive ways.

Your code makes assumptions. You assume message.content is a string because it always was before. You assume a tool call arrives with clean, parseable JSON. You assume finish_reason signals what you think it signals. These assumptions are invisible until they are fatal.

Consider the crash that started it all:

const text = response.choices[0].message.content.trim();

This line looks innocent. It worked for weeks. Then the new provider returned a tool call. In that moment, message.content was not an empty string. It was null. The actual payload lived inside message.tool_calls, but the parser had already moved on, calling .trim() on nothing. The API did not throw. The network layer did not complain. Your own parser killed the request.

Where Providers Quietly Diverge

The differences do not announce themselves in changelogs. They sit in the margins of the response object, waiting for edge cases.

Tool-call formatting. One provider sends tool arguments as a pre-validated JSON object. Another sends them as an escaped string inside a field. A third might split a long tool call across multiple streaming deltas, forcing you to buffer chunks before you can even see if the structure is valid. If your application expects a single parseable blob, it chokes.

Finish reasons. OpenAI uses specific strings like "stop", "length", "tool_calls", and "content_filter". A compatible provider might return "end_turn" or simply omit the field when the model hits the token ceiling. If your retry or fallback logic waits for "length" to detect truncation, it will sit idle while the user sees a half-finished answer.

Usage fields. Some providers strip token counts from streaming responses to shave milliseconds off latency. Others append usage only to the final chunk, or omit it entirely in non-streaming calls. If you charge customers per token and your accounting code expects usage.total_tokens to exist in every response object, your billing pipeline will silently record zeros.

Streaming behavior. Server-sent events are supposed to be standard, yet providers flush buffers at different frequencies. Event boundaries vary. One provider terminates a stream with a [DONE] signal. Another drops the connection cleanly with no sentinel at all. If your client blocks waiting for a specific closing marker, it hangs.

Errors and timeouts. A rate limit might arrive as a 429 with a retry-after header from one provider, and as a vague 502 from another. Some providers accept the request and then go quiet for two minutes before a network timeout. The OpenAI SDK will not magically normalize these into the exception types your logs expect.

Defensive Parsing for Unpredictable Shapes

The fix is not to trust the schema. The fix is to treat every response as a suspect.

Do not assume content is a string. Check it before you touch it.

const content = response.choices?.[0]?.message?.content;
const text = typeof content === "string" ? content.trim() : "";

Do not assume tool arguments are valid JSON. The model proposes an action. Your code must decide whether that proposal is safe enough to execute. Wrap every tool argument parse in a try-catch. If JSON.parse throws, treat the tool call as malformed garbage and route it to a failure handler. A hallucinated bracket or a missing quote should never bubble up as an unhandled exception.

If tool_calls exists but content is missing, your application should recognize a state transition. The user did not get a chat reply. The system got a work order. Those are two different paths, and your router should know the difference before it attempts string manipulation.

Behavioral Tests Before You Deploy

Melakukan ping ke endpoint dengan pesan "hi" membuktikan bahwa jaringan berfungsi. Hal itu tidak membuktikan apa pun tentang aplikasi Anda.

Sebelum Anda mengalihkan trafik produksi, jalankan rangkaian uji perilaku (behavioral test suite) yang ditargetkan terhadap penyedia baru:

  • Respons teks normal. Verifikasi bahwa content ada, berupa string, dan dapat dilewatkan melalui pipeline sanitasi Anda tanpa kesalahan casting.
  • Pemanggilan tool secara paksa (Forced tool call). Atur tool_choice menjadi required. Pastikan penyedia mematuhinya, dan periksa apakah content datang sebagai null, string kosong, atau kunci yang hilang. Setiap kondisi tersebut memerlukan handler-nya sendiri.
  • Argumen tool yang malformed. Masukkan skenario di mana model mengembalikan JSON yang rusak di dalam argumen tool. Pastikan parser Anda menolaknya dengan baik alih-alih membuat worker crash.
  • Respons mendekati batas token. Dorong batas context window. Periksa finish_reason. Jika penyedia mengembalikan sesuatu yang tidak terduga saat terjadi pemotongan (truncation), logika ringkasan atau retry Anda harus tahu cara bereaksi.

Ini adalah uji integrasi, bukan uji unit. Uji ini menguji hubungan nyata antara kode Anda dan karakteristik penyedia. Lulusi uji ini sebelum Anda menyatakan migrasi selesai.

Bangun Kontrak Internal

Perbedaan penyedia harus berhenti di batas jaringan Anda. Jangan biarkan hal tersebut bocor ke dalam logika bisnis.

Buatlah lapisan normalisasi yang mengonsumsi respons SDK mentah dan menghasilkan objek yang benar-benar dimiliki oleh aplikasi Anda. Petakan keanehan spesifik penyedia ke dalam format internal yang stabil. Jika Penyedia A mengembalikan argumen tool sebagai string dan Penyedia B mengembalikan objek, mapper Anda meratakan keduanya ke dalam struktur ToolRequest Anda sendiri. Jika penggunaan (usage) hilang, mapper Anda dapat mengestimasinya atau menandai celah tersebut, tetapi jangan pernah membiarkan undefined merembes ke dalam modul pelacakan biaya Anda.

Jika finish_reason tidak standar, terjemahkan ke dalam enum status terminal Anda sendiri: COMPLETE, TRUNCATED, TOOL_CALL, FILTERED. Aplikasi Anda harus memutuskan apa yang harus dilakukan berdasarkan abstraksi bersih ini, bukan dengan mengendus string mentah dari server pihak ketiga.

Lapisan ini mengubah pergantian penyedia dari permainan whack-a-mole menjadi perubahan satu file saja. Anda menulis ulang mapper, menjalankan uji perilaku, dan selesai. Aplikasi Anda tetap tidak tersentuh.

Pembaruan Dependensi, Bukan Sekadar Penyesuaian Konfigurasi

Mengganti penyedia LLM tidak seperti menukar endpoint CDN. Ini lebih mirip dengan mengubah database Anda dari PostgreSQL ke MySQL. Anda tidak akan pernah berasumsi bahwa connection string yang sama berarti perilaku kueri yang identik. Anda akan menguji semantik penguncian (locking semantics), jalur migrasi, dan keunikan pengindeksan. LLM layak mendapatkan rasa hormat yang sama. Mereka adalah sistem probabilistik yang menyamar sebagai API standar, dan respons mereka membawa asumsi tentang pemformatan, pemotongan (truncation), dan alur kontrol yang dapat menghancurkan aplikasi Anda tanpa memunculkan satu pun kesalahan jaringan.

Bug tersebut tidak pernah ada pada koneksi. Bug tersebut ada pada asumsi bahwa kompatibilitas berarti kesamaan. Kenyataannya tidak demikian. Validasi bentuknya. Uji batas-batasnya (edges). Kuasai kontraknya.

Sumber: The Bug Only Happened After I Switched LLM Providers

Komunitas: GyaanSetu AI di Telegram