Why You Shouldn't Hold DB Connections During LLM Calls

AI latency isn’t just the time the model spends thinking. It also includes how you manage database connections.

Holding a DB connection while waiting for an LLM or an embedding API can drain your connection pool. Slow external calls keep sessions open far longer than they need to be.

I dug into the Honcho repo to see their fix. They swapped a single, long-lived session for short, task-specific sessions.

Old Pattern

  • Open DB session
  • Read user settings
  • Call LLM (slow)
  • Call Embedding API (slow)
  • Save results
  • Close DB session

New Pattern

  • Open DB session for pre-flight checks
  • Read needed values into variables
  • Close DB session
  • Call LLM and Embedding APIs (no DB connection held)
  • Open a new, short DB session to save results
  • Close DB session

The aim isn’t to abandon the database; it’s to keep transaction consistency separate from network waiting.

Five steps to manage your connections

  1. Define your consistency boundaries.
  2. Pull all required values into variables before any external API call.
  3. Close the database scope.
  4. Perform the slow external tasks.
  5. Open a new, short write scope to persist the final results.

Note: If you use pgvector, the search runs inside the database, so you must keep the session open during that operation.

Shortening session life improves scaling, but watch for detached-object errors in your ORM and verify that data stays consistent across transaction snapshots.

Source: https://dev.to/junhyun-dev/neurin-llm-hocul-jung-db-connectioneul-jabji-anhneun-iyu-3abg

Optional learning community: https://t.me/GyaanSetuAi