Opening bank statements is nobody's idea of fun. They arrive as scanned PDFs, CSV exports, or XML files dressed up with

Opening bank statements is nobody's idea of fun. They arrive as scanned PDFs, CSV exports, or XML files dressed up with arcane acronyms like OFX. For accountants, bookkeepers, and fintech builders, turning these documents into clean, structured data is a constant headache. When large language models burst onto the scene, they seemed to offer an escape hatch. Just feed the machine a PDF and ask for JSON. What could go wrong?

I learned exactly what could go wrong while building StatementDecoder, a tool designed to convert bank statements into usable data. Like many developers, I assumed the hard part would be teaching the system to read diverse document layouts. I was wrong. Reading the documents was almost trivial. The real nightmare was recognizing when the machine had quietly invented a number or swapped two digits in a transaction amount.

The Demo That Worked Too Well

My first attempt was seductively simple. I piped bank statements directly into an LLM and requested structured JSON in return. The results felt like magic. The model handled different layouts with ease. It read scanned PDFs that standard parsers choked on. It seemed to understand tables, headers, and multi-page statements without explicit instructions. For a few glorious hours, I thought the problem was solved.

Then I tested it against real customer data, and the magic evaporated. UK banks each use their own statement designs, and the differences are not merely cosmetic. Wise statements carry their own formatting quirks. Revolut CSV exports look straightforward until you notice how they handle multi-currency transactions and metadata fields. Old OFX files, a format that genuinely looks like it belongs in the 1990s, throw archaic tag structures and encoding issues at any parser that expects modern markup.

The model still extracted data far better than any off-the-shelf template system. But far better is not good enough when money is involved.

When 99% Accuracy Is a Failure

Here is the fundamental problem with using AI for financial data extraction. If a model processes two hundred transaction rows and gets one hundred ninety-nine correct, the output looks pristine. The JSON is well-formed. The keys and values align. A casual review might show nothing suspicious. Yet if that single error transposes two digits in an amount, flips a deposit into a withdrawal, or shifts a decimal point, your bookkeeping is corrupted. You will not catch it by eyeballing a wall of structured data.

A human reviewing raw JSON rarely spots a swapped digit in a transaction amount. The formatting is perfect, which paradoxically makes the mistake more dangerous. You cannot ship a financial tool that is right most of the time. It must be right, or it must loudly announce that it is unsure.

My initial reaction was predictable. I engineered better prompts. I upgraded to more capable models. I experimented with chain-of-thought reasoning to make the model show its work. None of this fixed the core issue. I was asking the same probabilistic system to generate an answer and then asking that identical system to certify that the answer was correct. That is not verification. That is self-consistency theater.

Let the Math Decide

Bank statements have a feature that most documents do not: built-in arithmetic constraints. The opening balance plus the sum of all transactions must equal the closing balance. Running balances, when present, must line up row by row. These are not stylistic preferences. They are hard rules.

I rebuilt the architecture around this insight. Now, every extraction, regardless of origin, flows through a validation layer before any user sees it. It does not matter whether the data came from an LLM interpreting a fuzzy PDF, an OCR engine reading a scanned page, or a direct CSV parse. The validator treats all sources as equally suspect.

The check is brutally simple. Add every transaction to the opening balance. Compare the result to the stated closing balance. If the numbers do not match, something is wrong. Flag the statement for review. Reject the extraction. Do not let it reach the user.

This single change altered the entire character of the product. The language model no longer needed to be perfect. It only needed to be good enough to produce output that could survive a mathematical trial. The pressure shifted from achieving impossible accuracy in an unconstrained domain to building a tight feedback loop between generation and verification.

वैलिडेटर ने त्रुटियों में पैटर्न भी उजागर किए। कुछ दस्तावेज़ प्रकार लगातार गणितीय जाँच (math check) में विफल हो रहे थे, जिससे मुझे सटीक रूप से पता चला कि मुझे कहाँ प्रयास करने की आवश्यकता है। हर जगह बिना सोचे-समझे प्रॉम्प्ट इंजीनियरिंग में सुधार करने के बजाय, मैं देख सकता था कि विशिष्ट बैंक लेआउट के कारण व्यवस्थित गलतियाँ हो रही थीं।

जहाँ कोड की ज़रूरत है वहाँ कोड, जहाँ AI चमकता है वहाँ AI

शायद सबसे बड़ा सबक यह अहसास था कि पाइपलाइन के कितने बड़े हिस्से को AI की बिल्कुल भी आवश्यकता नहीं थी। जब मेरा सामना अव्यवस्थित ऑस्ट्रेलियाई OFX फ़ाइलों से हुआ, तो मेरी सहज प्रवृत्ति समस्या पर टोकन (tokens) खर्च करने की थी। मैंने संक्षेप में टूटे हुए XML को मॉडल में डालने और पार्सिंग से पहले संरचना को ठीक करने के लिए कहने पर विचार किया। इसके बजाय, मैंने बीस लाइनों का डिटरमिनिस्टिक कोड लिखा। इसने एन्कोडिंग की विसंगतियों और खराब टैग्स को तुरंत ठीक कर दिया, जिससे प्रति फ़ाइल शून्य लागत लगी और पूर्ण पुनरुत्पादकता (reproducibility) मिली।

उस अनुभव ने यह स्पष्ट कर दिया कि एक्सट्रैक्शन पाइपलाइन को कैसे व्यवस्थित किया जाना चाहिए। इसमें तीन अलग-अलग काम हैं, और उन्हें आपस में नहीं मिलाया जाना चाहिए।

• मॉडल अव्यवस्थित दस्तावेज़ों को समझता है। टेढ़े-मेढ़े टेबल, मिश्रित फ़ॉन्ट और हस्तलिखित