Case Study: Automating 40% of a FinTech's Operations with AI Agents

Most AI case studies are written like award submissions: a heroic before-and-after with the messy middle surgically removed. This one is written the other way. It is the full account of a six-week engagement with a mid-sized lending fintech — including the pipeline that failed its first evaluation, the week the operations team quietly stopped trusting the system, and the unglamorous data work that ended up mattering more than any model choice. If you are considering AI automation for your own operations, the messy middle is the part you actually need to read.

We have anonymized the client — call them NorthLend — at their request, but every number in this article is real, measured, and was signed off by their operations director. Where a detail would identify them, we have generalized it and said so.

The client and the bottleneck

NorthLend is a lending company with around 90 employees, of whom 14 worked in operations. Their business depends on processing loan applications quickly: every application arrives with a stack of supporting documents — bank statements, identity documents, income proofs, business registrations — in every format imaginable. PDFs, phone photos of paper documents, spreadsheets, occasionally a fax. Yes, a fax.

When we ran a time audit in week one, the numbers were stark. The operations team's day broke down into four repetitive task families: classifying and filing inbound documents (roughly 28% of team hours), manually keying data from those documents into their loan management system (24%), triaging the shared operations inbox — customer queries, broker updates, document resubmissions — into the right queues (19%), and running compliance pre-checks against a rulebook that lived partly in a wiki and mostly in two senior employees' heads (11%). That is 82% of a 14-person team's time spent on work that follows patterns, with the actual judgment calls — approve, decline, escalate — squeezed into what remained.

The cost was not just payroll. Application turnaround time averaged 3.4 days, and their own funnel data showed that every additional day of waiting cost them roughly 7% of applicants to competitors. The bottleneck was not underwriting judgment. It was everything before the judgment.

Scoping: what we said no to

The first meeting produced a wish list of eleven possible automations, including an ambitious 'AI underwriting assistant' that would draft approval recommendations. We cut the list to four and pushed the underwriting assistant off the roadmap entirely — not because it was impossible, but because it failed our error-tolerance test. A misfiled document gets caught and refiled; a biased or hallucinated lending recommendation is a regulatory incident. First projects should be built where mistakes are cheap, because there will be mistakes.

The four survivors were exactly the four task families from the time audit: document classification, data extraction, inbox triage, and compliance pre-screening. Each was frequent, painful, well-defined, and — critically — each produced output a human would review during the initial rollout, which meant the system could earn trust incrementally instead of demanding it upfront.

The architecture, in plain language

Everything runs on a single orchestration layer that connects to NorthLend's existing systems through their APIs — their loan management system, their email platform, and their document store. This matters more than it sounds: we changed nothing about the tools the team already used. Documents still landed in the same system; the queue still looked like the same queue. The AI works behind the interfaces people already knew, which is a large part of why adoption succeeded.

On top of that layer sit four pipelines. Each follows the same internal pattern: validate and normalize the input, call a language model with carefully engineered context, force the output into a strict schema (no free-text answers allowed anywhere), attach a confidence score, and route by that score — high confidence flows through automatically, low confidence lands in a human review queue with the model's reasoning attached. Every decision is logged with its full input, output, and confidence, which later proved to be the single most valuable design choice in the entire system.

Pipeline 1 — Document classification

Every inbound file is identified (is this a bank statement? an ID? an income proof? page 3 of a multi-part upload?), matched to the right application, and filed. The hard part was not classification accuracy — modern models are excellent at this — it was the compound documents: a single 40-page PDF containing a bank statement, two utility bills, and a photographed passport. We added a document-splitting stage that segments files before classification, which took the pipeline from 81% to 96.5% end-to-end accuracy on the evaluation set.

Pipeline 2 — Structured data extraction

Once classified, documents flow to extraction: names, account numbers, balances, income figures, dates — around 60 fields across document types, each validated against format rules and cross-checked against the application form. Any mismatch (the applicant says one income, the bank statement suggests another) is not an error to fix silently; it is a signal, flagged for the underwriter. This reframing — extraction as verification, not just data entry — is what turned the ops team from skeptics into advocates, because the system started catching discrepancies humans had been missing at the end of long days.

Pipeline 3 — Inbox triage

The shared inbox received about 350 messages a day. The triage agent reads each one, identifies the intent (document resubmission, status query, broker escalation, complaint, spam), links it to the relevant application, drafts a suggested reply for the routine categories, and routes it to the right queue. Humans send every reply — the agent only drafts. Median time-to-first-response dropped from 5.1 hours to 41 minutes, which had a second-order effect nobody predicted: resubmission chase-up emails fell by a third, because applicants who get fast answers send fewer 'just checking' messages that themselves need triaging.

Pipeline 4 — Compliance pre-screening

The most sensitive pipeline, and deliberately the most conservative. It runs every application against NorthLend's compliance rulebook — which we first had to actually write down, extracting it from the wiki and from structured interviews with the two senior staff who carried it in their heads. The agent never approves anything; it only flags, with a citation to the specific rule and the specific evidence. Its job is to make sure nothing reaches a human reviewer unexamined, not to replace the review. In the first month it flagged three genuine issues that the manual process had historically missed — a sanctions-list near-match and two document inconsistencies — and that fact alone ended the internal debate about whether the project was worth it.

“We expected the AI to be fast. What we didn't expect was for it to be thorough. It reads page 34 of the bank statement with exactly the same attention as page 1, which is more than I can honestly say for any of us at 6pm on a Friday.”

— Operations Director, NorthLend

What went wrong (three things, specifically)

First: the extraction pipeline failed its initial evaluation, badly. On the first 100-document test set it scored 72% — nowhere near deployable. The cause was not the model; it was that NorthLend's historical 'ground truth' data was itself wrong in about one case in ten, because the humans keying data had been making errors nobody measured. We spent four unplanned days having senior staff re-verify the evaluation set. The model had been more accurate than the humans it was being graded against. This is now the first thing we check on every engagement.

Second: in week four, the ops team quietly stopped trusting the triage agent after it misrouted one angry complaint into the low-priority queue. One visible failure outweighed thousands of silent successes — which is how trust in automation always works. The fix was social, not technical: we added a daily five-minute review where the team looked at the previous day's lowest-confidence decisions together. Seeing the system say 'I am not sure about this one' rebuilt trust faster than any accuracy statistic, because uncertainty honestly expressed reads as competence, not weakness.

Third: we underestimated document volume growth. The system worked so well that NorthLend's intake team started accepting application formats they had previously turned away, and volume rose 22% in two months. Good problem, real problem — it pushed our cost projections up and required a routing optimization we had planned to skip: cheap, fast models now handle the easy 70% of classifications, with the frontier model reserved for the genuinely ambiguous cases. That change cut inference costs by more than half.

The rollout: shadow mode, then tiers

For two weeks, all four pipelines ran in shadow mode — producing decisions, having them logged and compared, with zero effect on real operations. Shadow mode surfaced 30-plus edge cases (foreign-language documents, a broker who emailed screenshots of PDFs, statements from a bank whose layout broke the splitter) and generated the accuracy evidence that made the go-live decision boring rather than brave. We then cut over in confidence tiers: automation started with only the top 60% most-confident decisions, and the threshold was lowered every week as the evaluation numbers held. There was never a big-bang moment, and that was the point.

The numbers, six months on

• Operations capacity reclaimed: 40% — measured by the same time-audit methodology as the week-one baseline, not estimated
• Application turnaround: 3.4 days → 1.1 days median, with the funnel improvement worth more than the payroll saving
• Extraction accuracy: 98.2% on the continuously-growing evaluation set, versus a re-measured human baseline of 96.4%
• Exceptions routed to humans: 13% of all decisions — deliberately conservative, and trending down about half a point per month
• Running cost: under $2,100/month all-in, against a conservatively-estimated $38,000/month in reclaimed capacity

The headline the client cares about is none of these. It is that nobody was laid off: the four roles most affected moved onto the underwriting support and broker relations work the company had been unable to staff, and employee satisfaction in ops — which they survey quarterly — went up. Automating the work people hate is a very different project from automating the people, and the difference is a choice you make at scoping time.

The six weeks, week by week

Timelines in case studies usually get compressed into a single reassuring number, so here is the uncompressed version. Week one was entirely non-technical: the time audit, process shadowing (we sat with four ops staff for two days each and wrote down every decision they made), and the data audit that pulled 100 real applications with their full document stacks. By Friday of week one we had the four-pipeline scope, the baseline numbers, and — importantly — the discovery that the compliance rulebook existed nowhere in writing.

Weeks two and three were the feasibility spike and the rulebook extraction running in parallel. The spike is where the extraction pipeline failed its first evaluation and we lost four days re-verifying ground truth — the schedule absorbed it because we plan spike phases with slack for exactly this kind of discovery. Week three ended with a go/no-go meeting where we presented per-pipeline accuracy numbers on real data. Classification and triage were clearly ready; extraction was ready after the ground-truth fix; compliance pre-screening was downgraded to flag-only mode (it had been scoped with a suggested-decision feature that the spike showed was not reliable enough — so we cut it, and told the client why).

Weeks four and five were the production build: integrations with the loan management system and email platform, the schema enforcement layer, confidence routing, the audit log, and the evaluation suite wired into deployment so that no prompt or model change could ship without passing it. Week six was shadow mode — which we have described — plus training sessions that were deliberately run by the ops team's own senior members rather than by us. The system went live on the Monday of week seven at the 60% confidence tier. Total elapsed time from kickoff to first automated decision in production: 43 days.

Questions other clients ask about this project

What did it cost?

The build was in the mid five figures — inside the $30k–$90k layer-2 bracket we publish in our integration playbook — and running costs are under $2,100 a month, of which model inference is less than a third; the rest is hosting, monitoring, and document storage. Against the measured $38,000/month in reclaimed capacity, the build paid for itself in eleven days of operation. We share these brackets because cost opacity is the single biggest source of stalled decisions in this market, and we would rather lose a prospect to honest numbers than win one on vague ones.

What about hallucinations?

This is the right question, and the answer is architectural rather than hopeful. No pipeline in the system ever emits free text into a system of record. Every output is forced into a strict schema and validated against format rules and cross-references before it goes anywhere — an extracted account number that fails checksum validation, or a date that doesn't parse, is automatically an exception for human review, not a value that flows through. Where the model drafts prose (the triage replies), a human sends every message. Hallucination risk is real; it is also exactly why the boring engineering exists.

What happens when the models change?

Twice during the first six months, we migrated a pipeline to a newer, cheaper model. Each migration was one day of work: run the full evaluation suite against the new model, compare, switch the config, watch the dashboards. This is the payoff of treating the evaluation set as the system's source of truth — the growing library of verified real cases (over 4,000 by month six, fed by every human correction) means model upgrades are evidence-based and boring. NorthLend's team runs these migrations themselves now, which brings us to the last question.

Are you still involved?

Lightly, and by their choice rather than by contractual hostage-taking. At handover NorthLend received everything: source code, prompts, evaluation sets, infrastructure access, and a runbook. Their own engineers operate the system day to day. We do a quarterly review and remain on call for architecture questions. We structure every engagement this way — it costs us recurring revenue and wins us referrals, and we consider that a good trade.

The six-month retrospective: what we'd do differently

Every project deserves an honest retro, including the successful ones. Three things we would change. First, we would run the ground-truth verification in week one rather than discovering the need for it in week three — it is now a standard item in our data audit, but on this project it cost us four days of schedule and a tense Wednesday. Second, we would introduce the daily low-confidence review from day one of shadow mode rather than inventing it as a trust-repair measure in week four; it turned out to be the single best adoption mechanism in the project, and we stumbled into it.

Third — and this is the counterintuitive one — we would automate less in the first release. The triage pipeline launched with draft replies for six message categories; the team really only trusted and used three of them for the first month, and the other three drafts created review noise that slowed people down. We now scope first releases to the narrowest version the payback math supports, because a system people trust at 60% of its potential beats a system people distrust at 100%. Capability is easy to add to a trusted system; trust is hard to add to a capable one.

What we would not change: the shadow-mode discipline, the flag-only compliance scope, the decision to change nothing about the team's interfaces, and the week of process anthropology before any code. Every one of those choices looked slow in week one and paid for itself by week six. The client's operations director put it best in the retro: the project felt boring the whole way through, and that was apparently the point.

What you can steal from this project

If you take nothing else from this case study, take these: measure the baseline before you build, because the win you cannot prove is a win that did not politically happen. Distrust your ground-truth data until you have verified it. Put confidence scores and human escalation in the architecture from day one, not as a retrofit. Run shadow mode until go-live feels boring. And route by difficulty — most of your volume does not need your most expensive model.

None of this is exotic. It is operations discipline applied to a new kind of software component. The teams that treat AI that way are compounding a 40% capacity advantage right now, quietly, while their competitors are still watching demos.