The proof-of-concept phase of enterprise LLM adoption is over. The companies still running pilots in 2025 are falling behind the ones that have moved to production. This is what production looks like — and what it costs to get there.
The architecture pattern that won
After three years of experimentation, a clear architecture has emerged for most enterprise LLM workloads:
- A gateway layer that handles authentication, rate limiting, prompt logging, and PII redaction
- A retrieval layer that pulls relevant context from your enterprise data, typically a vector store backed by your existing knowledge sources
- A model layer that routes requests to one of several models based on task complexity and cost
- An evaluation layer that scores outputs against defined quality metrics in production
Companies that skip any of these four layers regret it. The gateway is where governance lives. Retrieval is where competitive advantage lives. The model layer is where you avoid vendor lock-in. The evaluation layer is how you know whether the system is improving or degrading.
Pick a model strategy, not a model
The single-model approach — “we standardized on GPT-4” — looks reasonable on day one and looks naive twelve months later. Models change. Prices drop. New providers emerge. Your strategy needs to assume that the dominant model in 2026 may not even exist yet.
The pattern that holds up is task-based routing. Simple classification and extraction tasks go to small, cheap models. Reasoning-heavy tasks go to frontier models. Internal-only workloads with sensitive data go to self-hosted open weights. The router is the durable architectural decision; the models behind it are interchangeable.
Retrieval is where the work is
The dirty secret of enterprise LLM deployments is that 80% of the engineering effort goes into retrieval, not modeling. Getting the right context in front of the model at the right time is harder than it looks because:
- Enterprise documents are messy (PDFs, scanned images, tables, slides)
- Permissions matter (a user shouldn’t see chunks they can’t access in the source system)
- Freshness matters (yesterday’s policy doc shouldn’t override today’s update)
- Multi-hop questions require iterative retrieval, not a single nearest-neighbor lookup
Treat retrieval as a first-class engineering discipline. Hire for it. Measure it. The model is commodity; the retrieval system is yours.
Evaluation is non-negotiable
You cannot improve what you don’t measure, and you cannot govern what you don’t measure. Production LLM systems need:
- A frozen evaluation set that grows with reported failures
- Automated scoring against that set on every prompt or model change
- Sampled human review of production outputs for regression detection
- A clear escalation path when evaluation scores drop
Companies that skip evaluation discover six months later that their assistant has been confidently wrong about the same five things the entire time, and nobody noticed because nobody was looking.
Cost is a design decision
The economics of LLM deployment can be brutal if you let them. A poorly designed retrieval system that sends 30K tokens of context with every request will cost an order of magnitude more than a well-designed one that sends 3K tokens.
Cost discipline starts at design time:
- Cache aggressively where determinism allows
- Compress context before it hits the model
- Use small models where they’re sufficient
- Stream responses to reduce timeout pressure
- Monitor cost per resolved task, not cost per token
A good rule of thumb: if you can’t tell me the cost-per-task for your top three workloads, you don’t have a deployment, you have a science project.
Compliance and the data question
Regulated industries — finance, healthcare, government — have additional constraints that change the deployment shape. Most of them resolve to two questions: where does the data go, and who can prove what was sent.
Self-hosted models, private endpoints, and audit logging are no longer optional in regulated environments. The good news: the open-weights ecosystem is now strong enough that self-hosting a capable model is realistic, where it would have been a research project two years ago.
What we tell new clients
Don’t start by picking a model. Start by picking three workflows where the value is obvious and measurable. Build the gateway, retrieval, and evaluation layers properly. Treat the model itself as the most replaceable part of the stack.
The companies that follow this pattern ship in quarters, not years. The ones that lead with the model end up rebuilding the whole thing eighteen months later.