How should I abstract the LLM provider (Ollama → Bedrock/OpenAI)?
Question
We're adding an AI feature that does code analysis and text summarization. To avoid cost on developer machines we use Ollama + a local LLM (Llama 3); in production we're planning AWS Bedrock or OpenAI. Which design pattern should I apply to abstract the LLM provider? How do I tolerate, in the architecture, the structural differences between local token limits/response times and production?
Answer
Short answer
Build a narrow port/adapter (Strategy) around your capability, not around the provider. A small interface plus real implementations selected by config; nothing more.
Context
Your real danger here is over-abstraction: people try to turn this into a generic “AI framework” and burn months on it. All you need is one thin interface. In Laravel you already have the mechanism that decides which implementation that interface resolves to — the binding I describe in using service providers and the container correctly exists for exactly this.
Why
-
Abstracting the capability rather than the vendor keeps the interface small. A tiny contract like
complete(prompt, opts): Resultis enough; Ollama, Bedrock and OpenAI become three implementations of it. The moment you leak provider-specific params into the core logic, the abstraction stops doing its job. -
Differences belong in configuration, not business logic. If a distinction like “small context window locally, big in prod” turns into an
ifblock, every new provider adds another branch to the codebase.
What to do
-
Define the interface by capability and let config pick.
config/envdecides which implementation runs; the caller shouldn’t know which provider it’s talking to. -
Put explicit timeouts + a cheap fallback in place. In production the provider slows down or hits a rate limit; put a clear timeout on every call and fall back to a cheap/local option when needed.
-
Budget tokens at the edge. Each model has a different context window; truncate/summarize the input at the edge so it fits the model.
-
Put streaming behind the same contract. If you need it, keep streaming support optional in the same interface; the caller shouldn’t know the provider’s stream API.
-
Keep a golden-prompt test suite. Run the same prompt set against each provider; that’s what catches what broke when you swap providers.
Bottom line: I’d build one small interface, three real implementations and config-based selection — anything more is YAGNI. Local Ollama is for dev cost and fast iteration; treat the production differences (context window, latency, rate limits, cost) as configuration, not as branches in business logic. The narrower the abstraction stays, the sturdier it is.
Compiled from questions I have run into in the field or been asked.
Related Reading
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.