# AI systems engineer: I treat the model as one component of the larger software system that carries it

> I design, customize, orchestrate and securely operate AI models as components of a software system; seven areas, from architecture to security.

- Technologies: Python, TypeScript, Rust, PyTorch, CUDA, OpenAI Triton, NVIDIA Dynamo-Triton, Kubernetes, pgvector, Milvus, Chroma, Apache Kafka, TensorFlow, Model Context Protocol, Retrieval-Augmented Generation
- Experience: Since 2024
- Posts: 7
- Projects: 16
- Updated: 2026-10-25
- Source: https://muhammetsafak.com/expertise/ai-systems/
- Language: en-US
- Author: Muhammet Şafak

---
## Me in this ecosystem

I treat AI not as a standalone feature but as a component of a larger software
system, and I solve the feature not in the model itself but in the system that
carries it in production. To me AI Systems Engineering is not just AI
development, ML engineering, agentic AI or AI infrastructure; it is a
systems-focused engineering approach tied to all of them. LLMs, RAG, MCP, agents
and fine-tuning are today's technical components of this discipline; when model
architectures and protocols change, the questions of architecture, integration,
security and operation stay where they are.

This page covers four topics in its own right: AI Systems Engineering,
Retrieval-Augmented Generation, AI Agents and Model Context Protocol. I keep
two things apart: the seven areas below are the scope of the discipline, and the
links in this section are the examples visible on this site. There is also work I
do not detail here because it is commercial; I wrote it up separately in the
cards and in the note at the end of the page.

All four have a visible example on this site. The gateway side is in the
[AI gateway product](/portfolio/ragmux/) and [its labs record](/labs/ragmux-lab/),
the work left after the RAG decision is in
[a separate post](/blog/the-real-work-starts-after-you-choose-rag/), and the MCP side is in
[the documentation server product](/portfolio/contextator/) and the
[one-server-per-project post](/blog/one-mcp-server-per-project-stops-scaling/).
Where an LLM enters the development loop is in [the code-review CLI](/portfolio/commitbrief/)
and the [AI self-review post](/blog/ai-self-review-what-it-speeds-up-where-it-falls-short/),
and the question of abstracting the provider is in
[a Just Ask answer](/just-ask/abstracting-the-llm-provider-from-ollama-to-bedrock-or-openai/).
The book-length version of the agent side is
[Engineering Reliable AI Agents](/books/engineering-reliable-ai-agents/);
[a skill that drafts a fix plan from a Sentry error](/labs/sentryfixer-lab/) is a small example that touches no file
without approval. Evidence for my non-AI event-driven and queue work is in
[a separate book](/books/designing-reliable-event-driven-systems/).

### The seven areas of the discipline

I treat AI as a component of a larger software system, not a standalone feature; the seven areas are one whole, from designing that component to operating it.

- **AI Systems Architecture** — Architecture is the backbone of AI Systems Engineering. In AI-native and AI-enabled systems I separate the model, application and infrastructure layers and keep components replaceable. In a provider-agnostic architecture, changing the model should leave the application untouched; I build that with a layer such as an AI Gateway, with fault tolerance and scalability in mind.
- **Model Engineering & Customization** — I own model selection and evaluation, and domain-specific adaptation. Fine-tuning and efficient methods such as LoRA/PEFT, model training with TensorFlow and PyTorch, and Machine Learning Research sit inside this area. I reduce latency and cost with Inference Optimization and put the model behind an API with Model Serving; small, local and open-weight models are among my options.
- **Agentic AI & Orchestration** — I treat AI Agents as unreliable decision-makers and run them inside control systems: the model proposes, code decides. In Tool Calling I design the tool contract, permissions and state; in Human-in-the-Loop I decide where human approval sits. Protocols such as the Model Context Protocol and the Agent2Agent Protocol, and orchestration and authority boundaries in Multi-Agent Systems, belong to this area too.
- **AI Integration & Data Pipelines** — In Retrieval-Augmented Generation the whole pipeline is mine, from parsing a document and drawing chunk boundaries to producing Embeddings and fusing Vector Search with full-text search. I build third-party and enterprise integrations, ETL/ELT flows, event-driven workflows and queues as parts of the data pipeline around the model.
- **AI Infrastructure & Platform Engineering** — I build AI Gateways, model routing and multi-provider integration, self-hosted systems, containerized deployment and AI delivery pipelines. Resource management for inference workloads, Model Serving on GPUs and Kubernetes, and infrastructure automation are the everyday work of this area.
- **Reliability, Security & Observability** — An unreliable AI system is a risk, not a feature. I design guardrails and policy enforcement, Prompt Injection and tool execution security, data privacy and access control. I manage tokens, latency and cost as a budget; I watch requests with LLM Observability, catch regressions with LLM Evaluation, and contain failure with retry, timeout, fallback and circuit breakers.
- **Applied AI Engineering** — I apply AI to real problems: extending an existing software product with AI capabilities, AI-assisted automation, developer tools and moving a research prototype into a production system belong to this area. The measure is not how impressive the model is but whether the problem is solved and the system stays operable.

### Decisions I make in an AI feature

Before which model gets chosen, these four decisions around the model set how long the system lives.

1. **Where the model sits** — I put the model behind an interface instead of embedding it in the application. The cost is an abstraction layer and giving up provider-specific features; in return, changing the model or the provider is a configuration change, not a rewrite.
2. **Budget** — I set tokens, latency and cost when the request is designed, not when the invoice arrives. Because output tokens are only final once the response is complete, a budget limit cannot stop a single request midway; the overshoot is at most one request, and the one rejected is the next.
3. **Authority** — Every tool I give an agent is an authority. I define tools narrowly, tie irreversible actions to human approval, and treat any text that enters the model's context as input that can steer it.
4. **Measurement** — I do not judge output quality by looking once; I measure it with an evaluation set that can be rerun. When the model, the prompt or the retrieval changes, the regression should show up there first, not with the user.

### The limit of the discipline

AI Systems Engineering is not only AI development, only ML engineering, only agentic AI or only AI infrastructure; it is the systems-focused engineering approach where they meet. That is why I do not tie the expertise to one technology, provider or trend. LLMs, RAG, MCP, agents and fine-tuning are today's technical components; when model architectures and protocols change, the questions of architecture, integration, security and operation stay where they are.

## Frequently asked

### How long have you worked with AI systems?

Since 2024, 2 years. This site has 7 posts tagged with this area.

### What do you mean by AI systems engineering?

Integrating AI models into real software systems, customizing and orchestrating them, and running them securely and sustainably. The model is only one component; where the provider, the budget, authority, context and evaluation live is the system's question.

### Is this expertise only LLM integration?

No. Calling an API or writing a prompt is a small part of the discipline. The seven areas run from model customization to agent orchestration, from data pipelines to infrastructure, from security to observability.

### What is left after you decide on RAG?

The questions retrieval does not answer: where the provider key lives, which application uses which model, where spending is cut off and how quality is measured. I solve them in the system layer outside the model.

### Why do you not trust an AI agent?

I assume the model will sometimes be wrong and design around that: the model proposes, code decides. Tools are defined narrowly and irreversible actions are tied to human approval.

### Should I run one MCP server per project?

What should multiply is not the number of servers but the endpoint and the authority boundary. Connecting an MCP server opens a trust boundary, so each project should get its own endpoint and its own authority; one server can carry that with several endpoints.