Expertise
Model serving: putting a trained model behind an API, since 2024
Training a model is not the end of the job: serving it behind an API, running it on a GPU and scaling it is its own engineering work.
- years
- 2
Me in the Model Serving ecosystem
Model serving starts where training ends: nobody can call a finished model until it runs behind an API. This page covers a category rather than a single tool; the rest of the system is on the AI systems page.
There are several ways to serve a model, and which one I pick depends on the job. NVIDIA Dynamo-Triton is the server I use to put trained models behind an API. Ollama runs open-weight models on a local machine. FastAPI comes in when I want to write the serving logic myself in Python. For custom GPU kernels in inference optimization I turn to OpenAI Triton; despite the similar name, it is a separate project from NVIDIA’s server.
I package the serving layer with Docker and deploy and scale it on Kubernetes. On GPU workloads the CUDA side belongs here too.
What is the Model Serving: Putting Models Behind an API role?
Model serving keeps a trained model running behind an API that applications can call. This page covers the category rather than a single tool: NVIDIA Dynamo-Triton, Ollama and FastAPI approach the same job from different places.
What does the Model Serving: Putting Models Behind an API do?
I decide where and how a model is served: a model serving server running on a GPU, Ollama on a local machine, or a FastAPI service written in Python. Then I package it as a container and deploy and scale it on Kubernetes.
Technologies I pair it with
-
NVIDIA Dynamo-Triton
The model serving server, formerly NVIDIA Triton Inference Server, that I use to put trained models behind an API.
-
OpenAI Triton
The language and compiler I use to write custom GPU kernels for inference optimization.
-
Ollama
A tool that runs open-weight models on a local machine and serves them behind an API.
-
FastAPI
The Python web framework that puts a model behind an HTTP API.
-
Kubernetes
The orchestration platform I use to deploy and scale model serving and inference workloads as containers.
-
CUDA
The NVIDIA platform I run training and inference workloads on the GPU with.
-
Docker
Development environment and deployment unit.
-
PyTorch
The deep learning framework I use for model training, fine-tuning and adaptation with LoRA/PEFT.
-
TensorFlow
The deep learning framework I use in my model-training work.
Frequently asked
8 questions
-
How long have you worked with model serving?
Since 2024, which makes 2 years.
-
What is model serving?
Keeping a trained model running behind an API that applications can send requests to. Loading the model, passing the request to it and returning the answer is the job of this layer.
-
What does a model serving engineer do?
Decides which server, which hardware and which API a model is served with, then packages, deploys and scales it. In my work that means choosing among NVIDIA Dynamo-Triton, Ollama and FastAPI and operating the result on Kubernetes.
-
What is NVIDIA Dynamo-Triton?
The model serving server that puts trained models behind an API; it was formerly called NVIDIA Triton Inference Server. NVIDIA's product page now lists it under the new name.
-
Are NVIDIA Triton and OpenAI Triton the same thing?
No. The names are similar but they are two separate projects. NVIDIA Dynamo-Triton is a server that serves models; OpenAI Triton is a language and compiler for writing custom GPU kernels. I use the first to serve models and the second for inference optimization.
-
When do Ollama and FastAPI make sense?
Ollama runs open-weight models on a local machine and serves them behind an API, which is practical for development and experiments. FastAPI is the Python web framework that puts a model behind an HTTP API, and it fits when I want to write the serving logic myself.
-
Do I need Kubernetes for model serving?
No. For a model running on a single machine it can be overkill. I use Kubernetes when I need to deploy and scale model serving and inference workloads as containers across machines.
-
Does model serving need a GPU?
It depends on the model. For models that have to run on a GPU I work with CUDA; not every model needs it.
Let us work together
If you have work in this ecosystem, tell me what you are building and we will talk about how to build it.