# Model serving: putting a trained model behind an API, since 2024 > How I work with model serving: putting models behind an API with NVIDIA Dynamo-Triton, Ollama and FastAPI, and scaling them on Kubernetes. - Highlights: 2 years - Experience: Since 2024 - Updated: 2026-11-08 - Source: https://muhammetsafak.com/expertise/model-serving/ - Language: en-US - Author: Muhammet Şafak --- ## Me in the Model Serving ecosystem Model serving starts where training ends: nobody can call a finished model until it runs behind an API. This page covers a category rather than a single tool; the rest of the system is on the [AI systems](https://muhammetsafak.com/expertise/ai-systems/) page. There are several ways to serve a model, and which one I pick depends on the job. NVIDIA Dynamo-Triton is the server I use to put trained models behind an API. Ollama runs open-weight models on a local machine. FastAPI comes in when I want to write the serving logic myself in Python. For custom GPU kernels in inference optimization I turn to OpenAI Triton; despite the similar name, it is a separate project from NVIDIA's server. I package the serving layer with Docker and deploy and scale it on Kubernetes. On GPU workloads the CUDA side belongs here too. ## What is the Model Serving: Putting Models Behind an API role? Model serving keeps a trained model running behind an API that applications can call. This page covers the category rather than a single tool: NVIDIA Dynamo-Triton, Ollama and FastAPI approach the same job from different places. ## What does the Model Serving: Putting Models Behind an API do? I decide where and how a model is served: a model serving server running on a GPU, Ollama on a local machine, or a FastAPI service written in Python. Then I package it as a container and deploy and scale it on Kubernetes. ## Technologies I pair it with - NVIDIA Dynamo-Triton: The model serving server, formerly NVIDIA Triton Inference Server, that I use to put trained models behind an API. - OpenAI Triton: The language and compiler I use to write custom GPU kernels for inference optimization. - Ollama: A tool that runs open-weight models on a local machine and serves them behind an API. - FastAPI: The Python web framework that puts a model behind an HTTP API. - Kubernetes: The orchestration platform I use to deploy and scale model serving and inference workloads as containers. - CUDA: The NVIDIA platform I run training and inference workloads on the GPU with. - Docker: Development environment and deployment unit. - PyTorch: The deep learning framework I use for model training, fine-tuning and adaptation with LoRA/PEFT. - TensorFlow: The deep learning framework I use in my model-training work. ## Frequently asked ### How long have you worked with model serving? Since 2024, which makes 2 years. ### What is model serving? Keeping a trained model running behind an API that applications can send requests to. Loading the model, passing the request to it and returning the answer is the job of this layer. ### What does a model serving engineer do? Decides which server, which hardware and which API a model is served with, then packages, deploys and scales it. In my work that means choosing among NVIDIA Dynamo-Triton, Ollama and FastAPI and operating the result on Kubernetes. ### What is NVIDIA Dynamo-Triton? The model serving server that puts trained models behind an API; it was formerly called NVIDIA Triton Inference Server. NVIDIA's product page now lists it under the new name. ### Are NVIDIA Triton and OpenAI Triton the same thing? No. The names are similar but they are two separate projects. NVIDIA Dynamo-Triton is a server that serves models; OpenAI Triton is a language and compiler for writing custom GPU kernels. I use the first to serve models and the second for inference optimization. ### When do Ollama and FastAPI make sense? Ollama runs open-weight models on a local machine and serves them behind an API, which is practical for development and experiments. FastAPI is the Python web framework that puts a model behind an HTTP API, and it fits when I want to write the serving logic myself. ### Do I need Kubernetes for model serving? No. For a model running on a single machine it can be overkill. I use Kubernetes when I need to deploy and scale model serving and inference workloads as containers across machines. ### Does model serving need a GPU? It depends on the model. For models that have to run on a GPU I work with CUDA; not every model needs it.