Building an Inference Server on AWS (EC2 or EKS) Using Ollama, vLLM, or Triton

Introduction

Inference servers are critical for deploying machine learning (ML) models at scale, allowing real-time or batch inference through APIs or microservices. When building an inference server on AWS, you can choose from several platforms and orchestration options, such as EC2 (Elastic Compute Cloud) or EKS (Elastic Kubernetes Service). The choice of the inference framework significantly impacts performance, cost, and scalability. Three popular frameworks for inference are:

1. Ollama — Known for rapid deployment and ease of use with LLMs.
2. vLLM — Specializes in optimized memory utilization for large language model inference.
3. Triton Inference Server — Offers multi-framework, high-performance inferencing for both CPUs and GPUs.

In this blog, we will explore how to set up an inference server using these frameworks, compare their capabilities, and recommend an appropriate choice based on specific use cases.

1. Overview of Inference Frameworks

Ollama

Purpose: Optimized for deploying language models.

Key Features:

• Designed for simplicity and LLM optimization.
• Native compatibility with popular transformer-based models.
• Provides containerized inference solutions.