Proposal: Add FastAPI + Ollama (Local AI/LLM Inference) Compose sample

[VimalN2005]

Proposed Sample

Add a new application sample: FastAPI + Ollama (Local AI/LLM Inference) under fastapi-ollama/.

Motivation & Value

Local LLM adoption is growing rapidly, and developers frequently look for a clean, reproducible Compose setup to run a Python backend alongside a containerized inference engine (Ollama) with model volume persistence. Currently, awesome-compose does not feature any AI/LLM sample.

Proposed Architecture

• ollama: Official Ollama container with persisted model volume (ollama_data:/root/.ollama).
• web: Lightweight FastAPI service (Python 3.11) exposing /generate, /chat, and /models endpoints with streaming support.
• compose.yaml: Coordinated services with volume persistence.
• README.md: Clear guide explaining how to start the stack, pull models (e.g. llama3.2:1b), and test endpoints via curl/Swagger UI.

Question for Maintainers

Would this be a welcome addition to the repository? If approved, please assign this issue to me so I can submit the implementation.

[stewartmbofana]

Thanks for contributing this sample! Having an Ollama + FastAPI sample in awesome-compose is a great addition for developers looking to run local LLM inference with Docker Compose.

Before this can be merged, please address the following changes:

1. Remove UTF-8 BOM (Byte Order Mark) from files

Several files have been saved with a UTF-8 BOM (\xef\xbb\xbf):

• fastapi-ollama/Dockerfile
• fastapi-ollama/compose.yaml
• fastapi-ollama/requirements.txt
• fastapi-ollama/app/main.py
• fastapi-ollama/README.md

In particular, the BOM at the beginning of fastapi-ollama/Dockerfile causes Docker BuildKit to fail to recognize the # syntax=docker/dockerfile:1.4 directive, because parser directives must strictly start at byte 0. Please re-save all files as standard UTF-8 (without BOM).

2. Remove --no-cache-dir when using pip cache mount in Dockerfile

In fastapi-ollama/Dockerfile:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir -r requirements.txt

Using --mount=type=cache,target=/root/.cache/pip alongside pip install --no-cache-dir is contradictory because --no-cache-dir instructs pip to bypass the cache directory entirely. Please remove --no-cache-dir so pip can utilize the cache mount:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

3. Streaming error handling & client lifecycle in app/main.py

• Streaming exception handling: In /generate and /chat, the try...except httpx.RequestError block does not catch connection or network failures during streaming because stream_generator() is consumed asynchronously by Starlette after the route returns.
• Status code check on stream: If Ollama returns a non-200 status code (e.g. 404 when a model has not yet been pulled), client.stream does not raise an exception, and the error response body is streamed under an HTTP 200 response. Consider verifying response.status_code == 200 before yielding chunks.
• Client lifecycle: Rather than creating a new httpx.AsyncClient inside each request, consider managing a shared client via FastAPI's lifespan context manager (@asynccontextmanager async def lifespan(app: FastAPI): ...) to enable HTTP connection pooling and proper cleanup.

4. GPU Acceleration Note

Most users running Ollama in Docker will be interested in GPU pass-through. It would be very helpful to add a note or commented-out configuration in compose.yaml and fastapi-ollama/README.md showing how to enable Nvidia GPU support (e.g., via deploy.resources.reservations.devices).

5. PR Title & Description

Please update the PR title and description to reflect that this is an implemented Pull Request ready for review rather than an issue proposal.