Learning LLM Inference

A from-scratch inference engine for Llama 2 7B, built to learn and implement modern LLM inference techniques.

Progress

• KV caching
• Continuous batching
• Paged KV cache
• Quantization
• Speculative decoding
• Custom kernels + kernel fusion
• More to come