Native Metal local LLM inference runtime optimized for Apple Silicon.
Features
- Native Metal inference kernels
- Support for all Apple M-series chips
- Q2 through FP16 quantization
- OpenAI-compatible local server
- Multi-language SDK bindings
- Continuous batching and KV caching
Use Cases
- Local LLM chat on Mac
- Privacy-sensitive offline inference
- Local OpenAI-compatible APIs
- Apple Silicon inference benchmarking
- Embedding models in desktop apps
- Low-latency agent backends
FAQ
BaseRT is a local LLM inference runtime from Base Compute built directly for Metal and Apple Silicon unified memory. It ships a CLI, C API, and Python, Node, Rust, and Swift bindings for pulling models, chatting, or serving an OpenAI-compatible API. Official benchmarks report up to 6.4x faster prefill than llama.cpp and 3.9x faster than MLX in selected tests; real gains vary by chip, model, and quantization. Core capabilities include: Native Metal inference kernels, Support for all Apple M-series chips, Q2 through FP16 quantization.
Common scenarios include: Local LLM chat on Mac, Privacy-sensitive offline inference, Local OpenAI-compatible APIs.
