Native Metal local LLM inference runtime optimized for Apple Silicon.
Features
- Native Metal inference kernels
- Support for all Apple M-series chips
- Q2 through FP16 quantization
- OpenAI-compatible local server
- Multi-language SDK bindings
- Continuous batching and KV caching
Use Cases
- Local LLM chat on Mac
- Privacy-sensitive offline inference
- Local OpenAI-compatible APIs
- Apple Silicon inference benchmarking
- Embedding models in desktop apps
- Low-latency agent backends
FAQ
Native Metal local LLM inference runtime optimized for Apple Silicon. Core capabilities include: Native Metal inference kernels, Support for all Apple M-series chips, Q2 through FP16 quantization.
Common scenarios include: Local LLM chat on Mac, Privacy-sensitive offline inference, Local OpenAI-compatible APIs.
