vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

GitHub repository

License

Apache-2.0

Stars

96,196

Homepage

This is an independent tool overview. Check the project repository for current releases, compatibility, and installation requirements.

Alternatives and related tools

On-device AI assistant for terminal and Android tasks.

Desktop Companion for Hermes Agent

Hermes WebUI: The best way to use Hermes Agent from the web or from your phone!