fastllm

fastllm is a high-performance large model inference library with no backend dependencies. It also supports tensor parallel inference of dense models and mixed mode inference of MOE models. Any graphics card above 10G can infer DeepSeek at full capacity. Dual-channel 9004/9005 server + single graphics card deploys the original DeepSeek full-blooded and full-precision model, with a single concurrency of 20tps; the INT4 quantitative model has a single concurrency of 30tps, and multiple concurrency can reach 60+.