GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
-
Updated
May 27, 2025 - C++
#
llm-inference
Here are 2,244 public repositories matching this topic...
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
python data-science machine-learning reinforcement-learning deep-learning deployment tensorflow optimization parallel pytorch distributed hyperparameter-optimization ray hyperparameter-search serving rllib large-language-models llm llm-serving llm-inference
-
Updated
Jun 6, 2026 - Python
Find secrets with Gitleaks 🔑
git go cli golang open-source security secret ci-cd cicd hacktoberfest dlp security-tools devsecops data-loss-prevention gitleaks ai-powered nhi llm llm-training llm-inference
-
Updated
Jun 4, 2026 - Go
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
-
Updated
May 25, 2026 - HTML
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
-
Updated
Jun 5, 2026 - Python
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
llama mistral fine-tuning mlops bentoml vicuna llm model-inference llmops llm-serving llm-inference open-source-llm llama2 openllm llm-ops llama3-1 llama3-2 llama3-2-vision
-
Updated
Jun 2, 2026 - Python
Official inference library for Mistral models
-
Updated
Apr 20, 2026 - Jupyter Notebook
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
nlp natural-language-processing ai computer-vision deep-learning transformers inference speech-recognition yolo recommendation-system performance-boost good-first-issue openvino diffusion-models stable-diffusion generative-ai llm-inference optimize-ai deploy-ai
-
Updated
Jun 5, 2026 - C++
High-speed Large Language Model Serving for Local Deployment
-
Updated
May 11, 2026 - C++
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
python machine-learning deep-learning model-serving multimodal mlops ml-engineering ai-inference llm generative-ai llmops llm-serving model-inference-service llm-inference inference-platform
-
Updated
Jun 3, 2026 - Python
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
llama cuda-kernels deepspeed llm fastertransformer llm-inference turbomind internlm llama2 codellama llama3
-
Updated
Jun 5, 2026 - Python
Plano is an AI-native proxy and data plane for agentic apps — with built-in orchestration, safety, observability, and smart LLM routing so you stay focused on your agents core logic.
proxy routing gateway prompt proxy-server openai envoy envoyproxy llms generative-ai llmops llm-inference llm-proxy ai-gateway llm-gateway llm-routing ai-gateway-support
-
Updated
Jun 3, 2026 - Rust
Open-source implementation of AlphaEvolve
genetic-algorithm discovery optimize evolutionary-algorithms deepmind-lab deepmind iterative-methods genetic-algorithms evolutionary-computation alphacode distributed-evolutionary-algorithms iterative-refinement llm-inference llm-engineering llm-ensemble coding-agent alpha-evolve alphaevolve openevolve
-
Updated
Mar 18, 2026 - Python
FlashInfer: Kernel Library for LLM Serving
-
Updated
Jun 6, 2026 - Python
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
kubernetes machine-learning tensorflow cncf pytorch artificial-intelligence xgboost k8s service-mesh hacktoberfest istio model-serving kubeflow mlops knative model-interpretability kserve vllm genai llm-inference
-
Updated
Jun 3, 2026 - Go
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
rust machine-learning transformers api-server developer-tools llama command-line-tool inference-server webgpu rust-crate huggingface huggingface-transformers huggingface-models webgpu-shaders llamacpp llm-inference local-ai gguf ollama-api openai-compatible
-
Updated
Jun 5, 2026 - Rust
Low-latency AI engine for mobile devices & wearables
android ios arm mobile framework ai speech edge transformer smartphone whisper rag edge-ai on-device-ai mobile-inference quantiz llm llms llamacpp llm-inference
-
Updated
Jun 6, 2026 - C++
Superduper: End-to-end framework for building custom AI applications and agents.
python data database ai mongodb chatbot ml transformers inference torch pytorch pretrained-models semantic-search rag mlops distributed-ml vector-search llmops llm-serving llm-inference
-
Updated
Sep 1, 2025 - Python
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉
mla vllm llm-inference awesome-llm flash-attention tensorrt-llm paged-attention deepseek flash-attention-3 deepseek-v3 minimax-01 deepseek-r1 flash-mla qwen3
-
Updated
Apr 20, 2026 - Python
A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.
cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
-
Updated
Jun 6, 2026 - Python
Improve this page
Add a description, image, and links to the llm-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-inference topic, visit your repo's landing page and select "manage topics."