Developer docs
API playgroundTry for free, no card

Search company profiles

VLLM

Full company profile

uuid02lz8ic

Namestring
VLLM
Legal namestring
vLLM
Websiteurl
vllm.ai
Company typeenum
Private
Founded yearint
2023
Descriptiontext

vLLM is an open-source high-throughput LLM inference and serving engine originally developed in 2023 at UC Berkeley's Sky Computing Lab and subsequently transitioned to the PyTorch Foundation for governance in 2024. Its core technology centers on PagedAttention, a memory-management innovation that reduces KV-cache fragmentation, combined with continuous batching, speculative decoding, CUDA/HIP graph execution, and broad quantization support (FP8, INT8, INT4, GPTQ/AWQ, GGUF, etc.). The platform serves 200+ model architectures — including decoder-only LLMs, mixture-of-experts, hybrid attention/SSM, multimodal, and embedding models — across 10+ hardware platforms spanning NVIDIA, AMD, Google TPU, Intel Gaudi/XPU, Huawei Ascend, AWS Neuron, Apple Silicon, and emerging accelerators. It exposes an OpenAI-compatible REST API (plus Anthropic Messages API and gRPC) and integrates with the broader PyTorch, HuggingFace, and Kubernetes ecosystems via companion tools including LLM Compressor, Speculators, Semantic Router, Production Stack, and AIBrix.

The project's go-to-market is community-led and PLG-style: source code is distributed freely under Apache 2.0 via GitHub, PyPI, and Docker Hub, with developer acquisition through documentation, recipes, benchmarks, Slack, and forums. vLLM generates no traditional revenue; sustainability is funded through cash donations (a16z, Sequoia Capital, Skywork AI, ZhenFund via GitHub Sponsors and OpenCollective) and compute donations from NVIDIA, AMD, AWS, Google Cloud, Intel, Alibaba Cloud, IBM, Red Hat, Anyscale, and several smaller GPU cloud providers. Its customers are horizontally distributed across AI/ML developers and researchers building LLM applications, enterprises deploying production LLM serving, and cloud providers offering GPU infrastructure.

Short descriptiontext

vLLM is an open-source, high-throughput LLM inference and serving engine (PyTorch Foundation project) serving 200+ model architectures across 10+ hardware platforms, used by AI developers, researchers, and enterprises globally under Apache 2.0.

Operating statusenum
Operating
Ownership categoryenum
Headcount rangeband
51–100
akta.pro rankint
HeadquartersSan Francisco, United States
HQ citystring
San Francisco
HQ countrystring
United States
HQ regionstring
North America
Markets served

Serves global market

Offices1 record

Each record includes

City, Country, Type, Description, Source

Keyword5 values
LLM inference engine, open source AI, model serving, GPU acceleration, PagedAttention technology
Industry1 code
PRIMARYHDABAMAF
Hybrid Cloud Compute & Virtualization Stack (HCI/VMs/Kubernetes)
NAICS code1 code
  • Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services518
SIC code1 code
  • Services-Prepackaged Software7372
Product category
LLM Inference Software
GTM motion1 record

Each record includes

Type, Description, Source

Revenue model1 record
1Open Source Distribution
TypeFreemium
Description

vLLM is an open-source project distributed freely under Apache 2.0 license. The project accepts donations through GitHub Sponsors and OpenCollective to support development, maintenance, and adoption.

vllm.ai
Marketing channels8 records

Each record includes

Title, Type, Stage, Description, Source

Distribution channels4 records

Each record includes

Title, Type, Scope, Target buyer, Description, Source

Cost components5 values
Technology or R&D, Personnel, Infrastructure, Operations, Marketing or Sales
Pricing details1 tier
1Open Source (Free)
ModelFreemium
Notes

Free to use under Apache 2.0 license. Accepts donations via GitHub Sponsors and OpenCollective.

vllm.ai
GTM typeB2B
B2B
Offering typeSoftware
Software
Brand1 of 7 records shown
1vLLM Recipes
Description

Community-maintained recipes for deploying models on various hardware configurations including NVIDIA H100/H200/B200/B300, Grace-Blackwell, and AMD MI300X/MI325X/MI355X.

vllm.ai
+6 more records
Core offering1 text field

vLLM is an open-source, high-throughput LLM inference and serving engine that runs large language models efficiently on diverse hardware via its PagedAttention memory management technique. It exposes an OpenAI-compatible HTTP server with Chat, Embeddings, and Completion endpoints, supports continuous batching, quantization, speculative decoding, and distributed inference, and is distributed for free under the Apache 2.0 license via GitHub, PyPI, Docker, and Hugging Face.

Differentiator
Functional benefit
Problem solved
Quantifiable outcome1 of 2 values shown
  • 23x throughput improvement with continuous batching
+1 more record
Product overview1 text field

vLLM is a unified open-source LLM inference and serving platform built around the vLLM Core Engine, which provides high-throughput, memory-efficient inference via PagedAttention and continuous batching. The platform includes ecosystem tools for the complete ML lifecycle: vLLM Recipes for deployment guides, vLLM Performance Dashboard for benchmarking, and vLLM Roadmap for project planning. The ecosystem extends with specialized modules including LLM Compressor for model quantization, Speculators for speculative decoding acceleration, Semantic Router for intelligent request routing, Production Stack for enterprise deployment, AIBrix for Kubernetes orchestration, GuideLLM for performance evaluation, and vLLM Omni for multi-modal (vision, audio, video) inference support. The platform supports 200+ model architectures across decoder-only LLMs, mixture-of-expert models, hybrid attention/SSM models, and multi-modal models on NVIDIA, AMD, Intel, and other hardware.

Product and service1 record
1vLLM Core Engine
CategoryLLM Inference Engine
Description

The open-source high-throughput LLM inference and serving engine with PagedAttention memory management, continuous batching, CUDA/HIP graph integration, quantization, speculative decoding, multi-LoRA, distributed inference, and an OpenAI-compatible HTTP server for Chat, Embeddings, and Completion endpoints across 200+ model architectures.

Scale indicator5 records

Each record includes

Type, Value, Description, Source

Partnership12 partners
Strategic tierCoreTypeTechnology or Integration
Description

NVIDIA provides compute resources and is a primary hardware platform supported by vLLM with optimized CUDA kernels, TRTLLM integration, and Blackwell/Grace-Hooper support.

Strategic tierCoreTypeTechnology or Integration
Description

AMD provides compute resources and ROCm support for running vLLM on AMD GPUs including MI300X/MI325X/MI355X.

Strategic tierCoreTypeTechnology or Integration
Description

Google Cloud provides compute resources and vLLM supports Google Cloud TPUs and TPU inference.

4AWS
Strategic tierCoreTypeTechnology or Integration
Description

AWS provides compute resources and vLLM supports AWS Neuron Accelerator for inference on AWS infrastructure.

vllm.ai
Strategic tierCoreTypeTechnology or Integration
Description

Intel provides compute resources and vLLM supports Intel Gaudi accelerators and XPU GPUs.

6Alibaba Cloud
Strategic tierCoreTypeTechnology or Integration
Description

Alibaba Cloud provides compute resources for vLLM development and testing.

vllm.ai
Strategic tierFlagshipTypeStrategic or Co-development Partner
Description

vLLM was originally developed in the Sky Computing Lab at UC Berkeley and continues to have strong academic ties with ongoing research collaboration.

Strategic tierCoreTypeTechnology or Integration
Description

Anyscale provides compute resources and vLLM has integration with Ray for distributed serving.

9Red Hat
Strategic tierCoreTypeTechnology or Integration
Description

Red Hat provides compute resources and vLLM supports enterprise deployment via Red Hat ecosystem.

vllm.ai
Strategic tierCoreTypeTechnology or Integration
Description

RunPod provides compute resources and vLLM has deployment integration with RunPod for cloud GPU instances.

Strategic tierFlagshipTypeStrategic or Co-development Partner
Description

vLLM is a PyTorch Foundation project, indicating strong institutional backing and integration with the PyTorch ecosystem.

Strategic tierCoreTypeTechnology or Integration
Description

Seamless integration with Hugging Face models and ecosystem for model loading and deployment.

Recent move6 records

Each record includes

Date, Type, Title, Description, Source

Expansion highlight6 records

Each record includes

Type, Description

Peers10 records
TypeDirect peer
Description

NVIDIA's optimized LLM inference engine that directly competes with vLLM for high-throughput serving on NVIDIA hardware. Both target state-of-the-art throughput on open and proprietary LLMs, but TensorRT-LLM benefits from NVIDIA's tight hardware-software integration.

TypeBroad incumbent
Description

Anyscale provides Ray-based distributed serving infrastructure and is a compute partner and deployment integration for vLLM. It is partially competitive for production-scale LLM serving and offers Ray-native alternatives to vLLM's distributed inference features.

3llama.cpp
TypeDirect peer
Description

A lightweight, widely-used C++ inference engine for running LLMs locally on consumer hardware. llama.cpp overlaps with vLLM's CPU/edge support and is a partial competitor for on-device and quantized serving scenarios.

TypeEmerging player
Description

A serverless cloud platform with first-class GPU and LLM inference support. Modal is listed as a vLLM deployment integration and represents an emerging adjacent competitor for hosted LLM serving, especially for AI-native startups.

TypeDirect peer
Description

Hugging Face's Rust-based LLM serving engine. TGI is the most direct open-source alternative to vLLM for serving HuggingFace models, competing on throughput, latency, and ease of integration with the broader HuggingFace ecosystem.

TypeEmerging player
Description

A local LLM serving platform built on llama.cpp that simplifies model deployment for developers and consumers. Ollama overlaps with vLLM at the developer/edge tier but focuses on single-machine ease-of-use rather than multi-GPU production serving.

TypeBroad incumbent
Description

NVIDIA's broader inference server supporting multiple model types and backends. Triton overlaps with vLLM in LLM serving but covers a wider scope (vision, speech, custom models) and benefits from NVIDIA's enterprise-grade support and tooling.

TypeDirect peer
Description

An emerging open-source LLM serving framework co-developed at UC Berkeley Sky Computing Lab. SGLang targets similar workloads as vLLM with a focus on structured generation and agentic workflows, competing for the same developer mindshare.

9LMDeploy
TypeEmerging player
Description

An LLM serving framework from InternLM/Shanghai AI Lab focused on high-throughput inference. LMDeploy is a partial competitor offering similar tensor-parallel and quantization capabilities, primarily used in the Chinese open-source LLM ecosystem.

TypeEmerging player
Description

Microsoft's open-source LLM inference library built on DeepSpeed. DeepSpeed-MII targets similar low-latency, high-throughput serving workloads as vLLM, with tight integration into the Azure ML and ONNX Runtime ecosystems.

Market position
Strengths5 records

Each record includes

Headline, Details, Source

Weaknesses5 records

Each record includes

Headline, Details, Source

Competitive moat6 records

Each record includes

Type, Details

Key risks7 records

Each record includes

Headline, Details, Source

Key highlights7 records

Each record includes

Headline, Details, Source

Customer concentration

Classification, Details

Segment2 records

Each record includes

Title, Type, Primary, Description, Pain point addressed, Use case, Source

Ideal customer profile2 records

Each record includes

Profile, Firmographic size, Sales motion, Sales cycle length, Buying structure, Purchase trigger, Buyer persona, Geography, Industry vertical, Primary use case, Description, Pain points, Evidence proof points, Target buyer

Technology focused
Yes
API detail
Has APIbool
Yes

Docs URL, Description

Integration35 records

Each record includes

Title, Type, Description, Source

AI capability14 records

Each record includes

Type, Description, Source

AI maturity
App detail

Has app

Feature10 records

Each record includes

Title, Differentiator, Description, Source

Core technology
Revenue estimate
Valuation estimate
Number of profiles
No data
No data
Funding overview

Funding stage, Last funding date, Total funding USD

Funding rounds

Each record includes

Round, Amount USD, Date, Pre money valuation, Total investors, Investors, News

Investors

Each record includes

Name, Type, Date of entry, Rounds participated, Website

Funding detail is available on the Subscription and Enterprise plan.Contact sales →

M&A

Each record includes

Name, Acquisition type, Announced date, Completed date, Status, Website, News

Investment

Each record includes

Name, Round, Announced date, Lead investor, Website, News

M&A and investment is available on the Subscription and Enterprise plan.Contact sales →

VLLM

LLM Inference Softwarevllm.ai

vLLM is an open-source, high-throughput LLM inference and serving engine (PyTorch Foundation project) serving 200+ model architectures across 10+ hardware platforms, used by AI developers, researchers, and enterprises globally under Apache 2.0.

What VLLM does

vLLM is an open-source high-throughput LLM inference and serving engine originally developed in 2023 at UC Berkeley's Sky Computing Lab and subsequently transitioned to the PyTorch Foundation for governance in 2024. Its core technology centers on PagedAttention, a memory-management innovation that reduces KV-cache fragmentation, combined with continuous batching, speculative decoding, CUDA/HIP graph execution, and broad quantization support (FP8, INT8, INT4, GPTQ/AWQ, GGUF, etc.). The platform serves 200+ model architectures — including decoder-only LLMs, mixture-of-experts, hybrid attention/SSM, multimodal, and embedding models — across 10+ hardware platforms spanning NVIDIA, AMD, Google TPU, Intel Gaudi/XPU, Huawei Ascend, AWS Neuron, Apple Silicon, and emerging accelerators. It exposes an OpenAI-compatible REST API (plus Anthropic Messages API and gRPC) and integrates with the broader PyTorch, HuggingFace, and Kubernetes ecosystems via companion tools including LLM Compressor, Speculators, Semantic Router, Production Stack, and AIBrix.

The project's go-to-market is community-led and PLG-style: source code is distributed freely under Apache 2.0 via GitHub, PyPI, and Docker Hub, with developer acquisition through documentation, recipes, benchmarks, Slack, and forums. vLLM generates no traditional revenue; sustainability is funded through cash donations (a16z, Sequoia Capital, Skywork AI, ZhenFund via GitHub Sponsors and OpenCollective) and compute donations from NVIDIA, AMD, AWS, Google Cloud, Intel, Alibaba Cloud, IBM, Red Hat, Anyscale, and several smaller GPU cloud providers. Its customers are horizontally distributed across AI/ML developers and researchers building LLM applications, enterprises deploying production LLM serving, and cloud providers offering GPU infrastructure.

VLLM firmographics

Firmographics
Name
VLLM
Legal name
vLLM
Website
https://vllm.ai
Company type
Private
Founded year
2023
Operating status
Operating
Headcount range
51–100 employees
Short description
vLLM is an open-source, high-throughput LLM inference and serving engine (PyTorch Foundation project) serving 200+ model architectures across 10+ hardware platforms, used by AI developers, researchers, and enterprises globally under Apache 2.0.
Ownership category
akta.pro rank

VLLM industry classification

Industry
Product category
LLM Inference Software
NAICS
Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services (518)
SIC
Services-Prepackaged Software (7372)
akta.pro industry
PRIMARYHDABAMAF
Hybrid Cloud Compute & Virtualization Stack (HCI/VMs/Kubernetes)

Keywords

  • LLM inference engine
  • Open source AI
  • Model serving
  • GPU acceleration
  • PagedAttention technology

Where VLLM is headquartered

Location

Headquarters

HQ city
San Francisco
HQ country
United States
HQ region
North America

Offices1 record

Markets served

VLLM business model

Business model
GTM type
B2B
Offering type
Software
Cost components
Technology or R&D, Personnel, Infrastructure, Operations, Marketing or Sales

Revenue model

  1. Open Source Distribution: vLLM is an open-source project distributed freely under Apache 2.0 license. The project accepts donations through GitHub Sponsors and OpenCollective to support development, maintenance, and adoption.

Pricing tiers

ModelBillingPrice
Freemium—Open Source (Free)

Go-to-market motion1 record

Distribution channels4 records

Marketing channels8 records

VLLM product offering

Product offering

Core offering

vLLM is an open-source, high-throughput LLM inference and serving engine that runs large language models efficiently on diverse hardware via its PagedAttention memory management technique. It exposes an OpenAI-compatible HTTP server with Chat, Embeddings, and Completion endpoints, supports continuous batching, quantization, speculative decoding, and distributed inference, and is distributed for free under the Apache 2.0 license via GitHub, PyPI, Docker, and Hugging Face.

Product overview

vLLM is a unified open-source LLM inference and serving platform built around the vLLM Core Engine, which provides high-throughput, memory-efficient inference via PagedAttention and continuous batching. The platform includes ecosystem tools for the complete ML lifecycle: vLLM Recipes for deployment guides, vLLM Performance Dashboard for benchmarking, and vLLM Roadmap for project planning. The ecosystem extends with specialized modules including LLM Compressor for model quantization, Speculators for speculative decoding acceleration, Semantic Router for intelligent request routing, Production Stack for enterprise deployment, AIBrix for Kubernetes orchestration, GuideLLM for performance evaluation, and vLLM Omni for multi-modal (vision, audio, video) inference support. The platform supports 200+ model architectures across decoder-only LLMs, mixture-of-expert models, hybrid attention/SSM models, and multi-modal models on NVIDIA, AMD, Intel, and other hardware.

Differentiator

Problem solved

Functional benefit

Brands

  • vLLM Recipes: Community-maintained recipes for deploying models on various hardware configurations including NVIDIA H100/H200/B200/B300, Grace-Blackwell, and AMD MI300X/MI325X/MI355X.
  • vLLM Omni
  • vLLM Playground
  • LLM Compressor
  • Production Stack
  • Semantic Router
  • Speculators

Products and services

  • vLLM Core Engine The open-source high-throughput LLM inference and serving engine with PagedAttention memory management, continuous batching, CUDA/HIP graph integration, quantization, speculative decoding, multi-LoRA, distributed inference, and an OpenAI-compatible HTTP server for Chat, Embeddings, and Completion endpoints across 200+ model architectures.

Quantifiable outcome

  • 23x throughput improvement with continuous batching
  • +1 more outcomes

Companies that use VLLM

Customer profile

Segments2 records

Ideal customer profiles2 records

VLLM technology and API

Technology

Technology focussed Yes

API detail

Has API
Yes
API docs
API detail

Core technology

AI maturity

App detail

Integration35 records

AI capability14 records

Feature10 records

VLLM partnerships and signals

Strategic signal

Partnerships

Twelve partnerships are on record, tiered core and flagship.

  • NVIDIAcoreTechnology or IntegrationNVIDIA provides compute resources and is a primary hardware platform supported by vLLM with optimized CUDA kernels, TRTLLM integration, and Blackwell/Grace-Hooper support.
  • AMDcoreTechnology or IntegrationAMD provides compute resources and ROCm support for running vLLM on AMD GPUs including MI300X/MI325X/MI355X.
  • Google CloudcoreTechnology or IntegrationGoogle Cloud provides compute resources and vLLM supports Google Cloud TPUs and TPU inference.
  • AWScoreTechnology or IntegrationAWS provides compute resources and vLLM supports AWS Neuron Accelerator for inference on AWS infrastructure.
  • IntelcoreTechnology or IntegrationIntel provides compute resources and vLLM supports Intel Gaudi accelerators and XPU GPUs.
  • Alibaba CloudcoreTechnology or IntegrationAlibaba Cloud provides compute resources for vLLM development and testing.
  • UC BerkeleyflagshipStrategic or Co-development PartnervLLM was originally developed in the Sky Computing Lab at UC Berkeley and continues to have strong academic ties with ongoing research collaboration.
  • AnyscalecoreTechnology or IntegrationAnyscale provides compute resources and vLLM has integration with Ray for distributed serving.
  • Red HatcoreTechnology or IntegrationRed Hat provides compute resources and vLLM supports enterprise deployment via Red Hat ecosystem.
  • RunPodcoreTechnology or IntegrationRunPod provides compute resources and vLLM has deployment integration with RunPod for cloud GPU instances.
  • PyTorch FoundationflagshipStrategic or Co-development PartnervLLM is a PyTorch Foundation project, indicating strong institutional backing and integration with the PyTorch ecosystem.
  • Hugging FacecoreTechnology or IntegrationSeamless integration with Hugging Face models and ecosystem for model loading and deployment.

Scale indicators5 records

Recent moves6 records

Expansion highlights6 records

VLLM competitors and assessment

Company assessment

Direct peers

  • NVIDIA TensorRT-LLM: NVIDIA's optimized LLM inference engine that directly competes with vLLM for high-throughput serving on NVIDIA hardware. Both target state-of-the-art throughput on open and proprietary LLMs, but TensorRT-LLM benefits from NVIDIA's tight hardware-software integration.
  • llama.cpp: A lightweight, widely-used C++ inference engine for running LLMs locally on consumer hardware. llama.cpp overlaps with vLLM's CPU/edge support and is a partial competitor for on-device and quantized serving scenarios.
  • Hugging Face Text Generation Inference (TGI): Hugging Face's Rust-based LLM serving engine. TGI is the most direct open-source alternative to vLLM for serving HuggingFace models, competing on throughput, latency, and ease of integration with the broader HuggingFace ecosystem.
  • SGLang: An emerging open-source LLM serving framework co-developed at UC Berkeley Sky Computing Lab. SGLang targets similar workloads as vLLM with a focus on structured generation and agentic workflows, competing for the same developer mindshare.

Broad incumbents

  • Anyscale (Ray Serve): Anyscale provides Ray-based distributed serving infrastructure and is a compute partner and deployment integration for vLLM. It is partially competitive for production-scale LLM serving and offers Ray-native alternatives to vLLM's distributed inference features.
  • NVIDIA Triton Inference Server: NVIDIA's broader inference server supporting multiple model types and backends. Triton overlaps with vLLM in LLM serving but covers a wider scope (vision, speech, custom models) and benefits from NVIDIA's enterprise-grade support and tooling.

Emerging players

  • Modal: A serverless cloud platform with first-class GPU and LLM inference support. Modal is listed as a vLLM deployment integration and represents an emerging adjacent competitor for hosted LLM serving, especially for AI-native startups.
  • Ollama: A local LLM serving platform built on llama.cpp that simplifies model deployment for developers and consumers. Ollama overlaps with vLLM at the developer/edge tier but focuses on single-machine ease-of-use rather than multi-GPU production serving.
  • LMDeploy: An LLM serving framework from InternLM/Shanghai AI Lab focused on high-throughput inference. LMDeploy is a partial competitor offering similar tensor-parallel and quantization capabilities, primarily used in the Chinese open-source LLM ecosystem.
  • DeepSpeed-MII: Microsoft's open-source LLM inference library built on DeepSpeed. DeepSpeed-MII targets similar low-latency, high-throughput serving workloads as vLLM, with tight integration into the Azure ML and ONNX Runtime ecosystems.

Market position

Strengths5 records

Weaknesses5 records

Competitive moat6 records

Key risks7 records

Key highlights7 records

Customer concentration

VLLM social profiles

Digital presence

VLLM financial estimates

Financial estimate

Revenue estimate

Valuation estimate

VLLM leadership team

Management profile

Number of profiles

VLLM funding detail

Funding detail

Funding overview

Funding rounds

Investors

Funding detail is available on the Subscription and Enterprise plan.Contact sales →

VLLM M&A and investment

M&A and investment

M&A

Investments

M&A and investment is available on the Subscription and Enterprise plan.Contact sales →

Frequently asked questions about VLLM

What does VLLM do?

vLLM is an open-source, high-throughput LLM inference and serving engine that runs large language models efficiently on diverse hardware via its PagedAttention memory management technique. It exposes an OpenAI-compatible HTTP server with Chat, Embeddings, and Completion endpoints, supports continuous batching, quantization, speculative decoding, and distributed inference, and is distributed for free under the Apache 2.0 license via GitHub, PyPI, Docker, and Hugging Face.

Is VLLM a public or private company?

VLLM is a private company. It is classified as nonprofit foundation owned and is currently operating.

When was VLLM founded?

VLLM was founded in 2023. It employs 51 to 100 people.

Where is VLLM based?

VLLM is headquartered in San Francisco, United States, in the North America region.

How does VLLM make money?

One revenue line is on record: open Source Distribution.

Who are VLLM's main competitors?

Direct peers on record are NVIDIA TensorRT-LLM, llama.cpp, Hugging Face Text Generation Inference (TGI) and SGLang. Broad incumbents are Anyscale (Ray Serve) and NVIDIA Triton Inference Server. Emerging players are Modal, Ollama, LMDeploy and DeepSpeed-MII.

Does VLLM have an API?

Yes. vLLM provides an OpenAI-compatible API server, plus Anthropic Messages API and gRPC support. The API enables developers to serve open-source LLMs with drop-in OpenAI compatibility for instant integration. The documentation also shows MCP (Model Context Protocol) tool endpoints. Developer documentation is at docs.vllm.ai.

What industry is VLLM in?

VLLM's product category is LLM Inference Software. Its primary akta.pro industry code is HDABAMAF, Hybrid Cloud Compute & Virtualization Stack (HCI/VMs/Kubernetes). Its NAICS code is 518 and its SIC code is 7372.

Unlock the full company data

50 free credits on sign-up, no credit card required.

Contact sales
Live signals
TowardsdatascienceContinuous Batching vs. Static Batching in vLLM, BenchmarkedA benchmark on a single GPU compared continuous, static, and dynamic batching in vLLM, finding continuous batching improved throughput by 2.91× over static batching. The scheduler alone contributed 2.91×, while the engine contributed 1.33×, and the full HuggingFace-to-vLLM change measured 3.86×. The gain scales with output-length variability, reaching 2.91× at the highest variance.MarkTechPostvLLM Semantic Router Releases d3-ultra: A 27B Open Multimodal Decision ModelvLLM Semantic Router released d3-ultra, a 26.09B Apache-2.0 multimodal decision model that reads input once and returns probabilities for typed questions. It leads the team's SR-Text test at 87.2 but trails d3-turbo on box grounding at 58.2% versus 60.0%. The model is available via transformers, vLLM engine, free API, and demo.PandailyAnt Group and vLLM Open-Source Humming, Quantized Inference Kernels With Zhenwu 810E SupportAnt Group and vLLM open-sourced Humming, a JIT-compiled GEMM kernel library for quantized LLM inference, supporting NVIDIA GPUs and T-Head's Zhenwu 810E. Benchmarks show up to 14% higher throughput and 25% faster time-to-first-token on DeepSeek-V4.1-Flash. The library is included in vLLM 0.29.0.Tech TimesvLLM Disaggregated Inference Gets Push KV Transfer, Slashing First-Token Latency Three TimesvLLM's push-based KV transfer reduces first-token latency from 1,744ms to 540ms at 8 queries per second with 16,000-token inputs. The protocol pre-registers decode memory during prefill, moving coordination off the critical path. A pull connector with background threading is planned to narrow the gap.DoNewsvLLM 深度优化 Kimi K3 推理:基于 CuTe DSL 实现解码性能大幅跃升- DoNewsvLLM engineers optimized Kimi K3 on NVIDIA GB300 hardware using a CuTe DSL decoding pipeline, achieving 1.68x speedup. Batch-1 decoding reached 842 tokens/s on 16 GPUs, and interaction latency improved up to 1.8x. The optimization is being reused for other models like DeepSeek-V4.1-Flash and GLM5.3.HackerNoonTop Tools for Serving Open-Source Models on Kubernetes | HackerNoonThe guide compares Kubernetes tools for serving open-source models, covering runtimes like vLLM, SGLang, and TGI, and platforms such as KServe, llm-d, and SIE. It notes TGI is in maintenance mode and recommends vLLM or SGLang for new deployments. It also discusses routing, autoscaling, and GPU setup trade-offs.MarkTechPostWhat is Decision 3.0? vLLM Semantic Router’s New Open Decision ModelsvLLM Semantic Router released Decision 3.0, a family of five open multimodal decision models from 0.8B to 27B parameters. The 27B d3 model scores 64.1 on text and 71.6 on vision on internal evals, while the 9B d3-flash beats Decision 2.0's 27B. Latency on AMD Instinct MI325X ranges from 21.7 to 84 ms for text.HackerNoonBeyond the Model: The Unlikely Architecture of the AI WinnersThe AI stack's real winners are infrastructure layers, not model companies. Stripe's $8 billion OpenRouter acquisition and vLLM's ecosystem position them to control routing and billing. The article argues these plumbing layers will dominate as AI becomes commoditized.EIN PresswireAnt Group and vLLM Open-Source Humming, Quantized Inference Kernels With Zhenwu 810E SupportAnt Group and vLLM open-sourced Humming, a JIT GEMM kernel library for quantized inference, under Apache 2.0. It supports INT1-8 and FP4-FP8 mixes, including T-Head's Zhenwu 810E, and is integrated into vLLM 0.29.0. Benchmarks show up to 43% throughput gains on DeepSeek-V4.1-Flash.TowardsdatascienceSpeculative Decoding on CPUs: Nearly 4x Faster Token Generation with DFlashIntel has enabled DFlash speculative decoding support for CPUs in vLLM v0.25.0, demonstrating a nearly 4x increase in token generation throughput on Intel Xeon processors. Testing with the Qwen3.5-9B model showed a 74% reduction in cost per generated token compared to autoregressive baselines by utilizing block diffusion and KV injection techniques.