vLLM Benchmark
About
Benchmarks vLLM deployments by measuring throughput, latency, and token generation speed through natural language test configuration
Details
- Author
- eliovp-bv
- Repository
- Eliovp-BV/mcp-vllm-benchmark
- GitHub stars
- 4
- Categories
- Design, Developer Tools, AI, Infrastructure
- Tags
- #analytics
Jump to
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
vLLM BenchmarkCommand (node, npx, python, etc.)npxArguments-
Argument 1
-y -
Argument 2
@highlight/mcp-server
Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
-
Argument 1
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
1. Clone the repository
2. Add it to your MCP servers:
{
"mcpServers": {
"mcp-vllm": {
"command": "uv",
"args": [
"run",
"/Path/TO/mcp-vllm-benchmarking-tool/server.py"
]
}
}
}
Then you can prompt for example like this:
Do a vllm benchmark for this endpoint: http://10.0.101.39:8888
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"vllm benchmark": {
"env": {},
"args": [
"-y",
"@highlight/mcp-server"
],
"command": "npx"
}
}
}Linux
{
"env": [],
"args": [
"-y",
"@highlight/mcp-server"
],
"command": "npx"
}Macos
{
"env": [],
"args": [
"-y",
"@highlight/mcp-server"
],
"command": "npx"
}Windows
{
"env": [],
"args": [
"/c",
"npx",
"-y",
"@highlight/mcp-server"
],
"command": "cmd"
}MCP vLLM Benchmarking Tool
This is proof of concept on how to use MCP to interactively benchmark vLLM.
We are not new to benchmarking, read our blog:
This is just an exploration of possibilities with MCP.
Usage
1. Clone the repository
2. Add it to your MCP servers:
{
"mcpServers": {
"mcp-vllm": {
"command": "uv",
"args": [
"run",
"/Path/TO/mcp-vllm-benchmarking-tool/server.py"
]
}
}
}
Then you can prompt for example like this:
Do a vllm benchmark for this endpoint: http://10.0.101.39:8888
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.
Todo:
- Due to some random outputs by vllm it may show that it found some invalid json. I have not really looked into it yet.
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
