Multi-LLM API Gateway

by freedanfan

6 stars
Not rated
GitHub

About

Provides a unified FastAPI server for interacting with multiple language model APIs, enabling seamless switching between OpenAI and Anthropic models without changing integration code.

Details

Author
freedanfan
Repository
freedanfan/mcp_server
GitHub stars
6
License
MIT License
Categories
Design, Developer Tools, AI, Infrastructure, Security, API

- JSON-RPC 2.0: Request-response communication based on standard JSON-RPC 2.0 protocol
- SSE Connection: Support for Server-Sent Events connections for real-time notifications
- Modular Design: Modular architecture for easy extension and customization
- Asynchronous Processing: High-performance service using FastAPI and asynchronous IO
- Complete Client: Includes a full test client implementation

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

  1. Download and install Highlight from highlightai.com/download
  2. Navigate to the plugins tab and select "Add Custom Plugin"
  3. Configure the plugin with the settings below
    Plugin Name Multi-LLM API Gateway
    Command (node, npx, python, etc.) npx
    Arguments
    • Argument 1 -y
    • Argument 2 @highlight/mcp-server

    Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.

  4. Enable "Start Automatically" if you want the plugin to start when Highlight launches

From the repository

Run the client in another terminal:

python mcp_client.py

If the server is not running at the default address, you can set an environment variable:

export MCP_SERVER_URL="http://your-server-address:port"
python mcp_client.py

initialize

Initialize the MCP server. Parameters: protocol version (string), capabilities (object)

sample

Send a sampling request with a prompt. Parameters: prompt (string)

shutdown

Send a shutdown request to gracefully close the server. Parameters: none

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "multi-llm api gateway": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Linux

{
    "env": [],
    "args": [
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "npx"
}

Macos

{
    "env": [],
    "args": [
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "npx"
}

Windows

{
    "env": [],
    "args": [
        "/c",
        "npx",
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "cmd"
}

MCP Server

中文文档

Project Overview

Built on FastAPI and MCP (Model Context Protocol), this project enables standardized context interaction between AI models and development environments. It enhances the scalability and maintainability of AI applications by simplifying model deployment, providing efficient API endpoints, and ensuring consistency in model input and output, making it easier for developers to integrate and manage AI tasks.

MCP (Model Context Protocol) is a unified protocol for context interaction between AI models and development environments. This project provides a Python-based MCP server implementation that supports basic MCP protocol features, including initialization, sampling, and session management.

Features

- JSON-RPC 2.0: Request-response communication based on standard JSON-RPC 2.0 protocol
- SSE Connection: Support for Server-Sent Events connections for real-time notifications
- Modular Design: Modular architecture for easy extension and customization
- Asynchronous Processing: High-performance service using FastAPI and asynchronous IO
- Complete Client: Includes a full test client implementation

Project Structure

mcp_server/
├── mcp_server.py         # MCP server main program
├── mcp_client.py         # MCP client test program
├── routers/
│   ├── __init__.py       # Router package initialization
│   └── base_router.py    # Base router implementation
├── requirements.txt      # Project dependencies
└── README.md             # Project documentation

Installation

1. Clone the repository:

git clone https://github.com/freedanfan/mcp_server.git
cd mcp_server

2. Install dependencies:

pip install -r requirements.txt

Usage

Starting the Server

python mcp_server.py

By default, the server will start on 127.0.0.1:12000. You can customize the host and port using environment variables:

export MCP_SERVER_HOST=0.0.0.0
export MCP_SERVER_PORT=8000
python mcp_server.py

Running the Client

Run the client in another terminal:

python mcp_client.py

If the server is not running at the default address, you can set an environment variable:

export MCP_SERVER_URL="http://your-server-address:port"
python mcp_client.py

API Endpoints

The server provides the following API endpoints:

- Root Path (/): Provides server information
- API Endpoint (/api): Handles JSON-RPC requests
- SSE Endpoint (/sse): Handles SSE connections

MCP Protocol Implementation

Initialization Flow

1. Client connects to the server via SSE
2. Server returns the API endpoint URI
3. Client sends an initialization request with protocol version and capabilities
4. Server responds to the initialization request, returning server capabilities

Sampling Request

Clients can send sampling requests with prompts:

{
  "jsonrpc": "2.0",
  "id": "request-id",
  "method": "sample",
  "params": {
    "prompt": "Hello, please introduce yourself."
  }
}

The server will return sampling results:

{
  "jsonrpc": "2.0",
  "id": "request-id",
  "result": {
    "content": "This is a response to the prompt...",
    "usage": {
      "prompt_tokens": 10,
      "completion_tokens": 50,
      "total_tokens": 60
    }
  }
}

Closing a Session

Clients can send a shutdown request:

{
  "jsonrpc": "2.0",
  "id": "request-id",
  "method": "shutdown",
  "params": {}
}

The server will gracefully shut down:

{
  "jsonrpc": "2.0",
  "id": "request-id",
  "result": {
    "status": "shutting_down"
  }
}

Development Extensions

Adding New Methods

To add new MCP methods, add a handler function to the MCPServer class and register it in the _register_methods method:

def handle_new_method(self, params: dict) -> dict:
    """Handle new method"""
    logger.info(f"Received new method request: {params}")
    # Processing logic
    return {"result": "success"}

def _register_methods(self):
# Register existing methods
self.router.register_method("initialize", self.handle_initialize)
self.router.register_method("sample", self.handle_sample)
self.router.register_method("shutdown", self.handle_shutdown)
# Register new method
self.router.register_method("new_method", self.handle_new_method)

Integrating AI Models

To integrate actual AI models, modify the handle_sample method:

async def handle_sample(self, params: dict) -> dict:
    """Handle sampling request"""
    logger.info(f"Received sampling request: {params}")
    
    # Get prompt
    prompt = params.get("prompt", "")
    
    # Call AI model API
    # For example: using OpenAI API
    response = await openai.ChatCompletion.acreate(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    
    content = response.choices[0].message.content
    usage = response.usage
    
    return {
        "content": content,
        "usage": {
            "prompt_tokens": usage.prompt_tokens,
            "completion_tokens": usage.completion_tokens,
            "total_tokens": usage.total_tokens
        }
    }

Troubleshooting

Common Issues

1. Connection Errors: Ensure the server is running and the client is using the correct server URL
2. 405 Method Not Allowed: Ensure the client is sending requests to the correct API endpoint
3. SSE Connection Failure: Check network connections and firewall settings

Logging

Both server and client provide detailed logging. View logs for more information:

# Increase log level
export PYTHONPATH=.
python -m logging -v DEBUG -m mcp_server

References

- MCP Protocol Specification
- FastAPI Documentation
- JSON-RPC 2.0 Specification
- SSE Specification

License

This project is licensed under the MIT License. See the LICENSE file for details. 29:["$","div",null,{"className":"my-8 pb-8 h-full max-

No reviews yet — be the first

Sign in to leave a review

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.