Agent Evals by Galileo
About
Bring agent evaluations, observability, and synthetic test set generation directly into your IDE for free with Galileo's new MCP server
Details
- Author
- Unknown
- Categories
- Developer Tools, Infrastructure, AI
Jump to
What can you do with Agent Evals By Galileo MCP?
- Create synthetic datasets— UseCreate Datasetsto generate test data with prompt injections, off-topic, or toxic queries.
- Check dataset status— UseCheck Dataset Statusto track progress and preview the first 10 rows of a dataset.
- Create prompt templates— UseCreate Prompt Templatesto build reusable templates with model config and temperature.
- Set up experiments— UseSet Up Experimentsfor step-by-step guidance on running Galileo experiments with metrics.
- Get signals on Log streams— UseGet Signalsto analyze production log streams and get fix recommendations.
- Integrate with OpenAI or LangChain— Get step-by-step guides for adding Galileo observability and tracing to your apps.
Learn how to integrate Galileo's Model Context Protocol (MCP) server with AI-enabled IDEs like Cursor and VS Code
The Galileo Model Context Protocol (MCP) server enables seamless integration between AI-powered IDEs, such as Cursor, or VS Code with GitHub Copilot, and Galileo's evaluation and observability platform.
With MCP, you can access Galileo's capabilities directly from your development environment, including:
- Creating and managing datasets
- Running experiments
- Setting up prompt templates
- Getting signals on Log streams
- Integrating Galileo with your code
Before you begin, ensure you have the following:
The Galileo MCP server works with both Cursor and VS Code. Follow the steps below for your IDE:
<Step title="Open MCP settings"> Open the Command Palette (Ctrl + Shift + P on Windows/Linux, or Cmd + Shift + P on Mac) and search for "MCP: Open User Configuration" </Step> <Step title="Add the Galileo MCP server configuration"> Copy and paste the configuration below. Replace YOUR-API-KEY with your actual Galileo API key. This assumes you are using app.galileo.ai. If you're using a self-hosted Galileo deployment, replace the https://api.galileo.ai/mcp/http/mcp url with your deployment URL. The format of this URL is based on your console URL, replacing console with api and appending /mcp/http/mcp. For example: if your console URL is https://console.galileo.example.com, the MCP url would be https://api.galileo.example.com/mcp/http/mcp if your console URL is https://console-galileo.apps.mycompany.com, the MCP url would be https://api-galileo.apps.mycompany.com/mcp/http/mcp </Step> <Step title="Reload VS Code"> Reload VS Code by opening the Command Palette and running "Developer: Reload Window" for the changes to take effect </Step> </Steps>
<Step title="Open the MCP server configuration file"> Click on Add Custom MCP to open the mcp.json MCP configuration file, or New MCP Server if you already have other MCP servers configured. </Step> <Step title="Add the Galileo MCP server configuration"> Copy and paste the configuration below. Replace YOUR-API-KEY with your actual Galileo API key. `json Cursor MCP Configuration highlight={3-9} theme={null} { "mcpServers": { "galileo_mcp_server": { "url": "https://api.galileo.ai/mcp/http/mcp", "headers": { "Galileo-API-Key": "YOUR-API-KEY", "Accept": "text/event-stream" } } } } ` This assumes you are using app.galileo.ai. If you're using a self-hosted Galileo deployment, replace the https://api.galileo.ai/mcp/http/mcp url with your deployment URL. The format of this URL is based on your console URL, replacing console with api and appending /mcp/http/mcp. For example: if your console URL is https://console.galileo.example.com, the MCP url would be https://api.galileo.example.com/mcp/http/mcp if your console URL is https://console-galileo.apps.mycompany.com, the MCP url would be https://api-galileo.apps.mycompany.com/mcp/http/mcp </Step> <Step title="Save and restart"> Save the configuration and restart Cursor for the changes to take effect </Step> </Steps>
The configuration is the same for both Cursor and VS Code. Make sure to replaceYOUR-API-KEYwith your actual Galileo API key from theAPI keys page.
Once configured, you can verify your MCP setup by asking your AI assistant in your IDE:
Help me create a synthetic dataset for customer support queries
How do I integrate Galileo with LangChain?
Your AI assistant should now be able to access Galileo's capabilities and respond with information from your Galileo account.
The Galileo MCP server provides powerful tools that you can access through natural conversation with your AI assistant. Simply ask questions or make requests, and the AI will use these tools to help you.
What you can ask: <CodeGroup> `text Query wrap theme={null} Create a dataset with 50 customer service queries about billing issues ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Generate a dataset of 30 chatbot queries, including some prompt injection attempts ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Make a dataset with product recommendation queries and include off-topic questions ` </CodeGroup>
What you can ask: <CodeGroup> `text Query wrap theme={null} Check the status of my dataset that's currently generating ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Show me the preview of dataset abc-123 ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Is my customer service dataset ready yet? ` </CodeGroup>
What you can ask: <CodeGroup> `text Query wrap theme={null} Create a prompt template called "Friendly Assistant" for customer support responses ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Make a prompt template for summarizing technical documentation with lower temperature ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Set up a chat template for a code review assistant ` </CodeGroup>
What you can ask: <CodeGroup> `text Query wrap theme={null} How do I set up a Galileo experiment in Python? ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Show me how to run an experiment with my RAG application ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Guide me through creating an experiment for my agentic workflow ` </CodeGroup>
What you can ask: <CodeGroup> `text Query wrap theme={null} What issues do you see in my production log stream? ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Fetch the most recent signals from Galileo and propose fixes for them in my code ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Get insights about my chatbot log stream and suggest improvements ` </CodeGroup>
What you can ask: <CodeGroup> `text Query wrap theme={null} How do I add Galileo logging to my OpenAI application? ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Show me how to integrate Galileo with OpenAI in TypeScript ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Help me set up Galileo observability for my GPT-4 chatbot ` </CodeGroup>
What you can ask: <CodeGroup> `text Query wrap theme={null} How do I integrate Galileo with my LangChain application? ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Show me how to add Galileo tracing to my LangChain agent ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Help me log my LangChain RAG pipeline with Galileo ` </CodeGroup>
What you can ask: <CodeGroup> `text Query wrap theme={null} How do I set up data logging in Galileo? ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Find documentation about custom metrics ` </CodeGroup> <CodeGroup> `text Query wrap theme={null} Search for examples of agentic AI evaluation ` </CodeGroup>```
The MCP server will guide you through the dataset creation process and provide a dataset ID to track progress.
The MCP server will provide complete integration code examples and setup instructions.
The MCP server will analyze your Log stream and suggest improvements.
This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.
A secure sandbox for executing code in Docker containers, providing a safe environment for AI applications.
FutureAGI EvalOps MCP is a paid remote MCP endpoint for AI SDK eval dashboard. It exposes structured JSON tools, a public server card, token-based access, usage receipts, and audit-ready wor
Verify AI agent communication with session types and formal proofs
Step-by-step observability for MCP agent workflows — trace, inspect, and debug multi-step agent executions
Take screenshots and analyze mobile apps with AI assistance directly from your IDE.
Create mock MCP servers instantly for developing and testing agentic AI workflows.
An advanced penetration testing tool for automated, LLM-driven security assessments using tools like nmap and dirb.
Prompt optimization loops and regression test suites for Claude Code, with a companion web UI.
Cyber Host Artificial Intelligence (C.H.A.I) is Autonomous penetration testing MCP (Model Context Protocol) server with an integrated AI decision engine, multi-provider LLM support, and an extensible plugin architecture.
A MCP server that connects with LangChain Checkpointers, Memory Stores to aid in monitoring and observability during development of AI Applications
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
