Schema Search

Note: The project has moved to anew repo and is being maintained by SignalPilot Labs. The code in the current repo is only available for reference.

# Minimal install (BM25 + fuzzy only, ~10MB) pip install "schema-search[postgres]" # With semantic/hybrid search support (~500MB with PyTorch) pip install "schema-search[postgres,semantic]" # With LLM chunking pip install "schema-search[postgres,semantic,llm]" # With MCP server pip install "schema-search[postgres,semantic,mcp]" # Other databases pip install "schema-search[mysql,semantic]" # MySQL pip install "schema-search[snowflake,semantic]" # Snowflake pip install "schema-search[bigquery,semantic]" # BigQuery pip install "schema-search[databricks,semantic]" # Databricks

- [semantic]: Enables semantic/hybrid search and CrossEncoder reranking (adds sentence-transformers)
- [llm]: Enables LLM-based schema chunking (adds openai)
- [mcp]: MCP server support (adds fastmcp)

logging: level: "WARNING" embedding: location: "memory" # Options: "memory", "vectordb" (coming soon) model: "multi-qa-MiniLM-L6-cos-v1" metric: "cosine" # Options: "cosine", "euclidean", "manhattan", "dot" batch_size: 32 show_progress: false cache_dir: "/tmp/.schema_search_cache" chunking: strategy: "raw" # Options: "raw", "llm" max_tokens: 256 overlap_tokens: 50 model: "gpt-4o-mini" search: # Search strategy: "semantic" (embeddings), "bm25" (BM25 lexical), "fuzzy" (fuzzy string matching), "hybrid" (semantic + bm25) strategy: "bm25" initial_top_k: 20 rerank_top_k: 5 semantic_weight: 0.67 # For hybrid search (bm25_weight = 1 - semantic_weight) hops: 1 # Number of foreign key hops for graph expansion (0-2 recommended) reranker: # CrossEncoder model for reranking. Set to null to disable reranking model: null # "Alibaba-NLP/gte-reranker-modernbert-base" schema: include_columns: true include_indices: true include_foreign_keys: true include_constraints: true output: format: "markdown" # Options: "json", "markdown" limit: 5 # Default number of results to return

Integrate with Claude Desktop or any MCP client.

Add to your MCP config (e.g.,~/.cursor/mcp.jsonor Claude Desktop config):

{ "mcpServers": { "schema-search": { "command": "uvx", "args": [ "schema-search[postgres,mcp]", "postgresql://user:pass@localhost/db", "optional/path/to/config.yml", "optional llm_api_key", "optional llm_base_url" ] } } }

{ "mcpServers": { "schema-search": { // conda: /Users/<username>/opt/miniconda3/envs/<your env>/bin/schema-search", "command": "path/to/schema-search", "args": [ "postgresql://user:pass@localhost/db", "optional/path/to/config.yml", "optional llm_api_key", "optional llm_base_url" ] } } }

The LLM API key and base url are only required if you use LLM-generated schema summaries (config.chunking.strategy = 'llm').

schema-search "postgresql://user:pass@localhost/db" "optional/path/to/config.yml"

Optional args:[config_path] [llm_api_key] [llm_base_url]

The server exposesschema_search(query, hops, limit)for natural language schema queries.

from sqlalchemy import create_engine from schema_search import SchemaSearch # PostgreSQL engine = create_engine("postgresql://user:pass@localhost/db") sc = SchemaSearch( engine=engine, config_path="optional/path/to/config.yml", # default: config.yml llm_api_key="optional llm api key", llm_base_url="optional llm base url" ) sc.index(force=False) # default is False results = sc.search("where are user refunds stored?") # Default output is markdown - render with str() print(results) # Formatted markdown with schemas, relationships, and scores # Access underlying data as dictionary result_dict = results.to_dict() for result in result_dict['results']: print(result['table']) # "refund_transactions" print(result['schema']) # Full column info, types, constraints print(result['related_tables']) # ["users", "payments", "transactions"] # Override output format explicitly json_results = sc.search("where are user refunds stored?", output_format="json") print(json_results) # JSON formatted string # Override hops, limit, search strategy, and output format results = sc.search("user_table", hops=1, limit=5, search_type="hybrid", output_format="markdown")

sc.index()automatically detects schema changes and refreshes cached metadata, so you rarely need to force a reindex manually.

Schema Search uses SQLAlchemy connection strings:

# PostgreSQL engine = create_engine("postgresql://postgres:mypass@localhost:5432/mydb") # MySQL engine = create_engine("mysql+pymysql://root:mypass@localhost:3306/mydb") # Snowflake engine = create_engine("snowflake://myuser:mypass@xy12345.us-east-1/MYDB/PUBLIC?warehouse=COMPUTE_WH&role=ANALYST") # BigQuery engine = create_engine("bigquery://my-project/my-dataset") # Databricks token = "dapi..." host = "dbc-xyz.cloud.databricks.com" http_path = "/sql/1.0/warehouses/abc123" catalog = "main" schema = "default" # Optional # Without schema (queries across all schemas in catalog) engine = create_engine( f"databricks://token:{token}@{host}?http_path={http_path}&catalog={catalog}", connect_args={"user_agent_entry": "schema-search"} ) # With schema (limits to specific schema) engine = create_engine( f"databricks://token:{token}@{host}?http_path={http_path}&catalog={catalog}&schema={schema}", connect_args={"user_agent_entry": "schema-search"} )

Schema Search supports four search strategies:

- bm25: Lexical search using BM25 ranking algorithm (no ML dependencies)
- fuzzy: String matching on table/column names using fuzzy matching (no ML dependencies)
- semantic: Embedding-based similarity search using sentence transformers (requires[semantic])
- hybrid: Combines semantic and bm25 scores (default: 67% semantic, 33% bm25) (requires[semantic])

Each strategy performs its own initial ranking, then optionally applies CrossEncoder reranking ifreranker.modelis configured (requires[semantic]). Setreranker.modeltonullto disable reranking.

Webenchmarkedon the Spider dataset (1,234 train queries across 18 databases) using the defaultconfig.yml.

# Use fuzzy search instead of default results = sc.search("user_table", search_type="fuzzy") # Use BM25 for keyword-based search results = sc.search("transactions payments", search_type="bm25") # Use hybrid for best of both worlds results = sc.search("where are user refunds?", search_type="hybrid") # Override hops and limit results = sc.search("user refunds", hops=2, limit=10) # Expand 2 hops, return 10 tables # Disable graph expansion results = sc.search("user_table", hops=0) # Only direct matches, no foreign key traversal

output: format: "markdown" # or "json" limit: 5 # Default number of results

# Default markdown output - just print the object results = sc.search("user payments") print(results) # Formatted markdown # Access underlying data as dictionary data = results.to_dict() print(data['results'][0]['table']) # "users" # Override to JSON format json_results = sc.search("user payments", output_format="json") print(json_results) # JSON formatted string

- Table name and relevance score
- Primary keys and columns with types/constraints
- Foreign key relationships
- Indices and constraints
- Related tables from graph expansion
- Matched content chunks

Use LLM to generate semantic summaries instead of raw schema text (requires[llm]extra):
- Install:pip install "schema-search[postgres,llm]"
- Setstrategy: "llm"inconfig.yml
- Pass API credentials:

sc = SchemaSearch( engine, llm_api_key="sk-...", llm_base_url="https://api.openai.com/v1/" # optional )

- Extract schemasfrom database using SQLAlchemy inspector
- Chunk schemasinto digestible pieces (markdown or LLM-generated summaries)
- Initial searchusing selected strategy (semantic/BM25/fuzzy)
- Expand via foreign keysto find related tables (configurable hops)
- Optional rerankingwith CrossEncoder to refine results
- Return top tables with full schema and relationships

Cache stored in/tmp/.schema_search_cache/(configurable inconfig.yml)

An AI-powered gateway for managing over 40 data sources like Alibaba Cloud and mainstream databases, featuring NL2SQL, code generation, and data migration.

Query Apple Health data using natural language and SQL.

Interact with AskTable SaaS or local deployments to query data sources using natural language.

Interact with CrateDB using natural language for Text-to-SQL queries and documentation retrieval.

MCP servers that give AI clients safe, structured access to SQL databases. One installable package per database engine. Every package exposes the same minimal two-tool surface, execute_sql and search_objects, with guardrails on by default: read-only mode, row caps, and statement timeouts.

Interact with an Elasticsearch cluster via the Model Context Protocol (MCP), enabling clients to query, manage, and analyze data.

Enables AI assistants to query data from DataFocus using natural language.

Provides read-only access to Apache Iceberg tables via Apache Impala, allowing LLMs to inspect schemas and execute queries.

Data intelligence platform - query your database in natural language, build dashboards, and set up automated alerts that monitor your metrics 24/7.

Interact with MongoDB databases using natural language to query collections, inspect schemas, and manage data.