sqbyl

A natural-language-to-SQL surface is only as good as the accuracy number you can put in front of stakeholders and stand behind. sqbyl is designed end-to-end around the ML-systems principles that keep that number honest — the same discipline you'd want before deploying any evaluated agent at scale:

-

Deterministic-first measurement.The headline accuracy isresult-set correctness— execute the gold SQL and the generated SQL, compare the rows. No LLM sits inside the number, so it's reproducible and can't drift with a prompt. LLM judges are strictlyadvisory: they triage the ambiguous pile and explainwhya row is suspect, but they never move the reported accuracy. Only a human override is authoritative.

Real train/test discipline.benchmarks/test.yamlis asealed held-out set. The dev loop — synth, coach, optimizer — can never read it; that's enforced as acode boundary(an import-linter rule in CI), not a convention you have to remember. Even judge calibration is split-scoped, so dev feedback can't leak into the test judge.The headline number is always the held-out one, with the dev score shown beside it so the gap is visible.

Goodhart-resistance by construction.The Coach optimizes context against the dev set — but itstructurally cannotmove the deterministic accuracy number, it's steered away from memorizing benchmark answers (fix the semantics, not the prompt), and it warns you that dev gains areunvalidated until a held-out re-score. Optimizing and measuring on the same set is training on the test set; sqbyl makes that mistake hard to commit.

Calibrated, honest uncertainty.A small eval set is noise-prone, so accuracy carries aWilson confidence interval— a 1–2 question flip on 30 questions isn't dressed up as a trend. A livejudge↔human agreementscore tells you exactly how far to trust the judge, and it's labeled asselection-biasedrather than overclaimed. The model's own self-reported confidence is labeled"unverified"— never presented as calibrated.

Reproducibility and provenance.Every scored run is stamped with themodel version per roleand the calibration state that shaped it. A score is never divorced from what produced it — the release scorecard records the exact models the number was earned on, and the runtime warns on model or schema mismatch at load.

Human-in-the-loop, everywhere.One unifying pattern runs through the judge, the benchmark synthesis, and the Coach:the LLM proposes, the human disposes, and the correction improves the system.Every judge verdict, synthesized question, and fix is a reviewable proposal, not a fait accompli.

Cost honesty.Free, deterministic work runs first at$0(connect, profile, infer joins). Paid work is estimated before, metered live, and capped by--budget. The economics of the agent are as legible as its accuracy.

The short version:sqbyl helps you ship a text-to-SQL agent whose accuracy you can actually report— because the number is deterministic, held out, provenance-stamped, and defended against the ways evaluation loops quietly lie to you.

Architecture: two packages, one dependency arrow

sqbyl ships astwo packages, so what you develop with is not what you deploy:

- sqbyl-runtime— the minimal, dependency-light runtime you embed in production: load a release,ask(), structured logging. No web stack, no eval machinery.
- sqbyl— the full dev toolkit: introspect, profile, annotate, synth, the eval harness, the Coach, LLM judges, the review console, the optimizer, and the release builder.

sqbyldepends onsqbyl-runtime,never the reverse— a one-way boundary enforced in CI (import-linter). None of the dev/eval machinery can leak into what runs in your app. You iterate with the toolkit; you ship the runtime. Both are strict-typed (py.typed) and pydantic-backed, and the release interface is a documented,schema_version'd JSON that a third party can read without sqbyl at all.

You're putting a natural-language-to-SQL surface over your own warehouse — an internal analytics tool or a product feature — and you need an accuracy number you can defend, plus a system you can read, edit, and version. You work in git, a SQL database, and YAML. You have (or can provision) aread-onlydatabase role and an LLM provider key (Anthropic or OpenAI).

export ANTHROPIC_API_KEY=sk-ant-... export DATABASE_URL=postgresql://readonly_user@warehouse.internal/analytics # use a read-only role

Then run the guided setup. Nosqbyl.yamlto hand-write —initscaffolds it for you if it's missing. It does the free, deterministic work first (connect, read schema, profile every column with read-only SQL), verifies your provider key ($0), shows you acosted plan, and only spends after you confirm:

Production is "just a model with logs." The dev machinery (eval, synth, coach, console) doesnotcome along — you embed the lightweight runtime:

from sqbyl_runtime import load agent = load("revenue-analytics.v1.json", db=env.DATABASE_URL, model="claude-opus-4-8") @app.post("/ask") # your API, your auth, your scaling def ask(q: str): return agent.ask(q) # → {plan, sql, rows, used_assets, usage, latency}

It inherits your app's auth, connection pooling, and observability.sqbyl run <release>/sqbyl serveexist for non-Python callers and quick HTTP exposure, but areintentionally not hardened— don't putsqbyl serveon the open internet.

Natural-language answers (opt-in).ask()returns structured data —sql,columns,rows— which stays the authoritative answer. For a chat/assistant surface that also wants a sentence ("There are 1,284 orders."), enable narration:load(..., narrate=True)(or per call,agent.ask(q, narrate=True)). That addsonefinal summarization call, grounded strictly on the executed rows, and populatesresult.answer. It'soff by defaultso the deterministic,$0-by-default runtime is unchanged; the call meters as its ownnarraterole (pin a cheaper model withnarration_model=), and the narrated sentence is a convenience over the rows, never a substitute for them. The CLI mirrors this withsqbyl ask "…" --narrate.

agent.ask()issynchronous and blocking(an LLM round-trip plus DB queries), but a single loadedagentissafe to call concurrently— the DB engine pools per-thread connections, the provider client (Anthropic or OpenAI) is thread-safe, and trace writes are locked. So under a threadpool it serves concurrent requests correctly.

The one rule for anasyncserver: runask()off the event loop, don't call it inside anasync defdirectly (that blocks the loop for the whole request).

# FastAPI: a sync endpoint is auto-run in a threadpool — this is the example above, and it's correct. @app.post("/ask") def ask(q: str): ... # From an async endpoint, offload explicitly: from starlette.concurrency import run_in_threadpool @app.post("/ask") async def ask(q: str): return await run_in_threadpool(agent.ask, q) # or asyncio.to_thread(agent.ask, q)

Bound concurrency (threadpool + DB pool size) as you would for any blocking workload. A native-async runtime (async provider client + async DB) isn't provided — the threadpool pattern is the supported path.

A sqbyl project is a git-native directory of plain files:

my-project/ ├── sqbyl.yaml # manifest: db connection, model(s), defaults ├── instructions.md # the (small) global instruction block ├── semantics/ # one YAML per table: columns, profiles, joins, measures, filters ├── examples/ # NL → SQL few-shot pairs ├── trusted/ # vetted, parameterized "source of truth" queries ├── benchmarks/ │ ├── dev.yaml # iteration set: Coach/Optimizer tune against this │ └── test.yaml # held-out set: Coach/Optimizer NEVER see it └── .sqbyl/ # runs, traces, usage, caches (gitignored)

Project configuration lives insqbyl.yaml; secrets are referenced byenv:name, not inlined:

name: revenue-analytics database: dialect: postgresql # postgresql | duckdb | snowflake | bigquery | mysql | sqlite url: env:DATABASE_URL read_only: true # refuses non-SELECT; warns if the credential can write model: provider: anthropic # anthropic | openai — one provider powers every role api_key: env:ANTHROPIC_API_KEY default: claude-opus-4-8 # per-role models (agent/judge/coach/...) override default # base_url: env:LLM_GATEWAY # optional: route the provider through a proxy / AI gateway

To use OpenAI instead, switch the three provider lines (everything else is identical):

model: provider: openai api_key: env:OPENAI_API_KEY default: gpt-5

The section a security reviewer will look for:

- Read-only by default.sqbyl refuses non-SELECTat the SQL layer and, on connect, inspects the credential's privileges and warns (with a suggested fix) if it can write. The agent and the Coach never issue DDL/DML. Point it at a dedicated read-only role.
- Secrets by reference.Connection strings and API keys areenv:-indirected — never written to project files, releases, or traces.
- Your data stays yours.Query result rows are not persisted to committed project files or traces; imported SQL that carries literal values is flagged for review before it can land. A release is the agent'sbrain— semantics, prompts, examples — never rows.
- Local-first, exportable telemetry.Traces follow the OpenTelemetry GenAI conventions and are written under.sqbyl/; export them to any OTel backend when you want.
- CI never spends tokens.Every LLM path runs against a mock / record-replay seam, so continuous integration never calls the API. Dependencies are vulnerability-scanned (pip-audit), license-checked, and updated via Dependabot.
- Production hardening is yours.You embed the runtime in your own service, inheriting its auth, TLS, pooling, and rate limiting.sqbyl serveis a localhost dev convenience, not a production server — don't expose it.

The spec is thewhy, the journey is anarrated first run, and the plan is a record ofhow it was built:

This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.

Create crafted UI components inspired by the best 21st.dev design engineers.

Bring agent evaluations, observability, and synthetic test set generation directly into your IDE for free with Galileo's new MCP server

An MCP server to help AI assistants to answer questions and generate AccelByte Extend SDK code more effectively .

MCP server for AI Diagram Maker — generate beautiful software engineering diagrams directly inside Cursor, Claude Desktop, Claude Code, or any MCP-compatible AI agent

ALAPI MCP Tools,Call hundreds of API interfaces via MCP

AI-powered SVG animation generator that transforms static files into animated SVG components using the Allyson platform

MCP server that gives AI assistants on-demand access to 1,500+ amCharts docs, ~300 code examples, and 1000+ class API references.

APIMatic MCP Server is used to validate OpenAPI specifications using APIMatic. The server processes OpenAPI files and returns validation summaries by leveraging APIMatic’s API.

One shared context layer for AI agents and humans — live API specs, DB schemas, and versioned contracts across repos so every agent and teammate works from the same source of truth.