Context Pipe

The Universal Standard for Context Engineering.

context-pipe is a high-performance orchestration layer directly inspired by Unix terminal piping — the same philosophy that made cmd1 | cmd2 | cmd3 the most durable composition primitive in computing. Just as the terminal chains processes through stdin/stdout byte streams, Context-Pipe chains AI tool calls through context streams: each node does one thing, passes its output to the next, and the LLM only sees the final, refined signal.

This is not a metaphor — it is a literal extension. Context-Pipe supports both MCP piping (chaining MCP tool calls through the orchestrator) and terminal piping (any binary, shell command, or script that reads stdin and writes stdout is a valid node). The two modes compose freely in a single pipe definition. And through the mcp-pipe CLI, it extends the terminal itself: the mcp-pipe tool subcommand makes any MCP server — context-mode, serena, GitHub, Firecrawl, or any server registered in pipes.json — directly pipeable from the shell, loading on demand, with no wrapper scripts and no IDE required:

cat error.log | mcp-pipe tool semantic-sift sift_logs | rg "CRITICAL"
curl -s https://example.com | mcp-pipe tool firecrawl scrape | mcp-pipe run semantic-refinery

Today, mcp-pipe run <pipe> already gives the terminal first-class access to any named pipe defined in pipes.json, composing terminal binaries through the same orchestrator used by the IDE.

---

🚀 The Vision

The AI agent has a fundamental infrastructure problem: every tool call returns raw, unfiltered output directly into the context window. Logs arrive with timestamps. Search results arrive with boilerplate. Agent A's 40KB analysis gets passed verbatim to Agent B. The context window fills. Signal drowns in noise. The LLM degrades.

context-pipe solves this at the infrastructure layer — before the LLM sees anything.

In the Studio of Two philosophy, we build Systems, not Patches. A patch would be a custom filter per tool. A system is a universal protocol: any tool that reads stdin and writes stdout becomes a node. Any sequence of nodes becomes a pipe. Any pipe is named, versioned, audited, and reusable across every project and every agent framework.

The result is a context supply chain: data enters raw, passes through a sequence of refineries (normalize → filter → compress → distil), and arrives at the LLM as dense, high-signal content. Every byte saved is accounted for in the Context Balance Sheet. Every pipe run is traceable. Every A2A handoff is protected.

This is not a wrapper around semantic-sift. It is the orchestration layer that makes any refinery composable, observable, and production-grade. A node can be a binary, a shell command, a Python script, or a full MCP tool (Figma, GitHub, context-mode, or any server registered in pipes.json). If it reads stdin and writes stdout, it belongs in the pipe.

Example — crawl the web, research it, save it, and ship it:

trigger: tool:web_search | tool:web_fetch

[URL]

    → firecrawl/scrape             # MCP node: fetch live page as clean text     ~18,400 tokens

    → markitdown                   # binary node: convert to structured Markdown     ~16,200 tokens

    → rg 'security|vulnerability'  # shell node: surface only relevant sections      ~3,100 tokens

    → prettier --parser markdown   # shell node: normalize formatting                ~3,050 tokens

    → semantic-sift-cli doc        # binary node: distil to high-signal summary        ~420 tokens

          ↳ tee → research.md      # T-pipe: save raw distilled copy to disk

    → security-auditor             # script node: project-specific logic               ~380 tokens

    → github/create_issue          # MCP node: open a tracked issue with findings

Context Balance Sheet (illustrative)
  in:  18,400 tokens  →  out: 380 tokens  —  97.9% saved  ·  1.2s total

Every node is a real subprocess. The T-pipe saves a raw copy at any point without interrupting the chain. The LLM receives only what matters — and every byte in, byte out, and millisecond of latency is recorded in the Context Balance Sheet automatically.

---

🛠️ Core Components

1. The Context-Pipe Protocol (CPP)

A language-agnostic standard with one rule: a node reads stdin, transforms content, and writes to stdout. Any binary, shell command, Python script, or MCP tool that honours this contract is a valid node. The protocol is defined in doc/CONTEXT_PIPE_PROTOCOL.md and is deliberately simple — no SDKs, no registration, no framework coupling.

2. The Orchestration Spine (orchestrator.py)

The execution engine that chains nodes into pipes. Runs each node as a real OS subprocess with shell=False enforced (no injection surface). Features: per-node timeout guard (PIPE_NODE_TIMEOUT_MS), T-Pipe stream splitting (save raw input to disk before a node processes it), and full trace accounting (input/output size + latency per node).

3. The Universal Switchboard (pipes.json + mappings)

Data-driven routing that resolves the optimal pipe automatically based on three trigger types: tool name (tool:regex), payload size (size:>N), and default fallback. Pipe definitions live in pipes.json (project-level) and optionally ~/.mcp-pipe.json (global, merged with local precedence). No code changes required to add, modify, or re-route pipes.

4. The MCP Surface (server.py + mcp-pipe CLI)

Eight MCP tools expose every capability to AI assistants directly: pipe_run, pipe_run_dynamic, pipe_read_file, pipe_analyze_file, pipe_list_shadow_tools, pipe_agent_handoff, get_pipe_stats, and pipe_onboard. The mcp-pipe CLI mirrors the same surface for terminal-first workflows — no IDE required. Shadow Tool Discovery (pipe_list_shadow_tools) gives the agent a live capability manifest combining configured pipes and curated PATH tools (jq, rg, markitdown, pandoc…).

5. Subconscious Interceptors (pipe_hook.py + onboarding.py)

IDE hooks that apply pipes transparently after every tool call — without the agent needing to invoke pipe_run explicitly. Supported: Cursor (postToolUse), VS Code/GitHub (hooks), Claude Code/Qwen/Codex (PostToolUse), Windsurf and Cline (pre-read security gateway), OpenClaw (native plugin), and pi.dev (native TypeScript extension). For OpenCode, the AGENTS.md SOP mandate is the active strategy (see Known Limitations). pipe_onboard injects all hooks, slash commands (/pipe-run, /pipe-dynamic, /pipe-handoff, /pipe-stats), and the full agent SOP in one command.

6. The A2A Bridge (a2a.py)

pipe_agent_handoff() distils Agent A's output before it enters Agent B's context window. Framework-agnostic — no monkey-patching. Works in CrewAI task callbacks, Google ADK transfer hooks, LangGraph edge functions, or any custom handoff point. Available as both a Python function and an MCP tool. Returns the original output unchanged on any error, so the agent chain is never interrupted.

7. The Native Rust Core (crates/cpipe)

cpipe is the high-performance Rust heart of the Context-Pipe ecosystem. It ports the full orchestration engine — config merging, placeholder resolution, stream routing, and the self-aware bypass guard — to a pre-compiled native binary with <2ms startup latency (500× faster than the Python runtime). It coexists with the Python server: MCP tools stay in Python (FastMCP), while the Rust binary is available as a Tauri sidecar, a standalone CLI (cpipe run, cpipe list, cpipe serve), or a Cargo library for direct embedding in Rust applications. See crates/cpipe/README.md for the full API.

---

✨ What Makes This Different

| Feature | What it does | Where |
|---|---|---|
| Unix pipe model for AI | Chain any stdin and stdout tool into a named pipe. Binary, shell, script, or MCP tool — same contract. | Advanced Node Types |
| MCP Node Type | Call any MCP tool (Figma, GitHub, context-mode) as a first-class pipe node — no wrapper scripts. | doc/MCP_NODE_SPEC.md |
| Compilation-free topology | Routing lives in pipes.json, not in node code. Reroute, branch, or swap a node by editing the map — no code changes, no recompile, no redeploy of any node. | doc/ARCHITECTURE.md |
| Protocol-first MCP composition | Swap any MCP server by changing a server key. No imports, no dependency declarations, no build cycle. Every MCP server speaks the same protocol — the entire ecosystem is a drop-in capability layer. | doc/ARCHITECTURE.md |
| Dynamic Pipes | AI agents construct and execute ad-hoc node lists at runtime via pipe_run_dynamic — no pipes.json entry required. | Dynamic Pipes |
| Shadow MCP Registry | Keep utility MCP servers invisible to the agent's tool list until needed. pipe_list_shadow_tools queries them on demand. | Shadow MCP Registry |
| A2A Agent Handoff | Distil Agent A's output before it enters Agent B's context window — framework-agnostic, no monkey-patching. | A2A Handoff |
| Version Awareness | Proactive GitHub-backed update alerts in pipe_verify and pipe_onboard to ensure environment parity. | Health Checks |
| Stream Integrity | Hardened orchestration engine with non-UTF8 robustness (errors="replace") and null-safe reading. | doc/ARCHITECTURE.md |
| T-Pipe Stream Splitting | Save a raw copy of any node's input to disk before it is distilled — for audit, debugging, and quality measurement. | 3. T-Pipe Nodes (Stream Splitting) |
| Adaptive Window Pressure | Signals remaining context headroom to every node; semantic-sift auto-adjusts --rate accordingly. | Environment Variables |
| Global Config | Share pipe definitions and MCP server registries across all projects — local pipes.json always wins. | doc/ARCHITECTURE.md |
| Shell Alias Injection | pipe_install_aliases writes mcp-pipe / cpipe into your shell profile — terminal-ready without venv activation. | Terminal Usage |
| Git Protection | pipe_onboard automatically updates .gitignore to protect internal artifacts from being committed. | Auto-Onboard |
| Context Balance Sheet | Every pipe run is accounted: chars in, chars out, latency per node, agent attribution, net ROI. | Telemetry & ROI |

---

🧠 The Architecture: Semantic Enums (Solving Schema Bloat)

In standard MCP setups, exposing multiple capabilities (PDF parsing, log searching, HTML cleaning) means exposing multiple tools. This causes Schema Bloat: the LLM's system prompt fills with thousands of tokens of complex tool instructions. For Small Language Models (SLMs), this pushes out chat history, overwhelms the context window, and leads to hallucinations.

context-pipe solves this through Semantic Enums.
Instead of teaching the AI how to use complex command-line utilities, you expose a single tool: pipe_run(input, pipe_name). The pipe_name parameter is simply an Enum of your predefined pipelines (e.g., ["parse-and-clean-pdf", "extract-critical-errors"]).

This perfectly separates Intent from Execution:
The LLM provides the Intent: "I need the clean text of this PDF, so I'll call the parse-and-clean-pdf pipe."
pipes.json provides the Execution: [pandoc -> jq -> semantic-sift]

By using brief, concise pipe names, you achieve extreme prompt compression. The AI gets a menu of high-level "buttons to push" rather than reading an instruction manual for every utility on the host machine. Better yet, if you upgrade your backend tooling (e.g., swapping pandoc for markitdown), you never have to update the LLM's prompt. The AI still calls the same pipe; the engine behind it just gets faster.

---

🔧 Three Independent Axes of Change

A CPP pipeline separates concerns across three layers that evolve on completely independent cycles:

| Layer | What it is | How you change it |
|---|---|---|
| Nodes | What each step does — a dumb stdin/stdout tool, unaware of the pipeline around it | Swap the binary, script, or MCP tool |
| pipes.json | The topology — how steps connect, branch, and route | Edit the map. No code change. No recompile. No redeploy. |
| MCP servers | The capability behind each tool call | Change the server key. No imports, no dependency declarations, no build cycle. |

Improving a node's quality does not change the topology. Restructuring the routing does not touch any node. Upgrading an MCP server improves every pipe that uses it automatically — with no pipeline changes.

For MCP nodes specifically, this dissolves the traditional dependency model entirely. Every MCP server speaks the same protocol: JSON-RPC, tools/call, text response. The pipe does not depend on what implements the service — it depends on what speaks the protocol. The entire MCP ecosystem is therefore the pipe's capability layer. Every current and future MCP server is already a valid drop-in replacement for any node that serves the same semantic purpose.

> In a script, you depend on what you import. In a pipe, you depend on what speaks the protocol.

This separation also means routing is compilation-free. In a traditional script, safeguards and recovery logic are embedded in code — changing how a workflow recovers requires changing, testing, and redeploying the script. In CPP, routing lives in pipes.json. A branch, a reroute, or a node swap is a configuration edit. The feedback loop between "what if I reroute this" and "let me observe what happens" collapses to near zero.

---

🚀 Quickstart (60 seconds)

```bash