🔍 🤖 🌐 Osoba

A powerful, modern UI that integrates local and hosted LLMs with intelligent web search and content extraction, SQL, YouTube transcript analysis, HubSpot actions, Python data analysis, Canva design creation, Figma design inspection, Poe multi-model AI (text, image, video, audio generation) — and a Codex MCP server for safe code scaffolding — all via the Model Context Protocol (MCP). Features include personalized AI assistance through user profiles and conversation context, a Document Knowledge Base for persistent reference material, provider settings, multi‑provider model picking, streaming chat, persistent conversations, a robust Tasks system, and Scheduled tasks with timezone‑aware timing.

Overview

Osoba showcases how to extend both local and hosted models through MCP tool use. It combines locally running LLMs via Ollama with intelligent web search and content extraction, SQL querying, YouTube transcript ingestion, HubSpot business actions, Python-based CSV analysis/visualization, Canva design creation and export, Figma file/design inspection and export, Poe multi-model AI for text chat and media generation (image, video, audio) — and a Codex Workspace server for code generation inside an isolated workspace. A multi‑provider layer adds OpenAI, Anthropic, Google, OpenRouter, Groq, and SambaNova. Conversations and tasks persist in MongoDB.

The project consists of several key components:

- Backend (FastAPI): Manages chat logic, model provider routing, MCP service communication (including starting and managing MCP services like web search, SQL, YouTube, HubSpot, Python, Canva, Figma, Poe, Codex), tasks/scheduler, and persistence.

- Frontend (React): A modern, responsive web interface for users to interact with the chat application.

- MCP Smart Web Search Server Module: Provides intelligent web search via the Serper.dev API with automatic content extraction from top results. Uses multi-method content extraction (trafilatura, BeautifulSoup) with URL prioritization, robots.txt compliance, and polite crawling practices.

- MCP SQL Server Module: Read-only querying against a MySQL database (with schema resources and query safety).

- MCP YouTube Transcript Server: Robustly fetches transcripts using multiple strategies (youtube-transcript-api, Pytube, yt-dlp) with optional proxy support, then persists transcript context to the conversation for follow-ups.

- MCP HubSpot Business Tools: Create/update marketing emails via HubSpot APIs using OAuth, with a JSON-first prompting flow.

- MCP Python Data Analysis Server: Comprehensive data analysis toolkit including CSV loading, data profiling, filtering, grouping/aggregation, outlier detection, data type conversion, statistical hypothesis testing, and visualization (base64 images) for complete analytical workflows.

- MCP Canva Design Server: Create, browse, export, import, resize, and autofill Canva designs directly from chat or autonomous tasks. Supports standard size presets (Instagram, YouTube, presentation, and more), custom dimensions, brand template autofill (Enterprise), asset uploads, design imports (PDF/PPTX/DOCX/PSD), and export to PNG, JPG, PDF, PPTX, SVG, MP4, or GIF. Requires a Canva Connect API access token (OAuth 2.0).

- MCP Figma Design Server: Read Figma files, extract specific nodes, export frames and components as images (PNG/JPG/SVG/PDF), manage comments, and extract the full design system (color tokens, typography, spacing, components) — all from chat or tasks. Requires a Figma Personal Access Token.

- MCP Poe AI Models Server: Access hundreds of AI models on the Poe platform through a single OpenAI-compatible API. Generate text with frontier LLMs (Claude, GPT, Gemini, Grok, Llama), create images with models like GPT-Image-1.5, Flux, and Stable Diffusion (with aspect ratio control), generate video and audio — all from chat or tasks. Requires a Poe API key.

- MCP Codex Workspace Server: Creates per‑run workspaces and launches the Codex CLI within an isolated directory, persists artifacts (JSONL events, manifest), enforces a configurable output policy, and exposes async run APIs for a streaming‑friendly UX. Gated on a valid OpenAI API key.

- MongoDB: Stores conversation history and user data.

This architecture demonstrates how MCP enables local models to access external tools and data sources, significantly enhancing their capabilities. The backend starts and supervises all MCP services for a scalable, feature-rich setup.

Demo

- Watch the demo on X: https://x.com/redbuilding/status/2010124029936427450

Features

- 🧠 Smart Web Search: Intelligent web search with automatic content extraction from top results. Goes beyond search snippets to fetch and analyze full webpage content using advanced extraction techniques.
- 🔎 URL Prioritization: Smart ranking of search results based on relevance scoring, including title/snippet matching, domain authority, and search position weighting.
- 🤖 Polite Web Crawling: Respects robots.txt, implements rate limiting, and uses proper User-Agent identification for ethical content extraction.
- 👤 User Profile & Context: Configure personal information (role, expertise, projects) and pin conversations for contextual AI assistance. The AI understands your background and can reference previous work for personalized responses.
- 🎯 Goals & Priorities: Define your short-term, medium-term, and long-term goals in a structured document (up to 2000 characters) to help the AI understand your objectives and provide relevant assistance.
- 🔔 Proactive Agent Heartbeat: Enhanced background service with context gathering (semantic memory, git, project files, system health), automated task creation, and file-based configuration (HEARTBEAT.md). Configurable intervals, context sources, and two-way sync between file and UI.
- 📌 Conversation Pinning: Select specific conversations to include as context for future chats, enabling the AI to build upon previous discussions and maintain continuity across sessions.
- 🧠 Semantic Memory: Unlimited conversation storage with intelligent semantic search powered by ChromaDB and nomic-embed-text embeddings. Automatically indexes conversations with 5+ messages, searches by meaning (not keywords), and injects relevant past conversations into new chats. Includes Memory Browser (Ctrl+Shift+M) for searching and managing your conversation history.
- 📚 Document Knowledge Base: Upload PDFs, Word documents, Markdown files, plain text, or web URLs as persistent reference material. Documents are chunked, embedded, and stored in a dedicated ChromaDB collection. Relevant document snippets are automatically injected into every chat alongside Semantic Memory. Open with Ctrl+Shift+K or the "KB" header button.
- 🧾 AI Chat Summaries (On‑Demand): Generate concise, LLM‑authored summaries for conversations and use them as the context payload for pinned chats (no heuristics). Summaries are user‑triggered, model‑selectable in Settings, and pinning enforces a cap of 5 chats.
- 🎯 Personalized AI Responses: AI adapts its communication style and suggestions based on your profile information and pinned conversation history for more relevant assistance.
- 💾 Persistent Conversations: Chat history is saved in MongoDB, allowing users to resume conversations.
- 🧠 Multi‑Provider LLMs: Ollama (local), plus OpenAI, Anthropic, Google, OpenRouter, Groq, SambaNova — with a Settings screen for API keys and a unified model picker.
- 🔌 MCP integration: Backend manages multiple MCP tools (web, SQL, YouTube, HubSpot, Python) as background services.
- 💻 Modern Web Interface: Built with React for a responsive and interactive user experience.
- 📊 Structured search results: Clean formatting of web search data for optimal context.
- 🌐 Enhanced Content Extraction: Multi-method content extraction using trafilatura and BeautifulSoup with graceful fallbacks for maximum reliability.
- ⚙️ Backend API: FastAPI backend providing robust API endpoints for chat and conversation management.
- 🗃️ SQL Querying Tool: Read-only MySQL querying with schema introspection and retry logic.
- 🔄 Conversation Management: List, rename, and delete conversations.
- 📺 YouTube Transcript Tool: Paste a YouTube URL; transcripts are fetched and stored for multi-turn follow-ups.
- 🧰 HubSpot Tools: OAuth-connect, then create/update marketing emails via guided JSON prompts.
- 🐍 Python Analysis Tool: Upload a CSV and run comprehensive analysis including data profiling, filtering, grouping, outlier detection, statistical testing, type conversion, and visualization — results stream back with images and detailed insights.
- ⚡ Streaming Responses: Frontend renders model output token-by-token and indicators for tool usage.
- 🗓️ Long‑Running Tasks (Plan & Execute): Create autonomous tasks that plan and execute multi‑step workflows, with budgets, retries, and verification.
- 📈 Live Task Progress: Tasks stream progress via SSE; per‑step outputs (tables/images/text) render in the UI.
- 🧩 LLM‑only Steps (No MCP): Tasks can include steps that run directly on Ollama (e.g., summaries/reasoning) without using any MCP tool.
- 🚦 Priority Task Queue: Memory-safe task execution with priority scheduling - scheduled tasks run first, user tasks queue behind them, only one task executes at a time to prevent system overload, especially important for local, memory-constrained systems.
- 🎨 Canva Design Tools (MCP): Create Canva designs from chat or tasks using standard presets (Instagram, YouTube thumbnail, presentation, A4, and more) or custom dimensions. Browse existing designs, upload assets, import files (PDF/PPTX/DOCX/PSD) as editable designs, resize designs for different platforms, autofill brand templates, and export to PNG, JPG, PDF, PPTX, SVG, MP4, or GIF — all without leaving Osoba. Requires a Canva Connect API access token.
- 🖼️ Figma Design Tools (MCP): Read Figma file structures, extract specific nodes by ID, export frames and components as images, list and post comments, and extract the full design system (color, typography, spacing tokens and component catalog). Useful for design review workflows, design-to-code handoffs, and automated design auditing. Requires a Figma Personal Access Token.
- 🤖 Poe AI Models (MCP): Access hundreds of AI models via the Poe platform — generate text with frontier LLMs, create images with GPT-Image-1.5, Flux, or Stable Diffusion (with 9:16/16:9 aspect ratio control), generate video and audio. Use from chat or autonomous tasks. Requires a Poe API key.
- ✨ Codex Workspace (MCP): Launch Codex to generate/edit files in an isolated workspace; inline run status in chat; artifacts persisted for review; gated on OpenAI key.
- 🗓️ Scheduled Tasks (Timezone‑Aware): Recurring cron or one‑time schedules computed in local timezone with DST safety; auto‑disable after first run for one‑time schedules; “Run now” with model override.

Task Execution & Memory Management

Task System MCP Tool Access

The task system has full access to all MCP tools available in the chat interface, enabling autonomous execution of complex workflows:

Available Tools (55 total):
- Web Search (5 tools): Basic search, smart content extraction, image search, news search, direct URL fetch
- Python Data Analysis (17 tools): Data loading, inspection, cleaning, transformation, statistical analysis, visualization
- Canva Design (10 tools): Create designs, list designs, get design details, export to file, upload assets, autofill brand templates, get template fields, import designs, resize designs, get design pages
- Figma Design (6 tools): Get file structure, get nodes, export images, list comments, post comment, extract design system
- Poe AI Models (5 tools): List models by modality, text chat, image generation (with aspect ratio), video generation, audio generation
- HubSpot Business (2 tools): Create/update marketing emails (requires OAuth)
- Codex Workspace (7 tools): Code generation and workspace management (requires OpenAI API key)
- Database (1 tool): Read-only SQL queries
- YouTube (1 tool): Transcript extraction
- LLM-only (1 tool): Direct LLM generation for reasoning steps

Advanced Task Capabilities:
- Research with smart content extraction (full webpage content, not just snippets)
- Advanced data analysis with outlier detection and statistical hypothesis testing
- Canva design workflows: research a topic then automatically create and export a design
- Figma design workflows: extract design tokens, export assets, and review file structure for design-to-code handoffs
- Poe AI workflows: generate images with aspect ratio control, then upload to Canva or use in multi-step creative pipelines
- Marketing automation with HubSpot integration
- Code generation with fine-grained Codex workspace management
- Multi-step workflows combining search, analysis, design, and generation
- KB context attachment: Attach 1–2 indexed Knowledge Base documents at task creation to inject reference material into the planner and every LLM step

Priority Queue System

- Priority 1 (Highest): Scheduled tasks always run first
- Priority 2 (Standard): User-created tasks queue behind scheduled tasks
- One Task at a Time: Only one task executes simultaneously to prevent memory crashes
- Queue Position: Users receive feedback about their position in the queue

Memory Safety

Task Scheduling

Scheduled Task Reliability

Catch-up Logic (Default):
When the backend starts after being stopped or the computer wakes from sleep, overdue scheduled tasks execute immediately. Tasks delayed by more than 5 minutes are marked with a warning indicator in the UI showing how late they ran (e.g., "⚠️ Last run: 45m late"). Recurring tasks calculate their next run from the current time to prevent cascading delays.

Delay Breakdown:
The system distinguishes between two types of delays:
- System Delay (sleep): Time the computer was asleep or backend was stopped
- Queue Delay (queued): Time spent waiting for other tasks to complete

The UI shows the breakdown: "⚠️ Last run: 15m late (sleep: 10m, queued: 5m)". This helps you understand whether delays are due to system sleep or task queue congestion. Since only one task executes at a time (for memory safety), tasks may queue behind catch-up tasks after system wake.

Running as a Service (Optional):
For guaranteed execution and automatic startup, you can run Osoba backend as a system service. This ensures the backend is always running and ready to execute scheduled tasks on time. Setup scripts are provided for macOS (Launch Agent), Linux (systemd), and Windows (NSSM). See scripts/README.md for detailed setup instructions.

Wake Scheduling (Advanced):
On supported systems, you can configure the computer to wake from sleep specifically to run scheduled tasks. This requires administrator/root access and is recommended primarily for desktop machines or when plugged in due to battery impact. Platform-specific instructions:
- macOS: Uses pmset to schedule wake events
- Linux: Uses RTC wake (hardware support required)
- Windows: Uses Task Scheduler with wake timers

See scripts/README.md for complete wake scheduling setup and troubleshooting.

Requirements

For Semantic Memory & Document Knowledge Base:
- Ollama with nomic-embed-text model: ollama pull nomic-embed-text
- ChromaDB, tiktoken, and pdfplumber (automatically installed via requirements.txt)

Optional (enable additional tools):
- Smart web search: trafilatura, beautifulsoup4, lxml (automatically installed)
- Python data analysis: pandas, numpy, matplotlib, seaborn
- YouTube transcripts: youtube-transcript-api, pytube, yt-dlp, requests
- HubSpot OAuth: valid OAuth app (client ID/secret) and redirect URL
- Canva Design: Canva Connect API access token (see Setup instructions)
- Figma Design: Figma Personal Access Token (from Figma Account Settings → Security → Personal access tokens)
- Poe AI Models: Poe API key (from poe.com/api/keys, requires Poe subscription)
- Codex Workspace: Codex CLI available on PATH (or set CODEX_BIN), OpenAI API key configured

Installation

1. Clone the repository:

    git clone https://github.com/redbuilding/osoba.git

    cd osoba

    

2. Set up Backend:
Navigate to the backend directory:

        cd backend

        

        python3 -m venv mcp

        source mcp/bin/activate  # On Windows: mcp\Scripts\activate

        

        pip install -r requirements.txt

        

    

        # For encrypting API keys entered into model provider Settings modal

        SETTINGS_ENCRYPTION_KEY=<paste_generated_fernet_key_here>

        

        # For Smart Web Search (server_search.py)

        SERPER_API_KEY=your_serper_api_key_here

# Smart extraction configuration (optional)

        SMART_EXTRACT_MAX_URLS=3

        SMART_EXTRACT_MAX_CHARS_PER_URL=2000

        SMART_EXTRACT_MAX_TOTAL_CHARS=5000

        SMART_EXTRACT_REQUEST_DELAY=1.0
# For MongoDB (main.py)

        MONGODB_URI=mongodb://localhost:27017/

        MONGODB_DATABASE_NAME=mcp_chat_db
# For MySQL Database Querying (server_mysql.py)

        DB_HOST=localhost

        DB_USER=your_db_user

        DB_PASSWORD=your_db_password

        DB_NAME=your_db_name
# Optional: Canva Design Tools (backend/server_canva.py)

        # Get your token from: https://www.canva.com/developers/

        CANVA_API_TOKEN=your_canva_personal_access_token_here
# Optional: Figma Design Tools (backend/server_figma.py)

        # Get your token from Figma Account Settings → Security → Personal access tokens

        FIGMA_ACCESS_TOKEN=your_figma_personal_access_token_here
# Optional: Poe AI Models (backend/server_poe.py)

        # Get your key from https://poe.com/api/keys (requires Poe subscription)

        POE_API_KEY=your_poe_api_key_here
# Optional: HubSpot OAuth (backend/auth_hubspot.py)

        HUBSPOT_CLIENT_ID=your_hubspot_client_id

        HUBSPOT_CLIENT_SECRET=your_hubspot_client_secret

        HUBSPOT_REDIRECT_URI=http://localhost:8000/auth/hubspot/oauth-callback

        FRONTEND_URL=http://localhost:5173
# Optional: YouTube transcript server (backend/server_youtube.py)

        # YTA_PROXY=https://user:pass@host:port

        # YTA_LOG_LEVEL=INFO
# Optional: Backend defaults

        DEFAULT_OLLAMA_MODEL=llama3.1

        OLLAMA_REPEAT_PENALTY=1.15

        

        # Codex MCP debugging state

        CODEX_DEBUG=false

        

Install optional dependencies for additional tools (if you plan to use them):

        # Smart web search content extraction (recommended)

        pip install trafilatura beautifulsoup4 lxml

# YouTube transcript tool

        pip install youtube-transcript-api pytube yt-dlp requests
# Python analysis tool

        pip install pandas numpy matplotlib seaborn scipy

        

3. Set up Frontend:
Navigate to the frontend directory:

        cd ../frontend

        

        npm install

        # or

        # yarn install

        

        echo 'VITE_API_URL=http://localhost:8000/api' > .env

        

4. Ensure Ollama is installed and a model is available:
The application might default to a specific model (e.g., llama3.1). Pull your desired model:

    ollama pull llama3.1

    # or your preferred model like llama3, mistral, etc.

    

Security Considerations

> ⚠️ Osoba is designed for local, single-user use. The API has no authentication layer. Never expose port 8000 to untrusted networks (e.g., via ngrok, port forwarding, or binding to 0.0.0.0).

- Network Exposure: The backend binds to 127.0.0.1 by default. Do not change this to 0.0.0.0 unless you understand the risks.
- Encryption Key: Set SETTINGS_ENCRYPTION_KEY in backend/.env to a stable Fernet key. Without it, provider API keys are lost on every restart.
- MongoDB: The default connection uses no authentication. For sensitive data, enable MongoDB authentication and use a connection string with credentials.
- Provider API Keys: Keys are encrypted at rest with Fernet and never returned by the API. Keep your .env file secure and out of version control.
- MySQL Queries: Queries are validated as read-only SELECT statements and automatically limited to 1000 rows. Connection and read timeouts prevent runaway queries.
- CSV Uploads: Uploads are capped at 50 MB decoded size, and the in-memory DataFrame store is limited to 10 datasets (oldest evicted first).

Usage

1. Ensure Prerequisites are Running:
Ollama: Must be running.
MongoDB: Your MongoDB instance must be accessible.
* MySQL Server (if using the SQL tool): Your MySQL server must be running and accessible with the credentials provided in .env.

2. Start the Backend Server:
Navigate to the backend directory and run the FastAPI application:

    # From the backend directory

    uvicorn main:app --reload --port 8000

    

3. Start the Frontend Development Server:
Navigate to the frontend directory and run:

    npm run dev

    # or

    # yarn dev

    

Interacting with the Application

- Open your browser to the frontend URL (e.g., http://localhost:5173).
- Use the chat interface to send messages; responses stream live with indicators when tools run.
- Click the ✨ Tool Selector to enable one of: Smart Web Search, Database, YouTube, HubSpot, Python, Canva, Figma, Poe, Codex (requires OpenAI configured).
- Use Settings (header) to configure provider API keys and unlock non‑Ollama models and Codex.
- Configure User Profile: In Settings → User Profile, add your role, expertise areas, current projects, and communication preferences for personalized AI assistance.
- Set Your Goals: In Settings → Goals & Priorities, define your short-term, medium-term, and long-term goals (up to 2000 characters). The AI will use these goals to provide contextual assistance and proactive insights.
- Enable Proactive Insights: Configure the heartbeat service in Settings → Goals & Priorities → Heartbeat Settings to receive periodic AI-generated insights about your progress, blockers, and suggestions. Click the bell icon (🔔) in the header to view insights.
- Pin Conversations: Hover over conversations in the sidebar and click the pin button to include them as context for future chats.
- If a conversation lacks a summary, you’ll be prompted to generate one before pinning. The UI blocks while the summary is created, then completes the pin.
- Up to 5 conversations can be pinned. Attempts beyond the cap are blocked and the sidebar shows “X/5 pinned”.
- Summaries are generated on‑demand by your chosen model (see Settings → Summaries) and are stored for reuse; only stored summaries are used for context.
- For YouTube: paste a video URL. The transcript is fetched and saved to the conversation for follow-ups.
- For HubSpot: click “Connect HubSpot” to complete OAuth, then describe the email to create/update.
- For Canva: set CANVA_API_TOKEN in .env, then ask the AI to create a design (e.g., “Create a YouTube thumbnail called 'AI in 2025'”), list your designs, upload assets, import files, or export a design to PDF.
- For Figma: set FIGMA_ACCESS_TOKEN in .env, then ask the AI to read a Figma file (e.g., “Get the structure of Figma file ABC123”), export specific frames as images, extract the design system tokens, or list/post comments.
- For Poe: set POE_API_KEY in .env, then ask the AI to generate images (e.g., "Generate a 16:9 landscape image of a mountain sunset"), chat with frontier models, or generate video/audio.
- For Python: upload a CSV file when prompted; follow-up questions reuse the loaded DataFrame for advanced analysis including filtering, grouping, outlier detection, statistical testing, and visualization.
- Manage conversations using the sidebar (create new, select, rename, delete, pin for context).

Long‑Running Tasks (Plan & Execute)

- Open the Tasks panel (Tasks button in the header) to:
- Create a new task by entering a high‑level goal.
- Monitor progress (live SSE stream), view step outputs (tables, images, text), and Pause/Resume/Cancel.
- “Promote to Task” from any user chat message to pre‑fill the goal and link the task to the conversation.
- Copy entire task results or individual step outputs using dedicated copy buttons.
- Delete completed, failed, or canceled tasks to clean up the task list.
- The backend plans each task as structured JSON (steps, tool, parameters, success criteria), then executes steps sequentially with:
- Budgets: max wall‑time and max tool calls.
- Per‑step timeouts and capped retries with backoff.
- Output verification against success criteria.
- Completion: When the final step finishes, the task status becomes COMPLETED. On failure/timeouts/budgets exceeded, status is FAILED. A concise summary is posted back into the linked conversation.
- When an OpenAI key is configured, the planner may propose a codex.run step for scaffolding/creation goals; otherwise such steps are automatically gated off.

APIs:
- Create task: POST /api/tasks { goal, conversation_id?, dry_run? }
- List tasks: GET /api/tasks
- Task detail: GET /api/tasks/{id}
- Task stream (SSE): GET /api/tasks/{id}/stream
- Pause/Resume/Cancel: POST /api/tasks/{id}/pause|resume|cancel
- Delete task: DELETE /api/tasks/{id}
- Status: GET /api/status includes tasks.active count

LLM‑only steps (no MCP):
- The planner supports llm.generate steps that run directly via Ollama, without any MCP server. If a prompt is omitted, the step’s instruction is used. These steps still respect budgets, timeouts, and verification.

Legacy Clients (Removed)

The original chat_client.py (terminal) and chat_frontend.py (Gradio) have been removed from the repository as they are no longer compatible with the current FastAPI backend architecture. All functionality is now provided through the modern React frontend.

Python Data Analysis Tools

The Python MCP server provides comprehensive data analysis capabilities through the following tools:

Core Data Operations

Data Quality & Cleaning

Data Manipulation & Analysis

Statistical Analysis