AC to Automation Converter

- Visual Understanding: AI analyzes screenshots to find elements visually
- Robust Automation: Works even when websites change their HTML structure
- Natural Descriptions: Use "click the blue login button" instead of CSS selectors
- Human-like Interaction: Sees pages exactly like humans do

- DOM Mode: Traditional CSS selector-based (fast)
- Vision Mode: AI visual understanding (robust)
- Hybrid Mode: Smart fallback - tries DOM first, uses Vision if needed

- Live Execution Logs: Watch automation steps in real-time
- Floating Log Panel: See progress without scrolling
- Color-coded Messages: Easy to spot successes, warnings, and errors
- Detailed Timestamps: Track execution timing precisely

- Real Browser Testing: Uses ChromeDriver for actual browser interaction
- AI-Powered Generation: OpenRouter AI converts natural language to automation
- Multiple Script Formats: Generate MCP Browser, Selenium, and Playwright scripts
- Visual Feedback: Screenshots and detailed execution reports

- Login/registration flows
- E-commerce checkout processes
- Form submissions and validations
- Multi-step wizards

- Click buttons by description ("red submit button")
- Find inputs by visual context ("email field in top-right")
- Navigate by visual landmarks ("menu button with hamburger icon")
- Verify visual states ("success message appears")

- Text presence verification
- Element visibility checks
- Page state validation
- Dynamic content testing

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# macOS with Homebrew brew install chromedriver # Ubuntu/Debian sudo apt-get install chromium-chromedriver # Windows: Download from https://chromedriver.chromium.org/

- Get OpenRouter API Key(Recommended):

- Get your API key (starts withsk-or-v1-...)
- 🎉 One key works for both AI generation AND vision models!

git clone <your-repo> cd ai-ac-automation cargo build --release

- Configure Environment Variables(Recommended):

# Create .env file with your OpenRouter API key echo "OPENROUTER_API_KEY=sk-or-v1-your-actual-key-here" > .env

- Start ChromeDriver(in separate terminal):
- Open Browser: Go tohttp://localhost:3001

You havetwo optionsfor configuring your OpenRouter API key:

# Create .env file in project root echo "OPENROUTER_API_KEY=sk-or-v1-your-key-here" > .env # Start the server - API key loaded automatically! cargo run --bin automation-ui

- Secure: API key never appears in UI or logs
- Convenient: No need to enter key every time
- Universal: Works for both AI generation and vision
- Safe:.envis in.gitignore- won't be committed

- Leave.envempty or don't create it
- Enter API key directly in the web interface forms
- Works for individual sessions

💡 Pro Tip:Use Option 1 for development, Option 2 for sharing/demos!

# Required: OpenRouter API key for all AI features OPENROUTER_API_KEY=sk-or-v1-your-actual-key-here # Optional: Default models (can be changed in UI) AI_MODEL=anthropic/claude-3.5-sonnet VISION_MODEL=openai/gpt-4o # Optional: Browser settings HEADLESS=false BROWSER_WIDTH=1920 BROWSER_HEIGHT=1080 # Optional: Server port PORT=3001

- Never commit.envto version control
- Keep your API keys secure and rotate them regularly
- Use different keys for development and production
- Enter URL:https://google.com
- Test Scenario:

- Click on search box - Type "browser automation" - Press Enter - Verify results appear - Click on first result

- Execute: Check "Execute immediately" → Click "Generate & Execute"
- Enable Vision: ✅ Check "Use AI Vision Mode (Like Skyvern)"
- Configure:

- Mode: Hybrid (tries DOM first, falls back to Vision)

- API Key: Auto-loaded from.envor enter manually
- Model: GPT-4 Omni (recommended)

Website: https://example.com/login Test: - Find the email input field - Type admin@test.com - Find the password field - Type mypassword123 - Click the blue login button - Verify the dashboard appears

Test the login functionality: - User should be able to log in with valid credentials - After login, dashboard should be visible - User profile should show correct information - Logout should work properly

If you haveOPENROUTER_API_KEYin your.envfile:

- ✅No API key entry needed- works automatically!
- ✅Same keyworks for both AI generation and vision
- ✅Secure- never appears in forms or logs
- ✅Fast- instant access to all AI features

👁️ Vision Models (Real AI Vision Integration)

🧠 Text Generation Models (AI Test Creation)

- One accountfor all AI providers
- Unified billingand usage tracking
- Rate limitingacross all models
- Easy model switchingin the UI
- No separate API keysto manage

- Element Not Found: Try Vision mode for robust element detection
- Timing Issues: Add waits for dynamic content
- Website Changes: Vision mode adapts automatically
- Anti-bot Detection: Use realistic delays and human-like patterns

┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Web UI │ │ OpenRouter │ │ ChromeDriver │ │ (Axum/HTML) │◄──►│ (Unified AI API) │ │ (Browser) │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Automation API │ │ Vision Engine │ │ Browser Actions │ │ (Workflow) │◄──►│ (Screenshots) │◄──►│ (Click/Type) │ └─────────────────┘ └──────────────────┘ └─────────────────┘

- Single API endpointfor all AI models
- Environment variableconfiguration (.env)
- Automatic failoverbetween providers
- Cost optimizationthrough unified billing

- automation-ui: Web interface and server
- automation-browser: Chrome automation with Vision support
- automation-api: Core workflow and data structures
- automation-integration: Pipeline orchestration
- automation-ai: AI model integration

🚀 NEW: Enhanced OpenRouter LLM Vision Demo

Experience the latest OpenRouter integration with cutting-edge vision models:

# Set your API key export OPENROUTER_API_KEY=sk-or-v1-your-key-here # Run the comprehensive computer vision demo cargo run --example computer_vision_demo -p automation-browser # 🔥 NEW: Run the enhanced OpenRouter LLM vision demo cargo run --example openrouter_vision_demo -p automation-browser # Run the enhanced Google search demo cargo run --example enhanced_google_search -p automation-browser

🎯 Enhanced Model Support (Updated 2024):

- anthropic/claude-3-5-sonnet-20241022- Latest Claude 3.5 Sonnet
- openai/gpt-4o-2024-11-20- Latest GPT-4o
- openai/gpt-4o-mini-2024-07-18- Budget-friendly vision
- google/gemini-pro-1.5- Google's vision model
- anthropic/claude-3-5-haiku-20241022- Fast HTML analysis

- Automatic model selection for each strategy
- Model-specific prompt engineering
- Enhanced error handling with detailed messages
- Performance benchmarks and comparisons

- DOM Inspection: AI analyzes HTML (faster, cheaper)
- Coordinate-Based: AI analyzes screenshots (more robust)
- Adaptive: Tries DOM first, falls back to coordinates

let mut engine = ChromeAutomationEngine::new(false) .with_vision_mode(api_key, None) .with_vision_strategy(VisionStrategy::Adaptive); // Use convenient model shortcuts engine.set_vision_model("claude"); // → claude-3-5-sonnet-20241022 engine.set_vision_model("gpt-4o"); // → gpt-4o-2024-11-20 engine.set_vision_model("gpt-4o-mini"); // → gpt-4o-mini-2024-07-18 // Auto-optimize for strategy engine.with_optimal_model_for_strategy(&VisionStrategy::CoordinateBased);

We welcome contributions! Here's how to get started:
- Forkthe repository
- Createa feature branch (git checkout -b feature/amazing-feature)
- Add testsfor your changes
- Update documentationas needed
- Submita pull request

- Vision Model Support: Add new AI vision providers
- Browser Support: Firefox, Safari automation
- UI Enhancements: Better visual design
- Performance: Optimization and caching
- Testing: More comprehensive test coverage

🚀 Built with ❤️ using Rust, AI Vision, and Real Browser Automation

This is a web browser that enables your coding agent, such as Claude Code, to visit websites on your behalf and assist you in identifying bugs or creating UI test cases.

A server for integrating with MagicPod, an AI-powered test automation platform.

Playwriter Session Cloud is a paid remote MCP endpoint for AI agent browser MCP session. It exposes structured JSON tools, a public server card, token-based access, usage receipts, and audit

A browser-agent QA swarm with an MCP interface for testing AI-generated websites.

MCP server for the Computer-Use Agent (CUA), allowing you to run CUA through Claude Desktop or other MCP clients.

Bring agent evaluations, observability, and synthetic test set generation directly into your IDE for free with Galileo's new MCP server

Browser automation via MCP for Chrome and Firefox

Bring the full power of BrowserStack’s Test Platform to your AI tools, making testing faster and easier for every developer and tester on your team.

Official Chrome DevTools MCP server for controlling and inspecting a live Chrome browser from coding agents such as Gemini, Claude, Cursor, and Copilot.