BuddyBots

Always-on AI teammates with their own computer — a parity-plus build of xAI's Grok Bot, self-hosted. Phase 1 + Phase 2 of docs/parity-spec.md.

Run it

Needs Node 22.13+ (uses the built-in SQLite).

npm run setup     # installs server + web deps and Playwright's Chromium
npm run dev       # API on :8787, UI on http://localhost:5173

Production style (one port): npm run build && npm start → http://localhost:8787 Docker (recommended once bots run terminal commands you haven't reviewed): docker build -t buddybots . && docker run -p 8787:8787 -v bb-data:/data buddybots

With no keys the app runs on a built-in simulator so every flow works offline (it is scripted, not smart, and every reply is labelled). Add real keys in Settings → API keys:

• any of Anthropic / OpenAI / DeepSeek / Gemini / xAI / OpenRouter, or an Ollama URL — each bot picks its own model
• Claude subscription on this computer (optional, personal use): Settings → *Claude subscription — this computer*. Press Connect my Claude plan: BuddyBots checks for Claude Code (offers to install it with npm if missing), makes one live call to see whether the stored sign-in really works, and if not runs Claude Code's own claude auth login — Claude's sign-in page opens in your browser, you approve, and the app continues by itself, tests, and switches on. If the page shows a one-time code instead of returning by itself, paste it in the box; it is piped to the program and not stored. An expired sign-in later shows Sign in again (same flow). Then choose which bots (or all) use it. BuddyBots starts the claude -p program for each model call (built-in tools off — the bot prompt spells out that BuddyBots tools are the text-JSON protocol, not native functions, so the model cannot talk itself into "no tools are exposed"; empty working folder, your ANTHROPIC_API_KEY stripped so it cannot fall back to API billing) and never reads or stores the login. Usage comes out of your plan's limits; cost shows as $0, so per-bot cost caps do not limit it. Slower than an API key (a process per call, 2 at a time — BB_CLAUDE_PARALLEL). Do not offer this to other users of a hosted BuddyBots: Anthropic does not allow products to offer claude.ai login or plan limits without approval, and its rules for this have changed more than once — check them before relying on it. Jev still needs its own key.
• Jev — browser navigation, approval risk-classing and room routing. Works with an OpenRouter key (/api/alpha/decisions, model ~typesafe/jev-latest) or a TypeSafe key. Without either, those decisions use simple heuristics.

Try: create a bot, open its computer, and send Open http://localhost:8787/testsite/ search flights from "Dubai" to "Lahore" then click Book now on a flight (the bundled SkyBench page is a safe target; "Book now" triggers an approval).

BB_HEADED=1 npm run dev shows the real Chromium window on your desktop.

Templates

• Library: Create bot → *My templates* + 8 built-ins. Any bot can be saved as a template (Share → Save as template): profile, skills, routines (they arrive paused) and role memory; preference memories only if you tick the box.
• Sharing: a template travels as a .json file, a single pasteable share code (buddy1.…, works with no server reachable), or an opt-in public link on your server (/t/<token>.json, unguessable, revocable, exposes only that template).
• Loading: Import template → file, share code, a link to template JSON, or a Grok Bot link.
• Grok Bot templates: paste https://x.ai/bot/<id> or a directory page such as https://gtemplate.net/bots/sable-game-art. What x.ai publishes for a shared bot is its name, author, description, colour and avatar shape — the actual skills and routines are delivered only inside the Grok Bot app after sign-in, so they cannot be read. The importer maps the public fields and, with a model key, drafts role, approval boundary, starter prompts and one working skill from the description. Every drafted part is labelled, the author and source link are kept, and you review a preview before anything is saved. Treat it as "inspired by", not a copy.
• Bring over your own Grok bot, in full: Import template → *My Grok bot*. Copy the export message, send it to your bot inside Grok Bot, paste its reply. The bot itself writes out its description, skills and routines, so they arrive word for word; plain-word schedules become cron and every routine arrives paused. Works with no model key when the reply is the JSON asked for; with a key it also converts messy pasted text. Limits: only as complete as the bot's own answer, its logins and history do not come across, and anything that used a Grok-only connector needs an MCP connector or the browser here. Only for bots you made.

When a site needs you (robot checks, sign-ins, codes)

Bots never try to get past a robot check, never type passwords and never retry a blocked page. When the navigator meets one it stops the task, marks the bot "Needs you" and asks in chat:

• Robot check / captcha and verification codes → *Take over the screen*, do it yourself, *Hand back*; the bot carries on from that page. Take-over passes through clicks, typing, scrolling, press-and-hold (keep the button down without moving; it is held until you let go) and drags (your own press, movement and release, as you make them — for slider checks, maps, drag-and-drop). A site may still reject a check done in a software-controlled browser; then use *Skip this site*.
• Sign-in → *Sign in with Google Chrome*. Google (and some others) refuse sign-in inside a software-controlled browser, even when you type the password yourself in take-over. So the bot's browser is closed for a moment and the same browser program is opened normally — no automation, no remote control, nothing reads what you type — on the bots' profile. Sign in, close the window, and the bot's browser reopens already signed in. The same button is in the take-over bar.
• Robot checks that reject the bot's browser even when you click (Cloudflare Turnstile "bot behavior detected", some sliders): use *Do it in a normal Chrome window*. Same mechanism as sign-in — the bot's software-controlled browser is closed and an ordinary Chrome opens on the same profile, with nothing attached to it. On your own computer that window appears on your desktop. With the bots' desktop (Docker/Linux) it opens on the bot's screen and you drive it from the app (clicks, drags, typing are sent as plain screen input); *Done — hand back* closes it the normal way and the bot's browser reopens with the session you earned. If the site challenges the bot's browser again afterwards, the bot stops and asks again; nothing is done to hide the automation.
• While you use a normal window, other bots that need the shared browser pause ("Paused — you are using the browser") and pick up where they were when it closes; nothing to stop or restart.
• Skip this site → the bot moves on to another source and says so in its report. If nobody answers for 20 minutes it does the same.
• "Access denied" pages that no person can fix are reported and the bot uses another source.

Installed Google Chrome is used for the bots when it is found (BB_BROWSER=chromium forces the bundled Chromium, BB_CHROME_PATH points at another install); both windows must be the same program because they share one profile. The hand sign-in window needs a screen: your own computer, or the Docker desktop. With the shared profile it is refused while another bot is mid-task in that browser. Not tested against Google itself — only against the bundled sign-in test page (/testsite/login.html) and robot-check page (/testsite/captcha.html).

Your Chrome — the BuddyBots extension (best answer to robot checks)

A fresh automated browser gets challenged because it *looks* fresh: no history, no cookies, launch switches, unusual timing; solving one check by hand does not change that, so the next page challenges again. The way out is not hiding the automation (BuddyBots never does that) but not using a fresh browser at all: with the BuddyBots for Chrome extension, bots set to Browser → My Chrome work in tabs of *your* real Chrome — your logins, cookies, history and home network. Sites treat those tabs like you, and when one does show a check you solve it in your own tab and it sticks, because it is your session.

In the app, a bot set to My Chrome shows its computer panel as a mirror of its tab; *Go to <bot>’s tab* switches your Chrome to the exact window and tab the bot is using (it keeps working; the window is raised even if minimised). *Take over in my Chrome* pauses the bot and brings the tab to the front — you act in Chrome itself and press *Hand back*. Options that only make sense for the bot's own browser (normal-window sign-in, on-screen driving) are not shown in that mode, and a banner says so when the extension is not connected and the bot has fallen back to its own browser.

Setup (one minute): Settings → Your Chrome shows the steps and a pairing code. In Chrome open chrome://extensions, turn on *Developer mode*, *Load unpacked* → the chrome-extension folder of this repo; click the BuddyBots toolbar icon, paste the pairing code, *Connect*. Then in a bot's Settings tab choose My Chrome. Chat says "Browsing in your Chrome" when it is in use; the computer panel shows the tab live, and *Take over* / "Needs you" bring the tab to the front so you act in Chrome itself and press *Hand back*.

How it works: the extension opens one BuddyBots window with a "BuddyBots" tab group — one tab per bot, always inside that group, never your own tabs — and runs small page helpers (chrome-extension/content.js, the same file the server evaluates in its own Playwright browser) in them via chrome.scripting. No DevTools debugger is attached, so Chrome shows no "is debugging this browser" bar. The server does all the thinking (server/src/chrome.ts wraps a tab in the same page surface the navigator already uses, so Jev, vision, take-over, teach mode and screenshots work unchanged); clicks and typing are synthesized DOM events, screenshots come from captureVisibleTab, which is why a bot's tab is kept active in that window while it works (the window is never focused unless you press *Show the tab*). Chrome asks for "read and change all your data on all websites" at install — that is what acting in a tab needs; nothing is sent anywhere but 127.0.0.1. The pairing token lives in data/chrome-extension.token. Limits: Chrome must be open on the computer running BuddyBots (a bot falls back to its own browser and says so when it isn't); if the BuddyBots window is fully covered or minimised, Chrome may pause the page — keep it visible somewhere; press-and-hold captchas are for you to do in that tab (synthetic events cannot hold a button); on the Docker/desktop image the extension is not available. Honest expectation: far fewer challenges, not zero — aggressive sites can still challenge *you*; the difference is that your answer holds.

Updating the extension. Chrome keeps running whatever files it loaded until you reload it. After updating BuddyBots, open chrome://extensions and press the reload (↻) button on BuddyBots (or remove it and Load unpacked from the current chrome-extension folder). An old extension shows as "out of date" in Settings → Your Chrome and in the bot's computer panel, and bots use their own browser until it is reloaded; the server requires extension protocol version 1.2.0 (GET /api/chrome → minVersion).

Formatting of bot replies

Replies render as markdown: headings, lists, GitHub-style tables (| a | b | with a |---|---| line; numeric columns right-align), and charts — a fenced block ``` `chart `` holding {"type":"bar|line|pie","title":"…","labels":[…],"series":[{"name":"…","values":[…]}],"unit":"$"} renders as an SVG chart with hover values; a malformed block shows as code. Charts and code fences survive the end of streaming (the reply parser no longer strips `` inside the final text). Bots are told to use a table for tabular data and add a chart for numbers over time / by category / as shares. Messages with a table or chart take the full chat width.

Liveness and long steps

• Task cards show what is happening right now under the active step: "Model is thinking · ~2,300 tokens · 47s", "Writing the next step · 1,240 chars", "MongoDBAtlas-gmg-uae: query — db.orders… · 12s of 90s", "Reading the result (8,120 characters)". The line turns amber when a step is running past 80% of its budget or past 2 minutes of thinking.
• No timer ends a task. Model calls fail only on *inactivity*: no bytes for BB_MODEL_IDLE_S (default 120 s) on the API path, no stream events for the same period on the Claude-subscription path (which streams thinking progress too). A 6-minute answer that keeps streaming is fine.
• Tools have their own budgets (connector call 90 s, terminal 60 s, database statements 60 s server-side, browser goal 6 min); the card shows the countdown; a connector call that overruns fails with "narrow the query" instead of hanging.
• Old tool results are trimmed in the transcript after the model has used them (the last 3 stay complete; older ones keep their first 1,200 characters and a note), which keeps later steps fast — especially on the subscription path, where every call resends the whole transcript.
• Every 10 minutes on one task the bot posts one line ("Still working — 12 min in, 7 tool calls, now on …") so you know it is alive; Stop is always available.

Workspaces

The workspace switcher sits under the BuddyBots name in the sidebar. Each workspace has its own crew and rooms; switching swaps them everywhere (sidebar, home, rooms). A bot only sees teammates in its own workspace. Settings, API keys, connectors, skills and the /workspace files folder are shared across workspaces (of one organisation — see Hosted mode). Deleting a workspace deletes its bots and rooms; the first workspace cannot be deleted.

Hosted mode — accounts and organisations (BB_MULTI=1)

By default BuddyBots is a single-owner app: no accounts, one implicit organisation, everything in data/. Start it with BB_MULTI=1 and it becomes multi-tenant:

• People sign up with email + password (scrypt-hashed, 10+ characters) and get their own organisation with a first workspace. Sessions are HttpOnly cookies (bb_sid, 30 days). Invite teammates from Settings → Team: the invite is a one-time link bound to that email, with a role (admin or member; the creator is owner). Members switch between their organisations, or create another, from the menu under the BuddyBots name; Sign out is there too.
• Everything is scoped to the organisation — workspaces, bots, rooms, conversations, tasks, approvals, routines, connectors, skills, files, settings and API keys — and the live socket only carries that organisation's events. An id from another organisation answers 404, never 403, so it cannot be probed. Each organisation has its own folder under data/orgs/<orgId>/workspace (files, skills, uploads, screens) and its own browser profiles and Chrome-extension pairing code.
• Platform keys: provider keys in the server's environment (OPENAI_API_KEY, OPENROUTER_API_KEY, COMPOSIO_API_KEY, …) act as the platform's keys for every organisation and show as "provided by BuddyBots"; an organisation can add its own in Settings, which then win. Connector logins (conn:*) never fall through.
• Never offered in hosted mode: the Claude-subscription provider (a personal plan must not serve other people), whole-server snapshots/reset, and the BB_TOKEN page.
• Set BB_PUBLIC_URL=https://app.example.com so invite links, OAuth callbacks (/api/connectors/oauth/callback, /api/composio/callback) and Chrome pairing codes point at the public address. Put HTTPS in front; cookies are marked Secure when the request arrives over https.
• Not done: Postgres (SQLite with WAL is fine for one machine; see docs/deploy.md), SSO, multi-region.

Tests: cd server && npx tsx --test test/multi.test.ts boots a BB_MULTI=1 server (with fake Stripe, Resend and app APIs) and checks sign-up/in/out, invites and roles, isolation between two organisations, metering and limits, Stripe checkout and webhooks, sharing, emails, password reset, audit, export, deletion, the sandbox and the public site.

Plans, credits and billing (hosted mode)

Everything a bot consumes is metered per organisation into usage_events and priced in credits (1 credit = one US cent of model/tool cost at list price; a browser step ¼, a terminal command ½, a connector call 1, a media generation 5 — model tokens at list price, free with the organisation's own key). Plans: Free $0 (300 credits, 3 bots, cheap models), Pro $29 (3,000 credits, 10 bots, every model, overage $12/1k up to a budget you set), Team $99 (12,000 credits, 50 bots, 5 seats + $15/seat, overage $10/1k), Enterprise. When credits run out the bots pause with a clear reason (banner, Usage page, email) instead of failing silently. Stripe Checkout starts a subscription, the customer portal manages it, webhooks (/api/billing/webhook, signed) drive plan, period and status, and overage is pushed as invoice items every 6 hours. Env: STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, STRIPE_PRICE_PRO|TEAM|SEAT. Platform admins (BB_ADMIN_EMAILS) get a Platform admin screen: every organisation, plan overrides, credit grants, suspend/restore, metrics and funnel.

Onboarding, crews, built-in connectors, email, sharing

• Crews: sign-up → pick a crew (E-commerce ops, Sales follow-up, Content studio, Research desk) → the bots and their room are created, the apps they expect are listed, and the first task runs while you watch. GET /api/crews, POST /api/crews/:id/apply.
• Built by BuddyBots connectors for apps no aggregator covers: MoEngage (events, users, transactional, campaign reports), Branch (links, analytics, exports), Salesforce Commerce Cloud (SCAPI products, orders, customers, inventory, coupons) and Fluent Commerce (orders, fulfilments, events, inventory, GraphQL). Each is a small MCP server under server/connectors/<id>/ started with the credentials you type (kept in the vault, passed only as that process's environment). Tested against mocked APIs in test/fixtures/fake-apis.mjs.
• Email (Resend, RESEND_API_KEY, EMAIL_FROM): welcome, invite links, "needs your OK", bots paused, password reset, email confirmation. Throttled so a paused organisation gets one email a day.
• Share links: a finished task, a bot chat or a room transcript becomes a public read-only page at /s/<token> (server-rendered, no app script, revocable from /api/shares).
• Public site: node scripts/site.mjs builds the landing page, pricing, status and docs (from README, API.md and docs/*.md) into site/dist; hosted mode serves them at /, /pricing, /docs/…, /terms, /privacy, /status, with the app at /app.

Learning — it gets better the more you use it

Show me (teaching by demonstration). The fastest way to make a bot better is to do the thing yourself once. Press 🎓 Show me — next to any step of a finished task, on the task card (*Show me how you'd do it*), in the chat composer, or in the computer panel — and the bot pauses, brings its tab to the front in your Chrome (or on its own screen) and watches what you do: clicks, typing, navigation, across as many sites as you like. Press Done and it writes down what it understood as *procedures* — site-scoped how-tos with a title, when to use them, the steps, and the values that vary (dates, cities, counts) — and posts a review card in the chat. Nothing is bound to the task you were on: if you set flight filters on Google Flights and then send an email in Gmail, you get two procedures, each used later when a task on that site fits. You rename, drop or confirm them; from then on every bot in the team follows them (the planner sees them as knowledge, the navigator replays their steps first and Jev gets them as hints), and they are listed and editable under Learning → *What you showed it*. The understanding step uses the bot's model (or the cheapest available one); with no model key the steps are split by site without a written summary.

Show me how (teach one step). When a bot gets a single step wrong on a site (it keeps clicking a button that does nothing, picks the wrong control, misses a passenger count), open its computer panel, find that step under Activity and press 🎓 Show me how. The bot pauses, brings its tab to the front (in your Chrome, or on its own screen), and records exactly what *you* do — clicks, typing, Enter — until you press Done. That becomes a lesson for that site (visible and editable under Learning) and a replayable shortcut, and every bot in the team follows it from then on. It is the fast alternative to *Teach a task*, which records a whole workflow as a skill.

Real clicks in your Chrome. Some sites (Google Flights among them) react to hover but ignore the synthetic click events an extension can send from a page script — the button shows its tooltip and nothing happens. The extension now checks whether a click changed anything; when it did not, it repeats it as a *real* click through Chrome's input path and remembers that the site needs real clicks. Chrome shows its "BuddyBots is debugging this browser" bar while that path is attached (about 20 s after the last real click); this is Chrome's own notice, not a hidden mode. The extension needs its debugger permission for this — reload it after updating (chrome://extensions → BuddyBots → ↻, then accept the new permission).

No model is fine-tuned; the crew learns the way a good team does — by writing things down and reusing what worked. After every browser goal (server/src/learn.ts) the run is analysed and four kinds of knowledge are kept per organisation, shared by every bot in it:

• Site notes — rules distilled from what went wrong or right on a domain, each with an evidence count: deep links that fail to load, robot checks and sign-in walls, scrolling that reveals nothing, dialogs that trap navigation, controls that do nothing when clicked, vision steps that ran for a minute. They are shown to the planner before it plans ("LEARNED FROM EARLIER RUNS") and to Jev on every decision on that site (site_notes_learned_earlier). People can add, edit, disable or delete them on the Learning screen — the crew can be taught a rule in one line.
• Learned paths — a goal reached with a clean Jev-only run is saved as a replayable trace (selectors + typed text). The next time the same goal comes up on that domain it replays in seconds; Jev only steps in to repair a step whose selector no longer matches, and the normal loop then verifies the goal. A path that starts failing is retired automatically.
• Jev examples — every successful Jev action on a site becomes a few-shot hint (actions_that_worked_here_before) for later goals on that site, so Jev's accuracy climbs per domain without touching the model.
• Site stats and trend — visits, steps, Jev accuracy, vision share, load failures, wall time and replays per domain, plus "last 5 runs vs the 5 before" so improvement is measured, not felt.

Also in this change: the navigator bans scrolling after three scrolls that revealed nothing and bans a control that has been opened three times on the same page — the two loops that turned a 4-minute flight search into a 24-minute one.

GET /api/learning returns it all; notes are exported with the organisation's data and purged with it. Tested end to end: the second run of the same goal replays the learned path with fewer Jev decisions, a failed deep link produces a note, notes can be added/disabled/deleted, paths deleted.

Trust and safety (hosted mode)

Per-organisation sandbox for shell commands (bubblewrap: read-only system, only the organisation's workspace writable at /workspace, no server secrets in the environment, CPU/memory/process caps; /api/status reports sandbox), audit log (sign-ins, invites, key and plan changes, exports, deletions), data export (zip of JSON + workspace files), organisation deletion with a 7-day grace period, rate limiting (failed sign-ins per IP, API per IP), security headers (CSP, HSTS over https, frame-ancestors none), password reset and email confirmation by one-time links, vault key from BB_VAULT_KEY in production. Legal drafts in docs/terms.md, docs/privacy.md, docs/security.md; deployment in docs/deploy.md (Fly.io reference config in fly.toml); launch plan in docs/launch.md.

Connectors — MCP and databases

App catalogue (Composio). Connectors → *App catalogue*: paste a Composio API key once (free tier: 100,000 calls/month) and BuddyBots shows Composio's catalogue — Salesforce (CRM + Service Cloud objects), Shopify, Google Analytics, Jira, Notion, Google Drive, Slack, HubSpot, Tableau, Databricks, Power BI and ~1,000 more, with logos and tool counts, listed in full (most used first, connected and featured apps at the top) with search and "Show more" paging. *Connect* on an app: OAuth apps open Composio's hosted sign-in (your own account), the browser returns to http://127.0.0.1:PORT/api/composio/callback, and the app becomes an ordinary connector (transport composio) whose tools — real schemas, "important" tools first — go to the bots like any MCP server; calls run through Composio's execute endpoint. No-auth apps connect at once. One connector per app per workspace: Composio's user id is buddybots-<workspaceId>, so each business links its own accounts. Bots can do it from chat too ("connect Salesforce" → the app name is looked up in the catalogue). The sample connectors older builds seeded (GitHub/Slack/Notion/Stripe/Filesystem with placeholder tokens) are removed on start-up; the paste box, MCP servers, OAuth MCP servers and databases still work exactly as before. Set COMPOSIO_BASE_URL to point at a stand-in (the tests do). Apps Composio has no managed credentials for (Tableau PAT, Databricks token, Shopify and Power BI with your own OAuth app…) show a small form on their card with exactly the fields Composio declares (server URL, token, client id/secret…); BuddyBots creates a custom auth config from them and connects — or, for own-OAuth-app modes, opens the sign-in. Verified live against Composio on 22 Sep 2026: catalogue search, no-auth app (Hacker News) connect + tool call, Salesforce hosted sign-in link, Tableau/Shopify field discovery. Not covered: triggers/webhooks and the tool-router.

Generated media shows in the chat, live. When a bot calls a connector's generation tool (Kling, Higgsfield, any image / video / voice / music MCP), the chat gets a card at once — "Generating video with Klingai · 45%" with a progress bar (percentage when the server reports one, an indeterminate bar otherwise), the prompt, and elapsed time. Async servers (a task id first): the bot calls the status tool once and the app keeps polling it every 15 s itself (BB_POLL_S), updating the card, for up to 10 minutes — no steps or tokens are spent waiting, and the bot only sees the finished file or the failure. A run with a generation in flight also gets extra step room, so a long render cannot end in "ran out of steps". When the file arrives it is shown inline: images open in the preview, videos and audio play in place, with download links. MCP image/audio content items (base64) are saved to workspace/generated/; URLs in text or JSON results are picked up by extension or field name. The bot is told to reply in one or two lines instead of pasting URLs or JSON; bare media URLs and !… in any reply render inline too. Generation calls get a 10-minute budget (other connector calls 90 s). Bots see each connector tool's exact argument shape ({model:string, arguments?:[{name:string,value:string}]}), and a tool error comes back with that shape attached, so a wrong shape (an object where Kling wants an array of name/value pairs — the cause of "service error") is corrected on the next call instead of being reported as an outage; a corrected retry updates the same card. The honesty audit also rejects "I filed a report/ticket" claims that no tool made. Tool search. Above 40 connected tools in total (BB_TOOL_SEARCH_MIN), the bot prompt lists each connector with its tool count and first few names instead of every schema, and bots call find_tools {"query":"update a Jira ticket"} to get the matching tools with exact argument shapes (lexical ranking over names and descriptions with verb synonyms; no model call). A call to a tool name that does not exist comes back with the closest real ones. Keeps prompts small and accurate with Salesforce (185), HubSpot (262) and GitHub (874) connected together. Tool descriptions go to the bots at full length (600 chars), and when a server has an argument-free "call this first" tool (who_am_i, describe, get_instructions, capabilities, list_models…) BuddyBots calls it once at connect time and puts the answer in the catalogue as that connector's conventions, so the first real call already follows the vendor's format.

OAuth MCP servers (Kling, Notion, Linear, Atlassian, …) work like in Claude or Cursor. Paste the URL, the docs' JSON (even a fragment like "klingai": {"url": "https://kling.ai/mcp"}), or the vendor's "help me set up … https://…/mcp" sentence into Connectors → Add — or ask a bot in chat ("connect mcp https://kling.ai/mcp", the connect_mcp tool). BuddyBots follows the MCP authorization spec: protected-resource discovery, authorization-server metadata, dynamic client registration (no client id to configure), PKCE, then opens the vendor's sign-in page in your browser (also a Sign in button on the card). Approve; the vendor sends the browser back to http://127.0.0.1:PORT/api/connectors/oauth/callback, BuddyBots exchanges the code, stores the tokens in the vault and connects — refreshes happen by themselves. Verified against kling.ai's real endpoints (registration + sign-in URL) and end to end against a stand-in OAuth server in the tests. The sign-in tab must land on the computer running BuddyBots (the callback is on 127.0.0.1) — for a cloud install, do the sign-in from a browser on that box or tunnel the port.

Connectors → Add a connector is one box. Paste any of: an MCP server URL, the JSON block from a server's docs ({"mcpServers": {…}}, with command/args/env or url/headers), a database address (postgresql://, mysql://, mongodb+srv://, sqlserver://, sqlite:file.db — Supabase, Neon, pgvector, PlanetScale and Atlas all use these), or a command line. BuddyBots recognises which it is, connects, and shows the tools or the reason it failed. Passwords, tokens and env values go into the encrypted vault and are masked everywhere. Remote MCP servers try Streamable HTTP then SSE; servers that need a browser (OAuth) login are not supported yet — use a header token. Databases connect read-only (list_tables, describe_table, query; SQL runs in a read-only transaction, one statement at a time, 200 rows max); a per-connector switch adds execute, and every data change goes through the approval gate. MongoDB is read-only in this version; when the address names no database, list_tables shows every database on the cluster and the other tools take a database argument. Database cards have Try: list tables to check a connection without a bot. Connector names are kept short and dot-free (e.g. MongoDBAtlas-gmg-uae) because bots type them, and the lookup forgives case, spaces and dashes. Tested live against Postgres and SQLite; MySQL, SQL Server and MongoDB drivers are wired but untested.

The bots' computer

Same model as Grok Bot: one persistent computer that keeps working when your laptop is closed — browser with saved logins, a terminal, and the shared /workspace — and you can watch it and take over.

• In Docker / on Linux the computer is a real desktop: an X display with a window manager, a real Chromium window and a terminal window. The live view shows the whole desktop; Take over sends real mouse and keyboard input to it, so you can use anything on that screen, not only the web page. Needs xvfb openbox xdotool xterm imagemagick (the Docker image has them). BB_DESKTOP=0 turns it off.
• On macOS / Windows without Docker it falls back to a headless browser: the live view shows the page only. Use Docker to get the desktop.
• Shared vs own screen: by default all bots share one desktop and one set of logins (each bot has its own tab) — that is how Grok Bot works. Turn on *Isolated browser profile per bot* and every bot gets its own desktop and its own logins.
• Always-on: BB_TOKEN=$(openssl rand -hex 24) docker compose up -d on any small VM (2 vCPU / 4 GB is enough for a few bots). Put HTTPS in front before using it over the internet. Snapshots, recover and reset are in Settings → Computer.

What is built

Area	Status
Bots: create (8 templates, generate-from-description), edit, pin, hide, duplicate (optional memory copy), export/import template, delete	done
Per-bot model (any provider) + weekly cost cap that actually stops the bot	done
Editable memory (preferences / role / work summaries), auto work summaries	done
Chat: tasks with live steps, approvals, handoff cards, file cards, threads, reactions, attachments, /skill, @mention, redirect while working, Stop now	done
Computer: persistent Chromium profile, live screen, takeover (click/type/scroll), masked secret entry, screenshots, activity timeline	done
Jev-first navigation (shadow DOM + iframes), small-LLM typing, vision fallback incl. coordinate clicks for canvas, per-step method/cost/success logging	done — real Jev untested, see below
Approvals: Allow once / Deny / Always allow, ask-first & allow rules (ask wins), Jev risk class, audit log + CSV export	done
Rooms: bots decide who answers, claim order, value gate (holds echo/agreement), turn cap, per-room budget, warnings bypass the cap, talkativeness per bot	done
Tool calls that come back as broken JSON (almost always cut off by the output limit) are never shown as messages: the bot is told to redo the call in smaller parts (write_file has append), and a final reply that still looks like a tool call is replaced with a plain warning	done
Bot-to-bot handoff (depth-limited, deadlock-safe). The delegating bot decides whether to wait; background handoffs report back on their own — the delegator gets a new turn with the result (or it is fed into its running turn) and carries the plan forward; capped at 12 callbacks per chain. Writing-only work orders are not forced to use tools (Jev decides)	done — verified live with gpt-5: plain request → researcher → file → copywriter → summary, and a background job that reported back unprompted
Your Chrome — the BuddyBots Chrome extension: bots browse in a BuddyBots tab group of the owner's real Chrome (content-script relay, no debugger bar, no stealth)	done — tested end to end with a stand-in extension (Your Chrome test) and with the real extension loaded in Chromium 141 (window + tab group + task); UI: Settings → Your Chrome, bot Settings → Browser → My Chrome, chat/computer-panel "Show the tab in my Chrome"
App catalogue — Composio-backed: key → search → Connect → hosted sign-in → tools; per-workspace accounts; app names in the paste box and connect_mcp	done — tested end to end against a stand-in Composio API (app catalogue test); and live against Composio: catalogue, no-auth connect + execute, Salesforce link, credential-field discovery
Honest-progress audit on every final reply: Jev checks (1) does the reply claim work is underway / coming later, (2) is each done item backed by a tool call; a handed-off bot that used no tool is sent back to do the job. Up to 2 corrections per run, ~$0.00003 each. Handoff cards show the teammate's live step and a Watch link	done — verified live 20 Sep 2026 with gpt-5 + Jev: a Chief of Staff created two specialists, handed research over, the specialist's fabricated "file saved" reply was caught (file did not exist), she then browsed Amazon Best Sellers for real and saved research/pets.md; the card showed her live steps. Open dialogs / popovers (filters, date pickers, cookie walls) own the page: the navigator lists only the dialog's controls, its text comes first, and scrolling moves the list *inside* the dialog (it reports when the list is at its end) instead of the page behind. Bots are told to apply a site's filters (airlines, stops, price, category) before reading results. Checks are: claimed files must exist, "briefed X" needs a real handoff to X, a teammate the plan or reply says was created must exist (no create_bot → sent back to create it or say so), no quitting after one blocked site, save partial findings, Jev for "work is underway" claims. Task cards only tick plan steps a tool actually ran; a step skipped in the middle shows not run and the Done line says how many
Bots staff their own team: create_bot / create_room tools (new bot inherits the creator's model, cap ≤ creator's, crew limit 24, logged in Activity); side-thread replies run the full agent with tools; a reply that announces work without having called a tool is bounced back once ("act or say you haven't")	done — verified live with gpt-5
Skills as markdown files; Teach-a-task recording → deterministic replay with Jev repair	done
Routines: cron + timezone, webhook trigger, dry-run, test run, full history, auto-pause after 2 weeks away	done
MCP connectors (stdio + HTTP), tools exposed to every bot, writes gated	done
Usage: real cost per bot / task / model; Jev savings (estimate)	done
Local-only mode, isolated browser profile per bot, encrypted key vault, browser notifications	done
Light/dark, responsive, command palette, onboarding	done
look tool: bots check their own work visually through a vision model	done
Template library: save/share/import (file, share code, public link), Grok Bot template import with conversion	done
Replies stream in as they are written (all providers; rooms wait for the value gate)	done
Audio and video attachments are transcribed (OpenAI key; ffmpeg for video)	done
Email trigger: one read-only IMAP mailbox, same match rules as webhooks	done — not tested against a live mailbox
desktop tool: a vision model drives the real mouse/keyboard on the whole screen, with zoom-refined clicks	works by keyboard; small click targets are unreliable with general models
Navigator hardening: anti-loop (dead actions are withdrawn), Enter key, autocomplete hints, blank-page and robot-wall detection, deep-link guidance	done
PDF / Word / Excel / PowerPoint attachments read inline; previews for them	done
Structured results (facts / assumptions / done / pending / questions) on tasks where the bot acted	done
Computer maintenance: daily auto-snapshot, manual snapshot, recover, reset browser profiles	done
Proactive follow-ups: unanswered approvals (30 min), unfinished tasks, routines failing twice	done
Webhook routines understand Slack Events + GitHub webhooks, with a narrow match rule	done
Handoffs can carry files; delete bot can wipe its files; mark as unread; error toasts with request ID	done

Not built (Phase 3): native desktop/mobile apps, teams/admin policy, DLP/proxy, egress routing, hardware keys. Also not built: multi-user accounts (there is one shared access token), a separate screen per bot while logins are shared.

Read this before trusting it

• Verified with real keys (20 Sep 2026): Jev via OpenRouter + OpenAI (gpt-5-mini, gpt-4.1-mini). SkyBench flight search 5/5 correct, ~9 s and ~$0.001 per task end to end; 26 Jev decisions, 100% step success, median 238 ms, ~$0.00003 each. Also passed: a real site (Wikipedia search → article → facts), approval allow/deny, a canvas-only page via the vision fallback (1 vision click), a 3-bot room with the redundancy gate holding a reply, a bot-to-bot handoff, memory, PDF+XLSX reconciliation with a structured report. 16 integration tests (npm test) run in simulator mode.
• Live-site reality check (20 Sep 2026, from a cloud datacenter IP): flight search failed everywhere — Skyscanner, Kayak, Wego and Google Flights served CAPTCHA / "unusual traffic" pages, airline sites returned 403. The bot stops at a robot check, says so, and tries another source; it never tries to get around one (many of these sites forbid automation in their terms — check before pointing bots at them). Hotels worked through a Google Hotels deep link (real names, ratings, nightly prices). Driving date pickers and guest steppers on Booking-style forms was unreliable: the bot fell back to a landing page and said its prices were indicative. Expect better odds from a home IP with BB_HEADED=1 and you taking over for checks, but treat travel/e-commerce search as "works sometimes", not solved.
• Real-desktop browser, live sites (20 Sep 2026, cloud IP, gpt-5-mini + Jev): YouTube search → top 3 titles/channels/views ✔. Amazon.com search → open product → Add to cart → cart shows 1 item, $9.99 ✔ (no sign-in, no checkout). X public profile → bio, followers, latest post ✔. Kayak deep link → real DXB–LHE return fares (PIA $363, Emirates $371 as read from the page while it was still loading more) ✔. Google Hotels → list returned, but the bot asked before finalising ◐. Refused from this network: Instagram (HTTP error), Google Flights (429), Skyscanner, Hotels.com (429), Booking.com (403), amazon.ae (proxy error in the build sandbox). Anything behind a login (posting on X/Instagram, your Amazon account) needs you to sign in once through Take over — the bot never types passwords; the login then persists in its browser profile.
• Whole-screen control (desktop tool), tested 20 Sep 2026: the plumbing works — screenshot with a coordinate grid, approval gate on every action, real mouse/keyboard, self-healing if the browser window gets closed or minimised. Driven by keyboard it succeeded (Ctrl+J → Chrome's Downloads page, read correctly). Clicking a 28-pixel toolbar button failed in every attempt with gpt-5-mini and gpt-5, even with a zoom-and-refine second look: general vision models are not trained to point at pixels. It needs a computer-use-trained model (Anthropic computer use, OpenAI's computer-use model) to be dependable; neither was available to test. Jev + DOM remains the default for web pages and is far cheaper.
• Streaming + transcription, tested with real keys: a 9-second video memo was transcribed and summarised correctly ($0.0003); replies streamed into the chat (first text ~3.5 s with gpt-5-mini, which thinks before it writes).
• Website building: a bot wrote a 3-file responsive site in 47 s for $0.009 and verified its FAQ by clicking it. Asked to self-review with the new look tool (a vision model inspects a screenshot) it improved contrast and buttons for $0.07 — and broke the FAQ by changing markup without updating the script, which it partly admitted. With gpt-5-mini as the brain, review what it ships; a stronger model per bot is the fix.
• Small sample. One test site, one real site, a handful of runs. Long multi-app tasks, logged-in SaaS, and other providers (Anthropic, DeepSeek, Gemini, xAI, Ollama) are wired but untested. Treat the Analytics tab as the source of truth as you use it.
• Found and fixed during real testing: identical buttons ("Book now" ×3) are now labelled with their row text so Jev picks the right one and the approval card says which; the navigator receives your original request so constraints like "do not book" survive; a Deny ends the browsing run; vision is capped at 8 steps per run; the room gate asks "is this redundant?" (the "adds value?" phrasing let echoes through).
• Prices in server/src/models.ts are a starting table; the Jev price is a placeholder. Override both in data/models.json. Cost numbers are only as right as that table.
• Security: on localhost there is no login. If you bind to anything else set BB_TOKEN (the app then asks for it) and put HTTPS in front — the token is a single shared secret, not user accounts; for real accounts run with BB_MULTI=1 (see Hosted mode). Bots can run shell commands in data/workspace on the host (gated by risk class, but a model can be wrong) — use Docker for anything beyond experiments. Bots share one browser profile unless "Isolated browser profile per bot" is on.
• Sites with bot protection (LinkedIn, Google, banks) will often block automation; the bot pauses and asks you to take over for CAPTCHAs, passwords and 2FA — it never types secrets itself.

Layout

server/ Fastify + node:sqlite + Playwright + MCP client · web/ React + Vite + zustand · shared/types.ts the contract · API.md endpoints · design/ the design handoff · docs/parity-spec.md feature checklist.