Inside a 23-Agent AI System We Built for a Software Agency Client

Running a software agency means juggling five departments at once: sales, finance, engineering, marketing, and client success — often with a team far smaller than the workload demands. One of our clients, a software agency, came to us with exactly that problem. Instead of hiring for every gap or duct-taping together a dozen SaaS subscriptions, we built them a multi-agent operating system: 23 specialized AI agents, each responsible for one job, coordinated by a self-healing orchestration layer that runs on a standard laptop.

This is the architecture end to end — how the agents are organized, how the system recovers from failures automatically, how cost stays controlled, and how we rolled it out in production.

The core design decision: hybrid, not all-cloud

Most multi-agent frameworks assume everything runs in the cloud, or everything runs locally. We split the difference deliberately:

• Orchestration stays local. The workflow engine, the relational database, the vector memory, browser automation, and code-verification sandboxes all run on standard hardware — an Intel i7 laptop, 16GB RAM, no GPU required.
• Reasoning happens in the cloud, on open-source models. Every agent's actual "thinking" — drafting a proposal, writing code, generating a blog post — routes to hosted open-source LLMs via an OpenAI-compatible API (Together AI and OpenRouter), not a proprietary black-box API.

This split matters for two reasons. Cost: local orchestration means no server bills for the parts of the system that don't need heavy compute — SQLite, not a managed database; local Docker, not a cloud sandbox service. Control: the client's code, proposals, and CRM data never leave the machine except as a stateless text prompt to a model — nothing sits in a third party's application layer.

Model routing: two clusters, two jobs

Not every agent needs the same kind of intelligence. Work routes to two model tiers based on what the task actually demands:

• The Technical Cluster (Solutions Architect, Lead Engineer, DevSecOps QA, License Auditor, Security Auditor, Corporate Memory Manager) runs on qwen2.5-coder:32b-instruct — tuned for code generation, debugging, and system design, where syntax precision and architectural reasoning matter most.
• The Growth Cluster (Account Executive, PM comms, SEO, blog writing, client onboarding, feedback routing) runs on llama3.1:8b-instruct — smaller, faster, and dramatically cheaper for high-volume text synthesis and structured output where raw coding precision isn't the bottleneck.

Every call through this router is metered in real time against a hard per-session cost ceiling. Cross the budget, execution halts immediately and an alert fires — no runaway bill, ever.

The 23-agent matrix

The agents are organized into five departments, mirroring how the agency itself is structured:

• Inbound Growth & Lead Lifecycle — Webform Ingestion Handler, Upwork Playwright Prospector, Upwork Proposal Writer, LinkedIn Portal Intake, Lead & CRM Sync Officer.
• Finance, Legal & Corporate Memory — Account Executive, Contract & NDA Generator, Invoicing & Billing Tracker, Corporate Memory Manager.
• Engineering, QA & Security — Solutions Architect, Project Manager/Supervisor Hub, Lead Engineer, DevSecOps QA Engineer, License Auditor.
• Automated Marketing, Content & SEO — SEO Strategy & Keyword Analyst, Technical Blog Writer, Autonomous Web Content Publisher, Automated LinkedIn Publisher.
• Maintenance, Success & Revision — Technical Writer, Client Onboarding Agent, Client Feedback Router, Maintenance & Upkeep Monitor, Security & FinOps Auditor.

The orchestration layer: a cyclic state machine

Instead of a rigid pipeline, the system runs as a cyclic, checkpointed graph (LangGraph). Several loops are load-bearing to the design:

• QA self-heal loop — code that fails static analysis or tests inside the Docker sandbox routes straight back to the Lead Engineer for an automatic fix, capped at three attempts before escalating to a human.
• Revision loop — client feedback becomes new tickets and re-enters engineering directly, capped at five rounds per project so contradictory feedback can't spin indefinitely.
• Two human approval gates — the graph physically pauses, not just logs a warning, at Project Scope Approval and Architecture Sign-off, and won't proceed until a human explicitly approves.

Built to survive crashes, not just handle success

• Durable checkpointing — every state transition writes to SQLite before any external action. A crash mid-step resumes exactly where it left off on restart, no lost work, no duplicate actions.
• Browser recovery — a hung Playwright automation gets its stale process killed, its lock cleared, and a fresh authenticated session relaunched automatically.
• Docker sandbox timeouts — every QA run gets a strict 60-second wall clock, hardened container flags (no network, dropped capabilities, process limits), so a broken snippet can't do anything beyond waste its own 60 seconds.
• Heartbeat monitoring — an agent that goes silent gets cancelled and automatically revived with exponential backoff, up to three times, before it's escalated to a human.

Rolling it out: one agent at a time

We didn't flip on all 23 agents simultaneously — risk stayed proportional to trust earned. Lead intake and CRM went live first (zero external side effects, immediate pipeline visibility), then sales drafting (always human-reviewed before reaching a client), then the engineering core, then the QA safety net once the coding loop was trusted, then delivery polish, then memory and marketing once there was a real backlog worth indexing. Anything external-facing — scraping or posting under the agency's name — went live last, only after every upstream step had been reviewed by hand for weeks.

What this actually buys the client

Not "AI replaces the team" — a structured, auditable layer between "a lead exists" and "a project ships," where every drafting step is AI-assisted but every consequential action — approving scope, signing off architecture, sending a contract — stays a deliberate human decision. The system drafts everything; a person still owns every send.