Building Agents without Harness-Engineering

Do not build your own agent. Host Hermes and give it tools, skills, and a system prompt. We're launching an API that makes this process easy.<br>For prismvideos.com, we shipped a media generation agent built on Vercel AI Agents SDK. Our agent understood which model to recommend to users, could generate images and videos, and could analyze videos and tell users how to recreate them. It was beautiful.<br>To my horror, days later, Higgsfield, a competitor of ours and a leader in the AI media generation space, launched an agent called Supercomputer. Supercomputer has observational memory (memory across sessions), skills, automations, a computer, and a filesystem. It would have taken us weeks to add all of these features. Supercomputer wasn't built with Vercel AI SDK, Claude Agents SDK, or OpenAI Agents SDK; it is built on Hermes, the open-source personal agent with 185k+ GitHub stars (at this time of writing).<br>I thought Hermes was a fad for nerds (like myself). But I realized if we used Hermes as a primitive for our agent, we could get session management (per-session memory and compaction), built-in tools (web search, browser, file system navigation), skills, self-learning, and automations for free. Customers could ask our agent, "every week look at our top-performing influencer video from last week and make five variations" - a true magic moment.<br>We deleted our existing agent, and we launched an EC2 instance with a Hono server. The server created a Hermes agent in a Docker container for every customer. It also acted as a reverse proxy for passing messages between our app and the Hermes gateway. Now, we communicate with every user's Hermes agent over a WebSocket connection.<br>Rather than building observational memory, skills, self-learning, automations, and a persistent filesystem, we only needed to focus on the engineering relevant to prismvideos.com. We can give the agent our system prompt, our tools for creating media and determining which models to use via MCP, our skills files (how to create UGC videos, storyboards, visual effects), and our connectors (Meta Ads Manager, Google Drive, Resend).<br>As consumer-facing agents get better - Claude, ChatGPT, Manus - customer expectations rise (for B2B software too). The Claude app has memory, so now my CEO wants it. What about self-learning? Steering? Can we add the Ralph Wiggum loop?<br>Companies are pouring billions into research and development on agent harnesses. I have no doubt that there will be a new agent harness after Hermes with a new feature everyone wants (it appears the new thing right now is Hermes' built-in learning loop). It is highly unlikely that an AI agent startup becomes wealthy by creating the best harness for a particular use case. If anything, they only expose themselves to the risk that a competitor ships a more feature-complete agent when the next harness arrives. AI agent startups are most likely to create differentiated value by integrating with their customers' proprietary data and learning their preferences.<br>The agent is the new primitive. Existing agent frameworks require developers to set up:<br>session management (in some cases)<br>tools (in some cases)<br>memory<br>self-learning<br>automations<br>persistent filesystem<br>container or sandboxed deployment<br>skills<br>MCP servers<br>But one through seven are part of any agent application.<br>By programmatically creating Hermes instances, developers get the agent and the infrastructure in a single API call:<br>POST /v1/deployments<br>Authorization: Bearer $PRISM_API_KEY<br>Content-Type: application/json<br>"customer_id": "cus_123",<br>"name": "Acme Creative Agent",<br>"runtime": "hermes",<br>"model": "anthropic/claude-sonnet-4.5",<br>"system_prompt": "You are Acme's media generation agent. Help the user plan, create, and iterate on high-performing short-form videos.",<br>"sandbox": {<br>"enabled": true,<br>"type": "docker",<br>"persistent_filesystem": true<br>},<br>"mcp_servers": [<br>"name": "prism-media",<br>"url": "https://api.prismvideos.com/mcp",<br>"tools": [<br>"search_models",<br>"get_model_schema",<br>"get_pricing",<br>"generate_image",<br>"generate_video",<br>"generate_audio"<br>],<br>"skills": [<br>"name": "ugc-video-creation",<br>"source": "file",<br>"path": ".prism/skills/ugc-video-creation/SKILL.md"<br>},<br>"name": "storyboarding",<br>"source": "inline",<br>"content": "---\nname: storyboarding\ndescription: Create shot-by-shot storyboards for short-form videos\n---\n# Storyboarding\n..."<br>},<br>"name": "social-media-visual-effects",<br>"source": "url",<br>"url": "https://example.com/skills/social-media-visual-effects/SKILL.md"<br>],<br>"secrets": {<br>"META_ADS_TOKEN": "sec_meta_ads_token",<br>"GOOGLE_DRIVE_TOKEN": "sec_google_drive_token"<br>},<br>"features": {<br>"memory": true,<br>"dreaming": true,<br>"automations": true,<br>"steering": true,<br>"filesystem_webhooks": true<br>Response:<br>"deployment_id": "dep_7xK9s2",<br>"customer_id": "cus_123",<br>"runtime": "hermes",<br>"status": "ready",<br>"model": "anthropic/claude-sonnet-4.5",<br>"thread_id": "thr_default_8a1",<br>"filesystem": {<br>"workspace_path":...