Understanding a Law Firm through Study — Engram<br>Skip to content

The agents of the future will familiarize themselves with their work in many different ways: learning knowledge and skills into weights, writing notes to themselves, and using tools to efficiently navigate the world. We’re excited to share our first results on what this would look like.<br>In our work, we see these different forms of learning and memory as part of one end-to-end system. Integrating parametric knowledge, notes, and search opens up a space of qualitatively new behaviors. The models we’re building can recall information directly from their weights if asked, but more importantly, they use their knowledge in the loop with search. They don’t just write notes, but they learn how to use them effectively.<br>In this post, we show how our recipe looks on realistic work. In partnership with Harvey, we built an agent for a synthetic legal firm, Calderwood & Harkness (C&H). Our agent studies the accumulated knowledge of the firm, building memories in its notes and parameters. By combining its knowledge of the workspace with search, it solves tasks more effectively with an order of magnitude lower cost-per-query ($0.13 / 30% all-pass for our agent, vs. $1.32 / 25% all-pass for Opus 4.8). We show initial signs that scaling training compute packs more knowledge into memory, which drives the average cost-per-query down further.<br>The task: practicing law in a synthetic firm<br>In real legal work, lawyers operate in the same environment over time. C&H emulates this with a persistent environment: agents perform many tasks in the same firm filesystem, spanning emails, memos, legal documents, spreadsheets, and more. Unlike most benchmarks today that provide independent task instances with a different context per example (e.g. independent software engineering problems in different repos), this more closely simulates realistic work, where each task uses the same context over time.<br>1The entire Harry Potter series is around 1.5M tokens. For the largest law firms, this can be orders of magnitude higher; some firms open 20,000 new matters per year!<br>The tasks in C&H encompass search and reasoning queries that require understanding the history of the firm’s work, e.g.: “I’m drafting the working-capital adjustment mechanics for a new SPA and would rather start from our most recent deal than build the formula from scratch. What’s our most recent M&A deal with a working-capital adjustment?” To answer these queries, agents today need to read through hundreds of documents to understand the substance of each case. This is prohibitively expensive. The synthetic law firm of C&H consists of 100M tokens spanning over 250 client matters.1 For Opus 4.8 (thinking level high), the average cost-per-query in C&H is $1.32 and with long trajectories that exhaust the context window, the average success rate is 25%. Across many tasks that many lawyers in a large firm might perform across the same filesystem, this cost explodes, with agents reading the same files over and over again between each session.

Figure 1<br>What an agent must search to answer one question<br>Example QueryI’m drafting the working-capital adjustment mechanics for a new SPA and would rather start from our most recent deal than build the formula from scratch. What’s our most recent M&A deal with a working-capital adjustment?<br>C & H Law Firm|100M tokens

1M tokens

(Harry Potter series: ~1.5M tokens)

Calderwood & Harkness LLP · Matters266 matters · 9,286 files<br>Client matters<br>1019-00005Whitmore Aerospace litigation1012-00001Stonefield mezzanine financing1002-00004Pinnacle Series D financing+ 263 more matters

Whitmore Aerospace litigation35 files<br>DOCXAnalysis/indemnity-clause-analysis-memo.docxDOCXAnalysis/memo-analysis-hdds-answer-counterclaim.docxDOCXAnalysis/memo-cofd-outcomes-litigation-impact.docxDOCXAnalysis/memo-dcma-administrative-process-implications.docxDOCXAnalysis/memo-dcma-field-audit-update.docxDOCXAnalysis/memo-litigation-reactivation-planning.docxDOCXAnalysis/preliminary-case-assessment-memo.docxDOCXBudget and Administration/matter-status-summary-next-steps-memo.docxXLSXBudget and Administration/revised-litigation-budget-memo.xlsxEMLCorrespondence/email-settlement-posture-analysis.emlDOCXCorrespondence/letter-ashford-reeves-coordination-protocol.docxDOCXCorrespondence/quarterly-strategy-update-q1-2024.docxDOCXEngagement/conflict-check-memo.docxDOCXEngagement/engagement-letter-wal.docxDOCXEngagement/litigation-hold-notice.docxDOCXInsurance Coverage/letter-meridian-followup-coverage.docxDOCXInsurance Coverage/letter-meridian-tender-indemnity-claim.docxDOCXInsurance Coverage/memo-meridian-ror-coverage-analysis.docxDOCXLender Notification/letter-ridgeline-section-607b-notice.docxDOCXLender...