DeepEval for TypeScript, now fully open-source | DeepEval - The LLM Evaluation Framework

🔥 DeepEval for TypeScript is now in beta. Read the announcement.<br>Search

BlogAnnouncements<br>DeepEval for TypeScript, now fully open-sourceWe're releasing TypeScript in DeepEval's Python monorepoIntroducing DeepEval 4.0 - Evaluation Harness for Vibe Coding AgentsDeepEval Got a New LookCommunity<br>LLM-as-a-Judge in 2026: Top evaluation techniques and best practicesEval harness: What it is, how to use it, and why you should careEval Driven Development: What it is, how to do it right, and real examples to learn fromBuild and Evaluate a Multi-Turn Chatbot Using DeepEvalEvaluate a RAG-Based Contract Assistant with DeepEvalHow Cognee Used DeepEval to Validate Their AI Memory Research: A Case StudyTop 5 G-Eval Metric Use Cases in DeepEvalComparisons<br>Top 5 LLM Evaluation Frameworks in 2026, ComparedAll DeepEval Alternatives, ComparedDeepEval vs ArizeDeepEval vs LangfuseDeepEval vs RagasDeepEval vs TrulensTop 5 LLM Evaluation Platforms in 2026, Compared

DeepEval for TypeScript, now fully open-source<br>DeepEval's TypeScript SDK is out in beta. Every metric, model, and tracing integration you know from Python, running as a gate in your CI/CD pipeline.<br>Copy MarkdownOpen<br>First authorJeffrey Ip

Announcements<br>Two months ago I wrote about why we put TypeScript inside DeepEval's Python monorepo instead of giving it its own repo. At the time it was still a client wrapper around Confident AI — it couldn't run a single metric.

Today it can run all of them. DeepEval for TypeScript is out in beta:

npm install -D deepeval

The headline isn't the metrics though. It's where they run: in CI/CD, as a gate on your pull requests.

Evals belong in your Typescript CI pipeline

The reason I keep pushing this is that an eval you run by hand is a nice number, and an eval that runs on every PR is a decision. That's the whole point of an eval harness — a regression in your agent should block a merge exactly like a regression in your business logic does, without anyone remembering to check.

For that to happen, evals have to live where your pipeline already looks: your test suite. In Python that's Pytest. In TypeScript it's Vitest, and the surface is a single matcher — toPass() — on top of a test file you'd recognize without ever having used DeepEval:

llm_app.test.ts

import { EvaluationDataset, Golden } from "deepeval/dataset";<br>import { AnswerRelevancyMetric } from "deepeval/metrics";<br>import { LLMTestCase } from "deepeval/test-case";<br>import { it, expect } from "vitest";<br>import "deepeval/vitest";

const dataset = new EvaluationDataset({<br>goldens: [new Golden({ input: "What is pi rounded to 2 decimal places?" })],<br>});

it.each(dataset.goldens as Golden[])(<br>"answers correctly #%$",<br>async (golden) => {<br>const testCase = new LLMTestCase({<br>input: golden.input,<br>actualOutput: await yourLlmApp(golden.input),<br>});<br>await expect(testCase).toPass([new AnswerRelevancyMetric()]);<br>);

Then run it:

npx deepeval test run llm_app.test.ts

Plain vitest works too, but you'd be leaving most of the value on the table. npx deepeval test run captures a trace per test, caches metric results so a re-run doesn't re-bill you, and gives you the same flags Python users have — --official, -i/--identifier, --max-concurrent, -c/--use-cache, --ignore-errors.

That one command is the whole integration story. It exits non-zero when a metric falls below its threshold, so any CI provider that runs a shell step already knows what to do with it:

.github/workflows/evals.yml

name: LLM App `deepeval` Tests

on:<br>pull_request:<br>branches: [main]

jobs:<br>test:<br>runs-on: ubuntu-latest<br>steps:<br>- uses: actions/checkout@v4<br>- uses: actions/setup-node@v4<br>with:<br>node-version: "20"

- run: npm ci

- name: Run evals<br>env:<br>OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}<br>run: npx deepeval test run llm_app.test.ts

Note what isn't in there: no CONFIDENT_API_KEY, no hosted service, no vendor in the critical path of your merge queue. Give it a judge key and the whole thing runs on the runner. Add the key later if you want shared reports and regression tracking across commits, and add --official on main to mark the baseline that future runs get compared against. The full walkthrough is in unit testing in CI/CD.

Every metric, open-source

This is the part I care most about, because a second language that only ships the easy half of the metric library is worse than no second language at all.

47 of DeepEval's 49 metrics are ported and open-source in TypeScript. Not a curated subset — G-Eval with log-prob weighted scoring, DAG decision graphs, the multi-turn suite, the MCP metrics, the multimodal ones, arena comparisons. The only two that haven't landed are AgentLoopDetectionMetric and ToolPermissionMetric, and they'll be in shortly.

They agree with Python because they aren't rewritten from memory. Both SDKs compile the same language-neutral prompt templates — one JSON bundle of judge prompts, shared by the two...