Create a LangChain Deep Agents Harness Profile for NVIDIA Nemotron 3 Ultra to Improve Performance | NVIDIA Technical Blog

Technical Blog

Subscribe

Related Resources

Agentic AI / Generative AI

Create a LangChain Deep Agents Harness Profile for NVIDIA Nemotron 3 Ultra to Improve Performance

Learn how to use harness engineering to improve agent accuracy without fine-tuning.

Jul 08, 2026

By Sean Lopp, Matthew Penn and Sukrit Rao

Like

Discuss (0)

AI-Generated Summary

Like

Dislike

Fine-tuning and harness profile engineering for agentic systems have enabled open-source models like NVIDIA Nemotron 3 Ultra to approach the accuracy of proprietary frontier models, using evaluation benchmarks and per-model customization entry points such as LangChain agent harness profiles.<br>The iterative improvement looprun evaluation, analyze failures, propose harness profile changes (e.g., prompt modifications, middleware insertion like ReadFileContinuationNoticeMiddleware), verify fixes, and re-run the full suiteminimizes regressions and overfitting while mechanizing agent optimization.<br>Automation via agentic proposers, exemplified by LangSmith Engine and the ralph loop, allows self-correcting harness profile refinement by constraining agent edits, verifying repeated test passes, and ensuring generalizable rather than overfit solutions, supporting adaptation of NVIDIA Nemotron 3 Ultra across different agent frameworks.

AI-generated content may summarize information incompletely. Verify important information. Learn more

Agentic systems often face a trade-off between accuracy and cost. The highest-performing proprietary frontier models and harnesses provide top accuracy but are expensive. Fine-tuning offers one way to address this problem. Smaller or more efficient open models starting with lower accuracy are taught to perform better with specific agents. However, fine-tuning requires expertise and hardware for training and hosting custom models.

While tuning prompts for specific use cases is common practice, formalizing this process for agent harnesses—and verifying that it produces significantly better results—is more recent. Two developments are making this possible:

Evaluation benchmarks built specifically for a given harness, making it possible to verify whether a change improves performance.

Per-model customization entry points, such as LangChain’s agent harness profiles, are a first-of-their-kind example, enabling teams to adapt models to specific agent workflows.

In this tutorial, you’ll create a LangChain Deep Agents harness profile for NVIDIA Nemotron 3 Ultra that matches proprietary frontier model intelligence. All adjustments are made to the agent harness using existing NVIDIA Nemotron 3 Ultra endpoints available from NVIDIA cloud providers.

Video 1. Create and validate a LangChain Deep Agents harness profile for Nemotron 3 Ultra using manual and automated evaluation

Prerequisites

A host with Python and LangChain Deep Agents installed.

An API key for NVIDIA Nemotron 3 Ultra. For testing, build.nvidia.com provides free API access. For production, consider an endpoint from NVIDIA Cloud Providers such as Baseten, Crusoe, Fireworks, Nebius, or Together AI.

Recommended: LangSmith account for collecting agent traces

Run evaluations to create a harness profile

LangChain Deep Agents is a popular open-source agent harness. To tune the agent harness for use with a specific model, LangChain provides two important tools:

An open-source evaluation benchmark.

Agent harness profiles that provide a first-class extension point for developers to change the agent harness behavior based on the model.

The procedure for tuning a deep agent for NVIDIA Nemotron 3 Ultra is:

Establish a baseline by running the evaluation benchmark using a deep agent and NVIDIA Nemotron 3 Ultra without a harness profile.

Analyze the failures.

Propose changes to the harness profile that address the failures.

Re-run the benchmark to verify the changes improve the evaluation without introducing regressions.

The types of changes available for the agent harness profile are:

Prompts: Change the deep agents’ base system prompt, apply a prompt suffix, or change tool descriptions. For example, adding instructions for Nemotron Ultra to prefer clarifying questions or preferring tool results over model recall.

Exclusions: Tools or middleware to remove.

Additions: Developers extend the harness with extra middleware or sub-agents, such as middleware that checks for truncated model responses or incorrect tool names.

The goal of harness engineering is to make the calls from the agent to the model more closely resemble what the model saw in the training data.

Run the evaluation

NVIDIA Nemotron 3 Ultra fails a test for the built-in read_file tool.

Failing test: tests/evals/test_file_operations.py::test_read_file_truncation_recovery_with_pagination[nvidiahub:ultra-tme]

Failure:<br>success check failed: Expected final text to contain...