Evaluating job search ranking with LLM judged NDCG | Corvi Careers

Skip to main content

Menu

✕Close

Corvi Careers

Explore

Resources

About us<br>Privacy<br>Terms

This content is for informational purposes only.

Resume attached

Location<br>Daly City

Search

Filters<br>All filters

All filters

Done

Titles

Companies

Distance

Any distance<br>25 miles<br>30 miles<br>50 miles<br>100 miles

Posted within

Any time<br>1 day<br>3 days<br>7 days<br>14 days<br>30 days

Job type<br>Full time<br>Part time<br>Contract<br>Internship

Level<br>Intern<br>Entry<br>Mid<br>Senior

Job categories

Job categories

Loading categories…

Selected categories<br>Clear all

No matching job categories.

Refine query with Corvi Careers

Show jobs

Cancel<br>Select locations<br>Done

Search for locations

Edit Categories<br>Done

Resume Upload

Upload resume

Only plain text .txt files are accepted.

Sent to OpenAI to extract skills, titles, and experience highlights.

Resume file is not stored on our servers.

Only extracted skills, titles, and experience highlights are saved locally in your browser and appear under Saved.

Cancel<br>Continue

Job search queries come in different shapes. Some are broad, such as product manager, software engineer, or sales development representative. Some are narrower role or skill queries, such as SQL Server DBA, biomedical engineer, or data platform engineer. Some are rare exact match queries, such as haskell, elm, or ocaml. Others combine role, skill, seniority, domain, or other constraints, such as senior backend engineer fintech python.

We wanted one scoring method that works across these cases. The question we care about is simple: for this query, did the product rank the strongest available matches near the top?

To answer that, we use LLM judged NDCG.

Evaluation case

Each evaluation case starts with a frozen eligible corpus: one query, one location filter, and one frozen snapshot of eligible jobs. The location filter defines the set of jobs that could have been shown. Once that set is fixed, the evaluation is about ranking quality.

For each case, the evaluator loads every eligible job from the frozen corpus, sends each job to an LLM judge with compact evidence from the posting, gets a 0 to 100 relevance score, sorts all judged jobs by score to create the ideal ranking under the judge, and compares the product's top results against that ideal ranking with NDCG.

Eligibility constraints, such as location and active state filters, are handled before judging. The judge scores job intent relevance. For example, with k8s, location: San Francisco, the evaluator scores the eligible corpus, identifies the strongest Kubernetes related matches under the judge, and checks whether the product ranked them near the top.

Why NDCG

Precision and recall force an extra decision: what score counts as relevant? In this setup, that means choosing a cutoff such as 70 or 90, or constructing some other target set for each query.

That loses information because the judge already gives us a graded 0 to 100 score. If one page has scores 95, 94, 93, 92, 91 and another has 75, 74, 73, 72, 71, both pages get precision@5 = 1.0 with a cutoff of 70. Those pages are clearly different.

NDCG preserves the score differences and the ordering. It uses the full graded score to construct the ideal ranking under the judge, then compares the product's ranked page against that ideal. The scoring question becomes: how much of the ideal top k score did the product capture, with earlier ranks weighted more heavily?

Why keep the 0 to 100 score

Coarse relevance buckets lose too much information for this use case. A 0 to 3 label scheme may be enough when there are only a few highly relevant results. Job search often has hundreds of strong matches. If every excellent job collapses into the same bucket, the metric cannot distinguish weaker excellent matches from stronger excellent matches.

The biggest impact range for us is often around 75 to 85. Those jobs are usually plausible matches, but not all plausible matches are equally good. If that whole range collapses into one "good" bucket, the metric loses the ability to tell whether the product put the stronger good matches ahead of the weaker ones. With linear gain NDCG over the full 0 to 100 score, that ordering still matters.

We do not treat every one point difference as exact truth. The score is a judge signal, not a physical measurement. But preserving the full score gives the evaluator resolution inside the large set of good and excellent matches.

Why linear gain

NDCG needs a gain function. A common formulation uses exponential gain, gain(rel) = 2^rel - 1. That works when relevance labels are small ordinal values like 0, 1, 2, 3.

It does not work directly with 0 to 100 scores. Exponential gain would make the top end dominate the metric unless we first bucketed or rescaled the scores, which would undo the reason we kept the 0 to 100 signal.

So we use linear gain: gain(score) = score. A score of 100 contributes twice as much raw gain as a...