How HackerRank Scores Engineers | Grandimam
← Grandimam
I have been doing my goal-setting, KPIs, and feedback assessments through LLMs. The hardest part was designing the scoring mechanism. When HackerRank open-sourced their Hiring Agent, I wanted to understand how they had solved the same problem — and what biases their rubric encodes.
Hiring Agent is an LLM-based resume scoring tool that parses a PDF, enriches it with GitHub and blog data, and returns a score. It’s straightforward to run — clone the repo, point it at a PDF:
$ git clone https://github.com/interviewstreet/hiring-agent<br>$ cd hiring-agent<br>$ python3 score.py /examples/java-engineer.pdf
The tool returns a structured score:
📊 RESUME EVALUATION RESULTS FOR: Senior Java Engineer
🎯 OVERALL SCORE: 71.0/100<br>🌐 Open Source: 12/35<br>🚀 Self Projects: 18/30<br>🏢 Production Experience: 25/25<br>💻 Technical Skills: 8/10
✅ KEY STRENGTHS:<br>1. Microservices Architecture<br>2. Event-Driven Systems<br>3. Cloud Technologies<br>4. Database Optimization<br>5. API Design
🔧 AREAS FOR IMPROVEMENT:<br>1. Lack of Open Source Contributions<br>2. Limited GitHub Activity<br>3. No Publicly Available Projects
I first provided it with a Senior Java Engineer resume. Though the profile has a near-perfect score on production experience and technical skills, the tool still evaluated the candidate at 71. I wanted to get a deeper understanding of how this tool was doing the scoring:
1. Resume Parsing
It starts in score.py where the PDF is handed off to a PDFHandler:
score.py:251-252
pdf_handler = PDFHandler()<br>resume_data = pdf_handler.extract_json_from_pdf(pdf_path)
Inside extract_json_from_pdf, the PDF is parsed using PyMuPDF and converted to Markdown:
pdf.py:54-57
resume_text = to_markdown(<br>doc,<br>pages=pages,
The Markdown content is then normalised into structured JSON - an internal schema (basics, work, education, skills, projects, awards) that represents the content of the resume. The normalisation step is important as it allows the tool to take in any resume regardless of the layout and represent it into a strict internal format. Now, parsing the resume from PDF to Markdown is not a foolproof step, in case of multi-column sections direct Markdown conversion is unreliable.
Let’s take an example of this resume:
The Markdown parser reads left-to-right, so the content is incorrectly extracted as:
Coding, Framework, Messaging, Database, Observability, Containers, Patterns, Cloud, Java, Python…
How well the LLM recovers from this depends on the model used. A more concrete solution would be to screenshot the resume and let the model parse it directly, rather than converting to Markdown first - similar to how Claude/Codex parses websites. The sections are then extracted individually using specialized prompts, each being an LLM call to the model.
2. Scoring Criteria
Almost all the intelligence is prompts, available in prompts/templates/ directory. Python is merely used in orchestration and plumbing.
"basics": "basics.jinja",<br>"work": "work.jinja",<br>"education": "education.jinja",<br>"skills": "skills.jinja",<br>"projects": "projects.jinja",<br>"awards": "awards.jinja",<br>"system_message": "system_message.jinja",<br>"github_project_selection": "github_project_selection.jinja",<br>"resume_evaluation_criteria": "resume_evaluation_criteria.jinja",<br>"resume_evaluation_system_message": "resume_evaluation_system_message.jinja",
resume_evaluation_system_message.jinja and resume_evaluation_criteria help evaluate the scoring. The other prompts are primarily for resume parsing capability. The scoring criteria in resume_evaluation_criteria is heavily biased towards individuals with open-source projects and contributions:
### Open Source (0-35 points)<br>**HIGH SCORES (25-35 points):**<br>- Contributions to popular open source projects (1000+ stars)<br>- Significant contributions to well-known projects<br>- Google Summer of Code (GSoC) participation<br>- Substantial community involvement
There are explicit rules that mention what projects are considered open-source and what are not.
**CRITICAL RULES:**<br>- Having personal GitHub repositories does NOT constitute open source contribution<br>- True open source contribution means contributing to OTHER people's projects<br>- When GitHub data shows all projects are 'self_project' type, open source score MUST be 10 points or less
And one that interested me most was the following rule:
**SPECIAL CONSIDERATION FOR STARTUP EXPERIENCE**: Give extra points for founder roles, co-founder positions, or early-stage engineer roles (first 10-20 employees) at startups, as these demonstrate exceptional initiative, technical leadership, and ability to build products from scratch.
Testing the Bias
The prompt says to reward founder and early-stage roles. I wanted to know: does it actually? I took the same resume and changed only the job titles. Three versions:
Senior Java Engineer
Founding Engineer - same resume, title swapped
Co-founder / CTO - same resume, title swapped again
Everything else...