What We Know We Don't Know: Empirical Software Engineering

Skip to Content

New: Logic for Programmers is now content-complete and in the hands of the copy editor! Get the current beta for 20% off here.

Hillel Wayne

Posted on<br>Nov 04, 2022

Empirical Software Engineering is the study of what actually works in programming. Instead of trusting our instincts we collect data, run studies, and peer-review our results. This talk is all about how we empirically find the facts in software and some of the challenges we face, with a particular focus on software defects and productivity.

Talk doesn&rsquo;t seem to be online yet; in the meantime, you can see a recording of an older version of the talk here. Slides are here.

Sources

I referenced a bunch of papers in my talk. These are links so you can read them yourself:

Intro

Big Data is slower than laptops<br>Scalability! But at what COST?

Why we care

Section references<br>The German Tech Sector

The Pragmatics of TDD

TDD is dead. Long live testing.

Methods

Controlled Trials<br>Comparing syntax highlightings and their effects on code comprehension

Natural Experiments<br>Simple Testing Can Prevent Most Critical Failures

Observational Studies<br>Towards the Creation of Interspecies Digital Games: An Observational Study on Cats&rsquo; Interest in Interactive Technologies

Errata: In the talk I said it studied &ldquo;cats on keyboards&rdquo;, but the paper was actually on cats on tablets (digital projections).

Software Developers&rsquo; perspectives on productivity

Defects

Programming Language Effects<br>A Large Scale Study of Programming Languages and Code Quality in Github (original faulty study)

On the Impact of Programming Languages on Code Quality (replication study)

My 6,000 word writeup of the whole fiasco

Code Review<br>Best Kept Secrets of Peer Code Review

An Empirical Study of the Impact of Modern Code Review Practices on Software Quality

First Come First Served: The Impact of File Position on Code Review

Modern Code Review: A Case Study at Google

Human Factors<br>Crunch Makes Games Worse

STRESS…At Work

Impact of a Night of Sleep Deprivation on Novice Developers Performance

Sleep deprivation: Impact on cognitive performance

Scheduled Overtime Effect On Construction Projects

Overtime and Extended Work Shifts

Additional Sources

Recommended Reading<br>Teaching tech together

Leprechauns of Software Engineering

The Programmer&rsquo;s Brain

Free research<br>Making Software

It Will Never Work In Theory

ACM digital library

ArXiv

Newsletter<br>(Article updates & weekly+ essays)

Powered by Buttondown.

&copy; 2026 Hillel Wayne