Software research must become more reusable

Sign in<br>Subscribe

This month I gave two invited talks that as paired pieces form a kind of bracket to hold together several big things I've been thinking about and working on lately: community-based research for developer science and what it gets us (to an audience of researchers), and what to do after you've measured the impact of AI badly in your organization (to an audience of leaders).<br>The first was to the UC Irvine inaugural forum on Developer Experience, invited by Tom Zimmerman (whose work you can read here). Speakers were invited across software research in both academia and industry, and included a full panel of the authors of the SPACE framework. You can read my live posting thoughts about the event.<br>One of the salient highlights from the interstitial, dinner and panel conversation included whether or not the Activity dimension of "SPACE" (which is really a general thought piece-style framework about topics to think about in developer productivity, rather than a set of specifically-defined, testable theories) is misaligned to the AI era. I believe that the vast majority of "activity" measures are badly used in engineering organizations, and that simplistic averages of things like PR throughput are likely to be wildly misleading given the obvious over-time variance in this work.<br>Without a real sense of the causal drivers of those changes, and very careful hierarchical multisystem statistical modeling of changes and variance over time (which I have to be honest never seen happen inside of an engineering organization – although tell me if you're doing this), counting up and averaging tiny chunks of output against some arbitrary threshold seems almost designed to ensure developers will be mismeasured at the very time that this "throughput" is becoming inherently decoupled from the processes of problem-solving. The meaning of "throughput" is clearly changing. The SPACE panel led me to believe I am in a minority view on this, though, as they all gave quite a lot of endorsement and defense of "Activity." For my part, I suspect characteristics of PRs that are entirely different from quantities and rates will prove to be critical to interpreting what we're seeing: e.g., types of tasks, features of PRs that allow for review and verification or not, and all sorts of characteristics of what a PR is actually doing. Activity measures are untroubled by such concerns.<br>I have always noticed that in the software teams I work with, the C of "SPACE" seems to go critically undermeasured, yet it is in relational and collaborative/coordinative elements of software work that we most clearly see some of the emerging impacts from AI– and those are also the activities from which we will probably need to derive the features of a "good PR" relative to the concerns of the team (stay tuned, because I actually have a few measurement answers I've been testing with real teams about how to better explore those features). I was delighted to hear Dr. Margaret-Anne Storey share that same perspective that we should pay more attention to collaboration/coordination from her experiences watching this framework have such a sizable impact on what the industry believes about itself.<br>For my talk, I have been thinking about how we get to make progress on understudied parts of people's experiences. I took the opportunity to give a talk that I've been meaning to give but hadn't had the right venue for before the UC Irvine event: Reusable Research.<br>Reusable research is simply, any research project that has a life longer than a short blip of attention when it gets published. That reuse could be shaping theory, providing data someone else builds on, or changing practice. It could also be the use we get out of evidence when we're able to translate it to a new domain or apply an underlying structural insight to a new version of the problem. Reused research is also research that makes it into those strange branching telephone games of tiktok summaries and instagram infographics (yes, we live in the modern world and should respect modern information flow, for better or worse!). Aiming for reusable research doesn't mean you can't value small, bespoke, and highly specific evidence. It just means you care beyond the conventional output of the study.<br>yes, you are the people who are here for the Weird WritingNot all reasons research gets reused are good ones. But almost every time a research project earns its keep in the library of human knowledge can be traced back to decisions and design choices that happened either before the study was run, or after the study is over. This can be exhausting for researchers when it's already a herculean task to continue to refine the specialized skills and domain knowledge of research itself. So it's easy for us to think this is unfair. Nevertheless, unfair realities are important – very important in the agentic coding era – to pay attention to the decisions and choices that...