Proof of Thought
Skip to content
Posted on
May 21, 2024
llms
•
generative-ai
•
meetings
•
management
•
reference
github
I’ve been thinking about a concept that I’ve been calling “Proof of Thought” for a while now. It’s the idea that certain processes in the modern workplace require that someone thinks about them for a while, and the best way to prove that they have done so historically is through written reports. You know the type: project plans, strategy documents, self-evaluations. The kind of documents that no one will ever read in their entirety after they’ve been written, but that still need to be well-written, relevant, and include enough plain English to make sense.
This system has been in place and worked more or less well enough for my entire career… that is until the recent rise of large language models (LLMs) and generative AI. Suddenly, for the low low cost of seconds of your time, you can have a report that passes all of those checks and definitely looks like someone put thought into it. But the thing is, they didn’t. And that’s a problem.
These Forged Proofs of thought look like the real thing. They pass all of our checks for a high quality version of these documents. But, as with other forgeries, when you go to cash them in you find they’re worthless. There’s a gap in the chain where thinking was supposed to happen and the whole thing falls apart.
Role of Documentation as “Proof of Thought”
Before I go into the implications of this idea and what we should do about it, I want to make sure I’m being super clear about what I mean when I say Proof of Thought.
Project Plans
One of the first things a project manager (PM) does when starting a new project is put together a project plan. This is a document that describes all of the things that need to happen over the course of the project and who is going to do them. In my experience, these plans are outdated basically the moment they are written and no one ever goes back and updates them. This is fine. The point of these documents is not to be a perfectly accurate representation of how the project is going, the point of these documents is that they force the project manager to think through the entire project from start to finish, think about all the ways it can go wrong, and expose any parts that are particularly fuzzy and undefined so they can start tracking down answers to those open questions.
Once it’s written, all of that work to generate it lives on in the PM’s mind. They mull things over in the shower, it keeps them up as they’re trying to fall asleep, and it becomes a part of the general world of things the PM knows. Coworkers will probably skim over it looking for their name so they know how much work they’re going to have to do, some parts of it will probably end up in Jira, and the rest will sit in that report until the end of time, a testament to a reality that only existed for a few hours before timelines shifted and stakeholders got aligned.
Self Evaluations
Self evaluations are another example of Proof of Thought. Once a year, employees are asked to write a document evaluating their own performance over the past year. The point of this document is not what they say - their manager is already aware of their accomplishments and setbacks. The point of the document is to force the employee to critically reflect on the past year, internalize the reality of their performance, and come into their performance review ready to discuss the upcoming year.
The danger of missing the point
It’s easy from the perspective of the person writing these documents to think that the point of what they’re doing is the document itself. “My manager told me to write a self-review,” they say. “Clearly they want a self-review. And the better a review I write, the happier they will be.” That’s been close enough to true until recently that the distinction hasn’t mattered. There’s been no way to write a great self-evaluation except to think really hard about the past year and write everything down in a logically coherent structure. If you answer all the questions in your project plan template, you’ve probably got a pretty good grasp of what needs to happen.
On November 30, 2022, with the release of ChatGPT, everything changed. We now have a way of generating a project plan that looks exactly like the real thing, without anyone involved in the project ever sitting down thinking about it. The thing that was supposed to be in the back of your mind isn’t there and you’re more likely to get halfway through and be surprised that no one’s talked to marketing about how we’re going to launch this thing.
Challenges Posed by LLMs
The problem with LLM-generated reports is that they can create a “Potemkin village” effect. On the surface, the reports may appear well-written, logical, and comprehensive. They look really good. All of the context cues that came with human-written content when...