AI and The Redmonk rankings
Welcome to my blog!
My name is Brian Douglas. I'm a husband to my wife and father of two girls.<br>We live together on our farm in the hills of Donegal, Ireland. We have sheep,<br>chickens, rabbits, and some dogs and cats.
I also have a hobby, which is computer science. I have a master's degree in the subject, and<br>it's handy for paying the bills.
This blog is dedicated to that hobby and my thirst for knowledge and understanding in regard to<br>the field of computer science.
I hope you'll find some of my writing interesting. If you do, then email me, and we can have some<br>correspondence, like it's 1998.
Every year I look forward to seeing the Redmonk language rankings. They are<br>the only language rankings I pay any attention too, cough Tiobe. In recent<br>years there has been very little movement in their top 20. It features all<br>the usual candidates. Java, Javascript, Python, C# etc... the languages you<br>may possibly get a job in. My interest lies in the outliers, seeing random<br>languages fly up the chart, often to fizzle out.
This year what caught my interest were three observations the writer made on<br>the impact of AI. Or rather the lack of impact.
Source article, redmonk.com
an anomalously low second half’s worth of pull requests from GitHub
This comment came after the author mentioned the declining relevance of<br>StackOverflow, which makes up one axis on the popularity grid. Github<br>making up the other.
The author expected a decline in StackOverflow tags, but was puzzled when<br>observing a decline in pull requests. This also surprised me. I see alot of<br>"hype" online about the supposed productivity gains related to generative<br>AI in software engineering. The lack of data to reflect this suggests to me<br>that there is little substance to the hype. Which would reflect my personal<br>experience. And before you suggest that Github pull requests are not a good<br>measure of productivity. Let me stop you. The sheer amount of money spent on<br>code generation should have increased pull requests and every other metric<br>related to "lines" of code.
The author surmised that it could be bad data, or that code creation went up<br>while code committed went down. The latter would suggest to me a decrease in<br>productivity. Are programmers just generating slop to get a dopamine hit without<br>actually producing anything of value? I suspect this is often the case. But<br>it's pure speculation. The data says pull requests have declined. That is a fact.
given that coding assistants make developers’ familiarity with a language less<br>relevant and the tools’ propensity to reflect the biases inherent in their training<br>data, it would be logical to expect some meaningful change in language usage...<br>however, these have not manifested
Again, this also surprised me. When I used generative AI, I no longer do, I used<br>it with languages I was not familiar. For example generating Ada code, and trying<br>out Zig. So my usage was exactly the usage the author expected to be reflected in<br>the charts. The problem I had was that the code generated often didn't work, and<br>I didn't have the knowledge of the langauge to fix it. For me, when people write<br>about code generation with AI they often seem to assume or suggest the code generated<br>is perfect. I have not found this to be the case. It was pot luck whether it worked<br>or not.
Using AI to write a language you are not familiar with also robs you of learning<br>that language. So even if you got AI to spit out Haskell code, why would you bother.<br>You would just be left with code that might work, in a language you can't understand.<br>So maybe it's not a surprise that developers are not jumping from one language to<br>another just cause they have a bot that can vomit up syntax.
coding assistance tools, in theory, should be lowering some of the barriers to<br>entry with Rust. If that’s happening, however, it’s not observable yet.
I don't care what anyone says, Rust is super hard. I've made multiple attempts to write<br>Rust and have been put off by the compiler constantly saying NO. One might think that<br>the strictness would lend itself to LLM's being able to generate working Rust code.<br>I've not found that to be the case and it looks like the data supports that finding.
So what are the benefits of generative AI for software engineers? The real benefits.<br>The ones that show up in data. Well going off my favorite twice annual report from Redmonk.<br>The data suggests that generative AI helps programmers make less pull requests, avoid<br>Rust, and doesn't help them learn new languages. What a tool. A great waste of time and<br>money.
Hip. Hip. Hooray. For AI.