Experiences with local models for coding
Experiences with local models for coding
Birgitta Böckeler
Birgitta is a Distinguished Engineer and AI-assisted delivery<br>expert at Thoughtworks. She has over 20 years of experience as a software<br>developer, architect and technical leader.
This article is part of “Exploring Gen<br>AI”. A series capturing Thoughtworks technologists' explorations of using gen ai technology for<br>software development.
08 July 2026
This is the second memo where I describe my recent experiences on running<br>small models locally on my developer machine for agentic coding. In<br>the first memo, I covered the many factors that can influence the viability of that setup — hardware, model choice, runtime, harness. Here I focus on the concrete experiences, the tasks I gave the models, what happened, and my final conclusions.
Scope
To recap: my focus is on agentic coding specifically, not just auto-complete. The machines I used were an M3 Max with 48GB RAM and an M5 Pro with 64GB RAM.
Viability funnel
Evaluating small models is quite tedious to be honest: downloading takes a while (on my non-fibre internet connection in the middle of Berlin), then configuring the new model in a harness, using it for tasks, interpreting the results, ...
I thought of it as working through a funnel of increasing viability:
Does it fit into the RAM? If I don't have enough RAM, I won't even be able to run<br>the model, period. For the general assessment of broader viability, let's assume as a<br>baseline of 48GB available RAM.
Does it run at reasonable speed? Once a model is running, the first smoke test is<br>to see how quickly it can respond to a simple request.
Can it handle tool calling? Once I saw that the request speed was bearable, I<br>gave it a simple task from a coding harness that involved reading and changing files, to<br>see if it can handle the tool calling.
Does it build functionally correct code?
Can it handle a continued conversation / more context? Once a simple task worked,<br>I continued the conversation for longer, to see how much back and forth it could take in<br>terms of length of the context.
Can it handle a larger or more complex task? If a setup survives 1-5, the next level would be to give it a more complex task to solve and see what happens.
Is the code quality acceptable? What is the balance like between speed of coding and review effort?
The journey
It has been quite a roller coaster journey, which is an observation in itself!
Phase 1 - manual evals: I started with picking a few tasks, and then doing them over and over again with different models and configurations, “manually”. I wanted to see what it feels like in terms of the user experience, which is hard to do with an automated setup.
Phase 2 - automated evals: However, after I shared those results with other colleagues in Thoughtworks, I immediately learned new things from their input and experiences. So to retry some of my tasks with different configurations, I vibe coded a small automated eval setup after all, which gave me more data.
Phase 3 - day-to-day use: After all of this, I worked with the most promising model (Qwen3.6 35B MoE) for a while on the tasks that kept popping up in my day to day, to see if I could integrate it into my workflows.
Tasks
Which task to give to the models makes a huge difference, and is one of the key challenges of evaluation. The main tasks I used in my more systematical comparison were both JavaScript/Typescript, so not a lot of tech stack diversity, keep that in mind.
In a less systematic way, I also wrote a few shell and Python scripts with it, which worked fine.
As the tasks are so important, I'm giving you some details here for two of them, to help inform your interpretation of my results. The choice of task is ultimately one of the biggest factors that determines viability of small, locally run models - it's all about expectations. It's about complexity of the task (How good is the model at reasoning?), about the number of files we estimate the agent will have to read and write (How good is the model at tool calling? How big is the context window?)
Task 1: Sort and cumulate an existing bar chart
I want to change the diagram in the frontend titled, “Messages per<br>anonymous poster”<br>- It should be “per poster”, not “anonymous”<br>- I want the bars in the bar chart to be sorted by number of messages, with highest bar<br>first on the left<br>- I want the x-axis to not show numbers (“#75”), instead I want it to show what<br>percentage of the overall messages a certain bar is at. E.g., if the first 10 bars add<br>up to 240 messages, and the overall messages in that time period are 1000, then I want<br>to see 24% at the x-axis at the 10th bar. I want this percentage to be shown every 10th<br>bar.
Needs some code search, albeit trivial (I give away the lede in the prompt, it needs to look for a particular chart title)
Needs changes to 1-2 files
The cumulation of values on the x-axis was a thing that models most frequently...