Languages and Logic: Self Hosting Coding LLMs
Sunday, August 9, 2026
Self Hosting Coding LLMs
The world is in an arms race to create data centers in 2026, to run and train the most intelligent AI. We spent close to $1 trillion on building data centers. Recently free open weight LLMs / AI models has shrunk so you can host them on your laptops and phones. I decided to try these for coding tasks to see if they work.
My local AI setup<br>I did a little research and found that I could test local LLMs with a setup consisting of three parts: LM-Studio, OpenCode and a compact open weight model.
Hardware<br>I do not have a computer with an Nvidia graphics card for testing. I used my laptops.
M2 MacBook with 16 GB RAM<br>PC with AMD Ryzen AI 7 350 w/ Radeon 860M and 32 GB RAM
LM Studio<br>LM Studio is a program that can download a lot of free LLMs models. You can use it to chat with the LLM, but it can also serve LLMs up for coding.
It is not the most efficient tool for models serving, but it is simple and versatile.
OpenCode<br>OpenCode is an open source AI agent harness that runs in a terminal or as an app. It has momentum and 195k GitHub stars.
I liked its minimal cursor based GUI. It connects to LLMs, from frontier labs like Anthropic, Google and OpenAI, but also cheaper models and local models.<br>We got used to the cheap AI, but all of a sudden AI cloud bills are growing dramatically. So I am worried about vendor lock in and rising cost<br>OpenCode has good ergonomics, similar to Claude Code. But in OpenCode you always see the number of tokens and money spent. I don't know when the LLM is doing work for cheap and when it burns tokens by reading through long logs or keeping sending a long context to the LLM. When you always see the meter running you notice waste fast.
Gemma 4 E4B open weight model from Google<br>Gemma 4 E4B is a compact LLM with 4 billion parameters using 4 bit quantisation, developed by Google DeepMind. My first impression was that it could handle coding test tasks, without use of web searches. I tested it with C++, Makefile and Python.<br>Gemma gets confused when I ask for more complicated work. I used a context window of 22k tokens due to hardware limitation, maybe what part of the reason for its limitations.<br>It is running at a good speed. It feels like using CoPilot a couple of years ago. But when you are using it in the terminal it has a contemporary feel. Gemma 4 E4B model ran fine both on the MacBook and the PC. It comes in different variations, and is around 5GB.
IP over the wire and privacy<br>Sending private information or intellectual properties over the wire has many problems. Especially now that AI easily can extract and use information. AI panic makes companies take big risks in order not to be left behind. This has turned software development into a minefield.<br>This and regulatory requirements will probably mean that local AI will gain some market share.
Local AI is not fool proof. When Microsoft announced a new AI feature called Recall in Windows 11, with screen captures every few seconds, I stopped using Windows. There was big backlash against Recall, so Microsoft had to change Recall and make it strictly opt-in.
AI race winner<br>I have no idea who will win the artificial intelligence race, but it is an epic battle, and there is never a dull day. Maybe the frontier lab can hold their lead due to better model training on bigger data centers. The hyperscalers could win on better cloud integration. ChatGPT was synonymous with AI. Now AI is a competitive market including AI startups with cheap LLMs and free open weight models.<br>For most tasks you do not need that much intelligence. I am not using the top frontier models for programming, cheaper AI models are enough.<br>Maybe the AI arms race is morphing into a race for super intelligence. Is super intelligence profitable? Could there be risks with super intelligence?
Is self hosting coding LLMs viable?<br>My first impression is that self hosting LLMs now have reached a level where they can do limited useful work. I was able to run coding LLMs tests on underpowered laptops. Your mileage may vary, but setup is easy.<br>Intrigued? Then take it for a spin.
Posted by<br>Sami Badawi
at<br>10:35 AM
Labels:<br>AI LLM selfhosting opencode lmstudio gemma
No comments:
Post a Comment
Older Post
Home
Subscribe to:<br>Post Comments (Atom)
Blog Archive
2026
(2)
August
(1)
Self Hosting Coding LLMs
January
(1)
2020
(4)
December
(1)
April
(1)
March
(1)
February
(1)
2019
(5)
October
(1)
September
(1)
April
(1)
February
(2)
2016
(1)
March
(1)
2015
(1)
September
(1)
2013
(5)
September
(1)
May
(2)
April
(1)
February
(1)
2012
(1)
March
(1)
2011
(6)
July
(2)
June
(1)
April
(2)
February
(1)
2010
(8)
December
(1)
November
(1)
October
(1)
June
(1)
April
(2)
March
(2)
2008
(11)
November
(1)
September
(1)
July
(1)
May
(1)
March
(1)
January
(6)
2007
(3)
December
(3)
About Me
Sami Badawi
Scientific...