Llama 2 LLM on DOS - YKM's Corner on the Web
English (en)
简体中文 (zh)
Maker, Coder, Private Pilot, Retrocomputing Enthusiast
Ever thought of running a local Large Language Model (LLM) on a vintage PC running DOS? Now you can!
Demos
This video shows DOS Llama 2 LLM client running on 2 vintage DOS machines.
Running on Thinkpad T42 (2004) and Toshiba Satellite 315CDT (1996). Their CPUs are Pentium M 735 1.7 Ghz and Pentium MMX 200Mhz respectively.
Running on FreeDOS 1.4 on Thinkpad X13 Gen 1 (2020) with Core i5-10310U 1.7Ghz.
Everything is open sourced with executable available here: https://github.com/yeokm1/dosllam2
Background
2 years ago I wrote a DOS ChatGPT client and many retrocomputing enthusiasts in the community wrote similar clients for other vintage platforms. They all had a similar issue, dependency on some remote service. What about running things locally?
Conventional wisdom states that running LLMs locally will require computers with high performance specifications especially GPUs with lots of VRAM. But is this actually true?
In 2023, famous machine learning expert Andrej Karpathy released his open-source llama2.c project which allows inferencing of Meta’s Llama 2 models with just a single C-file. Other developers then ported his project to run on many platforms including some vintage ones like Windows 98 and<br>Powerbook G4. Someone even ported it to run on a microcontroller.
This then inspired me to think, could the same thing be done for an even older platform like DOS?
Based on the demo video and screenshots, it is obvious I managed to do this so this blog post will show how I did it.
Technical details
The original code
llama2.c is written in a single C-file designed to inference only fp32 models of the Llama-2 architecture.
This repo still cares about efficiency, but not at the cost of simplicity, readability or portability.<br>https://github.com/karpathy/llama2.c
For ease of testing of smaller models, Karpathy trained several small models on the TinyStories dataset that only have sizes of 260K, 15M, 42M and 110M. This is to enable some basic level of LLM functionality on resource-constraint systems.
llama2.c although written for portability in mind, still has some challenges when it comes to making the codebase work for vintage systems.
Compilation and DOS Extender
The compiler I selected is Open Watcom v2 (OWC). I used this as I was familiar with it having used it for my DOS ChatGPT client as well.
OWC provides a GUI IDE but I prefer to use a Makefile to keep the project relatively lightweight and easier to manage.
This is the OWC Makefile I used to compile my project:
TARGET = dosllam2<br>OBJS = dosllam2.obj<br>CFLAGS = -za99<br>LDFLAGS = SYSTEM dos32a NAME $(TARGET)
all: clean $(TARGET).exe
$(TARGET).obj: dosllam2.c<br>wcc386 $(CFLAGS) dosllam2.c
$(TARGET).exe: $(OBJS)<br>wlink $(LDFLAGS) FILE $(OBJS)
The compiler toolchain uses wcc386 and wlink. I used -za99 to enable C99 support otherwise it will revert to C90.
Special attention is given to the dos32a flag. For most DOS programs, you generally can compile for 2 typical modes:
Traditional 16-bit architecture compatible with the Intel 8088 used in the first IBM PC
32-bit DOS extender support for Intel 386 CPUs or newer
Since the llama2.c codebase is written for modern systems of at least 32-bit, I must go for the latter option. The memory requirement even for the tiniest of LLMs demands at least a 32-bit system too.
One common 32-bit DOS extender used in the DOS-era is DOS/4G. This extender allows programs to access extended memory above the 640KB conventional memory and Upper Memory Area by switching the CPU to 32-bit protected mode. One very common use case is that of games which required gobs of memory. Famous examples using extenders are Doom and Descent.
OWC supports DOS/4G as a choice of extender as well as another newer and open-source DOS/32 by Narech Koumar released in 2006.
This extender can be embedded into your program as a runtime or run separately. Either way, most DOS-extenders will show a startup banner like the above.
Porting efforts
The original llama2.c codebase still assumes a reasonably modern-featured C compiler. Although the OWC project is still being maintained, there was a bit of porting effort required for me to get the code to compile.
Floating point operations
The inferencing requires heavy use of floating point operations which OWC does not support well.
#define sqrtf(x) ((float)sqrt((double)(x)))<br>#define powf(x, y) ((float)pow((double)(x), (double)(y)))<br>#define cosf(x) ((float)cos((double)(x)))<br>#define sinf(x) ((float)sin((double)(x)))<br>#define expf(x) ((float)exp((double)(x)))
So I had to define some macros using existing double functions.
Memory map
Modern compilers support the mmap or similar functions which allow files in storage to be mapped into memory. The contents of the file are not all read into memory at once but are lazy-loaded i.e only necessary portions are read...