GitHub - helasaoudi/llm-inspector: The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings. · GitHub

/" data-turbo-transient="true" />

Skip to content

Search or jump to...

Search code, repositories, users, issues, pull requests...

-->

Search

Clear

Search syntax tips

Provide feedback

--><br>We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel

Submit feedback

Saved searches

Use saved searches to filter your results more quickly

-->

Name

Query

To see all available qualifiers, see our documentation.

Cancel

Create saved search

Sign in

/;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

helasaoudi

llm-inspector

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

dev

BranchesTags

Go to file

CodeOpen more actions menu

Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit

History<br>24 Commits<br>24 Commits

.github/workflows

.github/workflows

docs

docs

src/llm_inspector

src/llm_inspector

tests

tests

.gitignore

.gitignore

LICENSE

LICENSE

README.md

README.md

pyproject.toml

pyproject.toml

View all files

Repository files navigation

LLM Inspector

The htop for LLM inference.

Measured. Not guessed.

Inspect → Understand → Optimize

LLM Inspector inspects live inference processes and shows exactly how GPU memory is being used, what model is running, how the runtime is configured, and where every reported value comes from.

Unlike traditional monitoring tools, it doesn't stop at inspection. It also analyzes the running workload and projects optimization opportunities—starting with quantization—to help you understand how different strategies would impact GPU memory before making any changes.

Example

Why?

Existing tools tell you that your GPU is using 18 GB.

LLM Inspector tells you why :

Weights 7 GB<br>KV Cache 8 GB<br>Workspace 1 GB<br>Other 2 GB

Then it tells you what would happen if you optimized:

FP8 would save ~3 GB of weights.<br>AWQ would save ~5 GB of weights.<br>Weight quantization won't fix an 8 GB KV Cache bottleneck.

That is the difference between a GPU monitor and an inference advisor.

Embedded goes deeper

Most tools stop here:

GPU<br>└── Process A<br>└── 17.3 GB

LLM Inspector goes one level deeper:

GPU<br>└── Process A<br>├── Weights<br>├── KV Cache<br>├── Workspace<br>├── Activations<br>└── Optimization Analysis (Projected)

That bridge—system observability + model internals , with an htop-style CLI—is the innovation. Deep metrics come from optional embedded attach() inside the inference process (see the install guide).

Measured vs Projected

Section<br>Kind

Process, Hardware, Model, Memory, Runtime<br>Measured from live sources (or Unavailable with a reason)

Optimization Analysis<br>Projected from measured inputs — never mutates the model

--verbose # show provenance for every field">llminspect inspect pid> --verbose # show provenance for every field

Install

Works on any NVIDIA GPU machine — laptop, workstation, cloud VM, bare-metal server, or DGX. Docker is optional.

pip install llm-inspector

For embedded deep metrics (Weights / KV / Activations) in the same Python env as the model:

pip install "llm-inspector[torch]"

From source (contributors):

git clone https://github.com/helasaoudi/llm-inspector<br>cd llm-inspector<br>python -m venv .venv && source .venv/bin/activate<br>pip install -e ".[torch]"

Try it

llminspect inspect --verbose">llminspect ps<br>llminspect inspect pid><br>llminspect inspect pid> --verbose

With Ollama on the host (no Docker):

ollama run llama3<br>llminspect inspect $(pgrep -f "ollama serve") --verbose

Supports

Ollama · vLLM · HuggingFace Transformers · FastAPI · custom PyTorch

macOS · Linux · any NVIDIA GPU server (including DGX)

Documentation

Guide<br>When to read it

Install & Integration<br>PyPI install, test on an inference service, Docker, attach(), troubleshooting

DGX notes<br>Extra detail from a DGX Spark inference-service setup

License

MIT

About

The htop for LLM inference see exactly where every GB of VRAM goes and get measured quantization savings.

Topics

inference

pytorch

pip

memory-profiler

htop

quantization

gpu-monitoring

llm

vllm

llm-inference

ollama

llminspect

llminspector

Resources

Readme

License

MIT license

Uh oh!

There was an error while loading. Please reload this page.

Activity

Stars

stars

Watchers

watching

Forks

forks

Report repository

Releases

v0.6.0

Latest

Jul 17, 2026

Packages

Uh oh!

There was an error while loading. Please reload this page.

Contributors

Uh oh!

There was an error while loading. Please reload this page.

Languages

Python<br>100.0%

You...