An approach to vulnerability observability of Go services leveraging Prometheus

ugabuga1 pts0 comments

GitHub - el10savio/vulnemetry: Vulnerability Observability For Go Services. Leveraging Govulncheck, Prometheus and OTel [Experimental] · GitHub

/" data-turbo-transient="true" />

Skip to content

Search/

Sign in<br>Sign upAppearance settings

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

el10savio

vulnemetry

Public

Notifications<br>You must be signed in to change notification settings

Fork

Star

main

BranchesTags

Go to file

CodeOpen more actions menu

Latest commit

History<br>4 Commits<br>4 Commits

Folders and files<br>NameNameLast commit message<br>Last commit date<br>.github/workflows

.github/workflows

demo

demo

docs

docs

e2e

e2e

vulncollector

vulncollector

vulncore

vulncore

vulnmatch

vulnmatch

vulnprofiler

vulnprofiler

.dockerignore

.dockerignore

.gitignore

.gitignore

.golangci.yml

.golangci.yml

Makefile

Makefile

README.md

README.md

go.mod

go.mod

go.sum

go.sum

View all files

Repository files navigation

vulnemetry

Currently, most Go programs have vulnerabilities scanned in CI, or even have their images scanned in production. GoVulncheck allows you to find out which function you are using, coupled with which package has a vulnerability, but there is no way to find this out for a live, running system in production.

This project aims to convert vulnerability scanning from a discrete process to a continuous observability system, which allows you to scan any Go service in your cluster. By importing a library to report the code executing, and then matching it against the vulnerability database that continuously updates itself. This gives you both code that is vulnerable and the fact that it is executing right now. As set of metrics in a Prometheus native fashion, which you can then create alerting and visibility for.

NOTE [EXPERIMENTAL] This is still an experimental project aimed at showcasing the feasibility of this aspect and is not to be used in production applications yet, since there is much future work to be researched and implemented.

Running the demo via docker compose

make up

Grafana at localhost:3000 (admin/admin) showcases observability of the vulnerabilities executing based on your service and others.

Applying it to your own code

p, err := vulnprofiler.New(vulnprofiler.Config{<br>Service: "checkout",<br>Window: 10 * time.Second,<br>})<br>if err != nil {<br>log.Fatal(err)

go p.Run(ctx)

mux := http.NewServeMux()<br>mux.Handle("GET /vulnexec", p.Handler())

How the system works

Go Service Your Go service imports a library that basically runs the profiler and debug info at regular intervals of time. This is then used to determine how often frames or parts of your code are executing. It is then exposed, similar to Prometheus metrics, at different time points, showcasing that these are the modules running of this version and this is the code running which has executed this many times.

OpenTelemetry The OpenTelemetry Collector is now used to regularly pull from your set of services and push it into the vulncollector, so it handles the transport from your services to the defined collector.

Vulncollector Similar to other Prometheus collectors, vulncollector receives, for your service, the frames of the programs that have occurred. It tries to translate the profiler output into a version that is compatible with or similar to Govulncheck's database naming of functions that are vulnerable. It also keeps a copy of the vulnerability database and, at regular intervals, updates it. It then performs this join of the two, which gets you both code that is vulnerable and how many times it's been executed, which it then uses to expose Prometheus metrics

Observability Prometheus then takes in the data from the vulncollector and stores it, which is then, in the case of our demo, used by Grafana to visualize in real time:

The clusters' vulnerabilities

How many services are affected

How vulnerabilities change with releases

The vulncollector's self-observability as well

vulnprofiler/

What is exposed

The producer , on the route you mount it at. vulnexec_frame_windows_total and<br>vulnexec_frame_samples_total carry the frame labels symbol · package · module · version · observer; the two info gauges carry service · replica · boot_id · goos · goarch · cpu_profiling.

metric<br>what it is

vulnexec_frame_windows_total<br>windows in which this frame was observed executing

vulnexec_frame_samples_total<br>time-weighted execution for this frame

vulnexec_producer_info<br>who is reporting, and whether the CPU observer got the profiling singleton

vulnexec_window_seconds<br>the sampling window

The collector , on its ops surface. The first two carry the full fleet key service · replica · osv_id · symbol · package · module · version · severity · match_kind · observer which is<br>affordable only because the...

service prometheus vulncollector vulnerability observability code

Related Articles