Turning your logs into metrics | FlowG
Skip to main content<br>Logs tell you what happened. Metrics tell you how much, how often, and<br>is it getting worse. Most infrastructures pay for both, twice, with two<br>different agents reading the same lines.
Since FlowG v0.62.0 , they can share the work: a pipeline is now able to<br>count the records flowing through it, and expose those counters to Prometheus.
Two questions, two costs
Show me the stack trace of the request that failed at 14:32
This is a question for logs. You need the full payload, and you need it exactly<br>once.
Did the error rate of the checkout service triple in the last ten minutes?
This is a different question entirely. You do not need a single log line to<br>answer it, you need one number, sampled regularly. And yet, in most setups,<br>that question is answered by running a text search over hundreds of gigabytes,<br>every thirty seconds, on a dashboard nobody is looking at.
That is the expensive way of doing it.
The interesting part is that FlowG already has the answer. Your pipeline<br>parses every record, splits it, classifies it, decides that this one is an<br>error coming from that component, and then routes it to a stream. All that<br>classification work is done, and then thrown away.
Metric nodes let you keep a cheap, numeric shadow of it.
The metric node
A metric node does exactly one thing:<br>it increments a counter for every record that reaches it. Its only configuration<br>is a name.
It is a terminal node , like routers and forwarders: records go in, nothing<br>comes out. So you do not insert it in your flow, you branch off it. Any node<br>can fan out to a router and to one or more metric nodes, and the record is<br>processed by all of them concurrently.
Here is a pipeline that stores everything, and counts a few things on the side<br>(syslog severity 3 being error):
SourceSYSLOG
Conditionseverity == "3"
Conditiontag == "nginx"
Conditiontag == "sshd"
Streamlogs
Nameerrors_total
Namenginx_logs_total
Namesshd_logs_total
The name you pick becomes the metric name, prefixed with the node namespace:<br>errors_total is exported as node_errors_total.
note<br>Prometheus conventions still apply, and nothing enforces them for you. Suffix<br>your counters with _total, keep them in snake_case, and pick a naming<br>scheme rather than naming each node ad-hoc — here, every per-component counter<br>is called _logs_total. We will exploit that scheme in a minute.
The per-pipeline exporter
Every pipeline exposes its own Prometheus exporter:
curl -s \
-H "Authorization: Bearer ${FLOWG_TOKEN}" \
http://localhost:5080/api/v1/pipelines/ingress/metrics
# HELP pipeline_logs_total Total number of logs processed by the pipeline since startup
# TYPE pipeline_logs_total counter
pipeline_logs_total 128374
# HELP node_errors_total Number of logs measured since startup
# TYPE node_errors_total counter
node_errors_total 812
# HELP node_nginx_logs_total Number of logs measured since startup
# TYPE node_nginx_logs_total counter
node_nginx_logs_total 96110
Two things to notice.
First, pipeline_logs_total is there for free. You do not have to draw anything<br>to get the total volume of a pipeline.
Second, the endpoint is authenticated , and requires the read_pipelines<br>scope. It lives on the regular API, not on the management interface. This is<br>deliberate: the /metrics endpoint on port 9113 is about monitoring<br>FlowG itself (goroutines, heap, ingestion counters), while the per-pipeline<br>exporter is about monitoring the infrastructure FlowG collects logs for .<br>Different audiences, different blast radius, different firewall rules.
Counters are in-memory<br>They start at zero when the pipeline is built, which happens on server startup<br>and every time you save the pipeline. Prometheus detects counter resets on its<br>own, so rate() and increase() stay correct across restarts — just do not<br>build anything on the raw counter value.
Wiring the whole thing
Let's build the stack. First, a scraper identity: a role with the single scope<br>it needs, a user bound to that role, and a personal access token.
# create the role and the user through the UI or the API, then:
flowg-client token create
Store the token where Prometheus can read it, and nowhere else:
install -m 0600 /dev/null ./secrets/flowg-token
printf '%s' "${FLOWG_TOKEN}" > ./secrets/flowg-token
Then the stack itself:
services:
flowg:
image: linksociety/flowg:latest
ports:
- "5080:5080/tcp"
- "5514:5514/udp"
volumes:
- flowg-data:/data
prometheus:
image: prom/prometheus:latest
command:
- --config.file=/etc/prometheus/prometheus.yml
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./rules.yml:/etc/prometheus/rules.yml:ro
- ./secrets/flowg-token:/etc/prometheus/flowg-token:ro
ports:
- "9090:9090"
alertmanager:
image: prom/alertmanager:latest
volumes:
- ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
ports:
- "9093:9093"
grafana:
image: grafana/grafana:latest
volumes:
-...