Error Log Pattern Analysis is Hard (and Important)<br>An error log is meant to signal something needs fixing. Unfortunately, in many legacy systems, error logs are noisy enough that they stop being useful.
My work as a Senior Software Engineer involves maintaining, refactoring and migrating legacy services — often those that had their fair share of ownership change in the past. They are heavy-hit production systems and so I thoroughly test before shipping: unit tests, docker isolated component tests, system tests and AB Diff tests. Post production, I lean heavily on error logs to determine system health, in addition to baseline APM metrics for the application.
Common Properties of Low Quality Error Logs#
Error logs at source locations that don’t have a real problem with the application. For instance, invalid input to the API that is well handled, or a cache miss that eventually uses a golden copy of data.
Misconfigured alerts. An error log implies a human needs to act at some point. For such legacy applications you likely find both: missing alerts on genuine errors (blindsided) and alerts set up on noisy error logs (alert fatigue).
The error rate is noisy. With so much noise, it’s tedious to weed out genuine failures. The error rate is a simple SLO of maximum acceptable frequency of failed requests in a given window.
From Low to High Quality Error Logs#
Conversely, high quality error logs can help you a lot with proactive monitoring of your application. In order to fix a legacy application that has low quality error logs, we must weed out the low quality noise.
Step 1: Error log catalog#
One approach I have been using is to start with an error log catalog. I set out to build a list of all unique error logs emitted by my application along with their daily counts and 7-day volume using the DRAIN algorithm. DRAIN clusters raw log lines into templates using a fixed-depth parse tree, matching on token position and length. I faced some challenges on step one:
Semi-structured or unstructured logs contain high cardinality fields that are difficult to parse and systematically handle during aggregation to log templates.
Linked libraries log into your application’s stream. Lacking your application’s context, they flag conditions at ERROR level that aren’t real errors for you.
Since algorithms like DRAIN are sensitive to token position, detecting new error log patterns leads to false positives. To reliably tell if an existing template has drifted or a new error has been emitted is a non-trivial problem.
Why This is Important (Even in the AI-Era)#
Logs are read by machine (AI/LLMs): many agentic tools allow connecting your log vendor to debug production issues. A noisy error log stops being merely a human annoyance. It’s garbage in — garbage out for AI and you get to pay for the noisy investigations.
Logs are written by machine (AI/LLMs): agentic coding tools ship a lot of code fast, and they instrument inconsistently: over-logging, wrong levels, duplicated catch-and-log. The rate at which low-quality log sites enter a codebase is going up, not down.
Alert coverage: having stable error log patterns means you have a catalog of all the different errors occurring in your system. This not only makes alerting on each category/pattern of error straightforward, it also gives a good indication of alert coverage on different types of errors.
What’s Next#
The catalog is Step 1 of 3. Step 2 is building a harness to allow AI to fix the noise at the source in the code rather than filtering it downstream, with AI-generated replication tests that reproduce the error condition before any log site is touched. Step 3 is measuring alert coverage against the catalog of genuine errors. Those are the next two posts.