My VP Eng Told Me to Stop Working on Observability. Here's Why He Was Wrong. · Jevin Maltais

Jevin Maltais

Fractional VP Eng & Principal Engineer for Series A/B companies. I fix engineering teams, implement AI that works in production, and build new product lines.

Unhinged Mode

© 2026. All rights reserved.

My VP Eng Told Me to Stop Working on Observability. Here's Why He Was Wrong.

I was a Staff Engineer at a small, quickly growing start-up. Our new VP eng pulled me aside:

it seems like you care a lot about developer experience and observability, but I need you to invest more in the feature work.

I understood where he was coming from, we were in a critical time of growing the business and the features were the clear path to growing revenue. However, the unseen part that he couldn’t see that were directly related to tech debt:

we had outages every week or two

users were getting issues connecting with SSE

AI agents were failing unreliably.

Everytime some issue happened, I had to take at least an hour (or two!) to find the issue then start in on the fix. It was a grind and it wasn’t getting better. Bugs and errors are common and can take a long time to resolve. Anything we can do to significantly cut down on the loop for discovering the issue, isolating the problem, understanding impact and fixing them will at the end of the day allow for a better user experience, and allow for more time to actually build the product .

Observability in my mind is: the holistic picture we have to understand the ongoing health of the system and finding issues. The quality of the observability is a combined measure of:

How quickly a dev can find the source of an issue.

Determine how widely that issue is impacting users.

How proactive issues bubble up by severity vs being brought to us by the users.

One of the first things I do when joining a team is getting the lay of the land of the scope of our responsibility and what our observability looks like. Here is the tier list I’ve seen:

1. Nothing apart from the out of the box tools - entirely complaint driven

you’re just using the Vercel logs or kubernetes logs you’re deployed with.

you don’t have any alerting because how could you? You aren’t emitting exceptions

all your bug reports come in from your users.

2. You added Datadog with integrations (Most teams I’ve been on)

Congratulations! If technology companies exist in part to give Datadog money, you’ve made it!

Bonus

Check out hosted Signoz. Similar functionality, open source, less expensive and Open Telemetry based.

At this stage:

you have auto-instrumenting telemetry enabled so you get basic logs and exceptions pushed

you added a bunch of your integrations and their default dashboards

you added some basic alerts

The good:

you have a solid picture when an error is emitted somewhere

you have a framework for alerting and paging your team

you have a single place for all your data.

MCP can do a basic investigation on its own since the data is there. It may miss a lot since the full attribution of a log isn’t there.

The bad:

all your services are disconnected: a request from the browser is disconnected from your routes which is disconnected from your db queries which is disconnected from your redis cache which is disconnected from your job runner. Tracking down where a request went wrong for a particular user is a nightmare

your logs are a dumpster fire. You haven’t done any curation at all. Lots of duplication and stuff you really don’t need.

logs aren’t structured: it’s impossible to able to isolate errors to particular users so when you’re digging into a user, you have no idea you try to correlate based on time. Which sucks. While the search tooling is good, you can only search by free text

no scope of impact since you have duplication of errors everywhere - you don’t have any grouping of errors so you can’t tell how often certain issues are happening or user impact.

cost - dumping all your data into something like Datadog can get expensive quick, especially if you don’t get the value you were hoping for because everything is still a mess.

3. Curated Distributed traces

Someone on your team has had enough frustration debugging that they bit the bullet and they have done the work to tie together. When a user has an issue you can see the entire path:

The browser request -> nginx/vercel -> router -> business logic and jobs -> database requests

Good:

out of the box metrics on what is slow - since all traces have start and end times, you can now have an objective window into what is slow in your system at a granular level

find that error with full context - remember that SSE error I mentioned at the beginning? With traces we could see all the requests coming in but we could also see no backend services being triggered. Without connecting the two, we only saw the incoming...