AI in Engineering Teams: ROI, Code Review, and Hiring

Skip to navigation<br>Skip to content

19.06.2026.

Engineering Management

CTOs Agree: Cognitive Debt Is the New Technical Debt

Ivan Brezak Brkan

Credit: CTO Craft Con

At a CTO Craft Dinner in Toronto, I sat down with engineering leaders from more than a dozen tech companies and asked where AI has actually landed. The free-for-all is over and we need to be realistic.

Our Shift CTO Craft Dinner format is built on candor, rather than slides or sponsor pitches. It’s just a group of senior engineering leaders talking about what’s actually happening in their organizations. At our Toronto dinner held on the outskirts of the CTO Craft conference, the theme was AI adoption in engineering

Within the first 5 minutes it was clear we’d spend the evening confirming one uncomfortable truth, similar to the one we heard earlier this spring in London: nobody has truly yet figured it out .

Note: We’ll try to figure it out at our Shift developer conference in September on the beautiful Croatian coast – tickets are on sale!

The free-for-all is over

Two years ago, the mandate was simple: spend on AI, no questions asked. That era is over.

What’s replaced it is a harder conversation, one several participants had clearly been having with their CFOs. The question has shifted from are you using AI to what are you getting for it?

The need for an ROI hasn’t changed. If anything, that window of just spending freely is dwindling. For larger organizations, the expectation is return within 12 months, or sooner.

The challenge, as one participant put it, is that the I in ROI is completely unmanaged. Engineering capacity used to mean headcount, something finance could model. Now it means tokens, and nobody controls how many tokens any individual engineer burns on a given day.

The CFO has no control once they’ve signed the contract over what the actual investment is going to be. And if you can’t tell me the investment, what does projected return even mean?

Several people around the table had run into the same wall: organizations that bought the tools, signed the contracts, and then realized there’s no financial model inside the company to manage what comes next . The comparison kept coming up: early cloud adoption, FinOps before FinOps existed. We’re in that same window. Costs are still a fantasy, usage is still undefined, and the metrics to measure it haven’t been invented yet.

One participant’s take: pushing for ROI too hard right now might mean measuring the wrong things entirely. The smarter move is to establish a baseline first.

I think eventually it will get to a more predictable state where we can say, approximately this many tokens leads to this much feature value. But I don’t think we can predict that yet.

The more pragmatic response, shared by more than one team: stop the free-for-all, start standardizing. This doesn’t mean we’re telling people to use AI less, but nudging from "use everything" to "use the same things, smarter."

Credit: CTO Craft Con

What you’re actually hiring for now

The hiring discussion exposed a split in how people in the room think about what an engineer actually is.

One participant drew a clean line between two different roles that often get conflated: the AI engineer who ships product , and the engineer who owns system design . Language doesn’t matter anymore: Python, Go, Rust, Node. But system design hasn’t changed. Someone still has to think about availability, budgets, and the architectural decisions that AI can’t make for you.

The new software engineer is a product leader. Someone thinking about what the product is, not just how it works. But we still need technical people who think about the design. Those are two different things.

On the interview side, the consensus leaned toward keeping technical fundamentals, but with caveats . One team hadn’t changed their process yet, still testing for systematic thinking, for the ability to break down ambiguous problems. Others were actively rethinking it. The most interesting take came from someone who’d shifted their interviews toward code review rather than coding, precisely because that’s what engineers actually do now.

I changed our interview process to focus on code review, because that’s what we’re actually doing. And implementation is now AI-assisted, however you choose to use your agents.

If your team can generate code faster than it can review it, you have a bottleneck. The constraint is human judgment , not output.

Team reactions and ‘recalibration moments’

Across the table, nobody described a team that was uniformly enthusiastic or uniformly resistant. The reality was messier.

One leader talked about late adopters at his company who finally jumped in after being gently pushed, then hit what he called "recalibration moments ": realizing that whole categories of work that used to take days now take hours, and having to rethink how their schedule works.

Another described...