Listen on Apple | Spotify | YouTube | etc…
Hey everyone,
I’ve gotten such interesting feedback on last week’s special Halloween / third anniversary episode featuring the anonymous CTO saying spooky things about AI and coding agents. So here’s a quick solo — voice memo style — episode for you. (Also typed out below)
The feedback ranges from people saying he’s “spot on” about the “insidious” problems that AI coding agents create, while others said “he’s holding it wrong”, i.e not using AI properly.
But first… I have a quick favor to ask you: Once a year I ask you to help me improve CRAFTED. by taking an annual survey.
Well... it’s been a year! And I have big plans for the show and some new things I’m working on, so I really want to hear from you.
And for one lucky survey-taker, I will make a $100 donation to the charity of your choice.
Please please please… It will only take five minutes…
Listen on Apple | Spotify | YouTube | etc…
This voice memo episode is best listened to, but also should read well if you prefer. Here’s the full transcript:
OK, back to the feedback I’ve been getting. And if you haven’t listened to the episode, stop and go do that first so you have the context for what the anonymous CTO said about AI coding agents and why he says they’re useful, but can also often be more trouble than they’re worth.
Here’s a short note I got from a very senior security expert at one of the biggest tech companies in the world:
“Complete agreement. I go very slightly further than your CTO, but only very slightly. They’re dead on and I wish it were safe for more people to publicly say so. Actually producing code is the least of what I do.”
But most people I’ve heard from take the opposite view, saying coding agents are really useful and getting better. Here’s what one senior engineering leader texted me:
“Hey Dan! He’s wrong :) Def limits to “vibe coding” and yolo coding, but the productivity is real and increasing - on the low end, maybe a 30% boost and for the right use cases it can be a 5-10x - but it’s a skill and needs to be done thoughtfully.”
A saltier version of this comes from a from a former colleague and very senior engineer who helps startups and enterprises get the most out of AI. He said:
“Just finished listening and the only thing I can agree with this person on is that AI won’t be taking our jobs right away. I’ve been using coding agents almost entirely exclusively for writing and reviewing code since April. I barely open an IDE. I’m mostly using them to also build AI.
The fact that their first comparison was to auto-complete indicates they don’t have current experience with coding agents. Coding agents aren’t omniscient, you have to expose it to your infrastructure for it to understand it, just like you would an engineer.
I hate to use the “your holding it wrong argument” but this persons experience is in holding it wrong. They seem to expect it to be able to do _more_ than an engineer and then blame it when it doesn’t.
I use coding agents every day to understand, review and to write code. Too big, bug ridden PRs are still the humans fault that is using the coding agent not the coding agents fault. You can produce high quality, fully tested code _if_ you guide it. Does it get things wrong and go down wrong paths, yes, but show me an engineer that doesn’t do that too.”
Another former colleague weighed in and broadly agreed with this take, but also pointed out that AI companies need to do a much better job of explaining what the agents can and can’t do.
He says that marketing from Claude and the like and comments from CEO’s like Sam Altman imply that the AI is “magic productivity sauce,” not a complex tool to learn that takes practice.
I asked for an example of this disconnect between what engineers think they can do and what the coding agents actually can do and he said:
“The agents in some sense are expected to know things without context provided, given they’re trained to seem like they contain encyclopedic knowledge and the companies aren’t doing much to qualify the caveats at the marketing level.”
[For example]: “you can ask Claude about, say, docker compose and it’ll say correct stuff about that, and then you ask it to do something with that knowledge and expect it to not guess wrong, but it too often will, because it isn’t a knowledge system, it’s a good guess system, and the training data is overloaded with N different versions of docker compose with api changes, incorrect stack overflow answers, etc.
You might also tend to expect it to remember the context of your project and adapt its solution appropriately but typically you have to do real work to give it enough context at the right level to be able to expect that, even though some of these tools do index your code automatically and do their own investigation when needed etc, it just doesn’t “just work”, it generally requires active direction and supervision.
Meanwhile, a senior product leader took issue with one particular point from the anonymous CTO. Posting on LinkedIn, he said
“As I was listening to the newest “Crafted” podcast episode, the concept of “adversarial AI” came up. The interviewee kind of brushed it off saying it’s not really a thing, meaning you can’t really use AI to govern/manage other AI because they are all trained on the same things.”
So another product chimed in to agree, adding:
“You absolutely can use AI to manage other AI. It’s happening with LLM-as-Judge evals right now; that’s how you scale up your evals as a product team.”
And here’s the last one I’ll share today. It comes from a senior engineering manager at a big platform that you probably interact with everyday. I thought he’d give me some version of “the anonymous CTO is doing it wrong”, but he shared a more nuanced take, saying:
“To be honest, I agree with a lot of what this CTO said... I don’t see AGI actually “thinking” anytime soon, it’s a lot of marketing hype for sure.
That said, agentic coding is definitely effective and if there are changes generated as big as he was talking about, that feels like a misuse of AI more than evidence we can’t review everything going in. If a PR is un-reviewable by a human, the AI should be tasked with breaking things down.
I think the effectiveness of AI coding is also SUPER domain dependent. Client facing web apps have so much material out there already that these models can pull from and be effective with, while esoteric backend systems are not gonna have that same training background.
The key to me that was mentioned in the chat is that it’s all about the data. High quality data is going to be what sets apart the best models from the rest, and it is interesting how much of that data we’re in some ways giving away for free to the AI companies. Super interesting times though for sure!
---
Amen to that!
Well, I would love to keep hearing from you on this. Please keep the feedback coming... DM me or post on LinkedIn and let’s discuss!
And once again, please give me feedback on CRAFTED. I’ve just launched the show’s annual survey and it would mean a lot if you’d take five minutes to fill it out.
These surveys are actually really important to podcasters and sponsors, so thank you in advance. And also for one lucky survey-taker I will give $100 to the charity of their choice.
Thanks — and see you soon!
And if you’re at WebSummit next week… me too!… let’s hang in Lisbon!
