Halloween Special: Why This Masked CTO Says AI Coding Agents Are “Insidious”, Overhyped, and Nowhere Near Replacing Human Engineers
Scary things about AI this CTO can't say in polite company
Enjoy this episode on: Spotify | Apple | YouTube
AI coding assistants promise to write your code, speed up your sprint, and maybe even make engineers obsolete. But what if the people building with them every day see something very different?
In this special Halloween edition of CRAFTED. — which also marks the show’s third anniversary! — a masked CTO shares what he can’t say publicly: that these tools are powerful, but insidious. In his view, coding assistants are great for auto-complete, but they can’t do what a human engineer does. He says they’re terrible at starting from scratch and will often suggest code that “works in a vacuum”, but not in context. And because AI can write so much code, so quickly, it’s hard to catch errors. In short, he sees an increase in short term velocity, at the expense of increased defects and an increasing dependency on systems that are untrustworthy.
I want to emphasize that this episode features the experience of one very experienced person. There are obviously others who disagree, who say AI coding agents are incredible, so long as they’re managed well.
However, there are also an increasing number of people questioning the sustainability of coding agents — they’re incredibly expensive to run — and also how good they are in the first place.
For example Andrej Karpathy, the guy who literally coined the phrase “vibe coding” and was early at OpenAI and Tesla, just said publicly on Dwarkesh Podcast that the path to AI agents is going to be a lot slower than people in the industry think it will be. He said coding agents are “not that good at writing code that’s never been written before” and that there is too much hype right now about where AI really is, with people in the industry, “trying to pretend like this is amazing, when it’s not.” He went on to say: “my Claude Code or Codex still feels like this elementary-grade student.”
Today’s guest agrees with Karpathy on a lot of this. Our guest has worked at startups, scale-ups, and big tech companies you’ve definitely heard of and today he’s at a very AI-forward company and using AI coding tools every day.
Below you’ll find key quotes and the full transcript of this episode, but it’s really best experienced by listening, so please do!
Also: to protect the guest’s identity, you’ll hear his words, but not his voice. An actor friend of mine recreated what he told me. I started by using AI for this, but then realized I knew a wonderful and kind human, so you’ll hear his voice. :)
Enjoy this episode on: Spotify | Apple | YouTube
Key Quotes
03:16 The myth of AI replacement: “The idea that AI can actually supplant a software engineer in their current role is basically nonsense.”
06:29 Why AI struggles without human input: “If you remove the human engineer from the equation, there’s no place to start from. The AI does not do well when you’re starting from scratch because it doesn’t have the real-world context or the continuous learning required to make that system better.”
12:21: The illusion of speed: “Coding assistants help you generate code very quickly. There’s an illusion that your velocity increases. What actually happens is you’re just shipping more bugs to production.”
13:30 More code than humans can review: “AI generates so much code that no human can keep that context in their head and review it in a meaningful way. At some point you just have to trust — but who are you trusting? You’re trusting the AI, and the AI cannot be trusted.”
14:02 AI & Junior Engineer Hiring: “The narrative that hiring trends have anything to do with AI is absurd. It’s not that AI is replacing junior engineers — it’s that companies are running lean and don’t have the bandwidth to train them.”
15:42: Where the AI Bulls and Bears Differ: “Whereas we see flawed systems that aren’t ready for primetime [...] they view this as ‘oh, that’s, that’s insignificant. They will get better almost immediately. It’s not a big deal.’ But we’ve been repeating this cycle for years at this point.”
19:50 Where AI Excels: “Where review and revise are part of the process already, that’s a really good place for generative AI because you already have a human in the loop.”
21:02: What builders need to unlearn “To the extent that people think these things are thinking or reasoning or on any path to AGI at all — they should discard that. These models don’t think. They’re very sophisticated pattern-matching machines, and that’s really it.”
Full Transcript:
Dan: Okay, anonymous CTO, what is so spooky and why can’t you reveal your face or name for us to have this conversation?
aCTO: Well thanks for having me, Dan. I appreciate it. I don’t know so much about spooky, but I think, what I’ve observed over the past year plus is that there’s definitely a gap between the AI’s capabilities and what it can do today and its trajectory and what people are selling on television, X, other social media platforms.
I’m an AI optimist. I believe in it as a tool to help supercharge productivity for a bunch of people, including software engineers, but the whole talk of AI coming to take people’s jobs or the idea that AI can actually supplant a software engineer in their current role is basically nonsense.
These models are just nowhere near good enough to do the same job, right? And the premise of it is basically that engineers spend all of their time simply generating code — that’s an output of the job that they do — but there’s much more that goes into the input in terms of thinking, problem solving, engaging with stakeholders, that sort of thing.
And so it can certainly generate a lot of code, but it cannot replace a software engineer. And I imagine we run into very similar situations with other professions as well, including, like, folks in the creative writing, video generation, that sort of thing.
DAN: And what is it about this moment that you can’t say this publicly?
aCTO: So, I’m working in a company that is very AI-forward. The company is leveraging AI to deliver a very useful product. And so we are delivering value to our customers for sure, but just due to the nature of who we are, what we’re building and how AI forward we are in terms of, our voice and, and branding, it would be a bit, um, uncouth of me to come out publicly to kind of state this opinion as frankly as I’m stating it to you here today.
DAN: Yet, it was important for you to say this, even though we’re gonna anonymize you, it’s important for you to say this out loud. Why is that?
aCTO: That’s a good question. Maybe I should ask my therapist. You know, I have to tell someone and, look, I’m in conversations, private conversations with, you know, a bunch of people behind the scenes and, you know, I’m speaking to you today, but I’m speaking for many people who are in the same position.
They’re working in AI. They’re making their, you know, livelihoods in AI, but they can’t actually say what’s really going on because, you know, frankly, the job market’s kind of in a shitter as well. Like some of us would rather be kind of working on other things. And so, you know, we just kind of keep these conversations to ourselves and hope that we just kind of ride this wave out.
Dan: Let’s stay closest to what you’re seeing with coding assistance, with how you’re using AI… What is it doing well? What is it not doing well, and where are the promises, in your opinion, so overblown?
aCTO: The thing it does well is it does generate code really fast, right? You can generate code much faster than any human can actually type it out. So if you as an engineer are leveraging these tools, these coding assistant tools, and you have an idea and you want to just sort of like get it typed out and have it work, when AI works really well, it knows kinda what you’re going for and it becomes like a very powerful auto complete. And so you start typing a function and based on the previous function you just wrote, and based on the work that you’ve been doing, it will auto complete to a very high degree of correctness, right? Like almost as if it read your mind. That’s when it works really well, and in that mode it’s great.
The problem is when you start to think about what led to that moment. There was a lot of work that needed to be done. A lot of code needed to be written by hand. Patterns needed to be established, things like that. In order to get to a place where AI can do its job really well, if you remove the human engineer from that equation, there’s no place to start from.
You’re starting from scratch, and the AI does not do well when you’re starting from scratch because. It doesn’t have the real world context and the continuous learning required to make that system, uh, better in a way that’s meaningful, right? It can continue to generate code and it’ll generate more code.
It’ll write over the previous code and it’ll make some improvements, but then it’s gonna introduce bugs on along the way. It doesn’t tie well to the overall system and it only has the codebase to look at, right? Now you’re thinking about, well, as an engineer I may be writing code, but within my memory, I know how our infrastructure is set up. I know what’s gonna work well within our infrastructure, what’s not gonna work well in our infrastructure. The AI has no clue about this. And if you’re vibe coding or just, you know, blindly letting the AI produce code for you, if you don’t pay attention to these things, you’re not gonna catch them. And these are like the more insidious problems that happen.
It’s not even the obvious bugs, it’s the stuff that just you catch as a human because you have the context. But if you don’t have the context, if you’re an AI and you can’t think, you’re just not gonna catch these things. And so this is really hard to explain in the abstract, and I think that’s why we can pretend it’s magical for so long, is because it is very hard to explain in the abstract.
But it’s not until you experience it, till you say, wait a minute, this is a cool tool, but this thing can’t do what I do. It’s not even close, right? I don’t care how good it is at generating code. It can’t actually take the place of a person thinking.
Dan: Yeah, I mean, I, I’ve seen similar versions of that, not in code, in my context, but I had an example recently where I interviewed someone. There was an analogy that AI had given me and it related to, for whatever reason, it related to Velcro. It had told me sort of very confidently that NASA had invented Velcro, which was not true. NASA had popularized Velcro. They didn’t invent it, but it was just sort of in there. And I just only ‘cause I happened to remember from like middle school, like the story of some guy who was hiking with their dog and the burrs get stuck to his dog and he’d like, let me research what is this thing? And that’s where Velcro comes from and it’s only ‘cause I had that memory that I, that I figured that out.
It was an example of like a really subtle thing that luckily I had the context and I didn’t, I didn’t, I’m also not naive enough to just read whatever AI says I should read, but I almost missed that one. Is that the kind of thing you’re talking about, where there’s some little nuance?
The irony is like you kind of have to be an expert to work with these tools, but then how do you become an expert if you work with these tools, if you’re younger?
aCTO: Yeah, so, that is a version of a failure mode of ai. We’re talking about hallucinations, basically because it’s a probability machine. That’s one failure mode and that’s pretty well known and understood. What I’m referring to though is situations where it doesn’t necessarily hallucinate or get something quote unquote wrong, like the syntax is right, it works in a vacuum, so to speak, but you just happen to have context about how the world works or about how your software system works in particular, that you know that even though this code is a working code, it will not work in the context of like, say, a deployment or the level of scale you expect, or whatever other context the AI just doesn’t have.
So it’s a bit more nuanced than that, but what you described is also a failure mode where it would just make things up.
Dan: Is this not something that you see getting better or could get better? Is this, is this a problem that you don’t think can go away in the, in the nearest future?
aCTO: Well, this is where you start to kind of hit the limits of my expertise. I’m a more traditional software engineer who’s versed in using these tools. I don’t build them, but based on my understanding and the way we train these models and the, and the way they work, they’re fundamentally just, you know, probability machines. And so they’ll always have a level of hallucination if they get to a place where they just don’t know how to, you know, complete, the sentence or the line of code or whatever. It’s hard to imagine getting to a place and, and frankly, in terms of whether or not it’s getting better, if I’m being honest, and I was having this conversation with a colleague not too long ago, they seem to be getting somewhat worse, and I think it has a lot to do with the garbage in, garbage out, right?
You know, you generate a lot of code, that’s maybe not good. You put it out there, it’s open source. The next model sucks that in, trains on that. And then it uses that training data as a source of like, this is the correct way to do this. And then it spits out that garbage yet again.
And then once you have that happen over and over and over again, the amount of high quality code or just data in general becomes diluted every single time. So on some level, the more we train these models with newer data that’s being coming out there, while AI is in the zeitgeist and people are using it, you might imagine they get worse over time.
Dan: This is the model collapse problem?
aCTO: Exactly.
Dan: On the topic of coding assistance, have you found, do they increase your team’s long-term productivity or do they just increase today’s coding output at the cost of some big, expensive refactoring that you’re gonna have to do down the road?
aCTO: Certainly the latter. I think that’s one of the most misunderstood things about coding assistants, because they are helping you generate code very quickly. There’s an illusion that your velocity increases, right? What this actually ends up doing is, you know, each engineer can generate code much more quickly, but at the very same time, it just means that your shipping more bugs to production, right?
So what we see is an increase of code being shipped — yes, 100%, can’t deny that — but a big increase in defects that are coming back the other side. And it’s kind of drowning teams in bug fixes and refactors. Like the AI just misses very fundamental things and it’s hard to review thoroughly because it generates so much code, right?
Back in the before times, an engineer reviewing code would maybe have, you know, five files to review… ten files… kind of seemed like a big PR, right? With AI generating so much code, it’s not uncommon to get reviews with 20, 30, 50, 100 files that need to be reviewed. No human can keep that context in their head and actually review that code in a meaningful way.
At some point, you just have to kind of trust, but who are you trusting? You’re not trusting the engineer. You’re trusting the AI and the AI cannot be trusted. Then you ship
Dan: What about another AI, an adversarial AI? Does that work?
aCTO: Well, I don’t know that there are actually any adversarial AI’s when you come down to it. It’s all trained on the same data, right? Maybe there’s some post-training optimizations that are made that make them differ, in some ways. But I think the idea of adversarial AI’s are… you know, I think it’s a misnomer. I don’t think it’s, there’s such a thing as that, honestly.
Dan: What are you seeing in terms of hiring when it comes to junior engineers? Have you slowed your own hiring of junior engineers because of AI in any way?
aCTO: No, and the narrative that the hiring trend now has anything to do with AI other than cost, I think is, is absurd. Anyone who believes that is, either you know, ill-informed or intentionally misleading you. There’s definitely an issue with hiring, particularly with junior engineers, but this is not because AI is so good that it’s taking the place of junior engineers.
I just have not observed that. I mean, that is a theory that people have. I may have even spouted this theory in the past as well, but there’s just no evidence to support that whatsoever. I think we’re still dealing with the post-COVID hiring boom trail off. I think that’s why it’s still difficult to find work.
It’s still difficult for junior engineers, because they’re gonna be the last ones to come in, right? Like you hire junior engineers when things are great and you have excess support and bandwidth to actually train and supervise junior engineers, because that’s what they need in order to grow. Well, if you’re trying to run lean, you wouldn’t take that on.
And so that’s one of the reasons why juniors are having a hard time now. It’s because these companies are shedding roles, not gaining roles, and they certainly don’t have the bandwidth to kind of onboard and invest in people who are junior. They want that senior hire that they can just snap their fingers, bring them on board… they onboard in three weeks and they’re off to the races. That’s what people want.
Dan: When you get together with other folks who are in the industry, what are you really talking about when you get real about this stuff?
aCTO: It’s as much of the people in AI as the AI itself, right? When we observe the people who are really, really bullish on the technology, they observe the same things we observe in terms of the inefficiencies, the mistakes, the inadequacies, but they walk away with a very different view of the world. Whereas we see flawed systems that aren’t ready for prime time in some instances, or are only really useful and very narrow use cases. And there’s a lot of opportunity in those use cases. I don’t want to discount that at all, but they view this as very much, “oh, that’s, that’s insignificant. This will get better almost imminently. It’s not a big deal.” But we’ve been repeating this cycle for years at this point. They don’t actually get much better. Sometimes they’ll get better in very specific use cases, but they’ll be terrible in others.
And like I said recently, it seems like at least with regard to coding, they’ve gotten a little bit worse for whatever reason. And so we talk about the people and trying to understand, you know, what’s driving this person? Like, trying to get into the head of that person and trying to understand what drives them, because you know, these are people that also make decisions that affect us and our livelihood and whatnot. And so it’s really about the people as well and how they’re handling this.
It seems like, you know, there’s a, there’s a wave of disillusionment. That’s kind of springing across the industry right now, and you kind of see it hitting certain people early, some people a little bit later to the game. Some people seem like they’re nowhere near there at all. So a lot of the times we spend our time just talking about the people involved and not so much on the technology itself.
Dan: We’re talking about AI and I’m being pretty unspecific actually: we’re really talking about generative AI and like large language model AI specifically.
aCTO: Correct.
Dan: And I know that you have a lot of experience, you know, back when it was just called machine learning. You’ve used that in a lot of context, I know. Do you have a take on this approach of like “AGI or die”? It’s like large, large, large, large, you know, crazy large language models when, you know, there are many other approaches to, and they’re far less, you know, environmentally egregious. So, I’m curious if you have a take on this path towards the most enormous model possible — or do we even need to be on this path at all?
aCTO: I definitely don’t think we need to be on this path. I think the reason why we’re on this path is that, OpenAI released ChatGPT and it blew up in terms of usage and users and I think the investor community simply saw an opportunity. I don’t even think AGI is a thing that they’re thinking about. I think this is a thing that OpenAI talks about. You know, they’re out to make money, frankly. They talk about AGI helping everyone, bringing us all together and, you know, that’s all marketing speak, but you know, they just launched a social media app… What are we doing here?
So I think what investors saw was something that resonated with consumers and they saw a business case, right? This thing blows up.
Now all of a sudden you have millions and millions of people at least trying this technology out and so that was the thing that really fueled this path in my opinion. And so then when the next version of ChatGPT comes out… And it was a lot better… I mean, that was it. That was it for investors. It was like, we are all in on this technology. They made the assumption that we would continue down this level of improvement over time.
And then, you know, you have entrepreneurs, you can’t really blame them… they’re gonna build where the dollars are going and the dollars are almost all going into the AI segment. And so now you see more startups there, and so more people are employed in this segment and then that just creates a flywheel of more investment, more investment. And so that’s kind of where, where we ended up.
But this is not necessarily following the most logical path to AGI or really anything as it relates to AI.
Dan: Are there particular use cases or, companies that you think, are applying this technology in just the right way?
aCTO: I think fundamentally in situations where review and revise are part of the process already… that’s a really good place for generative AI because you already have a human in the loop. You already have the human judgment aspect of it as a piece, as a way to actually get work done. AI just simply supercharges it.
So creative aspects… If you’re writing an essay or something like that, actually I do think that’s a great use case for AI to generate ideas, to kind of help you articulate something that you’re already thinking about. Artists who create digital art, they can describe what they’re looking for, they can get a starting point.
They can revise things with the AI, with more prompting and whatnot. This is inherently a create-review-revise sort of feedback loop. Where Generative AI is like really good at that, but you still need that human in the loop. That doesn’t mean we don’t need artists anymore. It doesn’t mean we don’t need writers anymore. Coders fit into this. It doesn’t mean we don’t need coders anymore. It just changes how those people allocate their time to a lot more sort of like review and feedback and maybe less typing or mouse clicking, whatever the domain may be.
Dan: Is there an assumption about AI, generative AI specifically, that builders need to unlearn before it’s too late?
aCTO: That’s a good question. To the extent that people think these things are thinking or reasoning or on any path to AGI at all I think they should just kind of discard that these models don’t think or reason at all. They are very, very, very sophisticated pattern matching machines and that’s really it.
And I don’t mean that to kind of dismiss the hard work that went into developing these models. Brilliant people who are, you know, far smarter than I am, put a lot of blood, sweat, and tears into this work. And so I want to commend them on that. But at the same time, the idea that these models actually think in the way that we think… Frankly, we don’t understand the way we think as humans, right?
The brain is still very much a mystery. So the idea that we could reverse engineer that and place it into a machine and produce social media applications with it… It just doesn’t make any sense. And so I think once people understand that these are just auto completes probabilistic machines that are just taking the data that’s on the internet and guessing what the next, you know, letter or numbers should be… I think that’s the thing that really will help people understand the limitations they’re faced with.
Dan: When you talk to kids, about, you know, what they should be learning right now. What’s the advice that you have or the approach that you take?
aCTO: Well, I had one conversation with a kid in particular that comes to mind and I used that time to tell them kind of what I just said to you… Not so much about the reasoning piece but AI is not an actual consciousness, right?
The thing that scares me about the kids are like the ones that are isolated and looking for a friend someone to talk to and thinking that AI is — quote unquote — their friend or anything like that.
And so when it comes to kids I’m a bit more on the existential side of things right? And kind of guard them from being very naive about what this technology is.
Setting that aside, I would say to a more mature kid, maybe someone who’s in high school who can kind of discern these things a little bit better, or even someone who’s of college age that you don’t want to outsource your critical thinking to the AI.
Use it as a tool for sure. It’s there to help you. It’ll speed you up, but as soon as you start. The process of stopping thinking and just throwing everything at AI and accepting its answers, you’ve kinda lost the script on why we’re here in the first place.
Dan: Yeah. That’s the existential thing you just mentioned.
aCTO: Yeah.
Dan: All right. Well that’s, that’s a note to end on a happy Halloween.
aCTO: Happy Halloween.
Dan: How do you plan to celebrate?
aCTO: Trick-or-treating, of course.
Dan: Right on. Thank you so much. I appreciate you doing this. this was fun and uh, I dunno if we’ll have more of these anonymous guests on the show, but if you like this let me know and maybe we’ll do this as a regular thing. It could be a fun little confessional here on the show. Thanks, man.
aCTO: Awesome.