If a system still needs a brittle harness, is it AGI? We dig into cybersecurity as an adversarial game, personal agents that become extensions of the self, and why the model layer wants to be a commodity platform — not a permanent monopoly.

Transcript

Pedro: And one of the ones that he thinks looks most promising does harnesses. And he was saying, in your neck of the woods, meaning academic researchers, harnesses are the big thing right now. And I don't know if I would agree with that. It is true that harnesses are a hot topic. At least among a segment of the AI research world, even. But not the only one. But why are they hot?

Because we have these LLMs, and then now there's other things on top of the LLMs. But we just call them models. And you can't quite get them to do what you want most of the time. And so all this harness industry, which it almost is at this point, is basically trying to channel that river into the banks you want it to be in. Which, if it was truly smart, you wouldn't need to do?

But the smarter researchers and companies know that harness is a temporary thing. If your horizon is the next few years, and it may be—you just need this stuff now. Then, by all means, do the harness. But don't put effort into it thinking it's going to pay off long term. It's a temporary measure. In the medium long term, we'll have better AI. And again, I think the Anthropic folks say, or some of them, that we try to be as lightweight in our harnesses as possible.

They are really true believers that they're on the straight shot to AGI. And if you think about it to those eyes, this harness stuff, we're not going to need it at some point. Now, where those improvements are going to come from is another question. What we've seen in the last three, four years is AI does these amazing things. But then what you see on top of it is the same old hacking that the next step of AI is going to make you irrelevant. And harnesses are just the latest incarnation of that.

Pablos: What made Anthropic the real contender it is was the coding harness. And you're right, it may be a moment in time and they may throw it out at some point where it's like, I don't even need any more because they are so good. But the opportunities in this window of time, which might just be a few years, might be the venture window that they're investing on, is a land grab.

I could switch from Claude to Codex to Grok or any other piece of crap any day and I don't really care. They're all amazing. And so I get that if you've memorized command-key combinations or built macros to do shit, whatever, and it fits into your old IDE, then you're stuck. But by and large, I think you're right. I'm not going to care. And by extension, it probably means that's going to be true for legal, for medical, for any other application.

Pedro: There's a hard distinction to be made here, which a lot of the future of these companies depends on, I think. There's really two aspects to a harness. One is I got to teach this thing to do proper or good software engineering.

That's the part that I think should die one of these days because Claude has presumably read every last book on software engineering and code development out there. Why do I have to do this harness? Clearly something is going wrong there. That shouldn't take a harness. That part I think is short lived. But there's another part you kind of like alluded to there, which is in the long run, much more important, at least from the point of view of value creation and value capture, which is a lot of the harness is me telling the system to act the way I like.

I want you to be this kind of code developer. I want you to work with me in the following way. I have these things that I like, good or bad. I don't think there's much capture, much moat in the latter. There's a lot. If I'm not able to easily port that, I actually think we could see a world in which the moat and the switching costs and the capture of AI is the biggest you've ever seen.

Pablos: Really?

Pedro: By how locked in you are. The only problem is that Anthropic and OpenAI, they have none of it right now. As you said. You see a little bit of that in Claude Code because people kind of like the way it works in a number of ways. And that's a big part of it. But if tomorrow something comes along that actually lets me do even more coding, even less trouble. Goodbye, Claude Code.

Pablos: It's kind of wild because I basically don't want to use any software anymore unless I have the code. There was a time when I had an idealistic version of that and was running BSD and shit. But now it's that I just want to be able to add features, change things by telling my AI to do it, which I'm doing all day long now. Just go add a button for this, change the way that works.

And I couldn't do that to the apps that I had before. I still can't add a button to Photoshop. And so I don't really need Photoshop. I don't use 90% of those buttons. Just rebuild the parts that I do and I'll use my own version. And that's kind of where I think it goes. And maybe you end up in a world where, even in an enterprise context, maybe you still have some Salesforce esque database behind everything. But every user has their own interface.

Pedro: Exactly. And the thing is this is where there's a lot of the capture and the value, if you will, which is once I have spent several years with a tool. And again, continual learning, as it's called these days. But the idea is like your system should be learning all the time. Why did this thing stop learning when it left the factory? Once I have spent years with the system, indirectly training it and sometimes explicitly, but also very much indirectly to do things as I want. Then it's become an extension of me. The threshold for me getting rid of it is going to be impossibly high for a company that is starting from scratch.

Pablos: Unless they can distill it out of you. The new company should be able to observe and distill out of you.

Pedro: There's always this cold start problem. I see LLMs and the whole internet as just solving the cold start problem. But now to give the other side of this. Because AI is interesting because it always has these multiple sides. You were talking about Salesforce and things like them, the SaaS companies. And is it the SaaS apocalypse coming and whatnot? On the one hand, yes, because all of these companies, including Microsoft and IBM and the 50 year old systems and the COBOL stuff—it's perverse, but it's like, it's the crap. That's the moat.

If it wasn't so crappy, the problem is that I've seen this, for example, at my university. It was like, they try to update their system that does whatever and it's their financials or their HR and it's a disaster. But here's the thing. The good news is AI dissolves a lot of that moat because the AI can understand all that little crap, the AI can go in and Oh, you're trying to do this and Let me do this better. That's the good news. AI doesn't do miracles. In science, they used to have this notion of the dusty deck Fortran.

It's a pile of Fortran punch cards that no one dares touch because it does what you need. But every time you try to mess with it, something goes wrong. And the unfortunate thing is like the whole SaaS world is full of this stuff. You're a bank, you can't afford to screw up or whatever. Or take any number of things.

Pablos: Using Fortran.

Pedro: But the thing is that the AI doesn't do miracles and a lot of that stuff, it's fragile. But in a way the fragility is part of their moat. Let me put this way. There are things that no amount of AI will figure out. I think the way a lot of this will happen is like, I'm a company that uses Salesforce and SAP and whatever. And I start using my AI system in parallel with the current one. And I do this for maybe years. And at some point, which will not be right away, the AI always does either the same thing or better. I can get rid of this crap.

But there might be, again, knowing how the world works, there will probably be years where that never happens.

Pablos: Or even just building the framework to do that A/B testing over time could be really powerful.

Pedro: There are people making that bet, and there's various versions of this and you could debate them. And I, again, this is all still very much in the beginning, but I would say if somebody came to me and said I want to do the next trillion dollar AI company, what should I do? Probably my first suggestion would be there's the current AI layer that can talk in natural language. But it's also very unreliable, opaque, prone to hallucinations. There's that. And there is in another completely different world up until now, all the corporate systems of record and all of that stuff that has all this crap, but also you can't live without.

And also that stuff isn't going. This is what the SAP Salesforce guys say. Oh, we're not going away because that's not going away. And they're right: the database has the ground truth. Another problem is that now the way the agents call this, it's a tool call. The fact that it's an AI doing the tool call doesn't do any miracles. In fact, the AI at this point for that purpose is stupider than the SQL maven. You need a layer that's going to do that for you. And that layer is going to learn from watching the people use it.

There's a whole sociological aspect to this. We don't know exactly how AI is going to turn out. But one thing is sure: you should be collecting data about how people talk about what they're doing. You should be paying your employees richly to do protocol thinking aloud continuously.

Pablos: How do you keep from, how do you keep from getting conscripted into one of these major AI projects?

Pedro: That's another big question that's playing out right now in multiple ways, including the open source and whatnot. And this is what they're dreaming of. I will be the owner of AI and everyone will be beholden to me and I will be worth a quintillion dollars. The Anthropic guys actually say this with a straight face. And if you follow that chain, it's like, well, the economy is going to become, it's not going to be the size it is now. With AI, it's going to grow faster. And we're going to capture 20% of it. A trillion dollars for Anthropic is cheap.

That's probably worth a quadrillion. But this is not the world you want to be in. And I think all of us need to fight that. What you actually want, the opposite of what they want is like, you want that layer to be commoditized. Just like the operating system, as commoditized as possible. It's just a platform.

Pablos: On one hand, in some sense, it is the most commoditized and most democratized technology of all time. Because in order for them to get that scale of revenue, they need as many customers as possible. That's why they got to sell it to everybody. They can't just keep it to themselves. And so even though they might end up with a sole source or a handful of providers at the bleeding edge of what's possible in AI, for them to make the revenue they want, they have to basically have the biggest customer base of all time.

Pedro: I partly agree. There's another very important aspect here: the time. With a couple of caveats that we can get to. I doubt that we're going to wind up in a world where there's a few big providers. And the reason is very simple. Those providers to their credit spent a lot of money and a lot of human effort coming up with these absolutely monstrous models. But 10 years from now, they're not going to seem monstrous. We will assign them to our students as a class project. Build an LLM, build, GPT-3 and then build whatever.

Pablos: Kids are doing it on Arduinos on YouTube now.

Pedro: There is this 200-line GPT-3, which is great. It kind of really demystifies the whole thing. And what mining the web is doing is giving the systems a base level of common sense knowledge and a few other things, but mainly that, which then allows them to understand language. But you only need so much of that. And again, there's a fundamental machine learning problem: our algorithms don't generalize well enough yet. You only need to acquire the capability of a regular human being once. And after that, this is the beauty of computers. You can copy it a million times.

That part right now is very valuable, but it's going to become very quickly, I think, very unimportant.

Pablos: I think that makes complete sense. If you got to the point where you understand language in a language model, then dumping in more fan fiction or something doesn't make it better at language.

It seems, at least for a lot of things, that's kind of the stage they're at. With language, we're certainly there. And coding, we're probably getting there. And I don't know what areas we're not getting there, but probably mostly the things where there's not good, data sets or the data sets are proprietary or something. And so they can't learn. I guess one thing you would have really unique insights on is from my position, what I see, I have a somewhat unique position in this because people pitch me on all their new AI breakthroughs.

And nobody's publishing any of this stuff anymore because they learned from the Transformer paper that you shouldn't just give it away. I see a lot of these behind the scenes breakthroughs. And my view is heavily affected by that. And you probably see some of that. What I see in the big labs is almost no breakthroughs. They have tried to lock up all the talent. All these people are calling them researchers, but largely I think they're just engineers building, different kinds of test models to see if they can get a little bit of an edge.

They've had little engineering advancements that are a mixture of experts or something as examples of ways to make the models cheaper and to run for inference. None of them seem to be able to hold on to any of these advantages for more than a couple of weeks before everybody else has the same thing. And so it's a very neck and neck kind of race. And it's not clear to me that we're really getting much value out of these geniuses at this stage. What do you see happening there?

Pedro: That is a great question with several aspects to it. I agree with your description of what's going on. And it's perverse. In a way there is now vastly more AI research than there ever was. And more resources being put in with justification. Because the payoff and the benefits are potentially enormous. But perversely at the same time, there is really less progress in AI than ever, except along certain very narrow directions that everybody's going in. What happened at Google, I think, is very interesting. Every company is a different case, but Google is a particularly interesting one because they've actually spanned a certain part of this. Google has the deepest and widest research teams of anybody, right?

Links

Recorded on July 24, 2026