AI, Actually – Episode 21: Getting Enterprise Agent Implementations Right: Completed Workflows, Not Just Smarter Models

Pete (00:36)

Awesome. Awesome.

And Ada is world class at that. And we’ll we’ll welcome her to the show for the first time. So we have been watching, you know, internally we we share a lot of videos we’re watching and and and so on and podcasts. And one of the ones that we watch a lot, we talk about a lot, is from a guy by the name of Nate B. Jones. And the one he did recently is called The Trillion Dollar Agentic Workflow Opportunity Is Here. And I think he he really has some interesting arguments. Is so some of the main

Concepts that he talks about. He makes some of the of these claims. He says, first, you know, it’s really not about agents. It’s about in general, specifically, it’s about the implementation, about implementation of AI and so on. He argues that a lot of the value is in completed workflows, not just maybe a chat or a quick task or something, but things that get help get real business work done end to end. Third, he talks says the hard part is the implementation layer.

You know, and it’s all you know, workflow design, data access, permissions, authority, evals, audit trails, and so on. And then and then fourth, he he talks about how, and I think some of this was the trigger for his con his ⁓ episode. There there’s a lot of convergence going on where the you know the model providers are moving toward deployment. We see them starting, you know, billion dollar you know, consultancies in partnership with.

Private equity companies, the consultancies are moving toward agents. SAS SA if you’re a SaaS company, where we have the whole SaaS pocalypse going on. And if you’re, you know, if you’re a SaaS platform, they’re having to open up again, you know, agent interfaces and and private equity companies, but in particular, if they have a ⁓ significant investment in SaaS companies, they they just see you know workflow automation as a major value lever. And frankly, I think a lot of them are are ⁓ concerned that they really need to lean in hard.

to sort of preserve the the the their investments. So and then finally he just he sort of warns that you know AI rappers are just gonna get squeezed unless they’re really close to the workflow. And to he he said he talks about close to the business object and the action layer. Right. So so look this is episode is just all about like is he right? What what should enterprise leaders actually do? And and think about things. So maybe on the first one

Nate says, ⁓ the model alone is not enough. And even and so they’re it’s interesting to me that they’re investing in deployment and forward deployed engineering and so on. So I guess Shanti, well I’ll go to you first. What does all that sort of tell us about the limits of the the model’s gonna do everything for me approach?

Shanti Greene (03:13)

Yeah, I him it’s a great point. In general, I have trouble agreeing with something more than the general points of this, partially because I wrote an article, gave a presentation called Your Workflow is not an agent, because they’re different things. Agentic workflows exist in the middle and they’re very important because you need those steps that you can validate in between to make sure things are happening well. But the model itself doesn’t have all of the access it needs.

Pete (03:43)

Mm-hmm.

Shanti Greene (03:43)

It

can become the orchestrator of your workflow. At any given step, it can orchestrate multiple other agents that are doing things, but a model on its own can create a plan, can execute components of it, but can’t do it all. It needs tools. It needs very specific access, ways of using that access to do specific things, and ways to evaluate whether it did those things well.

It’s very helpful if that’s not all the exact same model, both doing a thing, evaluating itself and planning.

Pete (04:20)

You know, it’s funny, we’re we’re spending a lot of time with our clients and there is so much work to be done in these enterprises to really t to tie these things together. Mike, what’s your as you as you look at this, what’s sort of your your take on is the model alone ever gonna get us there or what else do you need to do and how does that all look?

Mike (04:34)

Yeah, absolutely.

It’s

it’s you know, so it it’s kind of funny. It’s the the Mike Tyson quote, right? Like everybody has a plan till you get punched in the face. A AI is a world where there are twelve different hands coming at you all at once, right? If it’s not the governance people, it’s the internal IT. If it’s not internal IT, it’s a new model. If it’s not, you know, th there’s so many things that have to go right. And the and that’s exactly what Nate’s trying to say, is that in the delivery of getting all those things right is how you do how you get the value out of AI. And and the the problem is

Pete (04:43)

Yeah.

Mike (05:06)

You know, we’ve spent ⁓ three years now where a little bit of business knowledge and some vibe coding would get you a pretty good demo, right? And then you get punched in the face, right? That that that’s that that’s exactly what’s happening here. In fact, I I love this word workflow because it’s kind of like all things in AI, there’s these words that kind of come in and go out. Like we used to talk about tokens a lot. We don’t do that much anymore, but now we’re talking about workflows a lot. Well, here’s the trouble. There’s two kinds of workflows. There’s the deterministic, hey, I want to answer answer the user’s question, keep a record of it.

Pete (05:13)

Yeah. Yeah.

Ada Gil (05:17)

Yeah.

Mike (05:34)

Make sure they don’t spend too many tokens, check the truth of it, right? That’s a workflow. Then there’s another workflow, which is I want you to look online first, and then I want you to write that up into summary, and then I want to see bullet points, and then spin off, right? So these are both workflows. Now one of them is very objective and deterministic. One of them is very subjective, right? One of them is something you feed into a prompt, and the other one is something that you feed into software. So you so Nate’s absolutely right. And until we tease these things apart, we’re not even really talking about the same thing when we talk to our customers.

Pete (06:02)

Yeah, yeah. And Ada one of the reasons we love having you here is you you’ve got a different lens. I mean Mike and Shanti look at it very much from the from the technology perspective. What what perspective do you have on that on that topic from a

Ada Gil (06:07)

Yeah.

Yeah. So,

you know, first it’s amazing how quickly and nicely you can get to the wrong objective with ⁓ those models. It you can just build something amazing and it’s just not doing anything that that you want. so that that’s part of the risk. So I think those those models are really great and they’re giving us a lot of enablement to do things, but to me the key is first really to understand the business objectives.

What are the things that the business are trying to do? And then immediately try to build those things that are the evaluations. So that’s the way that I approach it. If you know what good looks like in your business, if you know how to evaluate that, that should be the beginning of what you’re building. And then all the models can then build up to that. But if you know the end, how are you gonna evaluate the success in your business? That’s the way that the models I think ⁓

should work today. And the other thing is that you really want to build solutions that are gonna live regardless of the models. So we build a you know solution for one of the customers and the models and the technology changed four times throughout the last six months. So what the solution is now not gonna be relevant? No, we need to build a solution that can live throughout those models over time as long as you understand what good looks

Pete (07:39)

Yeah, yeah. I do think it’s interesting, Mike. You y you know, we’ve been doing some of this stuff together for three years and there’s so many people that we work with that they sort of w well, we’re gonna sort of wait for the model to to catch up so that it can do the thing, you know? And I I think there’s a guy, his name is Ethan Molick. he wrote a book, I think what’s the name of the book? Somebody help me? Everybody remember? anyway, it’s a great book. You should read it. But but

Mike (07:51)

Yeah. Yeah. Yeah.

Shanti Greene (07:58)

Mm-hmm.

Ada Gil (08:04)

Now we know.

Mike (08:04)

Ethan Molick noted.

Pete (08:05)

But anyways, not

working working with ever I think it’s like collaborative I intelligence or something like that. But anyway, he’s he said if anything tells you that the model’s not gonna, you know, sort of take take everybody’s job and and do everything is the fact that anthropic and open AI are investing tens of billions of dollars in companies to deploy this stuff, right? ⁓ it’s because it’s it’s not just gonna happen, it’s not just gonna come off the shelf. So another point.

Mike (08:25)

Right. Yeah. To get it out there. Yeah. Right. You know, it yeah, the the analogy the

the electric car analogy is interesting here because right when they you know, you first understand about an electric car, well, yeah, it doesn’t need a big engine, it needs little ones, it doesn’t need, you know, a water pump, it doesn’t need an exhaust system, it’s better for the earth and all that, it’s still gotta get you there. Somebody still has to get the thing to go where it’s supposed to go. So the technology can can be whatever the technology used to be, whatever it is now, but somebody to Ada’s point

Pete (08:44)

Yeah. Right.

Mike (08:54)

still has to get it to the place where it’s going. It has to be put to use, right? And and that this is, you know, one of these crazy analogies, but to some extent, horses were better at that than cars, right? A horse would learn which way to go and get you there. And if you had a couple too many, right? And and we we when we lost that over over a hundred years. And now we’re finally getting that back, right? But but we’re not there yet. We still don’t have that full capability. And and so so it’s one thing to talk technology. It’s another thing to talk implementation and material delivery of value.

Pete (09:04)

Right? Yeah. A little more self drive, yeah, f if I self driving, yeah.

Shanti Greene (09:09)

Yeah.

Pete (09:24)

Yeah, yeah.

Shanti Greene (09:24)

Well,

there’s so many times where the business outcome is not well defined. So people are working on process and like, can I improve the process without thinking about what was the point of this process? What was the result we were trying to get towards? AI is very good at improving efficiency in process. Like I don’t think anybody is saying you can’t make that better, but to Ada’s point, you can make it a lot faster to get to a place you didn’t need to be. Like, cool.

Like now I’ve got this really quick way of doing a thing that’s not that important, or it gets me towards half of a goal. So the role as consultants, as service providers, is really helping companies define what’s the appropriate objective. What should you really be moving towards in the first place? And then how do you get there?

Pete (10:02)

Yeah. Yeah.

Yeah, yeah.

Ada Gil (10:15)

I th I think I will add to that that one of the things that we are recommending you know to our customers is just to start, because yes, there are gonna be areas in the workflow that the technology is not perfect now. The models are not perfect, but there’s so much around it, like how do you structure the workflow, how do you understand what the personas are doing? So even there is even if there is a core part in the in your project or it that is not yet ready.

start the technology will catch up and it will get better as as you go. I think the worst thing is just to sit and wait until it’s gonna get ready because then it’s just gonna be too late for it.

Pete (10:50)

Yeah. And I think too, it’s one thing to your the point of make sure you have the right objective, right? That the the goal is a valuable one and a worthy one. And then the other piece of this is knowing how to reinvent that process and not sort of pave over this this way that you were getting there that you really could just collapse down to n nothing as opposed to just sort of repeating repeating the old way. And that’s not always obvious to people that are doing.

Ada Gil (10:56)

Yeah.

Shanti Greene (11:15)

Yeah, we had an interesting discussion the other day and one of our lead engineering and architect folks said, like, you can’t be afraid to just rip out everything you’ve built and rebuild it now. And part of that is it’s so much easier to rebuild something from scratch than it used to be. I I used to be afraid of technical debt. Like, well, we can’t if we build it wrong, it’ll l last for years and we’ll never rebuild it. Rebuilding

Pete (11:32)

Yeah.

Yeah.

Right. ⁓

Shanti Greene (11:41)

Is easy, especially

once you’ve learned what the real problems are, how people are using it. We should probably, as technologists, start thinking about should we start a rebuild cadence? Like, do you just rebuild your software every six months, every year? It just say we’ll probably know more, could build it more effectively.

Pete (11:45)

Right.

Yeah. Yeah, it’s a good point.

Mike (12:02)

Yeah, your your

subject matter experts are the source code. Like that that’s really what it comes down to. You need the the SMEs documenting what they do is the source code. The code itself is disposable. ⁓ and then and and IT is there to provide scale and security and governance and provision, right? But but not to provide software, right? Because the the AI can generate that software. What you need is a partner that captures the process, that knows the right tech to apply, that puts it all together in a way that’s future proof.

Pete (12:13)

a good point.

Yeah.

Well, there’s even this concept I’ve heard Alon talk about called ghost repositories. And they don’t have necessarily the the source to your point, they don’t have the source code, but they have all the knowledge about the the thing you’re trying to do. And you can point point a model at it and build it in, you know, whatever model whatever language you need, forever whatever platform you need. It’s a really interesting concept. So Nate talks about this idea that SaaS is under pressure, right? And that

Mike (12:38)

Right? Mm-hmm.

Shanti Greene (12:41)

Yeah.

Mike (12:46)

Right. Yeah.

Pete (12:56)

And basically, and so Mike, I’m gonna kind of sort of maybe start with you is, you know, w do agents reduce the value of SaaS? Do they make systems of record more important? What’s gonna happen to companies like Salesforce and others? Help help us get what’s your mindset around that?

Mike (13:11)

Yeah.

Yeah, no, it’s a great question. Look, the ⁓ the fact is that SaaS companies ⁓ provide a lot of things, but one of them is that they they are the system of record for your data, right? Who can access what data, where does the data come from, what does clean data mean? What does it mean over time? That is the reason why you’re seeing partnerships between SaaS companies and foundation providers, not not one of them eating the other, right? So you’ll see like just recently, ⁓ Workday and Google connected, but we’ve seen it recently also with

Pete (13:37)

Okay.

Mike (13:41)

I don’t know, folks like Reddit ⁓ have have teamed up, you know, some of the ⁓ some of the online photography sources team up. And the reason they do that because they have proprietary knowledge that needs to get into the hands of users through the model. They they learn that bitter lesson, right? We talk about the bitter lesson where you try to race against the foundation models, they’re gonna spend a billion dollars, you’re gonna spend a hundred million. Guess what? They’re gonna win, right? ⁓ the Transformer AI is gonna win.

But they don’t have your proprietary data. And you know, in fact, they don’t want it. They don’t want the the dangers of the publicly ⁓ the sorry the the personally ⁓ private information. They don’t want the you know the dangers of compliance and regulatory ⁓ troubles. So the the partnership makes a lot of sense. And so in that context, now you would say, well, if my if my SaaS value sorry, my my SaaS provider encapsulates that important capability, some of the some of those human workflows.

of how to use the data and the permanent record of what the data is, the AI company doesn’t even want that, then it’s a great, you know, it’s a great way for these two parts to to partner up. Now, what I will say, and I do think this is an interesting one, ⁓ a prediction, I think we’re gonna reinvent the term SaaS. The problem is software as a service is what is what’s kind of weird, right? Because we just talked about a minute ago how it’s how software can get replaced easily, right? But I think it’s something more like system of record as a service or

Pete (14:45)

Mm.

Ada Gil (14:59)

Mm.

Pete (15:06)

Yeah.

Mike (15:06)

You

know, you’re basically your ⁓ job function core data as a service. I’m not in marketing, ⁓ you know, don’t play one on TV. But but I think that word is the problem, not the industries themselves. And I think we’re gonna continue to see that evolve.

Pete (15:16)

Yeah. Yeah.

Yep. That’s interesting.

Shanti Greene (15:20)

Yeah, I’ll

I’ll take a slightly more pessimistic view on the future of some of those SaaS companies and say that their their model of how they charge their clients is the thing that really needs to change. I don’t think I would ever pay as a company for seats again. Like nope, like the number of users I have on this platform, like now, maybe for utilization, like the amount of compute, because that’s directly controlled to how much.

I’m using of a platform. And then for those tiered features, like you gotta give people all of the features. We know how easy it is to add features. So unless those features are causing you to, as a provider, to incur more costs, there’s no reason they shouldn’t be available at every tier. Like just charge for utilization. I don’t, and I don’t actually like utilization-based charging, especially when it comes to AI, because I think we all know that we’re getting token discounts.

And the true cost of the compute is heavily subsidized. And I’d like to continue enjoying those subsidies as long as possible. I don’t know how long it’ll last. But those, yeah, that old way of charging just doesn’t seem to make sense. And I think that’s what turns off a lot of businesses, especially when they’re going to have to add customization around the initial application. I think Salesforce in particular is so large and

Pete (16:27)

Yeah. Yeah, right.

Shanti Greene (16:45)

I’ve used it for years, starting in like 2008 or nine, nine probably. The amount of customization, like this entire industry for Salesforce developers to customize instances because of it almost fits but doesn’t quite. And people are paying millions of dollars a year for a solution that only almost fits. For tens to hundreds of thousands, you could have something catered directly to your business that is not just.

Pete (17:08)

Yeah.

Shanti Greene (17:12)

Hey, I can store data here, but we’ll push back to you recommendations on what to do.

Ada Gil (17:17)

Yeah, I I agree to that. I think also, you know, what’s relevant in here is one thing that people need to understand is when it comes to after Go Live, ⁓ I I think after the assess you know, sort of solutions, there are lots of opportunities for enterprises to own their own solutions, improve them to where they want the solution to go, do all the features that they need. So I think it’s a lot of opportunities for the companies.

to actually take the solutions to where they wanna be and not the old way of, you know, waiting until the SaaS companies are, you know, improving all of the solutions for them. It’s a huge opportunity here for implementation.

Pete (17:56)

I don’t pretend to know exactly where it’s going, but there are a few, I think, interesting variables to think about. One is I do believe where companies want to end up, Shanti was something you talked about something very tailored, right? And there that’s gonna be increasingly the expectation. Okay. Now you could get there lots of different ways. So I think that’s number one. To your point, I actually researched this. Companies in the US spend twenty billion dollars a year customizing Salesforce.

I think, I think a lot of companies actually are going to say, you know, why would I spend another million dollars with Salesforce to customize the thing when I could probably build the thing? Now, that may, you know, depending on how big a company you are and how deeply embedded Salesforce is. Look, if you’re just using basic opportunity and pipeline management features in Salesforce and you’re not using all the rest of the platform, that’s a much lower lift than than than right, because you have all the human change sort of thing.

But definitely this idea of this more custom things. The other thing that I believe very much is that the way these are these systems are going to be mo most effective is if they they need all the context. Right? So am I going to now give to so in order for Salesforce to have all the context that it needs, am I going to send it my sales information, my employee commission information, my pay information? Like there’s a whole there’s a line somewhere.

And people aren’t going to send that information to Salesforce. They’re going to want all that to sit behind their firewall. Okay. So now what does that mean for this for the SaaS companies? Well, for depending on the SaaS company, maybe I just want to communicate with it but, you know, as a with an API. Just I wanna, you know, have an MCP connected to it and pull data from that. And, you know, maybe I’m still using the system, but I’m not, I’m not really gravitating, right? The the the the center of gravity is shifting. There is this, there is this saying I’ve heard many times, which is day

data has gravity, right? So I think a lot of the the the user interfaces and so on and and by the way I don’t think there’s much of a user interface in the future. I think I was talking to an agent and it’s using these systems. So to my to Mike to your point of you know software as a service is like you know is it really a user interface that I even care about? Not really. I just want the agent to sort of do the thing. And I think the SaaS companies are in a little bit of a pinch because on the one hand they’re not going to capture all the data for all the context. If they embed these features in their own platform

They’re gonna then turn have to turn around and charge the customers for the API fees. And we all know how much wait what when I’m paying for the for the through my API charges, that’s it gets very expensive. I have to pass that on and ideally make margin. Or I don’t have any of those features inside my product, and I just make the data available via MCP, but then I’m just sort of this database, you know, in the background. So I think I I don’t know where it’s gonna go. I do know in terms of the software, but I do know and believe that it’s gonna be.

heavily customized to my business and it’s gonna have to have all the context is that I could possibly give it so that it can really perform. And ideally it’s sitting it most of that is sitting behind my firewall.

Mike (20:56)

there is one question mark still in my mind, which is, you know, if you if you if you boil away the the fact that SaaS companies offer software, which is sounds funny, but take take that away for a minute. There’s there’s no UI. There there still is expertise in a vertical that is being provided by by these companies. And and we even see it in, you know, in some of our some of our portfolio companies where the customers who used to come to us and use us as essentially ⁓ a glorified system of record, they’re actually turning around and saying, Hey, I

Help me run my business better. Like, like don’t just help me build an invoice using, you know, ⁓ using AI, but actually tell me where am I missing something, right? I learned this word from you, white space, right? Where are my competitors doing something that I’m not? Right. you know, I my business grew 18%, but should I have grown 23%? Right? Like, like looking for those kind of meta answers in the in the in this picture. I think that gets back to that workflow concept where

Where the SaaS company maybe also knows workflows in this industry, not just data structures, right? And and what they offer that a as that end product isn’t isn’t like a click and drag experience like they used to offer, but it’s more of an advisory, right? That’s kind of built in, ⁓ access to a network, a partnership of other people like me, right? I think there’s yeah.

Pete (21:51)

Yeah.

Right. That’s right.

Yeah, yeah.

Ada Gil (22:07)

Mm-hmm.

Pete (22:09)

that’s a great yeah I

completely agree and it and I but you when I was earlier speaking I was saying you know if if I’m just using this little tiny piece of Salesforce I could see replacing that but the more embedded that system is in the workflow and especially as you’re pointing out these vertical companies I do think they have a tremendous edge and the more of the workflow that they own right like you said the more intelligence they’re bringing to this whole thing and the more holistic that is and the more it just feels like this

Mike (22:30)

In that space. Mm-hmm. Yep.

Pete (22:38)

This whole ERP for me, they they I think have a tremendous advantage. They do have to lean into all this, but I think that’s a really good

Mike (22:44)

Right.

Yeah, and you know, it’s interesting. Earlier this year, there was a standard that was created around this idea of skills in the AI world, right? So the the first standard really we ever had was open AI by default, the interface layer. they didn’t publish it as such, but everybody copied it. Second standard that came along was MCP, right? That was a huge wave. This this new standard of skills is really interesting because what it does is it it allows a third party that knows a lot about something, it’s almost like like prompts as a service, right?

Pete (23:03)

I

Mike (23:13)

They they they capture a whole lot of information about a specific kind of process and they label it for what it is and they put it out there into the world and charge for it. and all the models line up and are compatible with it, you know, including our own products. and I think that that is kind of a a a new way of thinking about things that’s part of part of embracing the fact that the that the foundation model providers are coming down the stack, right? They’re coming in to sort of do some of the software functions, but they’re gonna look to third parties to for that expertise.

⁓ and and it’s a just a a way for people that are in the SaaS business to basically engage in this new economy, but they have to change.

Ada Gil (23:48)

Yeah. I think I see a lot of our you know, ⁓ customers, their expectations from the solutions are changed dramatically in the last year. They really they’re not willing to accept the lowest common denominator across the industry. It needs to be really tailored to their solution. It needs to be a bespoke solution for their business. The business logic is very, very important. we worked with CPG companies a lot of them and it is

even unbelievable to see how a single metric like you know even market share or something like this can be interpreted in so many ways across the industries. And those are just critical things that, you know, that the companies that are implementing those workflows and and all of those things will to understand that they will need to just drive those type of value, specific values to their business.

Pete (24:39)

Yeah, it it’s a good segue into this next one where Nate talks about value sitting close to the business object, right? And talking about workflow knowledge mattering, right? And it a lot. And so I guess the question, and we’ll we’ll start with you, Ada, you know, if the value is in this completed workflows, do do cus do companies actually understand their workflows well enough to automate them? Just what’s your what’s your general take on this?

Ada Gil (25:05)

Yeah. I think they’re starting to understand more, but definitely not not enough. I think the the key thing that we need to today to help our customers is to really understand what is a repeatable workflow that is driving value for a business and it is a good sort of a process to automate for their business.

So one of the things is really to understand where the value is, what’s repeatable. the other thing that I I see a lot of challenges, they some of them are thinking that, you know, it’s gonna be usually fully automated. And I think one of the key things that is opportunity is really to understand how to design the human in the loop as part of the solution.

Because that’s the value that’s a lot of the value for the business. They have good employees, they know their business, they know what good looks like, they know how to change and and direct the model to do good things. So a lot of what we do is actually work with those customers to understand the use cases, understand what’s the value, but also to understand how to create a workflow leveraging ⁓ their employees. So

Pete (26:15)

Where are you finding that the the process lives today? Is it like in somebody’s head? Do they typically have them documented? Is it

Ada Gil (26:22)

barely somebody’s head. We usually help them to document that. Takes a lot of a lot of time. not even help it’s not even really available for them to understand what’s every step worth in terms of hours or money on who is doing that. It it usually doesn’t live very well in the system. So we are helping them to get that. We’re helping them to reimagine the ⁓ you know the future workflow.

Mike (26:23)

Mm.

Ada Gil (26:47)

And I think you mentioned it before. It’s it it requires a total reimagination. With one of our customers, we started with the end. So they worked for hours to deliver a solution that they just took it step by step to make sure that they’re not making mistakes because it was so expensive. And we actually delivered, you know, our solution delivered actually things from from the end. So we delivered the end product very cheap, very easily and let the users just

then as a human in the loop just change it a little bit. So the the you know current state and future state of how the workflow is very, very up to the users’ imaginations now.

Pete (27:28)

And and does that yeah, is that sort of dependent on the user in in their imagination? Do we have to typically educate them a little bit along the way? Like how is that working? Do they all they n natively sort of get it or no? Yeah.

Ada Gil (27:39)

we we ⁓ no. It’s still

we still need to push because the the boundaries of the imagination is still something that is not easy. I think it will come easy as you know, as ⁓ we are gonna get used to all of those capabilities over the AI agent, etc. But yeah, today we need to try to understand where the value is and start from there. So I’m really trying at the beginning to really understand where the core is and then build the process to that.

So that’s the key thing that we’re doing. Where is the core and how to get there?

Mike (28:09)

And that that’s exactly the lesson learned from the massive migration of software development in the last five months, literally from you know, people artisanally coding every line and every semicolon to the prompts do it now, right? And and the and the the reason it works in software is because there’s a way to test the result that’s automated, right? So Ada, as you’re pointing out, ⁓ if if the customer works to define what is the right result, then and obviously not for not for every case, but for some generalized case.

Pete (28:09)

What’s

Ada Gil (28:21)

Yeah.

Mike (28:37)

What are the rules? What are the guardrails? What are the boundaries? Then you’re giving the AI a way of testing itself, which means it can do a lot more work on its own. Right. So you still put the human in the loop, but but only when the model sort of runs off and says, Okay, now I have something I need help with. Otherwise it can always self check as it goes along the way and not get completely lost, right? Not build something silly, in the name of something that’s, you know, th that that’s intended to be really good.

Pete (28:59)

You know, we’re sort of a little bit related to your to this conversation and out of your comments, you know, he he Nate talks about forward deployed engineers, which, you know, is is a term that’s been made pretty famous at this point by ⁓ Palantir. And you know, so do enterprises, from what you can see and your your perspective, do they need new roles? Do they need new operating models to make agents work pretty repeatedly across their the business?

Shanti Greene (29:26)

Yeah, I think the anybody who’s building agents needs to start getting very familiar with the outcomes that their agents are working towards. So you do need a better understanding of what the business is trying to achieve, not just what the end result of a workflow should be or what those intermediary steps are. You really need to be much more goal driven, goal oriented at the kind of macro, what are we trying to do with this level? Less at the like codex now supports goals. So like, yeah.

Pete (29:33)

Mm-hmm.

Shanti Greene (29:55)

But that’s really a way to keep loops going, but you and you need good evaluation metrics and other things to make that useful. so like those are the types of things we had built in the past. We’re like, hey, if we’re able to define these intermediary things for the model to measure, then you can tell it, hey, keep going until these statistics get better.

Mike (30:11)

there was a a number of years ago we we with humans and with machine learning, right? We delved into a project to do a a market mix model, right? That’s where you say, Hey, if I’m I’m a manufacturer of goods, I have one dollar, what should I do with it? Make a new product, make a new package for an existing product, offer it as a discount, right? It’s a really interesting problem. And you know, everybody knows that well when you when you lower your price, you should sell more, right? That that’s that relationship.

Shanti Greene (30:19)

Mm-hmm.

Mike (30:37)

is sort of tried and true. It’s part of human psychology. Well, our model, because there wasn’t really enough data, it produced the opposite, right? It said, if you lower your price, you’ll sell less, which means if you raise your price, you have an infinite business, right? ⁓ and and so we very proudly presented these results to the user in the sense that we wanted to say, hey, look how easy it is to ask these questions and get these results. But there was no agent in the loop in that case, right? It was basically humans that were mathematicians doing their part, there were business people doing their part.

But there was nobody bringing it all together, right? And and it’s clearly an agent would have captured that and said, Hey, you can’t have you know a positive elasticity on price because then you have an infinite business. but unfortunately, the customer was the ⁓ agent in a loop and the conversation was over. They looked at it and said, Doesn’t matter how nice it is, it’s wrong. So you’re done, right? ⁓ and so those are the kind of situations where where now the we these agents coming into the picture.

Pete (31:25)

⁓

Mike (31:31)

Are are going to provide a much more sound approach to it. They’re going to see a negative or a positive elasticity on price and say, hang on, that can’t be. So first null it out. Second, make a note about it. Third, fix it later, right? The but the but the the the agent capturing that kind of thing, combining obvious human knowledge from psychology, from economics, from the marketplace with specifics of the business that are provided by ⁓ you know either a SaaS provider or by the business themselves.

All of that integrated by the the consultants that are providing that that forward deployed engineering capability, right? I think that’s what makes the winning solution. ⁓ and and it’s it’s the combination. Yep.

Pete (32:06)

What’s different about forward deployed engineers? Are they just consultants with like a like a fancy brand on or or what?

Mike (32:11)

Mm.

That’s an interesting question. so ⁓ consultants, a a lot of consultants have written a lot written a lot of software for a long time. I don’t think a lot of consultants would call themselves engineers traditionally. they certainly didn’t typically come out of computer science or computer engineering. and so but it’s it’s not in name only, right? It’s a it’s almost like a process and scalability kind of interest, right? So there’s more of a full stack for a forward engineer.

⁓ the the certainly the expectations of consumer engagement or customer engagement and market understanding are much higher for a forward engineer than they would have been for a software engineer five years ago, right? So ⁓ so it’s it’s sort of some new skills, some of the old skills overlapping ⁓ and in a lot of cases paid a whole lot more because ⁓ those skills are in demand.

Pete (32:50)

My

Ada Gil (33:00)

Yeah. I see you know, I I see a lot of forward deployed engineers now that, you know, in part of the work they’re actually, you know, doing all the engineering work and the coding and everything, and then they literally sit in a in a customer call and they wear a hat of a business and they just don’t talk technol they don’t talk technology. And that’s I think the the right thing to do because the customers are not buying technology. They’re buying a solution, they’re buying the value.

And when you explain to them what it is doing, I think that the good forward deployed engineers, they want that, you know, they build a technology, but then when they sit with the customer, it’s a business discussion. It’s not a technology. You hear what they want and then you go back and you just, you know, translate it to technology. Those are to me the very good forward deployed.

Pete (33:44)

Yeah.

And they need to be able to sit with the business person as they’re trying to understand the workflow in depth. That’s the other thing a lot of people don’t talk about. Like for a deployed engineer by themselves. Okay, I don’t I don’t know what you’re gonna do with it without the the mirror image on the other side of the business person that really understands here’s our objective, here’s what we’re trying to get done, here’s here’s where the opportunities lie and where we tend to get stuck and where we can might find an unlock.

Shanti Greene (34:00)

Yeah.

Pete (34:14)

that the the the engineer can actually bring to life. I think that’s the th that’s a part that people won’t talk about it.

⁓ the he also talks about Nate talks about the value is in these completed workflows, which is a lot of what we’ve been talking about. And so Ada, I’ll I’ll start with you on this one. You know, how do I think about what where what where should I start and which ones are valuable enough? Or maybe ⁓ do they have to be well documented to start, or you know, do we start with something that’s sort of low hanging fruit and safe? You know, what what’s your what’s your experience, Elliot?

Ada Gil (34:46)

Yeah, I think I I started to touch it ⁓ before. you know, I would not start too complex at the beginning. I think the key is to find something that is very valuable for the business. It’s repeatable, some something that you can de definitely document. And I think if you can really nail down the start to end of the process and how this will integrate into the overall organization and to the persona’s ⁓ day to day life, that’s something that I would start.

Definitely need to drive ROI for the business. So I I think that is the key. If there is, if this is not driving something that is measurable ROI, it’s gonna be hard to adopt the solution. So everything that is repeatable can be documented. Usually those things are not documented. Can be documented. The other critical thing is that it can be measurable. So things that you can really understand what’s the value of every you know.

Every time that you’re using the agent is something that I would definitely start with. I would not start with things that are, you know, hard in terms of regulations or necessarily something that are, you know, ⁓ modeling is very hard. Those are probably harder. I think definitely are possible to do, but I would not start with them as as the key, as the first use case in an enterprise.

Pete (36:06)

Yeah.

I I I I think in in general, I think that’s good advice. I think it varies wildly depending on who you are. Like if you’re a SaaS company right now, like you need to be thinking about leaning into what what revenue generating opportunities, completely differentiating your product. I’ve got, you know, VC f funded startups sort of sort of coming after my space. And so you have some strategic imperatives you need to you you’re you really need to chase as opposed to

you know, optimizing how the finance function works. but in general I do th I think that’s good advice to just start right start with something a little bit low risk and just get your get your feet under you. But some of these companies I think have to move much, much faster.

Shanti Greene (36:45)

Yeah, my my Yoda like version of agents is that it’s done or not done. There is no in between. Because if the job is not complete, I’m not using that agent anymore. Like I’ve got lots of personal agents and workflows and things running, most of which do not end in a complete output and they are worthless to me. So just like keep getting rid of them. And I’ve got a really simple workflow, it’s not very agentic, and all it does is look across.

Pete (36:53)

Yeah.

Shanti Greene (37:14)

Four different types of calendars and move events, copy them. It can do updates, it can do deletes and sync them in different ways. It is so valuable that I don’t have to think about where did that calendar event go? It’s like that is huge value because it completes the circle. Like everything that it needs to do is done, no matter where things are entered. And then

Pete (37:29)

Mm-hmm.

Yeah.

Shanti Greene (37:39)

Any of my more agentic ones, which like I’ve got one that’s supposed to like read the email newsletters, like extract the different stories, do some summarization and ranking them. And it’s so it’s 95% of the way there. It’s so close. But it’s not done. So I never use it. I’m like, nope, it runs, it just eats tokens. I’ve had to turn it off because my like Gemini API bill was too high. It’s like, nope, not use that one anymore. But and it just didn’t get to what I needed, which was I need to be able to see the top five AI stories for the day.

Pete (37:53)

Mm. Yeah.

Shanti Greene (38:07)

With nice quick summaries and the business impact, is this one actually important?

Mike (38:12)

I I think there’s there’s some interesting points to combining here, which is one is the tokens are subsidized right now and and the the sheer pace of demand, you know, the wall of demand is going to drive the prices of them up before they can be satisfied and the prices can come back down. So we are facing a very real hurdle here, which is saying, Hey, is this agent not only finishing something, but is it valuable enough compared to what I’m having to pay for it? Right. I mean, it’s it’s super interesting to actually get

Shanti Greene (38:15)

Mm-hmm.

Pete (38:25)

Mm-hmm.

Mike (38:40)

you know, get into the deep economics of that because you do realize pretty quickly that it’s possible w these things won’t be affordable ⁓ just on on that principle alone. And so that drives you to make your make your workflows even better. And then on top of that, it it’s combining both of that again, that that subjective workflow that is model driven, but also the outer objective workflow, which is how can I trust this and and how can I make sure that it’s being accessed by the people that that only that can access it.

and and how do I know that it’s repeatable in the future? All those things that we come to rely on when we, you know, when we count on part of our our enterprise on how we make money, but they’re not necessarily gonna come true unless they’re harnessed for the for the agentic world.

Pete (39:22)

Yeah, so that’s got to go into the equation. Another place I’ve heard people talk about starting, because we’ve talked earlier about it is hard very often to get the get the workflow out of people’s heads because that’s generally where it’s sitting. But one place to go look are ⁓ areas of the company that have maybe been outsourced. So maybe there’s some analytics work that’s been outsourced, or maybe there’s some other, you know, finance work that’s been outsourced. Well, to do that, you generally need to document a process, right?

And there’s a a contract about, you know, what’s going in and what’s coming out and so on. So that is one place that I’ve that I’ve heard to people talk about looking to start that’s a decent clue.

Mike (39:59)

Yep. Makes a lot of sense.

Pete (40:01)

All right. And so, I I guess we’ll sort of well maybe try to wrap it up here. So look, I think, you know, the the video was great. he’s a really good resource. You know, so we’re and we’re moving, you know, basically the aging conversation away from these demos and towards like real enterprise reality. I mean the the the real the real next wave of value is not just smarter models. By the way, we got we got a smarter model, what yesterday? Four point eight.

Mike (40:25)

Yeah, Claude Four Eight.

Shanti Greene (40:26)

Yep.

Pete (40:27)

Right. and it was funny because our s the Slack conversations were reflected this this idea that it’s like, well, you know, great, another model, but it’s it’s about completed workflows. It’s about, you know, ⁓ but but they’re much more than just a new model. It’s about, you know, workflow knowledge and trusted data and you know, and we’re we’re seeing this everywhere where just plugging these things in in a trusted way that adheres to company guardrails and security guidelines and so on.

It’s a ton of work to to pull this off to get all this context together. So, maybe just to sort of wrap it up, any any last thoughts from you guys? We’ll give and Ada, we’re gonna give you the last words. This is your first episode, but we’ll start with Mike and go to Shanti and then Ada. Just l maybe last words of wisdom here for the force.

Ada Gil (41:08)

Okay.

Mike (41:11)

Yeah. Sure. So so look, the

something that I I love to pick on, ⁓ again, I’m back to terminology. We talked about workflows earlier. I wanted to make sure it touched on harnesses, right? Because I feel like harness kind of came out of nowhere this year and and it and and all of a sudden everybody’s asking questions about it. And you know, when it happened, and Pete this ties to what you just said, ⁓ when when roughly when Claude Four Five came out, right, it did make a giant leap that we changed that model, right?

Shanti Greene (41:22)

Yeah.

Pete (41:24)

Yeah.

Mike (41:39)

but then all of a sudden after four or five, it’s like I’m just getting ⁓ it’s smarter and it’s smarter and smarter, but it’s already smarter than me about a lot of things, right? So, so so now it’s not about the model anymore. It now now that this word harness comes into play. Well, what exactly is the harness? And right, and as the word implies, it’s the thing that kind of pulls it all together. ⁓ but but where is it and what does it do and all that kind of stuff? Well, it turns out the harness is software, right? So we’re we’re having to sort of go back full loop. Now, the harness isn’t about

Pete (41:46)

Yeah.

Mike (42:08)

making specific business logic rules in software, right? The harness is about managing the interaction, the human and the loop that Ada was asking about, the agent that that needs to be something that we can change out over time, the workflow and how those come in and how they’re governed. Exactly, how they’re governed and how we track the results, right? So so I think that it’s just really important to to keep that that word harness and make sure that we’re giving it its kind of full due. Because when we we talk about you know tools and prompts and models and all these things that have workflows that we had in the past.

Pete (42:18)

Yeah. Where to get the contacts. Yeah.

Ada Gil (42:33)

Mm-hmm.

Mike (42:37)

the harness is going to become the thing that that governs and and and we’re sort of used to harness ⁓ in the sense of like the clawed code, that’s that’s actually a harness around different models, right? or or you know, codex is another one. There’s lots of different perplexity you could say is is a harness, right? and so so that’s going to become a point where the value needs to be added. And they’re going to advance really fast. Why? Because the models are creating them. So if that if that doesn’t twist over on itself and confuse you, you know, then you then you haven’t thought about it all the way, right? So

Pete (42:47)

Yeah.

Yeah.

Mike (43:06)

The the harnesses using models and the models are creating new harnesses. ⁓ and and that’s that’s part of this acceleration that we’re gonna see ⁓ as as these models gain value.

Pete (43:16)

Yeah. I be it used to be we all talked about cont like last year it felt like it was much more about context engineering. You heard that a lot. And now you’re hearing a lot more about harness engineering. Yeah. Shanti?

Mike (43:22)

That’s right.

That’s right.

Shanti Greene (43:28)

I think that where we’re going with agents is all about can we actually complete a task and harnessing appropriately to know that we’re getting close to allow the agents to self-correct and get us there. We put those things together and we’re gonna end up in a pretty good place and we’re very close. A lot of the harnesses are there, but thinking about how you eva do that evaluation and like what are the things you’re measuring to know you’re getting close.

is still a little bit of an art. So there’s not a full science of that yet. And that’s where I’d like to see the industry improve is how can we kind of standardize around knowing that we’re getting close.

Mike (44:07)

That’s right. We’re we’re all walking around with maps, but what we need is GPS. And and we know GPS is a good idea, but but the maps kind of work. So so what so it can’t be that much better, right? I think we’re gonna see when we do get GPS, we’re gonna really realize, yeah, maps, that that’s kinda funny, right? The the things with crinkles, no.

Shanti Greene (44:11)

Yeah.

Pete (44:21)

Yeah. Yeah. Yeah.

And I listen to us talk and we we’re so deep in this stuff every day. We forget that most of the clients that we talk to don’t even know there’s a map. You know, there there’s there there is Yeah, you know, they’re just they’re just getting started, right? And and I think ⁓ understanding some of the things we’re talking about that hey, pr you probably shouldn’t be waiting for the model. You probably should be jumping in.

Mike (44:35)

Right. They they just have turn by turn directions, right? Yeah. Yeah.

Pete (44:49)

You probably should be thinking about how do I organize a tea the right resources and team around delivering this and set the right expectations about like, look, this is really hard work. this is back like I’m old enough to remember there was a whole book around business process reengineering is what it what is what it was called. And we’re seeing a whole nother generation of that. And that was hard work around you know, around software and technology at the time that I think people really need to to to

⁓ embrace that as something that’s just not gonna pop off the ⁓ the shelf from a model at some point and really dive into how do we navigate this together. Inata, I promised you the last word.

Ada Gil (45:27)

Yeah. ⁓ yeah. So

luckily I’m thinking like the rest, but I think I will add, you know, to me the moat is I think we talked about evaluation. This is definitely very, very important. But if you have the combination of domain expertise, I think this is something that we always need to remember when we’re trying to build a workflow and a solution for customers, we need to make sure that we have first we have the people with domain expertise that understand the business, understand what’s good for the business. If we have the right evaluation, rigor.

And if we have the right, you know, people that have good development track records, I think that’s probably the the right combination to drive a true ROI for the business.

Pete (46:07)

Awesome. Well, Ada, great first episode. We’ll see you and some others. Guys, good to good to see everyone. It is Friday, so have a great weekend. See you on the other side.

Ada Gil (46:11)

Yes, thank you.

Shanti Greene (46:18)

Yeah, see you folks. Bye.

Mike (46:18)

Bye all. Take

Ada Gil (46:18)

Thanks.

Mike (46:19)

care.

Ada Gil (46:20)

Bye bye.