AI, Actually – Episode 25: Why AI Agent Pricing Should Be Value-Driven

And it’s not going away. And it starts with this idea of where’s the ROI associated with AI? And I think it requires, we’ve talked about this before, I think it requires a little bit of a reframing. our clients and and sort of all that we see and read out there are are are you’re buying tokens. you’re making use of large language models, and

I l I literally talked to the CEO of a company yesterday where they’re turning off their enterprise license for one of the large frontier model providers. They’re turning it off. And this is a company everyone watching this would recognize. And because they’re struggling over the ROI and the value, and I think it needs a little bit of a reframing. I’ve said this before, I’m gonna say it again. I think

That you have to look at this not as a technology. You’re buying intelligence, and that intelligence can with the right scaffolding, with the right kind of controls and governance and all the things around it, the right orchestration, the right harness, you’ve heard those words, can do work. That’s just it. You’re buying the ability to do work. The question is: what work are you gonna do with it?

And you gotta think about it a little i in many ways, it’s very much like a person or your team and the labor you buy there. This labor has differences though, and it can be applied in in great ways, but but there are lots of rough surfaces where it’s not gonna work either. And that’s the challenge. That’s where the ROI lives. What are you gonna do? What work are you gonna get done? And is that valuable to your business? And I’ve said this many times it’s work maybe you’re doing today.

That’s sort of low-hanging fruit and that’s sort of a cost takeout mindset. And I think that’s that’s sort of the bottom rung of the ladder. Then it’s what work do you wish you could get to, or maybe what work you haven’t thought of yet that will enable your enterprise and or or or your business or what you’re trying to do. And that’s the core of it. So let me let me stop with my rant here for a little bit and and turn it over to one of the team members. You know, Andy, pick up where I left off and and and

Take us to in into a little bit deeper domain.

Andy Sweet (03:23)

Yeah, no, it kind of felt like a senator when they ask questions. They kind of give you that five minute upfront. and then then then they leave you 30 seconds to answer. So do I get more than 30 seconds would be my question. Yes, so it’s so it’s interesting.

Jim Johnson (03:33)

Yes or no?

Andy Sweet (03:36)

I’m I I’m I’m gonna start maybe a little backwards, and and I think you’re exactly right, Jim. It’s interesting. Let’s say you’re a thousand person company. Would you hire a thousand people overnight and just leave them untrained and hope they deliver value to your business?

And that’s literally what some companies are doing. They’re flipping on these LLMs. It’s like hiring a thousand really smart people, but they have no idea where they’re going to do their jobs, how they can do their jobs, how they’re different from humans doing their jobs, right? And because we have this thing that has human-like intelligence in certain places, in other words, you ask Chat GPT to summarize Jane Austen’s pride and prejudice, and it sounds like a genius. You ask it.

without putting any scaffolding or anything around it, what Q4 sales last year in Brazil and what gross profit was, and it will give you an answer completely wrong. And so again, it’s not turning on these it it is putting these LLMs in a position where they can be successful, where they have guardrails, and even where it’s it’s activating for humans to be more productive, you have to have training, you have to have identified workflows.

And then it probably flows into, you know, topics like how do you how do you pay for these? And you know, I think that’s the rest of this podcast. But I just wanted to start there. It’s like I think there’s this idea and we get it all the time. We get people asking us all the time, can I just use Saul to do that? Can I just use Opus to do that? It’s like you can, but you have to do these other things, and that’s critical.

Nicole K (05:13)

Can can I add one thing before you jump in there, Jim? I I agree with everything that you just laid out, Andy. and I think the point about hiring a thousand new people is a is a great way to frame it. I think

Jim Johnson (05:25)

It is.

Nicole K (05:25)

there’s also another step to differentiate here or to distinguish. It’s

The the chatbot type making us as humans more efficient if we use them well, if we give them those guide rails. But there’s also the agentic solution, which is a different beast than this just turn on the frontier model. So the

a Gentec solution I think leads us into that conversation you were just referencing, Andy, where what is the output of that agent? What is the business value that I’m delivering from that agent? And how do I measure it? And then that gets into how should you pay for it?

Ada Gil (06:08)

I I think the biggest challenge that companies actually are seeing is that they have lots of employees and suddenly they are hiring a very talented but very different employee. And how do they actually integrate that person or or that agent into their workflow and how does it actually change

the workflow that everybody else in the company needs to do and how do you actually pay for for that and how do you evaluate the contribution that this you know sort of employee has. So that that’s that’s where things are starting to get a little bit challenged.

Jim Johnson (06:44)

We’ve

we’ve spent 200 years through the Industrial Revolution, et cetera, figuring out how to harness human labor.

Ada Gil (06:54)

Yeah.

Jim Johnson (06:54)

We’ve we’ve got organizational design and org structures and reporting structures. A million books have been built on it. We’ve got HR departments. We know how to manage when when our employees are having issues. you know, we deal with benefits, all these things.

For two hundred years we’ve we’ve we’ve evolved specialization, division of labor, just you you name it, and we’ve figured out how to harness that human labor in an economically productive way. By the way, and we still screw up, right? You you know, employees quit, and you know, employees do the wrong thing, it just is what it is. But we’ve we’ve greatly sort of enhanced and grown our ability to to harness that that that effort in in in a way that’s meaningful and and and and

Just good for the system. Now, we’re two years into agents or a year into sort of agentic solutions and LLMs, and we’re really early in figuring out how to harness that labor. And it fails in different ways. You know, your employee gets sick, that’s fine. they they don’t show up. you know, the the the GPUs are running slowly at your large language model provider, that’s a different sort of sort of failure and will alter.

what work gets done that day. We’re working through it. We’re, you know, Carpathy says we’re in the decade of agents, not the year of agents. I I totally agree. It’s gonna happen faster than two hundred years, but it’s this it’s the same problem, figuring out how to manage and get value out of that labor, but it’s gonna be a very different set of solutions.

Andy Sweet (08:31)

No, no, I love that point. I mean, look and I think we don’t throw away that two hundred years of history because I think we can apply it to agents. And I think one of the mistakes maybe the whole industry made early on is these things are so smart. They they do summarize Jane Austen’s pride and prejudice so well. Surely they’re going to be great in the enterprise. and and so we just we we start them off as a senior analyst.

Instead of let’s identify where this really smart intern could be valuable in the enterprise right now. Let’s start them as an intern. An example might be let’s say they’re doing they’re analyzing construction sites for safety violations. They just do simple evaluations. You give them a picture, they say yes or no. As they get better at it, then maybe they can get promoted to more of an associate where they start to alert the foreman on the site, hey, I see.

And then they start to become more automated, where they actually start writing safety reports and making recommendations. And then finally, maybe they start doing predictions and saying, okay, the weather’s gonna be bad. It’s a Monday after a holiday. These are traditionally high accident periods, safety violation periods. And so taking it on a path that you would do very much like that 200-year framework you talked about, Jim, I think is where again.

where the industry needs to go. And that’s the way you need to think about it. Identify those areas where this new thing that’s not a human, but it’s not traditional deterministic software can be effective. And then have a framework of knowing when it’s successful and how to promote it to do more and more within your organization to drive more value.

Nicole K (10:12)

And I think one of the things you said there, Andy, that I’ll build on is with each promotion that Andy just laid out with an employee, there’s either they are learning institutional knowledge to know how to go from step A to B, or they’re getting additional training from the company to know how to do step B better. and all of that needs to happen with your agent as well. You need to give it that context.

to be able to understand how to do that next task.

Ada Gil (10:44)

What we see basically in the market is that it is easier for people to understand and to define what is the outcome that they want from the agent. What is actually hard is to define how to measure what done means and to measure what’s what’s good enough. And the key to do that is, and there are some risk areas in here. First, you cannot let the agent

evaluate itself because that is just too risky. You you’re not supposed to do that. It is also very risky. Exactly. Exactly. But it is also very Yeah,

Jim Johnson (11:19)

When employees evaluate themselves, they think they did a good job.

Andy Sweet (11:24)

By the way, Jim, I’m doing amazing this year.

Nicole K (11:27)

Ha ha ha ha.

Ada Gil (11:28)

I agree. On the other hand, it’s very risky also to let humans evaluate evaluating an automated process because this is also something that is very risky. So

The best way to do that is to once you understand what are the outcomes that you’re trying to reach, is to understand what are the acceptance criteria that you have for that specific outcome. But the key is to define it at the beginning of the process, not as you go and change it all the time. To define it at the beginning of the process, and being able to evaluate it at the end, and then once you reach there, you can always continue and grow and have your inter becoming a

senior, etc. etc. But you have to define it at the beginning. Because we are like Jim said, we’re at the beginning of a very big journey here. And the key is to have a measurable start and then to grow as as we go.

Andy Sweet (12:23)

Yeah, no, and I I think you make a really interesting point because there’s certain agents that do things that are either right or wrong, right? My safety violation is it’s either right or it’s wrong, right? That’s either an OSHA violation or it’s not. But there’s other things agents are doing, things we’re doing with with them that are much more taste, much more, you know, is that a compelling story? It’s it’s looking at data. Is it really telling a compelling story? And com compelling’s in the eye of the beholder, oftentimes a senior leader.

Right. And so being able to again promote them along the way and in in coming up with that criteria of of good enough right up front. And in let’s get concrete. It’s like maybe those things start to get baked, like it’s going to produce a story or a deck that only requires one hour or less of a senior leader tweaking it. That’s good enough. That’s how we would grade an intern, or that’s how we would grade a mid-level associate. So

you’re you’re spot on coming up with that up front. And then maybe even, you know, how you pay for and how you think about the value becomes that output.

Nicole K (13:31)

think the other thing as I add on to that so so

What you guys are talking about is during the build, defining what are those acceptance criteria. I think if we look at once it’s built, once we have this agent and it goes live, what is my adoption? So I think that’s the other piece that you have to keep an eye on, even while you’re building it, is do the humans who the the users that I’m delivering this to, is this gonna fit inside their workflow? Is it easy enough for them to use? And I think back to your point of pricing as well.

Well, Andy, it’s driving the adoption of that new agent will drive the value. If you’ve done everything right up until the point you go live, driving the adoption will then drive the value delivered from that use case. And I think that’s where you have to tie the pricing of what is that unit of measure that Ada was just describing? And then did we meet all our acceptance criteria? And then are we using it? And you’re

value delivered should be tied to how much you’re paying for each unit.

Andy Sweet (14:39)

Yeah, that that that that makes complete sense. And you know, what’s interesting though, by the way, I think the ground you just covered there, Nicole, is is so brand new. And and and it’s

Nicole K (14:51)

Agree.

Andy Sweet (14:51)

definitely one sitting right in front of of of what we’re seeing with clients. And and how do you how do you measure the fact I I’m spending this much in tokens and what is the real business value I’m turning those tokens into?

What is that atomic? I think you called it an atomic output. Like, is it a deck? Yeah. Is is it

Nicole K (15:10)

Atomic unit. Yes.

Andy Sweet (15:12)

catching safety violations? Is it accurately mapping a milestone on a construction work site? And so you’re getting very concrete and and you can have that discussion at a board level and not a technology level. And the the last thing I would just say as you were talking, it triggered, and it’s kind of interesting. We’re interacting with these things that are

that are approaching human like intelligence, but they’re on a form factor, a laptop, a computer, that is very we’re accustomed to dealing with deterministic software. So when it takes seven minutes to do something or 45 minutes to do something that violates our instantaneous internet expectations. And so reworking how we think and the importance of of again, you know, you’re talking about that CEO gym, of making sure people understand

This is how these things work and and this is how you get value from And this is how they’re gonna work and this is how you have to interact with them. And so that that’s another component to this, I think, as well.

Jim Johnson (16:15)

W when you hire people today, if we sort of set all this aside, back to your you you’ve got a thousand employees, you’ve got ten employees, you’ve got a hundred thousand. There is an expectation of things getting done. You break the organization down, you define but you know, ultimately what is the the purpose of your company, what is your product, what do you do, whether you’re a digital business or you’re a you know or or or an Adams ATOMS business,

your you you and and you’d understand your labor cost and you understand your output. Those basics don’t change here. but understanding being able to observe what’s happening, especially given your point, Andy, that sort of living in the digital world, by the way, th that may not be indefinite, right? With with and

Andy Sweet (16:59)

Right.

Jim Johnson (17:00)

I’m I’m sort of laughing a little bit, but it’s gonna come out of the digital world very quickly here in the form of robotics and it already is.

Andy Sweet (17:06)

Yeah.

Jim Johnson (17:07)

So in those cases

the the observability may be more overt or maybe you know e even even easier to do. But but if you go back to the I got a thousand people on my payroll and am I getting the labor output that I expect for whatever your business is or I’m spending XYZ on tokens, am I getting the labor output and productivity that I expect?

But that that that’s the journey we’re on right now, helping people sort of figure out what it is it’s supposed to be doing, what is valuable effort, how do we measure it, how do we observe it, what is the atomic unit of output. And it’s going to be narrow right now. But we’re but this is but this is expanding. and it’s very different, right? You know, I I can, you know, hu humans can deal with sort of incredibly jagged surfaces.

In processes and interacting with each other. Large language models or AI isn’t there yet. It’s improving. I don’t know that it’ll ever map to sort of exactly the way a human would deal with surprises or jagged edges, maybe. But both have their ways they fail. Both have their ways they succeed. They’re different. And taking our experience from how we think about

observing work getting done in the one world. It it’s a it’s a brave new world in designing that in the LLM’s powered space.

Andy Sweet (18:38)

Yeah, and and you know, it’s kind of interesting that as you were saying that, it almost feels like you could replace anytime you use the word LM with intern. Like interns can’t handle the nuance initially when you bring them into an organization. And then you start to build the guardrails around that that allow them the semantic layer so that they understand what gross profit means in this context, et cetera. And so you’re exactly right. It literally is.

You know, if you had any mental model, think of LMs as interns that then need this training that that slowly builds up into time. And and so that goes again to the point I think Nicole, both you and Otta were making is you can’t put them in just any situation. You gotta start them where they’re gonna be successful and then grow them and then be able to measure their value. Are they delivering on their value and be willing to demote? We’re we’re we’re talking here about just endless promotions, but it might be a situation where, wow, we need to demote this or

You know, we don’t need that that intern from MIT, right? Because that’s that’s Fable Five. Maybe we can get that one. I went to Louisiana Tech, I turned out okay. I’m probably like a sonnet model, right? And so can I use Sonnet

Nicole K (19:50)

Okay.

Andy Sweet (19:51)

to actually do the thing and train it in the enterprise?

Jim Johnson (19:55)

Andy,

I disparaged a completely different university in a conversation with a client the other day when I this exact analogy. You may not need all MIT or Harvard educated frontier intelligence for the work that you would consider getting done. And I won’t name the university that I threw in at the time, it wasn’t Louisiana.

Andy Sweet (20:17)

Yeah, that that’s why I I I always yeah, anyway. Yeah. Yeah.

Jim Johnson (20:19)

You can hire cheaper intelligence because

of the nature of the work. And that’s a whole nother topic, sort of, and I think we’ve brushed into it in another podcast sort of optimizing for that.

Ada Gil (20:29)

Yeah and and the big step that I think can can be done here is to break down the workflow to

Smaller chunks of units of work that has behind them you know quality bars and and things that are able to measure that. And in you know, how how do you measure an intern? It’s very hard at the beginning to measure what’s the the contribution of an intern, but you can break down this very complex workflow and then measure each one of them, and then in the end you sum up the everything to to something that is bigger than than those.

Jim Johnson (21:04)

Ada, the

challenge I think is in front of us on that point, though, and I want to I want to hear your response to this is we break down a workflow, we find a a narrow slice where we think AI intelligence doing work makes sense, and that’s awesome. but as we potentially expand it, that workflow maybe needs to change. And I don’t think we have a view yet.

We talk about sort of re architecting, reengineering workflows and processes, but I don’t actually know what we should and I don’t think any of us do yet. Actually, what are the sort of rules about how to to design a w an end to end workflow that exactly fits with AI and humans working together? I think we’re figuring that out. sorry, I’m

Nicole K (21:52)

I’ll let you in.

Jim Johnson (21:53)

gonna kick it back.

Ada Gil (21:54)

Yeah, no,

I I I agree with you. So I I think today we lean on humans to close the loop, a lax between between those units. the key is to be very transparent when when you know the AI finishes the units and to be transparent with with the human how to complete the work and how to sort of communicate with with the next part of of of the workflow.

We will continue to close the gap there, but I agree with you at this point we’re actually letting the user do do a lot. And it’s it’s it’s where the art and the science are actually making a solution be very valuable for

Jim Johnson (22:35)

Most of it still looks

like a human workflow where AI is doing something. At some point we’re gonna invert it and it’ll be

Ada Gil (22:40)

Exactly.

Jim Johnson (22:41)

it’s it’s the the and and and sort of the the human role will need to evolve quickly into very very different it’s it’s the workflows may look different, may be harder to observe, etc. We’ll we’ll work through all that.

Ada Gil (22:55)

Agree. The key is to let the human trust every step of the process that the AI did. Because if a human needs to recheck and recheck everything, that’s where it’s becoming very frustrated. But if you build a very good workflow, very measurable, very trusted, then humans can actually expedite the process a lot with with trusting the AI behind it.

Nicole K (23:20)

And I think that’s where we come back to Andy’s model of promotions. Each maybe each promotion comes with the humans interacted with it and gained a level of trust. Now we can give it one more task. And then the human interacts and gains another level of trust. And then by the time it’s doing the entire process, we trust the output because we know we’ve checked it each step of the way.

Andy Sweet (23:43)

Yeah, the thing I would say, you know, just to build on the points, Jim, the key to solving this problem is is not is just starting to go live. You have to put these things and start to interact and use them and understand because there’s no playbook coming, right? You’re gonna have to create the playbook. We’re creating the playbook.

Jim Johnson (24:02)

Also.

Andy Sweet (24:04)

and and you know, the other thing I I I that triggered when you were talking is two things. One is

It’s interesting when we first started doing this, you saw a lot of industries creating new AI metrics. What is token consumption? Like your metrics of measuring value are the same ones you used in your business

Jim Johnson (24:20)

hasn’t changed.

Andy Sweet (24:21)

forever. Hasn’t changed. And so, you know, and then the other funny thing is the more things change, the more they stay the same. You know, business process reengineering was all the rage, right? Now understanding how business processes work, and now with the new nuance of okay, it’s not just re-engineering.

them to Jim’s point for all humans. It’s humans plus these new things that we have that are not quite software, not quite humans and and are somewhere in between, at least for now. And what do those new workflows look like and how do humans interact?

Jim Johnson (24:55)

I’m gonna I’m gonna bring us back home here. we talk about ROI a lot, Andy. I think you just made a super important point that the core elements in business don’t change. And ROI is either reduction in expense, improvement in efficiency, which is the ratio of expense to to revenue, or the growth in revenue. That’s in the inner circle. Maybe just outside that circle, we can talk about customer satisfaction.

employee satisfaction, the sort of competitive moat or strategic advantage, but now you’re sort of getting into a place that gets a little bit harder to measure or maybe takes a much longer time horizon. Those those things haven’t changed at all. And identifying valuable work that supports those things and how you’re gonna get that work done, whether it’s with

humans and the two hundred years of lessons we’ve learned on how to make that work, or whether it’s with AI and and you’re gonna use token intelligence to do work and support those things, it’s still the same game.

Andy Sweet (26:06)

It is. And you know, and to connect the dots, it it goes directly then what you just said to what Nicole was saying. What is that atomic output? How do you connect that output to these traditional business metrics? If we do this, this contributes to these metrics. And and then it it it connects itself together and it and it goes again, you know, out of your point of identifying that up front. Don’t don’t assume that’s going to magically happen. Be very deliberate about how that’s going to happen.

even track it during the development of your solution. Are we tracking toward that that ROI that we anticipated? Are we delivering those output, atomic outputs like we should be, or do we need to make adjustments or do we need to stop? The other thing, there’s nothing that says you have to march off the cliff with all your flags flying. If it’s not working, stop and pivot.

Jim Johnson (26:55)

Yeah.

Andy Sweet (26:57)

Right? And so

Ada Gil (26:57)

Yeah.

Andy Sweet (26:59)

I I I love how you tied that together.

Ada Gil (27:01)

If you if you know what’s what’s good enough for every step of the process, you can actually even define for every process what’s enough in terms of the models, the cost, the timing. So you you can actually even not only just measure the outcome and how good the outcome is, but you can actually also define how much you want to invest in in every step of the process to really reduce your cost, because not necessarily you need to use for every step the best model.

Andy Sweet (27:29)

That’s right.

Ada Gil (27:30)

If you measure if you know before that it’s it’s it’s that’s not supposed to be so expensive, then then do not spend too much on that salastap.

Jim Johnson (27:38)

Hey team. Everyone on this call, I think, and and this is the great thing about the the the world we’re in and the business we’re in and the work we’re doing. Everyone here, I think, has a call in three minutes, probably with clients. So we’re gonna go do that. We’re gonna wrap it up. Ada, Andy, Nicole, thanks so much for showing out. team running the podcast, Vivian and Ruth, thank you very much. let’s call it and we’ll see each other again in a week.

Andy Sweet (28:07)

Yeah, thanks all the production people behind the scenes. Our hundreds

Ada Gil (28:10)

Yes.

Andy Sweet (28:10)

of people I put this on. Thanks every single one of them.

Jim Johnson (28:11)

Yeah.

Nicole K (28:13)

Thanks, Dave.

Ada Gil (28:13)

Thank you.