AI, Actually – Episode 22: AI Intelligence Isn’t Free: Token Costs, TCO, and ROI Math for Enterprise AI

Then we’re gonna spend some time on the idea of just tokens in general and maybe more importantly TCL. because you know here as we’re halfway through twenty six, ⁓ our clients and and sort of the press and everyone sort of dialing into the idea that that that maybe this free intelligence isn’t quite free. And then

Third topic is we’re gonna spend a little bit of time on ROI in general and try and tie all these things to j ⁓ together in terms of what’s going on out there. So let’s let’s dive in, let’s start with Fable. you know, who wants to jump in there? Nicole, I know you’re sort of ⁓ been been experimenting with sort of the cost ideas and what’s going on and some of the value of Fable. So jump in.

Andy Sweet (01:31)

I just wanna say I think we ought to de dedicate this episode, number twenty two, to Emmett Smith, greatest running back of all time from the Dallas Cowboys. Sorry, Nicole, I didn’t mean to hop in there, but ⁓ I let’s agree. All right.

Nicole K (01:31)

Yeah, yeah.

You

No, that’s alright.

Jim Johnson (01:42)

Yeah. Highly

AI actually revel relevant. Alright.

Nicole K (01:47)

So gymnasts

don’t have numbers on their jersey, so I don’t have a jersey number to attribute to, so I’ll go with Emmett Smith. Okay, so yeah, my tie-in, Jim, I’m gonna hand it to Shanti in a minute to talk more about ⁓ fable, because I know both Shanti and Andy have opinions on ⁓ the value that fable delivers. I think my perspective is more around the ROI.

And it’s the fact that we’re talking about the input and the output tokens consumed by Fable and the list prices ratchet up a little bit for Fable. But the big difference when you’re looking at list prices, say GPT-55 versus OPUS 4748 or Fable is not necessarily the input or output cost per million tokens. It’s actually how many tokens are

are used for the same question. And then what is the cost of answering that question across them? And when you look across them, Fable consumes many more tokens. They’re actually, the jury’s still out on what the X factor is there. But if we look at GPT-55, we’re at a one X. If we’re looking at OPUS 47 and 48, we’re at about a three X for the same. And what we know so far is that Fable is a higher token consumption.

That’s still one part of your ROI story. So I’ll reserve more on ROI to the end because I think we really want to talk about the impressive results we can get from Fable first, but we’ll bring it back in the end to talk about ROI. And that’s just one part of your cost, your denominator in the ROI story.

Andy Sweet (03:27)

Yeah, so I you know, just to pick up, you know, I I don’t think all tokens though are created equal. And what I mean by that is, you know, a model like Fable can do things that we couldn’t do even with 4.8. And we’ll give some specific examples. So I think more and more the world’s gonna head toward what is the problem I’m trying to solve. But maybe more importantly, I can now solve problems that I couldn’t before. And so it’s again, we’ve talked about this quite a bit. It’s on the creativity of the end users.

of of these models to to make sure that you’re getting the ROI and you’re pushing the envelope of what these models can do. And you’re not living a year ago and assuming the limitations that were a year ago and just using the models the same exact way. So not all tokens are created equal. Thinking about what business problem are you trying to solve and I think is is going to be key. And it’s it it’s interesting now we we do almost have these different levels of employees. We’ve talked

traditionally about agents being digital employees. Well, well, now you have to figure out which digital employee do you want to apply to a particular problem. And you may compose multiple employees, your most senior one, f Fable five, coupled with sonnet, coupled with machine learning models. And so that that architecture becomes more and more important. And I’ll just say, you know, maybe we get to this later.

I think all those CEOs that were measuring productivity by token consumption are probably kicking themselves in the behind right now and not looking at the business value.

Shanti Greene (05:01)

I’ve I’ve read stories that a bunch of those are getting walked back after seeing the bills, including looking at our own personal bills last month and going, huh, overages and API based fees really get to you after a certain amount of time and some usage. So that’s definitely a thing that’s happening. I’ve talked to a lot of our engineers and we’re seeing a lot of good results from Fable Five. We’re seeing difficult problems being solved in fewer iterations, so less turns.

Still a lot of tokens, maybe not more tokens, potentially the same number, just being charged at a higher rate. But the user experience is better in that regard. You don’t have to prompt as many times to get to the solution you’re looking for. Now, sometimes it will spin for twelve, fifteen minutes and show you nothing in between other than like still thinking and getting your little anthropic verbs there of like what’s happening. It’s perplexing or burbling.

until I think there’s 150 different ones that were shown in like one of those ⁓ claud code leaks. But it’ll come up with the solution that you’re looking for at end. So in those senses of getting to the final work result in terms of how you’re measuring output, Fable’s doing a great job for a lot of the folks using it. I’ve had good results with it, seems to be thinking well, seems to use a very similar vocabulary and way of putting words together than that Opus did. So I was looking at some output, I was like, it.

Definitely still sounds like Claude. So whatever their company language is, that seems to still be the case. But it is expensive. And I think a month or two ago, I was writing a story about how people might want to use more efficient models in different parts of their workflow to save costs. Like, okay, we can use Opus for planning, use Sonnet for execution. And I believe there’s ways to do that with Fable directly, the way Fable can manage subagents and be like, Fable’s your orchestrator, your planner.

Let Opus, let Sonnet do some of that execution, or go even less expensive, depending on what that execution is. And maybe this becomes a place where ask Fable, how complex is this execution, and have it choose the right model from whatever you predefine as the models that it can use based on the level of effort that it needs to make. So there could be some pretty interesting subagent sub agent flows that we build.

Andy Sweet (07:19)

Yeah, the only thing the thing I would a add to that, Shanti, is it’s interesting now i when we talk about this stuff, it’s so funny to say, you know, way different six months ago, like literally six months ago. And in what we now allow these models to decide around tool selection, ⁓ actually doing reasoning versus very, very deterministic workflows. I’m not arguing that we we get rid of the deterministic guardrails at all.

Shanti Greene (07:30)

Mm-hmm.

Andy Sweet (07:45)

But now we’re starting to see solutions that are in market that we’re putting in market that we’re allowing the LLM to do a lot more reasoning within those guardrails, which are creating much more creative, human-like solutions and emulating and even exceeding what what humans can do in a particular job, freeing the humans up to do more interesting work that’s more suited for them. And so it it it’s just we live in an interesting time, and I think it’s gonna require creativity.

From all of us to take advantage of what we’re being given by Anthropic in this case, and really take advantage of those solutions. And we’re starting to see that.

Shanti Greene (08:22)

I think something really clever that Anthropic did is when they created that goal skill similar and Codex has a goal skill also. I don’t know quite as much about how the Codex one works, is that the Anthropic goal skill uses a different model to do the goal evaluation than it does to do the execution. Because we’ve seen that a model will think that of course it’s achieved the goal. I’m the one who was doing it. I’ve obviously done it. So by using a separate model, you get a much more objective judgment there.

Jim Johnson (08:49)

Nicole, you and I were both gonna jump in. I’m gonna ask you to go and then I’ll

Nicole K (08:53)

Okay, no worries, either way. So I was just gonna come back to Andy’s point about what you can do now that you couldn’t before. And I think that’s the critical thing when we think about ROI as the numerator and the denominator, the costs are almost irrelevant as long as you make that the numerator big enough, right? What can you do now that you couldn’t do before? Are there revenue lift opportunities that you can enable?

You can totally change the game. And to Andy’s point, your imagination is the only limiter here. So you need to know how to build the agent. You need to know how to put the guard rails on it. But if you can imagine it, we can build it.

Jim Johnson (09:34)

Let me play devil’s advocate just a little bit. So I I agree with Andy’s point. you know, in a in a perfect world, you would stop and say, what’s the right model to use here? And you know, you sort of understand what choices you’re making. But that’s a subset of the population that’s able to think that through right now. The combination of confusing pricing plans at the individual level that are changing.

literally weekly, the enterprise level, the API path, the various models,

The frontier model companies are making it harder for the enterprises that they want to serve and that they want to be their customers. They’re making it harder for them to embrace this. And this confusion in pricing and and sort of what you’re really gonna spend. I mean, we all learned the lesson during the the sort of the cloudification area. Yeah, we we’re going to the cloud, we’re going to the cloud, we’re going to the cloud. my God, I just saw my cloud bill.

Andy Sweet (10:33)

It’s like

Jim Johnson (10:38)

And there was a lot of pain and and scars from that era that are now, you know, sort of we’re sort of doing it again at some level. And yeah, there’s a lot of hype over, I think it was Uber, the development organization, you know, they they saw their bill, they consumed the year’s budget of tokens in their development organization in four months. ⁓ but I don’t sense that the frontier model companies are sort of

working yet to figure out how to embrace this and to make it simpler and to get past some of those scars and a and and to to help people sort of figure it out.

And I get it, right? I mean that they need to sell tokens and they need to be profitable. And at some level, eventually we’re we’re the the the you know the subsidies and all these things that are going on are gonna have to it’s it’s gonna have to work out and they are going to have to be companies that are in the black. I get that. but there is a certain level of confusion and trust issues that are being built up in this arena where where people just don’t know what they’re paying for.

And are getting surprised by the bill.

Shanti Greene (11:43)

Yeah, I think Uber is a great example there because Uber was subsidizing for years to get people used to the idea of app based ride sharing. And when they turned those off, prices doubled. And people had gotten used to it though. You’ve like, you’re like, well, the service has been working really well. I don’t want to stop using it. And I’m very scared that that’s exactly the playbook that the company, the frontier models are going with.

Andy Sweet (11:44)

But I can

Jim Johnson (12:05)

But it

Andy Sweet (12:08)

Well simply I’ve been

Jim Johnson (12:09)

But in your Uber

example, I know the price. In fact, I see pri I know the price before I commit to what I’m about to do. And I get it. There’s th these are not these are not good analogs. But that’s sort of so yes, it was a number before and now it’s double that. But in each case, I know that. In this case, it is completely a black box. And you scale that black box a thousand times a minute.

Shanti Greene (12:14)

You do, yeah.

Andy Sweet (12:29)

Yeah, but I I’m I’m gonna go back

to Nicole’s point. I’m gonna go back to Nicole’s point. I think what we’re seeing with our clients or seeing in the industry. I’ll say seeing in the industry is a lack of focus on the numerator. And and all it is, it’s literally we’re turning models loose and then praying. It’s just a deploy and pray, right? And when it doesn’t work, we we end up, you know, stomped. And it worked for a while when the subsidi you know

Jim Johnson (12:42)

I could.

Nicole K (12:51)

That’s

Andy Sweet (12:56)

Tokens were subsidized. But now this is a true, and it’s just basic blocking and tackling. The more the world changes, the more it stays the same. And so there’s a cost to these models and tokens, and you have to make sure you’re getting the value. And so it’s going back to the basics of building business cases. And and so you’re right, Jim. Maybe not everybody’s thinking about the exact model, but I would argue it’s even going back to just the basics of a business case of saying, okay, what are we going to get from this? And let’s only do it if we’re going to get the value.

That makes the investment worthwhile. And it’s not just an im anyway. I

Jim Johnson (13:30)

Well

Nicole K (13:31)

You’re exactly

right. And that’s exactly where I was going to go. I wrote down focus on priority business cases. You have to look at what are all the business cases you can consider. You have to go through that value viability proposition and decide which one am I going after. To your point, you can’t just go after everything and hope for the best. Pick your priorities and go after them and understand up front the approximate the approximate ROI. Game that out.

before you start to the point of the Uber pricing. Guest rate before you go.

Jim Johnson (14:03)

Well, I think there was a

there was a perception that we’re gonna roll this out and we’re just gonna get value and productivity from it. And you may or may not. And knowing whether you did or you did not is pretty hard to tell. ⁓ and largely that was based on the assumption that people will just be more productive. Maybe, probably, ⁓ but it lacks intentionality around your business.

Andy Sweet (14:11)

Exactly.

Shanti Greene (14:19)

Mm-hmm.

Jim Johnson (14:31)

And the technology is a way to deliver now. We’re sort of bleeding into the ROI story in general here, and I like your numerator and denominator thinking. That’s a great framing. but the business problems are still the business problems. We’re we’re trying to sell something, we’re trying to drive margin on it, we’re trying to grow our business, we’re trying to enter into new markets. None of that has changed. AI is a powerful way to improve.

any number of dimensions along the along sort of those things that you’re trying to optimize and and do that maybe you you you could now do it differently than you did it in the past. But that’s where the business case needs to start and drive from that. That’s what I think ultimately the enterprise use case is and and the series of enterprise use cases. And it maybe it starts with process optimization, but you know how you drive revenue lift and new product introduction

All of those opportunities are where the h I think the highest value is, not the hey, you know, Claude can help me with my or or or you know, Gemini can help me with my spreadsheet and I can free up some hours. ‘Cause it’s really hard to capture that time.

Andy Sweet (15:35)

And just think about this. I mean, w no organization would unleash a new CRM or ERP in their organization without massive planning and and thinking about in a business case. But literally we’re unleashing the most powerful technology you could argue that we’ve ever built and we’re turning it loose in the organization without the same level of rigor that we would with a basic software package. And then we’re stunned by the the outcome. So

I I I think it it’s because of the nature of the technology, it’s deceptively easy but it’s complex. And so I think the the strength of AI is also its potential weakness if you’re not thinking in these traditional kinds of ways around business cases.

Jim Johnson (16:19)

We sort of skipped over a little bit the the idea of TCO. And I know that TCO to me is sort of the full scoping of the denominator in in Nicole’s example. And tokens are certainly a hot topic to talk about, but maybe we ought to just broaden the there’s some other things in there that when you when you embrace this technology to drive solutions in your business that are part of that denominator. Who wants to jump in on that one?

Shanti Greene (16:46)

Yeah, I’ll throw in some thoughts there. One is that, you know, it used to be a slow enough development process that you’d be very thoughtful about it. You’d build a very finite number of features. You could test those features relatively easily. And that has really changed. We can now build an infinite number of features, and they’re nearly impossible to test. So the entire process of how do we know this thing is working has gone from, yeah, we’ll spend a few weeks testing it because there was

Only a few things to test to. We’re gonna spend months trying to test all of the different ways that people might use it. It’s very bespoke. You can customize it to no end. it’s really changed the way the entire process of software works because it’s not a solution, one solution for everybody. It is lots of similar but different little solutions that can solve very specific problems.

Andy Sweet (17:39)

And so that’s where the creativity comes in. Right. So so if we if we just say, okay, the way we test is how we always tested and make it human intensive, then then absolutely. But how can we now and we’re you know leverage agents to be that eval? So agents are testing agents at the speed of agents. And and so in in defining what done really should look like. And I think that again is the power of of a model like

fable where it can reason on what done looks like based on intent of the solution, where typically again that would be very human intensive. And I I’m not saying it’s a silver bullet, but again, that’s where the creative thinking has to be applied with these new models.

Nicole K (18:21)

think that’s spot on. think another thing that we need to think about here, and I think Shanta, you alluded to this, is who is that human in the loop that will own this? There is a cost to that human in the loop who will own it because despite the fact that we are building agents that are technology powered, that even do the auto evals, they do a lot of the things that our humans used to do. mean, heck, I used my agent to write a PRD this morning, right? That’s a very human task.

you can use the agents to do all the different things, but you still have a human who needs to manage it, who needs to own it, who needs to care and feed it so that ⁓ it continues to stay relevant. So that’s another thing you need to include in that denominator, that cost.

Andy Sweet (19:05)

I

I I love that point. And I think that’s one again the industry is gonna catch up on is okay, I can’t just put this in into production and then forget about it. It has to be cared for and you know, you call you call it whatever you want, agentic ops, but that that a notion of being able to support that in production, I think is a key to your question, Jim. And I I I love that point, Nicole.

Jim Johnson (19:30)

Agreed. So any yeah, so managing the agent, the the the cost of sort of ensuring that it’s doing what it’s doing. you know, Shanti, you I said I think the the organization’s ability to ingest change somewhere in there as the cost component.

Shanti Greene (19:45)

Yeah.

Andy Sweet (19:49)

I’m I’m gonna be

controversial on ⁓ well, I don’t know, maybe not controversial. By the way, you can tell I’m an all numerator guy in in whatever process cost. Yeah, there you go. but but I think I think the there there’s so much we have to invest in in because I think AI is adopted at the pace it’s trusted. Can you trust it in the organization and building that trust has a cost?

Nicole K (19:57)

It’s okay, I’ll balance you out.

Andy Sweet (20:14)

And and so being able to do that in a way that that’s optimizing that part of the denominator, I think is critically important. And it goes to being face to face in these processes at key points in in the whole development process of an agent up front, in the middle, maybe at the end when you’re doing UAT. And you’re able to tighten that cycle. so so again, I think that’s that’s a big piece of this as well, that that total cost of ownership.

Nicole K (20:43)

love that point and I was having a conversation earlier this week with a customer about what I call the trust gap for that adoption. So we talk about agents and the point of the word agent is that you’re giving an agency. And I think the gap between an implementation of some of our agents versus the agents we truly give agency, where we allow them to make decisions and take actions for us.

is that trust gap. And I think if you look at that percent adoption, the pull through, as people start to first use it as something that describes an action, that recommends an action, as they start to use that more and more and take human action accordingly, that’s when you want to pull that switch to turn it over to full on agent, where you’ve closed the trust gap and now you make it make the decisions for you.

And boy, did you just ratchet up that numerator and the value delivered in your ROI case. But you allowed the humans to come along that loop with you.

Andy Sweet (21:51)

Yeah, a lot yeah.

Jim Johnson (21:52)

Love it.

I I I mean I I think it’s spot on Nicole in terms of that the the idea that the numerator is fixed is erroneous. I mean the the opportunity for the numerator to evolve in the value story as you evolve the trust in your agenc solution, as you

Andy Sweet (22:04)

Exactly.

Jim Johnson (22:14)

manage and optini optimize the denominator side of the equation, tokens being one component, but sort of the agent managed services being another, the the the the you know the the the the change management of it, just sort of all of it. ⁓ you’re continu you I mean you’re sort of describing a world, if I’m hearing you correctly, where you’re thinking about evolving both of the both the numerator and the denominator as you mature simultaneously to continue to optimize the value of that solution, the ROI of that.

Nicole K (22:27)

infrastructure.

Andy Sweet (22:42)

And I think

the most important thing organizations can do is to start to build that trust is all the things Nicole described, but also you have to have the most senior leaders from the CEO on down leveraging AI so they have a sense of what’s possible, how it could apply to their business. I think when that happens, that’s a transformative

pivot in companies. And and and all of a sudden the the sense of urgency gets pushed down. People under the senior leadership see their senior sh leadership using AI. It’s inspirational. And it goes from, you know, being a thing on a PowerPoint with, you know, little arrows up up and left to to real examples of how you can move the business forward. And in in that and again, putting it in the context of the numerator and denominator. I I I love that. And this is

We’re gonna invest and here’s the outcomes we expect. so that personal innovation.

Jim Johnson (23:34)

Well,

a I mean, Andy, your earlier point that says business cases are still relevant and maybe maybe more relevant than ever. you know, and there was always sort of a world where people just said, I can’t I can’t put a numerator on this. I just know this is strategically important. That’s fine. And and a lot of companies let business cases go through like that. And maybe some of the most important business decisions ever made were based on, I just know this is strategically important. But

the process, the thinking all still apply. There’s incremental components to it. And I love Nicole’s point that wherever you start with that isn’t necessarily where you end. Both the numerator and the denominator can evolve over time along the way. So super relevant stuff.

Nicole K (24:22)

Yeah, I think we started out talking about Fable and maybe I’ll pitch it a little bit back to you, Shanti, but I think the ⁓ point about the numerator evolving is how do we take advantage of those more powerful models as they come out? And I think Shanti made the point earlier that we’ve seen early indications that Fable is great at the planning. know, aces in their places use the more powerful newer models

Jim Johnson (24:29)

Yeah.

Nicole K (24:49)

for what they’re good at and start figuring out what they’re good at so that we can continue to increase that numerator.

Andy Sweet (24:55)

Yeah, I don’t mean to interrupt that point. Sorry, Shanti, I’ll let you go really quick. So so it used to be we would say, Where where does AI fit and where do the human? I think what we’re articulating is there’s now multiple kinds of models. So where does these particular models fit plus these kinds of humans? So just like we don’t treat every human the same, or hopefully we don’t, we don’t treat every model the same. And and so I think it’s it’s that. And sorry.

Shanti Greene (24:56)

Yeah, yeah.

Andy Sweet (25:21)

Sorry, Shanti. It’s it’s the Emmett Smith episode, so I’m getting a little excited and I

Shanti Greene (25:26)

Hold on, let me do my Barry Sanders spin move because we’re going with running backs today. And ⁓

Andy Sweet (25:33)

I

won’t

Shanti Greene (25:33)

Clearly relevant. I don’t see what the policy. yeah, I I it gets a lot of that denominator for when you pick the right model is all about the consumption plan too. Are you on straight API consumption? Is there a subsidized subscription you can buy into? Because if you’re just monitoring your usage, you’re on a subscription, most people just stick with whatever the most powerful model is until they run out of credits. There’s no harm in doing it to them until you can’t anymore.

And that’s where some of the new enterprise pricing starts to really become interesting because that’s who’s paying the highest bills already. They’re switching to a higher-tiered plan where now it’s not a subscription anymore. It’s full consumption at a single price. It’s really forcing companies to think about how do we want this to be used? Like what kind of tool is it? Where does it belong? And maybe they will have to focus on.

Which use cases will actually bring us value and like is this a numerator part that makes sense?

Jim Johnson (26:34)

Guys, super interesting stuff. I you know, it’s just continues to amaze me and sort of a as I reflect on you know, every week we’re sort of talking about something brand new and different, but it still does tie back to sort of some tri some level of traditional business principles, which I I think we can’t lose sight of. And I I’m gonna keep Nicole’s sort of

Grow the numerator, focus on the numerator. Andy’s a numerator guy. Hey, I’ve always been a numerator guy, but I think in this I had to play a little bit of a little bit of devil’s advocate. So ⁓ awesome stuff. ⁓ let’s wrap. it’s it’s Friday afternoon and there’s a lot going on. you know, go Knicks, go USA in the World Cup, go North Carolina in the College World Series and maybe Georgia.

You know, there there’s like go Carolina Hurricanes in the Stanley Cup, go Freddy and all the other Europeans that I’m following on X now who are wandering around the USA as a part of the World Cup. so awesome stuff.

Andy Sweet (27:35)

Okay, lightning

round. Lightning round, who wins the NBA finals? I’m gonna go first, Spurs. And this will this will air after. So we’re we’re on the record.

Jim Johnson (27:45)

I hope

not. It’ll cause a billion dollars in damage to New York City.

Andy Sweet (27:48)

Spurs, shanty?

Shanti Greene (27:51)

I was rooting for the Spurs because I grew up too close to New York. So I like Boston sports teams, so I will put root for any team that’s not from New York. So I’m going Spurs.

Jim Johnson (27:55)

Yeah.

my gosh.

Andy Sweet (28:01)

Nicole.

Nicole K (28:02)

Born and raised in New York, baby. I’m going next.

Andy Sweet (28:04)

Nicks.

Jim Johnson (28:04)

the next

Andy Sweet (28:06)

All right. Nicks. All right. Two Spurs, two Nicks. Find out who are is the best procnosticator. Can’t say the word. On episode twenty-three.

Jim Johnson (28:06)

the next

Shanti Greene (28:15)

I I’m using zero

data on this one. I’m it’s really who I don’t want to win. So

Nicole K (28:18)

Yeah

Jim Johnson (28:19)

My

my daughter lives in Manhattan and she and her husband said they can hear without watching the game, they can feel and hear the city reacting to the game. Yeah. All right. Have a great week. Thanks.

Andy Sweet (28:30)

That’s awesome. Yep. See.

Nicole K (28:31)

Reverberating. That’s awesome.

Shanti Greene (28:35)

Alright, so

yes. Right.