AI, Actually – Episode 9: What’s Actually Working in Enterprise AI: Business Value, Success Predictors, and Agent Ops

Joey Gaspierik (02:21)

Yeah, listen,

it’s a good question. especially this week with the stock market and the AI bubble and the tech bubble and everything sort of going down and everybody’s going, there’s this big promise of AI and we haven’t seen any results. ⁓

Jim Johnson (02:35)

Hang on a

second, the market’s bursting, I’m go make a trade.

Joey Gaspierik (02:38)

Yeah, you might.

Shanti Greene (02:38)

You

Nicole Kosky (02:38)

you

Joey Gaspierik (02:39)

Yeah. You’re in trouble, Jim. ⁓ No, it’s this sort of everybody’s expecting to see these fast results. And I think that has been, you we haven’t seen this massive success and that’s being considered a failure or an early failure. But I’d say we are extremely early. I would also say that if you have used any of these AI, LLMs, any of the AI solutions that are out there sort of as a consumer,

You know that there is massive potential. think all these enterprises know there’s massive potential. And what we’re seeing are people are absolutely continuing to have budgets and increase their budgets for these AI projects. So we’re extremely early. don’t, wouldn’t say anything’s been a failure. I think that there’s definitely projects that have fizzled out or they’ve gone down and everybody’s learning a lot as they’re going. But I wouldn’t, I wouldn’t consider anything to be a failure so far. So.

To answer the question, do I think anything’s failing? I don’t think anything’s failing. I think we’re way too early to the game at this point.

Nicole Kosky (03:36)

to weigh in on that. totally agree with what Joey’s saying. I would also add, think one of the things in the market is people sort of think of AI as magical. And I feel like I want to dispel that. There is nothing magic about AI. It is still a project. And we need to focus on the fundamentals. Make a list. There’s lots of opportunities for Agentic solutions out there.

make a list of things you want to accomplish, then prioritize that list. Make a backlog just like you would with any project. Go through that backlog and look at two factors, I think, that we talk about with our customers all the time. What is the value and what is the viability of that solution? And let’s weigh out the value and viability. And I think those customers that are having success are looking at that high ROI for the value.

but also where the solution is viable. The data is ready to enable this solution. So I think, again, it’s just back to business fundamentals. AI is a tool. Agents are a tool. Let’s apply the business fundamentals to use them successfully, to have wins rather than failures.

Joey Gaspierik (04:52)

The one thing Nicole said that is absolutely true is everybody thinks it’s magical. being on with this week alone to the most well-known brands, consumer goods brands on the planet, speaking with specifically in these cases, some of their technical teams. And I think there’s this concept of an easy button when it comes to AI and people looking for an easy button.

And it goes back to what I said, we use these tools, the chat GPTs, the Claudes you know, when we’re writing emails and doing things and it feels very easy. We might even sort of, I I automate certain things all the time, whether it’s, you know, proposals or other small tasks. And it, does things so well that you think, ⁓ this should be easy to scale across the enterprise. But it’s, it’s not easy to implement agentic solutions. I don’t want to, you know,

It’s not impossible, but at this point it’s not something easy. It takes work and it takes skills that don’t exist commonly across the enterprise. And when people don’t have these skills, they go through these projects with this traditional sort of BI thought process, this business intelligence thought process of get all the data in one place and start creating something that you can push out to the business.

And I mean, for eight years here at Answer Rocket, it’s never been data first. It’s always business objective first and anybody in any AI project anywhere knows this. But you have to start there, that requires, and Jim, you say this all the time, the three important skills, you have to have heavy business context, some traditional IT skills, and some AI engineering skills. And you have to have someone that has all three of these skills implementing these agents. And that doesn’t exist a lot at these enterprises.

So whenever they’re trying to configure or implement or create these AI solutions, they do that maybe with one of those skills in mind, but not the other two. And at the end of the day, you get something that lacks business context that business users will look at and go, I can’t use this. It needs to go further or it’s giving me obvious answers or, you know, any of these things that I think we’ve seen with some of the early ⁓ AI projects.

Shanti Greene (06:55)

you get to a place where people say, my AI solution failed, but they don’t even know what an AI solution is. Is it your data science model that failed? Was it a ML piece of work somewhere? Was it a GenAI solution? And did it need to be a GenAI solution? Did you even try to fit the right type of technology for your use case? And did your use case make sense? Was it a use case that was going to deliver some ROI if you invested in it?

Or was it a really cool, nice to have show that technology could work here because it makes this workflow really fast, but it wasn’t a workflow that actually delivered a lot of value to the business. And you’ve got to answer a lot of those questions. So people will say something failed, but not even know what that something is.

Jim Johnson (07:38)

We talk about sort of at a detail level, this idea of deterministic systems versus non-deterministic systems or the non-determinism of generative AI. And it’s unsettling that something in the computer may come back with this answer today and with a little bit different answer tomorrow and even a different answer the next day. Shanti, I’m going to go back to you and sort of, I think that…

It’s just a different mindset about the results that come out of it and where it makes sense, what piece of the technology, where do you apply LLMs, where do you not apply LLMs, and where’s the right place to apply it. I think there’s a sense that in all the experimentation that’s going on out there, there may be places where the technology is being misapplied or at least being… ⁓

not just as it relates to the business use case potentially, and I agree with that, and I think Nicole, you were making that point, and Joey, but also even if it’s the right business use case, where do you apply the generative AI technology and where not? Shanti, you can pick up on that if you would.

Shanti Greene (08:43)

Yeah,

I think about that a lot. And we were thinking about a agent’s framework the other day. And one of our big questions was, well, do you need an agent here or is this a limited number of choices? And you can determine what path you’re going to go down with some simple, like if else logic that are very strict criteria, at which point you don’t need an agent. You just have an expert system that says, if this happens, then you go down this way. If this happens, you go down this other way. That’s a really great solution for a lot of problems.

So trying to fit a language model in to be the brains of that problem where it’s going to look at the context and decide which way to go doesn’t make sense. You don’t need all of that extra compute power when it’s really a very simplistic system that you’re going to look at where you say, did this happen? Great, I’m going to go down this path or this other thing. And you can do lots of complicated things on each of those paths, but you don’t necessarily need a model to be the thing that’s making that determination and doing the judgment of what you should do.

There’s other use cases where it is fuzzy, where you have a lot of inputs coming in. You might need to mix them together in different ways and do some interpretation. And you don’t have strong human rules about whether you want to do that. We worked on a billing system once and they were migrating from, so it’s like a cable company. And when they took over an apartment complex, they would migrate from individual billing to group billing. So some things you would get as part of your plan. And when they did this, had to move all of the individual billing codes over. They were like,

100 or something billing codes on individuals and 100 or something group billing codes. But there weren’t one-to-one mappings. So the first thing we did was say, well, let’s talk to the experts, the people who are doing this day to day. Maybe we can come up with some rules because they wanted a rules engine. Well, it turns out all of those people did them differently. Now, was 2018. We didn’t have strong language models that we were all privy to using. So what we ended up doing was actually building out some

different types of decision tree models and then averaging those models together and trying to create rules and then interpret them through a rules engine. But it was really hard because we would get to places that said, hey, if you did these seven things, then you would go from billing code A to billing code A1. And somebody would say like, but one of those things doesn’t make any sense. Why are you using this step? I don’t know why. I can tell you that there was a correlation between this step and this result in the far end, but there were six other steps after it.

and we had to keep splitting the thing and some information was gained. And that’s how the decision got made. But that’s right now, if you told me to do that, that might be a great LLM use case. There’s not any hard and fast rules and humans have trouble making the judgment.

Jim Johnson (11:12)

I mean, that’s sort of a funny thing because I’m sorry, this is a funny thing because, you know, if you have four people in your organization, in your department who are working on something, chances are there’s variance between them. In whatever the work is, you know, if they’re all sort of trying to do the same thing or generally doing the same activity and you accept that. But in a world where we may bring a language model involved into play to help do the work, we’re unwilling to accept

Shanti Greene (11:23)

Mm-hmm. Yeah.

Nicole Kosky (11:24)

Yes.

Yeah.

Jim Johnson (11:39)

some variation in the answers that come out of the language model. And it’s highly probable that we can get to a place where the percentage variation of answer is within the same acceptable boundaries that we would have with a team of people doing an activity. And I think that’s a little bit of a mindset shift for leadership in terms of the value that we can get out of a solution. Sorry, Nicole, I cut you off there.

Nicole Kosky (12:05)

No, no, you didn’t cut me off. I was going to interject. So I like to think of it as aces in their places. You use the LLM to do the things that it does well, but you use the deterministic code to do the thing that it does well. So I think of a lot of the analytical agentic agents that we build as an LLM sandwich, where I have the front end, where the LLM interprets the user question.

And then in the middle of that sandwich, the meat is actually deterministic code that analyzes data and produces a set of facts. And then the bottom piece of bread, that LLM sandwich, is then taking the facts that are fed from the deterministic code and telling the story to the user. So again, you put their aces in their places. The LLM does a great job with user interpretation. It does a great job of storytelling.

but the deterministic code can take data, do analysis, and turn it into a set of facts that you can rely on so that now your outcome might be phrased a little bit differently. Even on the incoming side, the question may be phrased a little differently, but at the end of the day, the solution, the response you get from the solution is reliable. And to me, that’s the aces in their place is that LLM sandwich.

Jim Johnson (13:27)

I’ll remember that all weekend, that catchy phrase. Thank you for that. It’s like putting a song in my head. Joey, I’m to go back to you here for a second. First conversation, second conversation with a potential client or customer that’s thinking about going down this path, you get a gut feel based on that conversation as to whether or not they’re ready and their probability of success, what their probability of

Nicole Kosky (13:29)

You’re welcome.

Shanti Greene (13:30)

You

Jim Johnson (13:53)

probability of success might be. Thoughts on that, sort of what are some early tells or indicators as to an organization that’s going to have a good outcome or not. And it may just be the use case, but I also sense that it’s the, you know, the individuals, the organization, the leadership, but thoughts.

Joey Gaspierik (14:10)

Yeah, I mean, a lot of it comes down to who are you talking to and at what level or who’s sponsoring the specific project. think a lot of times I look at have they done something already? Have they, you we come into the clients who have no idea what their AI journey would look like and where to even start. And they need help just picking out ⁓ use cases or developing that roadmap, Jim. But a lot of times we’ll come in and people have thought through where would we get the most value?

I think they need some help from a viability standpoint, understanding what’s actually possible. But when you walk in somewhere and it’s got senior level stakeholders who are engaged and folks who have said, hey, here are the areas in our business that we’ve identified that we can absolutely benefit if we were able to automate or apply some sort of AI to, and they need help figuring out, is this possible? How can we do this?

How fast can we get something like this implemented? So whenever you come in and there’s been something that’s been done already, whether even if it’s the first project or a pilot, even if that pilot has quote unquote failed or they’ve learned some things from it on why it didn’t succeed, need some more help. Or if they just even have a roadmap identified and the right level of senior stakeholders, think right there, think, okay, they’re serious and this is somewhere where we can absolutely help.

Jim Johnson (15:25)

Shanti, your thoughts on predictors of success or not when you have a first conversation?

Shanti Greene (15:28)

Yeah, I mean, I think

a lot of times it’s what are they going to measure? Like have they really thought about the business outcome that they’re trying to achieve and how are they going to know that they’ve achieved that outcome? If you’ve got something that’s difficult to measure, something like developer productivity, like it’s great. And LLMs can actually help with that. Like LLMs write good code, or at least they write code that works pretty well, depending on how you want to define good.

But if you’re not measuring that ahead of time, like you don’t have a long history of measuring productivity among your development teams, and you come in and say like, OK, we want to bring LLMs in and we want this to be our business outcome as more productive developers, you’re not going to succeed doing that because you didn’t have a good baseline measurement. You don’t know what it really takes. And even though this is a place that LLMs can succeed, and you could have a really strong use case here, the company is not set up well for success.

So thinking about that, like, what are you measuring? Like, what’s that ROI going to be? What’s the quantifiable outcome? What does your timeline look like? Like, are you targeting something that makes sense, like something within a year? You could probably get it. If you said within three months, maybe you can’t. So do you have any reasonable expectation of the timeline? And do you have a reasonable expectation of the total cost? A lot of machine learning models, data science models, these generative AI projects in general, they’re not.

build it once, it runs forever, there’s no maintenance cost. They’re kind of living, breathing things and they need nurturing. And you need to understand that total cost of ownership, that there’s ongoing things that need to happen. There’s change that happens, there’s drift that happens and you want to make those adjustments. That’s part of these systems. That has to be part of your plan and your budgeting and the process. Who’s going to do it? How much is it going to cost? How quickly does it need to run? I could build a great model.

But if your use case is it has to return in fractions of a millisecond and not one second, maybe it doesn’t work for this case. Like the model works well. It would help the use case, but we can’t hit the speed. Well, LLMs aren’t always fast. So there’s some other features in there where you have to think about, like, do we have the technology? Do we have the infrastructure?

Jim Johnson (17:33)

Nicole, you and I have had this conversation 20 times, which is that care and feeding of a generative AI solution. I’ll let you run with that one here for a minute and I’ll pile on top.

Nicole Kosky (17:39)

Absolutely.

Yeah, for sure. So first I

want to come back to what projects are successful. I want to go back to Joey’s first point about business sponsorship. We definitely find that the customers who have the use case identified with that strong ⁓ ROI, but also bring in that business sponsor who believes in the solution and not only believes in it, but is sort of reiterating to the business, repointing the business.

back to that solution, that is where you actually start to get traction and when you hit your ROI. But to your point, Jim, also agentic operations, like this is, I think something that we’re gonna start hearing a lot more about as we get from sort of these POCs, the actual live agents, to Shanti’s point, there’s drift over time. And not just drift over time, but you have new users come in and start to use it.

those new users interact with it in a different way. You better have someone on there recognizing how to enhance the context for that agent to make sure it’s responding according to the new vernacular that a new persona brought to the conversation. Agent tech operations cover so many different things and is so important to continue to drive the success. It’s not once in live and done.

As Shanti said, it’s actually get that agent tech operations team to operationalize the success, the ongoing success of your solution.

Joey Gaspierik (19:19)

I’ve got two points that obvious ROI. whenever I was answering the question, said, you want to find use cases that are valuable. We want to help people understand. Yeah, it’s valuable, but how can you execute on that? But that, again, the ROI piece is why having that senior level stakeholder is so important. Because I think you’ve got folks who see the ROI, but they’re scared to sort of.

They’re scared to commit it and scared to say, if we do this, then we’ll see this ROI.

Jim Johnson (19:45)

I

that’s sort of American business or global business. People are afraid to sign up and put their name next to ROI. But yes, and now this is a further manifestation of a 50-year problem. Who wants to put their name next to the return on investment? Sorry.

Joey Gaspierik (19:49)

Yeah.

Absolutely. So you go up a couple of rungs and get somebody to see if we can do this. It’s obvious. I’ll give you an example. I’m working with a large ⁓ manufacturer and they’re looking at skew rationalization use case and Shanti, we worked on this one together. We put together a proposal on an advanced modeling use case where we were basically going to help them. They’ve got like a hundred and a couple hundred thousand SKUs and they want to understand which ones can they eliminate from their portfolio.

and basically improve their working capital. That’s what they’re trying to do. And we put together a proposal to show them how we would do it. It was an opportunity where we felt very confident we could help them do that. They saw that, they saw the plan, they believed in it, but then they said,

then we’d have to actually go and execute on this. We’d have to go and implement the strategy. We’d have to go and make sure every single one of our salespeople globally didn’t sell this SKU or didn’t promise it to a customer. And that they didn’t have an idea of how would we actually go and implement that. So it’s one thing to say, hey, here’s the ROI and believe in it and we can answer the question that you asked us and propose on it and believe we can build the solution.

But at the end of the day, now you’ve got new workflows, you’ve got new rules, new business rules that you have to go implement and govern and make sure. And again, that all leads back to top down. If your CEO is saying, absolutely not, these SKUs are off the table, if you sell them, it will be impossible for us to deliver. You have to have that sort of mandate coming down from the top whenever you’re instituting these types of changes, or else you’re not going to realize that ROI. So again, it all comes back to me to that senior level stakeholder.

Jim Johnson (21:39)

Let me pick up, I want to pick up on Nicole’s point on agent ops, agent operations. Agent ops sounds much cooler. It like we’re really doing something dangerous in our line of work. know, Joe, you and I were in this same conversation, global CPG out of the UK, early in the dialogue. And there’s sort of two parts to this. Having a conversation, getting a client excited about the possibilities.

of enterprise use cases powered by what agents can do. Do work that you’re doing today, do work that you wish you could do, do work that maybe you haven’t thought of, and the value that that can be both in terms of efficiency, maybe cost takeout, and maybe revenue lift. But that was sort part A to the conversation, so we can all get excited about it. Part B to that conversation is you are bringing in something new.

I don’t know what you want to call it, but it’s effectively a digital employee. It’s not replacing an employee. It’s not a one for one. We’re nowhere near that right now. But you’re bringing in sort of a new style of how digital work gets done. And someone needs to supervise it. Someone needs to own it. Someone needs to check on that employee. Someone needs to own the work getting done correctly.

And Joey, it’s exactly what you said before. It’s a combination of skills. It’s some IT skills. It’s some deep business skills about what you’re trying to do. It’s some AI skills. And I think that we both collectively said to that company at the time in that discussion, either you’re going to have to do this, you’re going to have to pay someone else to do it, or we’ll do it. You’re going to pay us to do it. But it’s got to be done because it is not a set it and forget it.

kind of activity for sure and it evolves over time.

Joey Gaspierik (23:22)

And that honestly goes back to the easy button thing. People think, I hired this agent or I build this agent and I set it off and then I move on and it goes and does this magical amount of work. And will that happen? Like I don’t, I probably in the world of AI, I don’t, I don’t think anything won’t happen anymore. But it’s not happening right now. And I think again, it goes back to

We don’t have the skills to do that. And in an IT world, it’s the traditional IT mindset is build it and release it and move on and doesn’t exist here. And that’s where a lot of people then start to chalk it up as well. It’s too hard. maybe it’s just we’re too early. But absolutely these skills exist. Companies like Answer Rocket exist to create agents.

and continuously make them better until that technology catches up where it can sort of govern itself and start to do these things on its own. Well, there’s a future probably where that lives. I Mike Finley could talk about that for hours on a podcast. But right now it’s not. And hard does not mean failure. Hard just means there’s work to do. And there’s work that not a lot of people know how to do. And that’s why firms like ours are seeing a lot of success.

Jim Johnson (24:34)

Shanti or Nicole, this idea of sort of constructing these solutions and putting them in production, talk a little bit more about how it’s different than traditional deterministic systems because it mean all of us, we’ve seen it, we know, we’ve sort of seen the, we’ve seen successes, we’ve seen failures as a result.

and sort of we’ve seen that this idea of drift and things wandering away pick up on that a little bit and give me your thoughts.

Shanti Greene (25:02)

Yeah. And in a couple of ways, it’s actually similar in that even those deterministic systems also drift. They drift for different reasons. They drift because the world changes. So the same relationships that existed previously no longer exist or the relationships mean different things. Now that can happen to LLM, agentic generative AI based systems as well. But those generative systems drift for lots of possible reasons. I think Nicole mentioned you different users.

come up with different ways to talk to them. So now you’ve just got a whole different set of inputs and you might be doing something like few shot learning, something that will help kind of ground your answers, make them more consistent. But when users start talking to your models a little bit differently and prompting them differently, unless you’re doing like some P tuning or other things that take the user’s prompts, pre-process them and then feed them into the model, well, you’re gonna get different inputs. So, you know, I’ve thought about…

different technical ways that you can try to avoid that and things you can do. And very few people actually do that type of like P tuning, which I find to be interesting that it was something fun to talk about a couple of years ago and then nobody really picked up on it. But LLM is a great at writing prompts. So you could do it the simple way, which is take the user input, have a different LLM, turn that into a prompt and feed that back in to whatever your source system is. And you’d get a little more consistency, but then, Hey, they release a new model.

Like new version of the model comes out. Well, that new model is going to be better in a lot of ways, but it’s not going to be exactly the same. And because it’s not exactly the same, it will like lead to some inconsistency in your system. It’ll just be different than what you came to expect. And a lot of what we see are really expectation violations. You train something, you get it working, you expect it to just keep doing whatever it did when you first started. And it’s not going to do that. So you shouldn’t expect it.

But you do, it’s kind of human nature to just want it to be the same.

Jim Johnson (26:50)

wouldn’t expect it of an employee that they would always do it the same way.

Don’t expect it of a generative AI powered solution to always do it the same way.

Shanti Greene (26:59)

But

Jim, that’s logic and people don’t actually, they want to apply the same logic to people and to systems.

Jim Johnson (27:05)

You know, we have the opportunity to work with very large organizations. We have the opportunity to work with sort what I’ll call mid cap organizations. I think I’ve said this before. I mid cap organizations, this is a unique time in history where they have a technology available to them that will actually allow them to both compete with much larger competitors and to potentially leapfrog because they’re not encumbered by a whole host of legacy technologies and solutions that we could talk about here. But often,

Shanti Greene (27:26)

Yes.

Jim Johnson (27:33)

those mid-cap organizations are earlier in their journey and often very immature in their journey. And one of the first things that’s sort of interesting to share with them is this idea of don’t think of it like a system or a technology, think of it like a person. And sometimes you can see the light bulb come on and then they change over and it is a sort of dramatic difference in their thinking in the room.

about the possibilities once you make that leap. Sorry, Nicole, I jumped in front of you again there, but I wanted to get your take on some of that same topic.

Nicole Kosky (28:07)

Yeah, absolutely.

I think even tying it into that last point you made about a person, the big difference to me between POC versus live production implementation agent or agentic solution is that start small. When you’re training a new person, you give them one thing to do first, and then you add on to it and you add on to it and you add on to it. I think the same is true here.

Jim Johnson (28:31)

Totally agree.

Nicole Kosky (28:33)

take the smallest thing you can bite off, get that live, hand that off to agentic operations to now manage, and then start the next phase of your project and add more intelligence to it, build that on, hand that off to agentic operations, and just keep building wins on top of each other. I think that’s the way to get to successful production-ready solutions.

Jim Johnson (29:00)

I love it. Minimum viable, valuable work. I’m making this up as we go along, which is sort of tradition for me. The MVVW, minimum viable, I don’t know. Somebody will figure that out. I think you’re spot on with that. Sort what is the smallest piece of valuable work and then let’s keep adding to it. I think that’s spot on, Nicole.

Nicole Kosky (29:03)

Yeah.

Shanti Greene (29:22)

I was going say something we were seeing in that recent Wharton survey that came out where they talked to a whole bunch of tier one, two, and three companies about who’s being successful with AI and what’s not being successful is they’re seeing that successful teams are using that phased approach and they’ve got clear stage gates. So you do it, you train it to do one thing. So you build a model, you work on a use case, do something small, show that it works, move on to the next one or improve it, do better.

Joey Gaspierik (29:22)

ahead.

Shanti Greene (29:48)

going from phase to phase in small incremental steps is much more successful than trying to do these big bang implementations.

Joey Gaspierik (29:55)

Yeah. And that sort of plays in actually nicely to what I was going to mention sort of on your point and your point, Shanti and your point, Jim, the mid-market conversation around, you’re talking about a lot more immaturity in terms of, I don’t know, tech stack and sort of current processes. And I think that that is such a big advantage for these mid-market companies because now there’s less sort of hurdles for adoption because, well, where’s that?

Shanti Greene (30:15)

Huge.

Joey Gaspierik (30:21)

Where’s my dashboard that I looked at every day? And this doesn’t look like that. And usually I get this chart from Bob whenever he sends it. And where’s that? No, no, no. All of a sudden we’re just automating the thing that they didn’t have before. Or that was in an Excel spreadsheet that nobody really even knew how to do. So only 5 % of the people open it up. But now we’re getting to the actual insights that they’re looking for. And now we’re really breaking down those adoption barriers. I think, and then the next thing is, all right, you do this phased approach like Shanti mentioned. You can say, okay,

Mid-cap clients, let’s phase this out. Let’s start here on maybe something that’s going to be highly valuable that we could do very quickly. And there’s five phases to implementing this larger use case. Each one of those phases, if done right, could have massive benefit to your company to where the whole project’s paying for itself times 20, right? And this is really where I see these mid-cap companies having a massive advantage right now in this.

Jim Johnson (31:15)

Huge opportunity.

Joey Gaspierik (31:17)

huge opportunity

and you said the word leapfrog and I absolutely believe that’s true because they’re going to be able to serve as clients better than their their customers or their competitors in this case.

Shanti Greene (31:25)

They’re, they’re

already moving faster and adopting AI technology much faster than their tier one competitors, because they just don’t have the legacy things standing in their way. And they know they need something to give them that advantage. So they’re actually looking for what’s next. What could we be doing? A lot of the larger companies are very complacent in here’s where we are. Here’s how we’ve always done it, which is a terrible reason to keep doing things a certain way. And they’ve built up all of these gatekeepers around.

not doing things, that they stopped to forget about how they got there, and then what’s possible.

Joey Gaspierik (31:58)

A hundred percent.

Jim Johnson (31:59)

I don’t want my

job to ever be gatekeeper on how to not do things.

Shanti Greene (32:03)

You

don’t want to work in IT.

Joey Gaspierik (32:06)

Yeah, the

Jim Johnson (32:06)

Ha ha.

Joey Gaspierik (32:06)

famous foe here, if you want to talk to Pete Reilly about it, it’s the AI prevention department. These massive companies all have one and we all run into them. go, you got a group that’s excited. You might even have a senior stakeholders like we’re absolutely going to do this. We run into that AI prevention department. It’s, it’s donezo. You got all these politics and you got, well we can build this. We can build this. You know, we’ve got the team to do it. And that is going to allow these mid-market companies to catch up and leapfrog.

Jim Johnson (32:11)

Yeah.

Guys, we’re gonna bring it home here. I’m gonna go around the horn. One or two sort of core recommendations you would make to an organization if they’re starting their journey around how generative AI and AI in general might enable them or even if they’re sort of well on their way. I’ll go around the horn. Nicole, you go first.

Nicole Kosky (32:53)

I’ll go with, to me, the two key things, back to business fundamentals again. First, build a list of opportunities and then rank them by value and viability. then second, I want to start small and add on. Third, I want to have a business sponsor. So make sure your priority use cases have that business sponsor so that they have the right support within the organization.

someone who can even stamp out that AI prevention department that Joey mentioned.

Jim Johnson (33:25)

Shanti.

Shanti Greene (33:26)

Yeah. So one, I’m going to say companies should be investing disproportionately in data quality and data governance. And I don’t usually say I like data governance or governance in general. You know, yeah. But data quality is so important and the governance teams are often the ones who are actually making sure that data is high quality that really investing in that at an organizational level and making sure that you actually have the information to make decisions is going to enable all of the things you want to do down the line.

Jim Johnson (33:36)

You’re like a big data guy.

Shanti Greene (33:54)

So do that, and then pick a use case, any use case that has measurable ROI.

Jim Johnson (33:58)

Alright Joey, you were first up and now I’m going to give you the final word today. What do you got?

Joey Gaspierik (34:04)

Yeah, I mean, it goes back to a couple of things that I think both Nicole and Shanti already said. You have to have that ROI. You have to believe in that ROI and you have to get that.

aligned ⁓ to a senior stakeholder of the company, if not the CEO. And then you have to have an expectation, a realistic expectation about what it’s going to take to deliver on that project and ⁓ on go. And you can’t be looking for the quote unquote easy button. You have to know that it’s going to take some work. It’s going to take a handful of weeks. But if you pick the right use case and you implement that correctly, you can achieve that ROI.

Jim Johnson (34:38)

Joey, I totally agree. It’s probably worthy of another podcast at some point. I think there’s a gap. I think there is a gap, especially in large enterprises, between sort of how IT is thinking about it and the aspirations and goals and hopes of the business. And these two have not found a way to meet in the middle.

And IT is trying to treat this as sort of some traditional infrastructure. We can stand up some enabling technology, we’ll roll out chat, GPT, and they’re not sort of going far enough to meet the business. And I don’t know that I have the answer right now. This is probably worthy of more discussion. then the business, there’s just so many hopes and dreams tied to what AI can do for them. But they don’t know how to sort of get far enough over to the IT guys.

And there’s a conversation to have because so much of what we’re talking about today, the context, the prompting, the data, it sort of lives in the middle of these two worlds. And we talked about the need for the intersection of AI skills and IT skills and business skills. And I don’t know that I’ve ever seen a sort of a greater need for those groups to come together than right now and then add in the AI skills.

I’m not sure I’m sort of giving a recommendation there as saying there’s a gap. And we keep seeing it over and over and we’ve seen IT organizations fail and not get far enough. And I think that’s part of the disappointment that the two groups haven’t found a way to meet in the middle. So, all right, guys, it’s Friday afternoon. We did it. We knocked out number nine here. Thanks so much. Joey, thanks for joining as a first timer. Shanti and Nicole, thanks for coming back as always.

Shanti Greene (36:18)

Absolutely

Jim Johnson (36:18)

Bring in the energy and hopefully we’ll see you soon on number 10. Thanks so much

Shanti Greene (36:23)

Sounds good. All right,

Joey Gaspierik (36:24)

Thank you.

Nicole Kosky (36:24)

Thanks

Shanti Greene (36:24)

see

Nicole Kosky (36:24)

everyone.

Shanti Greene (36:24)

you. Bye.