AI, Actually – Episode 20: Is Your Data AI-Ready? The Semantic Layer and the Last Mile Problem

And we’ve never talked about that one thing that we should probably talk about, and that’s data and how that’s a part of the story. Maybe I’ll tee it up this way, thinking about it little bit. We seem to run into maybe two extremes. There are those clients and businesses that we work with who feel like data has to be perfect. We have to do everything we can with our data before we do anything else.

I’ll see you in a few years. Or at the other end of the spectrum, and we run into this one too, is can’t we just chuck everything we got all the data we have and the LLM is so brilliantly ⁓ intelligent that it’ll figure everything out. And neither extreme really seems to be the right answer. So that’s sort of the topic today is where we are in this journey. Let me tee up. We’ve got several regulars. Our friend Nicole Kosky.

⁓ Andy sweet and we’ve got a new participant today. Ben Titmus, Ben actually leads the data practice focused on data and infrastructure and data platforms, at AnswerRocket, So we thought we thought we would throw him into the fire. In fact, I’m going to throw them into the fire right off the bat. Ben hit me up with those two extremes and, maybe just talk a little bit about sort of what you’re seeing or what you think’s different now.

Benjamin Titmus (02:07)

Yeah, I mean,

I would say last year we had a lot of big numbers come out just around failures of AI projects, right? I think the number one that I quote is 87 % of AI projects never reach production. And I think what we’ve come to the conclusion is a lot of that is because of data, right? AI systems fail when they don’t understand what your data means. And this concept of the last mile problem, which is the semantic layer that I think the world is starting to get some traction on in terms of that language.

You know, that tied to the data itself is really where we’re seeing the rubber hit the road and the AI agents actually being able to do work reliably.

Jim Johnson (02:46)

Yeah, but I’m a client. I can’t do anything. At least that’s what I say. My data’s not ready. Somebody jump in on that.

Benjamin Titmus (02:53)

Yeah, I mean, I don’t,

Well, I mean, don’t think

that that’s true either, right? I think there are ways to essentially do projects iteratively. know, it used to be that you’d have to build a massive sort of data warehouse that would take six to 12 months with a large team to develop. Now you can be much more targeted. So if you were to take a pilot or a use case that has immediate ROI for the business, you can start to build the data foundation for the future.

⁓ while proving out that use case, right? And you can do that relatively quickly, you know, in terms of weeks, not months. And, you know, these modern data platforms that exist today, if you look across, like I would say the four main ones, right, you’ve got Microsoft Fabric, you’ve got Snowflake, Databricks, and then Google BigQuery, they’re all starting to get this, right, that, you know, you really need to have that strong foundation and that semantic layer to make agents work.

They’re making it really easy for folks to plug and play and build sort of iteratively and kind of turn on use cases as you go.

Andrew Sweet (03:58)

Yeah, and I would just say, well, before I say this, feel like, you know, we talk about this all the time. It’s almost like we’re hitting a tape recorder and hitting play. Start with the business outcome. And so when you say, is your data AI ready? Is it ready to achieve this outcome? Right. Versus it’s almost like asking you when people say, is my data AI ready? It’s like asking if a hammer is nail ready. It just depends on the task. Right. Are you hanging a picture?

I don’t know if that analogy is perfect, it just, it’s what came to me. came to me on this afternoon. So, you know, the key is to start with that outcome, understand what data is required to achieve that outcome and Ben, exactly to your point, build that use case and then use the business outcome, whether it’s increased revenue, cost savings, whatever the case might be to fund the next wave. And so you constantly have this use case.

Jim Johnson (04:27)

like it.

Andrew Sweet (04:52)

AI platform, data platform, working in tandem parallel and building out those use cases.

Jim Johnson (04:59)

you know, Nicole talked about value and viability and data readiness, but our clients are sort of consistently of the view that, Whoa, whoa, whoa, my data is not ready. Now, sometimes I think that’s sort of an excuse to not jump in and sort of figure out this big, bad, scary AI thing. ⁓ but maybe we could double click just a little bit on the idea of

quote, data readiness. What does that really mean? Let’s, let’s pull that apart a little bit.

Benjamin Titmus (05:23)

Well, I do think there are differences in

how AI is leveraging data towards what was done traditionally, right? If you think about traditional data warehouses, they were really built for BI dashboards, right?

And AI, for it to work, fundamentally needs ⁓ some different things. And some of it is around how you store the data. And some of it’s around the metadata that you’re supplying to it. But really, it comes down to a couple of things. ⁓ First, you want to be able to tie the facts that you’re getting to what we call data lineage or data providence. So where is the data coming from? So exactly what was the field, and what table, and what database, and what

Calculation was done and being able to track that audit trail all the way through the work that the AI is doing To the final output is very important especially as you think about working in heavily heavily regulated environments some of our clients are You know asset managers and financial services some of some of our clients are health care clients that are dealing with You know very prescriptive compliance needs and we they need to be able to track that audit trail

I also think that data from a metadata standpoint, when you’re talking about AI, it needs to be time aware, right? So AI models need to understand what data is available when, and that allows it to essentially start to do pattern recognition and understand when data might be stale. There’s a lot of transactional systems today where data has persisted and you don’t really have a time series. You don’t necessarily know.

when it was last updated. So being able to understand how data changes over time in systems is very important for these AI capabilities. And then I think there’s a couple of other things you got to think about. with that sort of time-aware structure, think AI really wants real-time data.

not necessarily batch processing, is where BI heavy platforms really leaned in and back in the past. I think you were also able to do more multimodal unstructured structured data type things. So you’re thinking about text, images, embeddings. This is very different from relational databases of the past. And then also, I think what’s important is this concept of the semantic layer, which I touched on earlier.

At the end of the day, humans are using these AI capabilities and language not always prescriptive in terms of what somebody is saying and it can, contextually can be taken in different ways. So being able to train your own eye on what the language of your business is and, and force that via a semantic layer is important for the AI to understand the context of your business and how it should actually interact with the data that you have.

Jim Johnson (07:59)

By the way, the phrase or the term semantic layer is one of the top six things listed as something people have no idea what the heck we’re talking about when we use that phrase. So maybe let’s pull up and define that just a little bit because it’s thrown around all the time and it can be a little bit confusing, particularly from people who are outside the AI bubble.

Andrew Sweet (08:26)

Yeah, so let’s talk some concrete specifics. So the semantic layer is not just your data, it’s how you use your data. It’s the knowledge of the business. So a concrete example might be, you know, when you ask a large language model a question like, how are we doing against our competitors? Well, let’s say you have competitors in one segment that may be medical wear, scrub suits, et cetera, but you’re also in the athleisure market as well.

And so it knowing semantically where you’re at in the conversation of where to go pull data in the context of the conversation, when you said, how are we doing against our competitors? It was in the medical segment. And so it knows exactly where to go and pull that data to understand those competitive pressures and then what good looks like. And so it’s giving you those rules. And then even beyond that, starts to give you thresholds around.

You know, if something happens at this level, it goes to Ben. If something happens at this level, it goes to Nicole. And so it goes beyond traditional semantic layers that we built for dashboards, like Ben was saying, where there’s humans on top of it. And now we have to make them machine readable for an agent to be able to take action. So it’s that embedding of the knowledge of your business so that models can act or agents can act on behalf of your business.

Jim Johnson (09:48)

Ben, you agree with that definition?

Benjamin Titmus (09:49)

more simply said, you know, it’s the language that your AI needs to understand your business. Right. And I like what Jim says a lot at our company and on these podcasts, which is if you think of AI as a really smart new hire, you wouldn’t give them, you wouldn’t just throw them into the company and say, Hey, good luck. You’re giving them training, right. ⁓ You know,

The AI deserves the same onboarding a full-time hire would in terms of understanding what urgent activities are, what does revenue mean in my department, who needs to be looped in in certain decisions, and how do different departments interact with data in different ways.

Jim Johnson (10:24)

Is building the semantic layer a one-time activity? Is it a forever activity? How do you get somebody going on that front?

Andrew Sweet (10:34)

So the beauty of the semantic layer is you get massive lift from an initial take. So it’s just good old fashioned traditional consulting going in and understanding what your processes are. So even what we’re seeing just empirically at a construction company is going in and just having them define what their processes are and realizing where there’s gaps and where they’re doing things inconsistently across the enterprise. And so even doing that,

that exercise of defining the semantic layer can identify gaps in your business that humans can take advantage of even before you start to build your agents. you know, the challenge we’re trying to avoid, you know, we used to say this all the time, like garbage in, garbage out. The problem with large language models is it’s not garbage in, garbage out. It’s garbage in, plausible, but wrong out. It sounds decent. It sounds almost right because they’re able to

operate at that level, but it’s wrong. So getting that layer right and not only that, but also defining the guardrails of what a model can deliver as output. So let’s just do a quick example. Maybe it’s an LLM powered HR assist, where you can ask it questions and that’s all fine and good until maybe somebody asks the question,

Nicole (11:46)

Thanks.

Andrew Sweet (11:59)

Why did Jane get promoted and I didn’t? All of a sudden it comes back with salary information, upside potential. It answers the question very truthfully. It did, it does. It’s answering truthfully just to the wrong person. That’s just another quick example.

Jim Johnson (12:10)

It would certainly create excitement in the company.

I think Ben sort of hit on it a moment ago, but we also get companies who simply adopt the view that says, hey, I can give it access to all of my data and it will magically make sense of that. And the short answer is it won’t. And that can be a tremendous point of frustration, I think for…

companies and people who maybe don’t have the understanding or maturity around what this technology is and isn’t right now. And the fact that there’s still work to sort of marry up what AI can do, what large language models can do with the available information and sort of what companies are aspiring to. there any comments or reactions about that gap?

Nicole (13:10)

jump in on that a bit. So I think one of the things that people need to realize is that ambiguity is the enemy of LLMs. If you throw all the data you have at it with very little context to go with it, you’re gonna get answers, but you’re not gonna know if those answers are accurate or if they’re hallucinations. So I think one of the risks of going at

hey, I’ve got my LLM, I’m feeding it all the data I have, is misleading answers, but with confidence. And I think that’s one of the things that a lot of people run into when diving in too fast without really understanding what are the specific use cases and what is the data related to those use cases and whether that value viability threshold is met.

Andrew Sweet (13:58)

Yeah, the other thing I would I love this point, Nicole. The other thing I would add is some some companies have made tremendous investments, for example, in machine learning models, right? And they can make all sorts of predictions around forecasting, loyalty, et cetera. And if you build a semantic layer appropriately, the model knows when to leverage those machine learning models. So when you ask it a forecasting question, it takes the output from a model that does forecasting and it doesn’t try to forecast itself.

And so you start to see investments you’ve made in capabilities start to be leveraged in a better way because now you’re democratizing that output from a machine learning model in a way that now can be interacted with. So it’s not just data, it’s taking advantage of capabilities you’ve built within your infrastructure that you can now take advantage of in a holistic workflow.

Benjamin Titmus (14:49)

I’d love to touch more on

also think that what the semantic layer can do.

is build constraints that govern what the AI is allowed to sort of make up or needs to leverage one of these capabilities you pre-baked, right? So what tools should be called to pull in certain numbers that you need to be the same every time that somebody is asking that question, you can build that into the semantic layer and make sure that the AI is calculating that appropriately. Because these things can be wildly creative. And I think the more that you build those guardrails around what your business logic actually means,

and you wanted to say, you know, I would put this akin to back in the day when you had to do a board report and there was a ton of massaging of the data and storytelling that you’re layering on top of that so that, you know, the deck is saying the right things. I think it’s very similar here where you need someone to extract that business tacit knowledge.

out of the people’s heads that used to curate these things and then build it into the guard rails that the AI can sort of utilize to play within a walled garden of how you expect it to behave.

Nicole (15:57)

I think that’s a great point and I want to add on to that, Ben. I think I always, I like to use the term ACEs in their places. And I like to think about when I need deterministic math, should I be using an LLM for a prediction for that? Or should I actually be using code? And so to me, you need to have the right framework as well, the right scaffolding for your agent to make sure you have the LLM where you need to use the LLM.

for optimum results and you need to use deterministic code pointing to the right data to deliver the factual results that the LLM can then describe with the creativity and the contextual understanding of your business appropriately.

Andrew Sweet (16:43)

Yeah, the interesting or the ironic thing is now AI assisted or agented coding help us build those deterministic guardrails much faster. So you almost get in weird meta kind of thing where you’re using LLMs to write the deterministic software that protects humans against LLMs in an enterprise context.

Nicole (16:54)

Yes.

Benjamin Titmus (17:03)

Well, also you don’t to build it yourself anymore. mean, last year,

a lot of these capabilities you had to build custom. And now those platform providers that I mentioned earlier, a lot of these have pre-baked capabilities to not only build a semantic layer, but also to implement the guardrails that you’re looking for in terms of your agents and other analytical capabilities that you’re building out. But they’re all sort of taking a different path. If you think about Microsoft Fabric,

They’re saying, hey, we already got a bunch of people on Power BI. Why don’t we just try to recreate that sort of the rules, actions, entity relationships that you did from an analytics standpoint. And that can now be the starting point for AI. there, you know, if you’re in that ecosystem, a lot of people are taking that route.

Or if you look at Snowflake, they’ve sort of got an auto discovery capability. So they’re looking at how the schema sort of interacts across tables and across systems and how is data actually being utilized. And that should be the starting point for your AI building capabilities. Databricks, totally different, right? They’re sort of saying, hey, we’re going to create an open book for you and you’re going to be able to define it all yourself manually. And we’re not going to…

really try to layer anything on top of that. We’re just gonna make it super easy for you to have a custom solution there. And then Google sort of has their own way of looking at it, right? So they’re looking across sort of their productivity capabilities, what they’re doing from an online perspective.

and they’re using the of the vast amount of knowledge that they have to create autonomous embeddings on top of your data. And then that’s your starting point for what you’re doing with them. So they’re all sort of taking a different approach. But the good news is this is a multi-billion dollar problem that you don’t have to finance yourself, that you can leverage a lot of the capabilities that.

Andrew Sweet (18:55)

I’m going to push back on that. I do think Snowflake and Databricks, et cetera, they are definitely investing in this. If we’re really talking about agents that are going to act, you have to pull knowledge out of people’s heads and then get that in. So there’s no silver bullet. And the thing I worry about is vendors make it sound like a silver bullet. just, you you go and auto discover. Well, you can auto discover some small percentage of what you need.

to actually make an agent act. The rest of it is really understanding how your business works so that an agent can operate within that context. so, you know, I, again, I love the fact that Snowflake and Databricks and Microsoft is catching up to us at Answer Rocket and they’re helping do this a little bit, but we still have a lot to do in the industry. so.

Jim Johnson (19:45)

I mean, the last mile is

incredibly complex and nuanced and there’s so much knowledge that’s in each person’s head and more often than not, it’s not all in one person’s head. It’s spread around and the way business gets done has grown up organically through trial and error and a thousand other things or a million other things over time. And it’s really hard to capture all that.

It’s possible. And you sort of start small, start narrow and focused on focus on a place where an AI powered solution makes sense. We should talk about the spotters guide at some point in the future. We’ll come back to that and, and, where value can be derived, but man, it’s hard to pull it out. ⁓ what’s let me, me, let me sort of come back to what’s separating those who are making it, who are sort of getting there right now. If we could sort of keep a data lens on this, if we could.

those companies who are achieving some success, getting return on investment, those that aren’t, what are some things that stand out to this group?

Andrew Sweet (20:50)

Yeah, I’m just going to go back. Yeah, I’m going to sound like a broken record, but it’s the ones that are starting with business outcomes. And it sounds so easy, when you have vendors out there announcing new semantic layers or entropic or…

Nicole (20:50)

I

Andrew Sweet (21:06)

Open AI dropping new models. It’s easy to get distracted by all the squirrels in the trees and lose sight of what you actually have to focus on. that is business outcomes. And it goes directly, Jim, to your point of starting small, starting with something that adds value and then keep humans in the loop. know, so, for example, if you’re doing something that reconciles how you should be paid commissions, for example, you can alert when the commission

rate is out of whack and take no action. Then as you build confidence, you can start to have that agent actually take action and start to resolve that commission problem on your behalf. And then even handle escalation. So it’s starting small. It’s letting it act just like you would a junior employee, right? You’re not giving them the keys to the kingdom right off the bat, because again, think of your LOM is that junior employee and you don’t want to give that person

know, full access to every system you have at admin level. So, so it’s, it’s starting with the business outcome is what I see people and then staying maniacally focused on being able to pivot, not getting locked into, okay, we set this direction and by goodness, we’re going to go down that path come hell or high water. I don’t care what happens in the market. I don’t care what models drop. We’re just going to just keep our heads down and march off the cliff.

Jim Johnson (22:28)

Andy, that just, by the way, I think that lock in point is a really interesting one and we’re at a place right now. And I think for the foreseeable future where I’ve said this many times, I think on this podcast that ROI ultimately is the only future proofing. And there’s a great chance that whatever you do today.

you will be able to do 12 months from now less expensively, better, faster, cheaper. And that’s okay. That’s the journey we’re on. This is sort of an unbelievable curve of capability, of revenue opportunities, of cost flattening opportunities. That’s awesome. But which means you want to do things now that are going to deliver value quickly. And they’re all over the map. They’re there to be had. But

but that way you have no regrets down the line. I do think though there is some risk of our clients and companies out there looking for the easy button, which says, I’m gonna pick everything from one vendor and I’m done and I’m rolling it out. And that’s just not gonna cut it right now because values unlock through putting these things together in just the right way with.

Bluntly with, again, of focused on the business value, the business opportunity, the business use case, and sort of take what’s there. Start with the business side. Back to your point, Andy. So I’m sort of coming back around a lot of words to say I agree with you. Other thoughts?

Benjamin Titmus (23:49)

I mean, also, mean, just to throw

into that, these tools are so like obsolete, so fast, you know, used to be you could, you could pick a platform like you’re mentioning there, Jim, and it would be around for the next couple of years. these, you know, next month, whatever I decided to get switched out for another model or new vendor. Yeah.

Nicole (23:51)

Yeah. Go ahead.

Jim Johnson (24:08)

what’s

causing clients to freeze.

Nicole (24:10)

100%. And I think perfect example, we just did some benchmarking across five different models using the same agent. And here’s where the scaffolding matters. Your scaffolding should allow you to throw out one model, replace it with the next, throw out that model, replace it with the next. You need to be able to iterate on that. And with the five different models, significant difference in the business outcome.

the value of the business outcome across the five different models and each one leapfrogs the other. So you have to make sure you have your data correct, but also your context. know we touched on this a bit earlier. Context in your agents is sort of the new institutional knowledge. You used to have institutional knowledge spread across 10 different people for a particular problem or hundreds or thousands.

you need to have that context in your agent to make sure you’re surfacing the right answers. And that context needs to be relevant to the data that it’s delivering. But again, I think it’s the key thing is keep that context updated as you update your models, make sure your scaffolding allows you to throw out the latest, the last model and replace it with the latest each time a new thing comes out so that you are future-proofing the value of your agent.

Jim Johnson (25:29)

You know, I know the model companies are not going to want to hear this, but at some point we’re in a place that says, Hey, it’s almost like electricity coming out of the wall. And maybe they’re slightly different, but plug in one, plug in the other, plug in the next. And depending on the agentic use case, you’ve got running one maybe better than the other, but the flexibility to swap out and sort of plug into the different outlet, if you will, is huge. All right, guys, so let’s, let’s bring it home.

Nicole (25:32)

I

Jim Johnson (25:58)

If you were going to make a recommendation or two, let’s circle back to the data story. If you’re going to make a recommendation or two to a chief data officer tomorrow, you’re sitting in the room, what would be your key advice in this environment given all that’s going on?

Benjamin Titmus (26:16)

I mean,

I think everyone’s been using this word moat this year, but I really think that your business to stay competitive is unlocking your data, right? And being able to put that in the hands of these AI capabilities, that allows you to do more with less, right? Or more with the same amount of people.

You know, I could do work today that would have taken me, three of me, a year ago to do. And I think having that unlock and having that tacit knowledge be in the hand of these tools is really important.

Nicole (26:47)

I totally agree with that. want to pile onto it, going back to that business value. You have to prioritize what is my business use case and what is the data tied to that business use case that delivers that value. And then you need to prioritize what’s ready so that you get those quick wins. And as Andy was talking about earlier, you get the value out of the first win to pay for the next one.

Andrew Sweet (27:11)

Yeah, so just I would do three things if I were a chief data officer tomorrow. ⁓ Number one, I would build ⁓ a roadmap of use cases and be very clear and work with the business on that. Number two, I would really understand where my data is at. The answer my data is good or my data is bad is insufficient. Like what is good? What does good look like? What how many duplicates does it have? Are there gaps in the data? What third party data would help augment?

our first party data. So having a very clear view of where your data is exactly at and how it can support that roadmap you just created. The third thing I do is create a relationship with a company like Anthropic and get as deep with them as I could to really thoroughly understand where they’re going with their roadmap and then potentially even a Snowflake or a Databricks. So those vendor partnerships

But the most important thing I would do as a chief data officer is engage Answer Rocket to help us with all of those activities.

Jim Johnson (28:13)

I love that.

I love that a lot. I’m going to give them a piece of career advice. I think I’ve intersected with a number of chief data officers who sort of have have extracted themselves and their organization to a point where they don’t sufficiently understand the business anymore. They’re sort of focused on the data and the tooling and we’ve got this much data and this many tables and this is going on.

You got to reengage the number one thing that you and your organization can do to be effective in helping the enterprise move forward. Somebody’s got to do it here by the way. And it might as well start there with the chief data officer is to sort of step up in terms of deeply understanding the business and the unlock that’s sort of buried within the data. And you can serve as the glue between the business and IT and AI.

and it’s just a critical role to fill. know, heck, you’ll be on your way career-wise. You’ll get a title that even bigger than the word chief. But that’s the, it’s sort of a missing piece right now. In many cases, I see CDOs who sort of step back, if you will, and it’s become more of an IT type role. And I think it needs to be much more than that. Guys, thank you. Thank you, thank you for showing up. I know we’ve all got day jobs.

I appreciate it, Ben, great job joining us. Based on that performance, invite you back at some point, another 20 episodes. Nicole and Andy, of course, will be here on a regular basis. So appreciate it, guys. Have a great Monday. ⁓

Benjamin Titmus (29:40)

Ha!

Thank you.

Andrew Sweet (29:49)

Yep.

Nicole (29:49)

Thanks, team.

Andrew Sweet (29:49)

See ya.