Co-authored by Andy Sweet, VP of Enterprise AI Solutions
We are asked to fix broken enterprise AI PoCs literally every day. After countless engagements, we’ve identified clear patterns in why AI initiatives fail and, more importantly, how to fix them. Here’s what we’ve learned.
Why Are So Many AI Proofs of Concept Failing?
Integrating AI is a little bit like learning to cook with a microwave. It’s a great tool, but things can really go wrong too. If you’re used to other infra like databases and cloud compute, AI looks like one more building block. And the demos are extremely easy! But mastering it is a long process that enterprises need help achieving.
One clear repeat offence is the excessive application of important but outdated techniques like RAG. Organizations implement RAG everywhere because it was revolutionary when it first appeared, but now they’re forcing it into use cases where it doesn’t belong.
Another good example is reliance on “flavor of the week” techniques that have become outmoded. Teams implement the latest trending approach across their entire stack, then wonder why results don’t match the hype from social media.
Misaligned expectations can also lead to PoCs failing. Not being clear on what success looks like leads to parts of the organization declaring success while others being disappointed, resulting in paralysis. These expectations must be rooted in business outcomes, not technical metrics or accuracy scores.
How Does AI Fatigue Affect Company AI Initiatives?
Two key ways that AI fatigue affects AI initiatives are:
Increased Risk Aversion. After failed PoCs, organizations can overcorrect and fall into excessive governance and approvals that further kill the rapid iteration that successful AI PoC development requires. This can lead to a downward spiral of failure.
Talent Retention Issues. High-performing AI teams get frustrated when organizations lose momentum and start getting in their own way especially when they see the risk aversion setting in. They leave for companies with a clear vision and AI approaches that lend themselves for success.
What Are the Impacts of Failed Proof Of Concepts On Broader AI Adoption?
In the long run, the only impact is delay. It’s just change management friction. In the short term though, heads roll, projects cancel. It’s unfortunate.
Failed PoCs can also potentially create misleading “lessons learned” that get shared industry-wide. These “lessons learned” can stifle momentum by propagating anti-patterns as “best practices”. One company’s failure with a specific approach becomes another company’s reason to avoid AI entirely.
The good news is that the tech works, a growing number of companies have figured out that there is professional help, and boards don’t need to be visionaries to keep the pressure on: this tech is redefining the economy, whether incumbent solutions like it or not.
How Can Companies Fix Proof of Concept Failures And Scale AI Deployment Faster?
There are common recurring problems that have simple fixes, but the “meta problem” is that enterprises think they know how to use this tech. It’s not hard, but it is different. It takes a mind shift, and often requires abandoning some contractually locked-in legacy components and service providers.
Here’s our approach to fixing broken AI initiatives:
1. Start With Proof, Not Promises
You can’t really believe anyone who tells you they have decades of expertise on a tech stack that was invented last year. We’re in a show-me state. Helping our customers through this starts with demonstrating that we know how to do it, then diagnosing the next round of issues they face. By the time we get through a design review, they know how to win again.
2. Address the Knowledge Gap
Even competent IT professionals can be baffled when their existing techniques, experts, and ninjas keep failing. The biggest obstacle is the knowledge gap. Educate the organization, starting at the most senior leadership, to learn what AI can and cannot do realistically. And this literacy extends to the technologists as well in how to map AI capabilities to business outcomes.
3. Debug at the Model Level
If we had to give a single point of advice on how to get a project unstuck, it would be to peel back all the layers and look at what is actually being sent to language models, and what is coming back from them, without all the clutter in between. Most of the time, we find that the model is capable of a lot more than the software wrapping it allows, and there are surprising things happening under the hood that have easy fixes, but nobody is looking at what’s going over the wire to the language model, so nobody knows.
4. Treat AI Like Mission-Critical Software
Recognize that AI software is still just software, and many companies know how to do that well. So, plan AI deployments like they are mission-critical enterprise software, hire the right expertise to fix the obvious problems, and launch.
5. Future-Proof With the Right Approach
As far as future proofing goes, the clear winner is to get on board with tool use and function calling. This is not the same as agents, though it is an important part of agent design.
The Path Forward
The enterprises succeeding with AI aren’t waiting for perfect conditions. They’re learning from these common failures, bringing in the right expertise, and moving forward with clarity about what AI can and cannot do.
The technology is ready. The question is: are you ready to approach it differently?
