Featuring Mike Finley, CTO at AnswerRocket
Our own Mike Finley, CTO at AnswerRocket, recently joined Matthew DeMello, Editorial Director at Emerj AI Research, on the AI in Business Podcast to continue the conversation about enterprise AI in CPG. Building on Jim Johnson’s earlier appearance, Mike dives deep into the technical and strategic realities of moving from impressive demos to production-ready agents that deliver business value.
Mike brings his extensive experience as a longtime AI practitioner (dating back to pre-transformer deep learning consortiums) to explain what actually makes an agent work, why most organizations water down their AI to uselessness, and how to avoid binding yourself to a single vendor’s ecosystem. From the “watered down effect” that kills promising projects to the concept of “vibe admin” for workflow automation, this conversation cuts through the hype to reveal what enterprises need to build agents that survive contact with reality.
In This Episode, You’ll Learn:
- (02:15) What Actually Makes an Agent: Agency, Not Just Chat
- (04:30) Business Strategy Over RAG: Teaching Agents Your 100-Year Process
- (06:45) The Watered Down Effect: How Guardrails Kill Agent Utility
- (09:20) Vendor Lock-In Risks: Why Latest Isn’t Always Greatest
- (12:35) Data Governance vs. Harmonization: The Network Effect
- (15:50) Six Weeks, Not Six Months: Rapid Iteration Over Perfection
- (18:25) Cross-Division Value: Leveraging Existing Tools as Agent Capabilities
- (21:40) Vibe Admin: Automating Workflows with Human Escalation
- (23:15) Measuring What Matters: Usage and Displacement Metrics
Key Insights from Mike Finley:
On What Makes a Real Agent: “What makes something an agent is its ability to have agency, its ability to actually make decisions and take actions on your behalf. It needs to understand what you want it to do in your business. Is it helping you set your prices correctly? Is it helping you allocate advertising dollars? Is it helping you reduce waste in your supply chain?”
On the Watered Down Effect: “Somebody has an idea, they build an agent, and it does something slightly different than it did the first time. So they get nervous and put guardrails on it. It was doing 100% of some scope, but they narrow it down to 30%. Then they repeat that two or three times, and now it’s doing 1% of what it used to do. Then people say, ‘At home, I can do so much more with ChatGPT. Why am I stuck with these inferior tools at work?'”
On Vendor Independence: “We’re in a rarefied space where these intelligences on tap from the major providers are very much neck and neck. They’re all trying to differentiate themselves by adding layers above their model, trying to lure you into their ecosystem. If you commit to that company’s API, you are almost certainly making yourself subject to not having the best language model at least in some time frame into the future.”
On Data Harmonization: “In the traditional world, there’s going to be an 18 month project involving five different scales of normalization. In the language model world that’s no longer necessary. Like a Harvard MBA, the LLM can assimilate data from multiple sources in its context and do useful things with it. I need to be able to add incrementally new data sets without adding an exponentially large amount of work.”
On Rapid Iteration: “You want to be in people’s hands in six weeks and see how they respond. Then in six months, hopefully you’re on the 10th iteration. If it’s not something you’re spending money on today, that’s not the problem you want to solve with AI, because you’re not going to save any money when you use AI to do it.”
On Vibe Admin: “There’s vibe coding where LLMs write code supervised by a human. Well, there’s vibe admin—if you’ve got a workflow like onboarding a new supplier, a 90 day process involving credit terms and vetting and quality tests, you can have an agent that watches over that process from soup to nuts. All it has to do is escalate to a human when something breaks.”
Resources and Concepts Mentioned:
- Key AI Concepts:
- Agency: The defining characteristic of true agents—ability to make decisions and take actions autonomously
- MCP (Model Context Protocol): Standard framework for how language models use tools
- The Watered Down Effect: Progressive over-constraining of agents until they become useless
- Vibe Admin: Workflow automation where agents supervise processes and escalate exceptions to humans
- Network Effect with Data: How value builds exponentially when multiple data sources can be related
- Context Window: The LLM’s “brain” where it assimilates information from multiple sources
- Model Management:
- Latest Model Trap: Risk of auto-subscribing to newest versions without governance
- Regression Testing: Essential practice for treating LLMs as enterprise software
- Model Specialization: Different models for reasoning, chat, embeddings, etc.
- Enterprise Patterns:
- Six Week Iterations: Rapid deployment and feedback cycles vs. six month waterfall
- Cross-Division Tool Leverage: Reusing existing Excel sheets, notebooks, and utilities as agent tools
- Human-in-the-Loop Authorization: Agents go as far as possible without requiring human approval
- Usage Displacement Metrics: Measuring what stops being used when AI is deployed
- Technical Standards:
- Microservices Architecture: Governed way to access enterprise data
- Enterprise Service Bus: Traditional communication patterns now applied to agent-to-agent interaction
Why This Matters for Enterprise Leaders:
Mike’s insights reveal a critical tension in enterprise AI: the same nervousness that makes organizations add guardrails is what kills agent utility. The “watered down effect” happens when teams see slight variations in agent behavior and respond by constraining the system until it’s barely useful. Meanwhile, employees experience powerful consumer AI at home and wonder why enterprise tools feel so limited.
The solution isn’t to eliminate constraints—it’s to architect systems properly from the start. This means understanding business strategy (not just dumping documents in RAG), providing valid data through proper tools, treating LLMs as software that requires testing and governance, and iterating in six-week cycles instead of six-month waterfalls.
Want to Learn More?
Mike Finley serves as CTO at AnswerRocket, helping enterprises build production-ready AI agents that deliver measurable business value. To discuss how to avoid the watered down effect and build agents that actually work in your organization, book a meeting with our team.
