Customer Support Unpacked (The Derby Mill Series ep 08)

In this unpacked episode, the team further expands its discussion of themes that came up in episode seven, which explored the automation of customer support with artificial intelligence. Our guests in that episode were a duo that is leading efforts in that space: CEO Mike Murchison and chief product and technology officer Mike Gozzo from Ada.

In this episode, Intrepid Growth Partners cofounder and partner Ajay Agrawal leads the discussion with Intrepid Senior Advisors Rich Sutton (Turing Award winner), Sendhil Mullainathan (MacArthur genius grant recipient) and Niamh Gavin (CEO, Emergent Platforms).

In the previous episode, we learned that Ada’s north star is “percent automated resolutions”, or the percentage of customer inquiries that are fully resolved by AI without human intervention. One challenge is that Ada relies on large language models (LLMs) rather than action-based goals, often requiring human agents to step in when confidence is low.

“It’s a mistake to think that [Ada’s AI agents] have goals,” says Sutton. ”What we have instead … is we have [AI agents] mimicking people.”

All of which raises the question of how customer support will evolve as this technology advances towards the limit.

Our team also debates the need for clear, objective measures of AI performance and the challenges of achieving true goal-oriented AI systems.

Our panel of experts:

Ajay Agrawal, co-founder and partner, Intrepid Growth PartnersRichard Sutton, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of AlbertaSendhil Mullainathan, MacArthur Genius grant recipient and professor, MITNiamh Gavin, Applied AI scientist, CEO, Emergent PlatformsSutton, Mullainathan and Gavin are all Intrepid Growth Partners’ senior advisors.

LINKS

Ada websiteThis episode extends the discussion from Derby Mill episode 07: Customer Support Rich Sutton’s home page. Follow Rich on XSendhil Mullainathan’s website. Follow Sendhil on XBe sure to catch every episode by subscribing on the following platforms:YouTube // Spotify // Apple Podcasts

DISCUSSION POINTS

00:00 Introduction and opening credit02:00 Ada refresher03:46 Clip: Testing harness06:50 Clip discussion begins10:15 What are goal-based objectives?13:30 Is this the year of the agent?17:40 What makes agents goal-oriented19:20 Decision-making fundamentals in AI21:27 Clip: Automating system improvement over time23:32 Clip discussion begins30:13 Automating the evaluation process34:08 What could Ada look like in the limit?41:03 Closing remarks

NUGGET 01: Vertical vs. Horizontal Competition

Fine-tuning used to be costly and impractical, pushing companies to open-source solutions—only to revert to OpenAI due to complexity. Now, companies like Ada build on top of model providers, offering flexibility while managing AI’s complexity. Niamh discusses the competitiveness between verticalized AI (industry-specific applications) and horizontal AI (broad sector models).

NUGGET 02: The Challenge of Interpretability

Ada's evaluations rely on human judgment. The challenge here is interpretability—determining whether an outcome is truly good without direct human input. Rich Sutton offers potential solutions, including using reinforcement learning with human feedback (RLHF) as a proxy measure trained on high-quality data.

NUGGET 03: Benchmarking vs. Deployment in the Field

Niamh and Sendhil discuss how, despite concerns about hallucinations in AI-powered customer service, CEOs adopt GenAI more for signaling competence than for real effectiveness.

DISCLAIMER

The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.