ABOUT THIS EPISODE
GenAI evaluation turns into a question of better or worse once every answer a model gives could be correct, and that is where Deepshikha and Anamika Mukhopadhyay begin. We talk about why one model needs three different definitions of good as a customer bot, a clinical summarizer and a coding agent, and why most teams they meet in workshops measure whatever their tools give them by default. A faithfulness score like 0.87279 left a room of CXOs asking whether it was a rounding problem; a 12 percent drop in issue resolution and 200 extra tickets did not. And a prepared Reddit post shows how an indirect prompt injection slips past an agent that had blocked every direct one.
"We measure what our tools give us by default." - Anamika Mukhopadhyay
Deepshikha leads Nagarro’s AI in Testing practice and brings over 13 years of experience at the intersection of software quality and emerging technologies. She specializes in designing AI-driven testing strategies that enhance reliability, accelerate delivery, and help organizations build systems that are not only efficient but also trustworthy and resilient when it matters most.
Anamika Mukhopadhyay is a QA Consultant with over 13 years of experience in software testing and automation and leads Nagarro’s Emerging and Special Tech Practice. Passionate about inclusivity and equal access to technology, she explores AI-driven innovations to advance testing practices. Her focus is on delivering sustainable, accessible, and high-quality solutions that empower diverse users across an evolving digital landscape.
Highlights:
- Anamika Mukhopadhyay hears the same answer in workshops with teams running production AI systems: they measure what their tools report by default, without asking which numbers they need.
- The same model needs different GenAI metrics per use case, from satisfaction and issue resolution for a support bot to groundedness and adoption rate for a clinical summarization tool.
- A dip in the faithfulness score left a CXO room asking about rounding; a 12 percent drop in issue resolution and 200 extra support tickets got their attention.
- GenAI evaluation runs as a loop of offline evaluation on a golden dataset and online evaluation with traces, because a model version can change every two to three weeks.
- Deepshikha's team planted an indirect prompt injection in a Reddit post, and an agent with guardrails against direct injection followed it through its web tool call.
🚀 Do you like the podcast? Then please rate and share it. And feel free to send feedback to podcast@testing-unleashed.fm