WEBVTT
1
00:00:00.120 --> 00:00:04.360
Welcome to Data Science Dot Show, the podcast where data
2
00:00:04.440 --> 00:00:09.039
science meets executive leadership. We talk with top AI and
3
00:00:09.199 --> 00:00:13.400
analytics thought leaders about turning data into real business impact.
4
00:00:14.080 --> 00:00:17.199
If you're a C level leader or data expert shaping
5
00:00:17.239 --> 00:00:19.359
the future, you're in the right place.
6
00:00:20.440 --> 00:00:24.079
Last quarter, a global pricing model sat unused for six months,
7
00:00:24.359 --> 00:00:26.839
not because it was wrong, but because the company never
8
00:00:26.960 --> 00:00:30.760
changed how pricing decisions were made. I'm Merco Peters. We
9
00:00:30.920 --> 00:00:34.399
explore how data science and AI drive real business impact,
10
00:00:34.640 --> 00:00:37.880
straight from the leaders making it happen. Today, we fix
11
00:00:37.920 --> 00:00:41.399
that gap between models and measurable value. Joining me is
12
00:00:41.399 --> 00:00:44.799
a chief data and AI officer who's run pricing, fraud
13
00:00:44.840 --> 00:00:47.799
and supply chain programs at scale. Thanks for being here,
14
00:00:48.159 --> 00:00:52.640
Thanks Merco. Happy to be here. Over the last eight years,
15
00:00:52.799 --> 00:00:57.000
I've led central data teams across multiple industries, hundreds of
16
00:00:57.039 --> 00:01:01.960
models live, dozens of decision workflows. The consistent lesson is
17
00:01:02.000 --> 00:01:07.359
blunt models alone don't change behavior. The organization around the
18
00:01:07.400 --> 00:01:13.680
model does. Quick question, When you say the organization, what
19
00:01:13.719 --> 00:01:16.959
do you mean in plain terms? I mean the KPIs.
20
00:01:17.000 --> 00:01:21.319
People are measured on their incentives, the approval workflows, and
21
00:01:21.400 --> 00:01:24.159
the feedback loops that teach the model and the team.
22
00:01:24.560 --> 00:01:27.439
If you don't change those, the model is just noise
23
00:01:27.519 --> 00:01:31.040
on top of old habits. Right, So the first cut
24
00:01:31.079 --> 00:01:34.959
is decision boundary, who acts and where automation runs? How
25
00:01:34.959 --> 00:01:40.120
do you map that quickly? We use three buckets advisory signals,
26
00:01:40.480 --> 00:01:47.120
constrained automation, and full automation advisory nudges a human constrained
27
00:01:47.159 --> 00:01:51.959
automation executes with human override. Full automation runs end to
28
00:01:52.200 --> 00:01:57.239
end for low risk flows. The bucket dictates metrics, audit needs,
29
00:01:57.400 --> 00:02:01.159
and cadence. Give me a one line operational rule I
30
00:02:01.200 --> 00:02:04.200
can use to decide which bucket a use case falls into.
31
00:02:04.719 --> 00:02:08.319
If a wrong action causes irreversible customer harm or major
32
00:02:08.360 --> 00:02:12.840
regulatory exposure, keep it advisory or constrained. If impact is
33
00:02:12.919 --> 00:02:18.000
reversible and well instrumented, consider full automation after validation. Let's
34
00:02:18.039 --> 00:02:22.800
make this concrete pricing. You mentioned a pilot earlier. What
35
00:02:23.000 --> 00:02:26.560
was different there? We launched in one product category and
36
00:02:26.680 --> 00:02:30.599
one channel. The model produced a single operable signal, a
37
00:02:30.639 --> 00:02:34.599
recommended price band with expected margin impact, and a confidence level.
38
00:02:35.000 --> 00:02:37.759
That clarity made it easy for pricing managers to act.
39
00:02:38.159 --> 00:02:42.560
You wrapped each recommendation with margin delta and confidence. What
40
00:02:42.639 --> 00:02:46.000
did you measure first to prove it worked? Adoption rate,
41
00:02:46.280 --> 00:02:50.560
how often managers accepted the recommendation, an incremental margin per
42
00:02:50.599 --> 00:02:55.560
accepted recommendation. Those two metrics directly answered the question did
43
00:02:55.599 --> 00:02:59.879
behavior change and did it improve outcomes? Now the politically
44
00:03:00.080 --> 00:03:04.639
hard part. Incentives. You changed pay or evaluation to drive
45
00:03:04.680 --> 00:03:08.960
adoption right. We shifted from a narrow accuracy target to
46
00:03:09.039 --> 00:03:13.719
an outcome metric realized margin lift per decision. We gave
47
00:03:13.840 --> 00:03:17.840
managers a small share of upside when they adopted recommendations,
48
00:03:18.199 --> 00:03:23.479
while preserving their accountability for the decision. It reduced defensive rejections.
49
00:03:23.919 --> 00:03:27.280
Hold up critics will say that shares of upside can
50
00:03:27.319 --> 00:03:31.840
create perverse behavior. How did you prevent gaming? We layered
51
00:03:31.879 --> 00:03:36.599
guard rails, cohort level checks, anomaly detection on accepted decisions,
52
00:03:36.800 --> 00:03:41.560
and periodic audits. If a manager's accepted recommendations diverged from
53
00:03:41.639 --> 00:03:45.439
cohort performance, we flagged it for review and froze incentive
54
00:03:45.439 --> 00:03:51.120
payouts until investigated. Incentives nudged adoption not reckless behavior. Good
55
00:03:51.560 --> 00:03:55.360
incentives plus audits. Switching to governance, how do you keep
56
00:03:55.360 --> 00:03:59.520
governance from becoming a compliance theater? We run two loops,
57
00:04:00.159 --> 00:04:06.280
design time, risk assessment, data lineage and test plans before production,
58
00:04:07.120 --> 00:04:14.280
run time monitoring, drift detection, rollback procedures. Most important clearly
59
00:04:14.360 --> 00:04:19.639
assigned ownership. The executive sponsor sets risk appetite, and the
60
00:04:19.720 --> 00:04:25.560
CDO enforces operational guard rails. You mentioned ownership twice. Who
61
00:04:25.600 --> 00:04:29.000
signs off on expansion to more categories or full automation.
62
00:04:29.720 --> 00:04:33.879
The executive sponsor approves the decision boundary and business risk.
63
00:04:34.279 --> 00:04:38.759
The CDO and product lead must demonstrate outcome, stability, uptake
64
00:04:38.920 --> 00:04:43.439
and observability first. If those converge, the sponsor signs off
65
00:04:43.480 --> 00:04:48.240
to expand. Let's talk signals and observability. What's the minimum
66
00:04:48.279 --> 00:04:52.439
measurement stack you'd require before any expansion. Start with adoption
67
00:04:52.600 --> 00:04:57.879
rate and signal accuracy against a labeled baseline. Add decision latency,
68
00:04:58.399 --> 00:05:03.759
error rate versus baseline, and the incremental business metric revenue, lift, cost,
69
00:05:03.759 --> 00:05:09.000
avoided fraud reduction. Tie these into daily ops dashboards, weekly
70
00:05:09.040 --> 00:05:14.199
business reviews, and quarterly executive audits that cadence prevents surprises.
71
00:05:14.680 --> 00:05:18.399
Operational integration is where projects die. What are the two
72
00:05:18.480 --> 00:05:23.279
most common execution mistakes? Number one shipping a model endpoint
73
00:05:23.319 --> 00:05:27.360
without changing the UI or workflow users ignore it Number
74
00:05:27.360 --> 00:05:32.120
two not instrumenting downstream outcomes, no audit trail, no feedback
75
00:05:32.160 --> 00:05:35.759
to improve the model. The fix is cross functional sprints
76
00:05:35.759 --> 00:05:38.920
that deliver the model plus the UX and telemetry. In
77
00:05:39.000 --> 00:05:45.120
one go, you said cross functional sprints. How short six weeks? Three?
78
00:05:45.560 --> 00:05:50.279
Six week integrated pilots are our suite spot model, UI, incentive,
79
00:05:50.319 --> 00:05:54.040
tweak and measurement. That period is long enough to collect signal,
80
00:05:54.279 --> 00:05:58.040
short enough to keep focus looking ahead. What should executives
81
00:05:58.040 --> 00:06:02.279
prioritize over the next twelve to twenty months? Three priorities
82
00:06:02.600 --> 00:06:07.920
build repeatable playbooks for decision integration, invest in observability and
83
00:06:08.000 --> 00:06:13.360
audit trails, and formalize incentive changes. Now, regulators and customers
84
00:06:13.399 --> 00:06:18.040
will demand traceability, competitors will push speed. Prepare for both
85
00:06:18.519 --> 00:06:22.000
before we close. Two tactical moves A sea level can
86
00:06:22.000 --> 00:06:26.560
sponsor this quarter quick and verifiable. First run a six
87
00:06:26.639 --> 00:06:30.279
week integrated pilot in one high impact area that includes
88
00:06:30.319 --> 00:06:34.240
the model, UI change and a clear incentive tweak report,
89
00:06:34.279 --> 00:06:38.720
adoption and outcome delta at week six. Second, institute a
90
00:06:38.839 --> 00:06:43.680
lightweight run time governance checklist, monitoring rollback, owner and an
91
00:06:43.720 --> 00:06:47.839
audit trail. No model influences decisions without it if you
92
00:06:48.000 --> 00:06:51.759
take nothing else. Two simple lines to repeat at the
93
00:06:51.800 --> 00:06:57.519
next executive meeting. One align incentives and KPIs to the
94
00:06:57.560 --> 00:07:05.000
business outcome. Two acquire operational guardrails before expanding automation period.
95
00:07:06.120 --> 00:07:10.639
That's the difference between models and value. If this episode helped,
96
00:07:10.839 --> 00:07:13.199
subscribe and share it with a colleague who owns an
97
00:07:13.240 --> 00:07:17.040
important decision, and let's connect on LinkedIn. I'm mereco Peters.
98
00:07:17.600 --> 00:07:20.319
We'll be back with more conversations on how data science
99
00:07:20.319 --> 00:07:24.439
and AI create real business impact. Until next time, thanks
100
00:07:24.519 --> 00:07:28.920
for listening to Datascience Dot Show. If you found value
101
00:07:28.959 --> 00:07:32.720
in this episode, subscribe, share it with your network, and
102
00:07:32.839 --> 00:07:35.959
join us next time as we explore how data science
103
00:07:36.079 --> 00:07:39.560
drives smarter decisions and better business outcomes.
1
00:00:00.120 --> 00:00:04.360
Welcome to Data Science Dot Show, the podcast where data
2
00:00:04.440 --> 00:00:09.039
science meets executive leadership. We talk with top AI and
3
00:00:09.199 --> 00:00:13.400
analytics thought leaders about turning data into real business impact.
4
00:00:14.080 --> 00:00:17.199
If you're a C level leader or data expert shaping
5
00:00:17.239 --> 00:00:19.359
the future, you're in the right place.
6
00:00:20.440 --> 00:00:24.079
Last quarter, a global pricing model sat unused for six months,
7
00:00:24.359 --> 00:00:26.839
not because it was wrong, but because the company never
8
00:00:26.960 --> 00:00:30.760
changed how pricing decisions were made. I'm Merco Peters. We
9
00:00:30.920 --> 00:00:34.399
explore how data science and AI drive real business impact,
10
00:00:34.640 --> 00:00:37.880
straight from the leaders making it happen. Today, we fix
11
00:00:37.920 --> 00:00:41.399
that gap between models and measurable value. Joining me is
12
00:00:41.399 --> 00:00:44.799
a chief data and AI officer who's run pricing, fraud
13
00:00:44.840 --> 00:00:47.799
and supply chain programs at scale. Thanks for being here,
14
00:00:48.159 --> 00:00:52.640
Thanks Merco. Happy to be here. Over the last eight years,
15
00:00:52.799 --> 00:00:57.000
I've led central data teams across multiple industries, hundreds of
16
00:00:57.039 --> 00:01:01.960
models live, dozens of decision workflows. The consistent lesson is
17
00:01:02.000 --> 00:01:07.359
blunt models alone don't change behavior. The organization around the
18
00:01:07.400 --> 00:01:13.680
model does. Quick question, When you say the organization, what
19
00:01:13.719 --> 00:01:16.959
do you mean in plain terms? I mean the KPIs.
20
00:01:17.000 --> 00:01:21.319
People are measured on their incentives, the approval workflows, and
21
00:01:21.400 --> 00:01:24.159
the feedback loops that teach the model and the team.
22
00:01:24.560 --> 00:01:27.439
If you don't change those, the model is just noise
23
00:01:27.519 --> 00:01:31.040
on top of old habits. Right, So the first cut
24
00:01:31.079 --> 00:01:34.959
is decision boundary, who acts and where automation runs? How
25
00:01:34.959 --> 00:01:40.120
do you map that quickly? We use three buckets advisory signals,
26
00:01:40.480 --> 00:01:47.120
constrained automation, and full automation advisory nudges a human constrained
27
00:01:47.159 --> 00:01:51.959
automation executes with human override. Full automation runs end to
28
00:01:52.200 --> 00:01:57.239
end for low risk flows. The bucket dictates metrics, audit needs,
29
00:01:57.400 --> 00:02:01.159
and cadence. Give me a one line operational rule I
30
00:02:01.200 --> 00:02:04.200
can use to decide which bucket a use case falls into.
31
00:02:04.719 --> 00:02:08.319
If a wrong action causes irreversible customer harm or major
32
00:02:08.360 --> 00:02:12.840
regulatory exposure, keep it advisory or constrained. If impact is
33
00:02:12.919 --> 00:02:18.000
reversible and well instrumented, consider full automation after validation. Let's
34
00:02:18.039 --> 00:02:22.800
make this concrete pricing. You mentioned a pilot earlier. What
35
00:02:23.000 --> 00:02:26.560
was different there? We launched in one product category and
36
00:02:26.680 --> 00:02:30.599
one channel. The model produced a single operable signal, a
37
00:02:30.639 --> 00:02:34.599
recommended price band with expected margin impact, and a confidence level.
38
00:02:35.000 --> 00:02:37.759
That clarity made it easy for pricing managers to act.
39
00:02:38.159 --> 00:02:42.560
You wrapped each recommendation with margin delta and confidence. What
40
00:02:42.639 --> 00:02:46.000
did you measure first to prove it worked? Adoption rate,
41
00:02:46.280 --> 00:02:50.560
how often managers accepted the recommendation, an incremental margin per
42
00:02:50.599 --> 00:02:55.560
accepted recommendation. Those two metrics directly answered the question did
43
00:02:55.599 --> 00:02:59.879
behavior change and did it improve outcomes? Now the politically
44
00:03:00.080 --> 00:03:04.639
hard part. Incentives. You changed pay or evaluation to drive
45
00:03:04.680 --> 00:03:08.960
adoption right. We shifted from a narrow accuracy target to
46
00:03:09.039 --> 00:03:13.719
an outcome metric realized margin lift per decision. We gave
47
00:03:13.840 --> 00:03:17.840
managers a small share of upside when they adopted recommendations,
48
00:03:18.199 --> 00:03:23.479
while preserving their accountability for the decision. It reduced defensive rejections.
49
00:03:23.919 --> 00:03:27.280
Hold up critics will say that shares of upside can
50
00:03:27.319 --> 00:03:31.840
create perverse behavior. How did you prevent gaming? We layered
51
00:03:31.879 --> 00:03:36.599
guard rails, cohort level checks, anomaly detection on accepted decisions,
52
00:03:36.800 --> 00:03:41.560
and periodic audits. If a manager's accepted recommendations diverged from
53
00:03:41.639 --> 00:03:45.439
cohort performance, we flagged it for review and froze incentive
54
00:03:45.439 --> 00:03:51.120
payouts until investigated. Incentives nudged adoption not reckless behavior. Good
55
00:03:51.560 --> 00:03:55.360
incentives plus audits. Switching to governance, how do you keep
56
00:03:55.360 --> 00:03:59.520
governance from becoming a compliance theater? We run two loops,
57
00:04:00.159 --> 00:04:06.280
design time, risk assessment, data lineage and test plans before production,
58
00:04:07.120 --> 00:04:14.280
run time monitoring, drift detection, rollback procedures. Most important clearly
59
00:04:14.360 --> 00:04:19.639
assigned ownership. The executive sponsor sets risk appetite, and the
60
00:04:19.720 --> 00:04:25.560
CDO enforces operational guard rails. You mentioned ownership twice. Who
61
00:04:25.600 --> 00:04:29.000
signs off on expansion to more categories or full automation.
62
00:04:29.720 --> 00:04:33.879
The executive sponsor approves the decision boundary and business risk.
63
00:04:34.279 --> 00:04:38.759
The CDO and product lead must demonstrate outcome, stability, uptake
64
00:04:38.920 --> 00:04:43.439
and observability first. If those converge, the sponsor signs off
65
00:04:43.480 --> 00:04:48.240
to expand. Let's talk signals and observability. What's the minimum
66
00:04:48.279 --> 00:04:52.439
measurement stack you'd require before any expansion. Start with adoption
67
00:04:52.600 --> 00:04:57.879
rate and signal accuracy against a labeled baseline. Add decision latency,
68
00:04:58.399 --> 00:05:03.759
error rate versus baseline, and the incremental business metric revenue, lift, cost,
69
00:05:03.759 --> 00:05:09.000
avoided fraud reduction. Tie these into daily ops dashboards, weekly
70
00:05:09.040 --> 00:05:14.199
business reviews, and quarterly executive audits that cadence prevents surprises.
71
00:05:14.680 --> 00:05:18.399
Operational integration is where projects die. What are the two
72
00:05:18.480 --> 00:05:23.279
most common execution mistakes? Number one shipping a model endpoint
73
00:05:23.319 --> 00:05:27.360
without changing the UI or workflow users ignore it Number
74
00:05:27.360 --> 00:05:32.120
two not instrumenting downstream outcomes, no audit trail, no feedback
75
00:05:32.160 --> 00:05:35.759
to improve the model. The fix is cross functional sprints
76
00:05:35.759 --> 00:05:38.920
that deliver the model plus the UX and telemetry. In
77
00:05:39.000 --> 00:05:45.120
one go, you said cross functional sprints. How short six weeks? Three?
78
00:05:45.560 --> 00:05:50.279
Six week integrated pilots are our suite spot model, UI, incentive,
79
00:05:50.319 --> 00:05:54.040
tweak and measurement. That period is long enough to collect signal,
80
00:05:54.279 --> 00:05:58.040
short enough to keep focus looking ahead. What should executives
81
00:05:58.040 --> 00:06:02.279
prioritize over the next twelve to twenty months? Three priorities
82
00:06:02.600 --> 00:06:07.920
build repeatable playbooks for decision integration, invest in observability and
83
00:06:08.000 --> 00:06:13.360
audit trails, and formalize incentive changes. Now, regulators and customers
84
00:06:13.399 --> 00:06:18.040
will demand traceability, competitors will push speed. Prepare for both
85
00:06:18.519 --> 00:06:22.000
before we close. Two tactical moves A sea level can
86
00:06:22.000 --> 00:06:26.560
sponsor this quarter quick and verifiable. First run a six
87
00:06:26.639 --> 00:06:30.279
week integrated pilot in one high impact area that includes
88
00:06:30.319 --> 00:06:34.240
the model, UI change and a clear incentive tweak report,
89
00:06:34.279 --> 00:06:38.720
adoption and outcome delta at week six. Second, institute a
90
00:06:38.839 --> 00:06:43.680
lightweight run time governance checklist, monitoring rollback, owner and an
91
00:06:43.720 --> 00:06:47.839
audit trail. No model influences decisions without it if you
92
00:06:48.000 --> 00:06:51.759
take nothing else. Two simple lines to repeat at the
93
00:06:51.800 --> 00:06:57.519
next executive meeting. One align incentives and KPIs to the
94
00:06:57.560 --> 00:07:05.000
business outcome. Two acquire operational guardrails before expanding automation period.
95
00:07:06.120 --> 00:07:10.639
That's the difference between models and value. If this episode helped,
96
00:07:10.839 --> 00:07:13.199
subscribe and share it with a colleague who owns an
97
00:07:13.240 --> 00:07:17.040
important decision, and let's connect on LinkedIn. I'm mereco Peters.
98
00:07:17.600 --> 00:07:20.319
We'll be back with more conversations on how data science
99
00:07:20.319 --> 00:07:24.439
and AI create real business impact. Until next time, thanks
100
00:07:24.519 --> 00:07:28.920
for listening to Datascience Dot Show. If you found value
101
00:07:28.959 --> 00:07:32.720
in this episode, subscribe, share it with your network, and
102
00:07:32.839 --> 00:07:35.959
join us next time as we explore how data science
103
00:07:36.079 --> 00:07:39.560
drives smarter decisions and better business outcomes.
