How Amazon Uses Telemetry and AI to Automate Physical Logistics

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 10/20/2026 00:27:34
Duration: 3006.000
Channels: 1
1
00:00:11.360 --> 00:00:15.440
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.075 --> 00:00:20.074
Today's episode is sponsored by Parallel, where agents find answers.
3
00:00:20.475 --> 00:00:27.195
Most engineers today closely follow new model releases, but don't pay attention to their agent's most important tool, web search.
4
00:00:27.675 --> 00:00:40.140
Parallel develops enterprise grade infrastructure for agents to retrieve high quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto optimal quality, cost, and speed.
5
00:00:40.780 --> 00:00:53.245
Whether you work on voice agents that need two hundred millisecond latency, chatbots that balance speed, depth, and quality, or long horizon agents to do thorough overnight research for you, Parallel is a single platform for all of your agentic research.
6
00:00:53.645 --> 00:00:57.805
Get started for free today at dataengineeringpodcast.com/parallel.
7
00:00:58.400 --> 00:01:02.239
Your host is Tobias Macey, and today I'm interviewing Keerti Kulkarni
8
00:01:02.239 --> 00:01:07.360
about how she is working on physical logistics automation at AWS
9
00:01:07.360 --> 00:01:13.200
and some of the work that she's done on managing that at scale. So, Kirti, can you start by introducing yourself?
10
00:01:14.135 --> 00:01:17.735
Yeah. Absolutely. Excited to be here. Thanks for giving me the opportunity.
11
00:01:17.895 --> 00:01:24.295
I lead the technical program for AI enabled operations automation at Amazon Key, a team in Sunnyvale.
12
00:01:24.375 --> 00:01:30.270
My day job is building the telemetry data and the AI layer that runs the physical logistics automation.
13
00:01:30.670 --> 00:01:33.470
This is a system that handles loading and loading
14
00:01:33.710 --> 00:01:38.750
stop across all of the North American logistics facilities owned by Amazon.
15
00:01:39.150 --> 00:01:42.190
And we handle about 10 k docs
16
00:01:41.725 --> 00:01:43.965
today at 100 plus facilities,
17
00:01:44.045 --> 00:01:46.524
and this has led to 77%
18
00:01:46.524 --> 00:01:50.604
adoption of the automated workflow as of March 2026.
19
00:01:51.244 --> 00:01:57.970
So what I do specifically is build the data plumbing, which lets the physical operations be automated and run themselves.
20
00:01:58.130 --> 00:02:03.810
And I do it at the Sims where the telemetry warehouses and now AI have to actually cooperate.
21
00:02:05.090 --> 00:02:10.875
So I would like to give also a little bit background of how I got here and what inspired that journey.
22
00:02:11.195 --> 00:02:17.915
I spent about fifteen plus years as a TPM, technical program managers, working at the crossroads of data engineering,
23
00:02:17.995 --> 00:02:19.915
basically distributed systems,
24
00:02:19.915 --> 00:02:20.875
and physical operations.
25
00:02:21.510 --> 00:02:23.510
Before physical automation,
26
00:02:23.510 --> 00:02:29.110
I led the BI platform at Amazon Key. I separated the KeyRedshift
27
00:02:29.110 --> 00:02:31.830
environment from the Ring's shared cluster,
28
00:02:31.910 --> 00:02:38.665
migrated in about six months with only two BI engineers at that time and no dedicated data engineers.
29
00:02:38.985 --> 00:02:44.905
So with this effort, we managed to cut our dashboard failures by 68%.
30
00:02:45.465 --> 00:02:52.185
And today, this environment supports about 20 plus BI and DE engineers and still
31
00:02:51.250 --> 00:02:53.250
is performing at scale.
32
00:02:53.489 --> 00:02:56.450
Before Amazon, I led a 18,500,000
33
00:02:56.450 --> 00:02:57.330
safety
34
00:02:57.330 --> 00:03:04.530
assessment program for Genentech. And before that, I was in Watson Health and Watson Drug Discovery as a TPM
35
00:03:04.610 --> 00:03:08.185
in IBM for four years and earlier
36
00:03:08.185 --> 00:03:10.905
in the... In IBM, but in a different team,
37
00:03:11.225 --> 00:03:13.065
which was the Asia Pacific
38
00:03:13.065 --> 00:03:15.385
master data management practice
39
00:03:15.545 --> 00:03:27.150
for about six years with building an eight person team across the continent. So that's my journey in data so far. And you mentioned that you're focused on physical logistics automation
40
00:03:27.230 --> 00:03:29.390
and the ways that that is manifest
41
00:03:30.030 --> 00:03:42.445
in terms of Amazon's distribution facilities. And I'm wondering if you can just unpack a little bit more of what that term means and some of the technical and organizational systems that feed into it.
42
00:03:43.564 --> 00:03:44.925
Absolutely.
43
00:03:45.004 --> 00:03:47.165
When I'm talking about physical automation,
44
00:03:47.405 --> 00:03:52.260
or physical logistics automation, it is usually paired with the software
45
00:03:52.260 --> 00:03:56.100
or warehouse management systems that basically tells the machines
46
00:03:56.340 --> 00:04:11.745
in the warehouses what to do. But the automation itself is the physical execution layer. That is the robots or the conveyor belts actually moving things, not just the data. So physical automation in Amazon is how we handle the workflow
47
00:04:11.985 --> 00:04:17.185
of basically the vehicles coming in to the warehouses. They getting secured
48
00:04:17.330 --> 00:04:18.610
in the docks,
49
00:04:19.090 --> 00:04:22.610
then doing all of the processes which actually
50
00:04:22.850 --> 00:04:28.130
make it locked and safe for the loading and loading associates to go inside
51
00:04:28.450 --> 00:04:32.770
and start that workflow where the bell starts moving and the
52
00:04:33.435 --> 00:04:37.275
goods start coming in or going out. So at... On the
53
00:04:37.675 --> 00:04:44.395
surface, it seems very simple, but there are multiple steps that make this more safe and more automated.
54
00:04:44.474 --> 00:04:49.160
So this is what my team specifically has been doing, but there are other teams handling
55
00:04:49.160 --> 00:04:52.440
various other parts that moving goods from
56
00:04:52.760 --> 00:05:02.695
one area to the other in terms of the category of the goods or the next destination of the goods and so on. So physical automation at this scale
57
00:05:02.855 --> 00:05:05.335
is happening at Amazon Logistics Center.
58
00:05:05.895 --> 00:05:08.455
So that's basically the details of
59
00:05:09.095 --> 00:05:11.495
what physical automation is. And then
60
00:05:11.975 --> 00:05:16.080
your other question was about, I think, the technical and organizational
61
00:05:16.080 --> 00:05:18.560
systems involved in the production of this data.
62
00:05:19.360 --> 00:05:20.480
So technical,
63
00:05:20.720 --> 00:05:21.520
systems
64
00:05:21.520 --> 00:05:24.800
are pretty much in one line. I use AWS
65
00:05:24.800 --> 00:05:26.320
CloudWatch Analytics,
66
00:05:26.320 --> 00:05:35.795
then I use s three buckets to transfer this data to via Firehose mechanism. So as we are an Amazon house, we are part of Amazon.
67
00:05:35.875 --> 00:05:37.875
We use basically the AWS
68
00:05:37.875 --> 00:05:39.075
ecosystem.
69
00:05:39.475 --> 00:05:41.475
We have Kinesis Data Streams as well.
70
00:05:42.220 --> 00:05:47.580
And then for the AI part of it, we use Amazon Quick, which is the Amazon AI toolkit,
71
00:05:47.740 --> 00:05:48.700
the Spaces,
72
00:05:48.700 --> 00:05:51.580
knowledge bases, and AI agents.
73
00:05:51.580 --> 00:05:54.860
In my case, it's also a chatbot conversational agent,
74
00:05:55.735 --> 00:05:59.015
which works with now the organizational systems.
75
00:05:59.255 --> 00:06:03.975
They are basically the site associates using the system, and then they are hierarchy
76
00:06:04.055 --> 00:06:09.735
for overriding any common issues, like multiple layers of management or their IT support
77
00:06:10.139 --> 00:06:17.340
or their maintenance folks, which have the authority or ability to take care of certain kinds of issues
78
00:06:17.660 --> 00:06:23.820
and basically canceling or pausing the automated workflow to take care of any safety or hardware issues.
79
00:06:24.875 --> 00:06:26.155
Another organizational
80
00:06:26.155 --> 00:06:26.955
system
81
00:06:27.115 --> 00:06:31.195
is basically the role based control over all of this,
82
00:06:31.595 --> 00:06:32.875
using this hierarchy
83
00:06:32.875 --> 00:06:36.235
and controlling each of the objects in this physical layer.
84
00:06:36.555 --> 00:06:38.475
There is also an integration
85
00:06:38.475 --> 00:06:40.074
with the yard where
86
00:06:40.270 --> 00:06:42.669
the yard basically integrates saying,
87
00:06:42.909 --> 00:06:47.389
this is where this vehicle is going to go, whether it's inbound, outbound,
88
00:06:48.030 --> 00:06:49.630
or basically mixed
89
00:06:49.870 --> 00:06:50.590
docks,
90
00:06:50.990 --> 00:06:55.135
which is the source of the trailers and then which are sending logistics
91
00:06:55.215 --> 00:06:56.655
workflows to the doors.
92
00:06:57.055 --> 00:07:00.575
So I'll stop here and see if there are any follow-up questions, Tobias.
93
00:07:01.375 --> 00:07:03.135
So given the
94
00:07:03.455 --> 00:07:04.335
variety
95
00:07:04.335 --> 00:07:05.055
of
96
00:07:05.215 --> 00:07:07.455
physical systems that you're integrating with,
97
00:07:08.050 --> 00:07:23.005
what are some of the types of data and some of the relative volumes of data, both at the kind of individual event level as well as in the aggregate and some of the ways that that influences the ways that you think about the overarching
98
00:07:23.005 --> 00:07:24.765
solution that you're focused on?
99
00:07:25.485 --> 00:07:30.445
Great question. So the scale of the data is pretty much in petabytes because
100
00:07:31.005 --> 00:07:31.725
every
101
00:07:31.965 --> 00:07:33.405
second there is
102
00:07:34.125 --> 00:07:36.765
some vehicle docking at some of the,
103
00:07:37.340 --> 00:07:39.420
one of the Amazon warehouse
104
00:07:39.500 --> 00:07:46.700
in The US or in the world. So currently we are live in The US, but we are planning to expand globally as well.
105
00:07:47.180 --> 00:07:50.060
So the data is definitely scale is petabytes,
106
00:07:50.354 --> 00:07:55.875
but the events are basically time streams, which are coming, as I said, every second.
107
00:07:56.115 --> 00:07:57.395
And we
108
00:07:58.194 --> 00:08:01.955
are not just maintaining the events, but we are also maintaining the history,
109
00:08:02.514 --> 00:08:11.380
in which we are saying that say forty seconds back, this door was active and this event was happened. And based on that context,
110
00:08:11.620 --> 00:08:14.500
we are actually going to rate the next event, whether
111
00:08:14.900 --> 00:08:18.660
it had a potential of causing any failure
112
00:08:18.900 --> 00:08:26.565
or it is the regular expected event. So maintaining all of that history as well as the context makes
113
00:08:27.685 --> 00:08:30.885
the scale even more complex and,
114
00:08:31.925 --> 00:08:32.645
more daunting.
115
00:08:33.820 --> 00:08:34.460
And
116
00:08:34.779 --> 00:08:41.340
the other aspect of this is because you're dealing with real world physical systems and because the
117
00:08:41.980 --> 00:08:44.940
operational capabilities of Amazon relies
118
00:08:45.675 --> 00:08:54.154
information that you receive, and you're also dealing with a wide number of people at various locations whose jobs are also going to be impacted.
119
00:08:54.555 --> 00:08:59.550
What are some of the failure modes or some of the effects of inaccuracies
120
00:08:59.550 --> 00:09:03.550
or delays in terms of the processing of the data that you're working with?
121
00:09:05.150 --> 00:09:07.150
Yeah, that's a good question.
122
00:09:07.790 --> 00:09:08.510
Because
123
00:09:08.910 --> 00:09:19.865
the jobs of the people at the warehouse who are plugging in to this workflows and doing the jobs which are needed there, even with the automated workflow,
124
00:09:20.185 --> 00:09:23.705
if there is any safety issue or if they do not follow any
125
00:09:24.040 --> 00:09:25.640
particular step guide,
126
00:09:25.880 --> 00:09:27.080
then this causes,
127
00:09:27.880 --> 00:09:37.480
basically a warning and then a future consequence for them. So this becomes a very high priority for the logistics leadership to track
128
00:09:37.720 --> 00:09:38.440
and manage.
129
00:09:39.935 --> 00:09:40.575
So
130
00:09:40.815 --> 00:09:47.615
we are making sure that the data is as detailed as possible, but some of the data is obfuscated
131
00:09:48.095 --> 00:09:51.855
at the user ID level. So if there is, like,
132
00:09:53.150 --> 00:09:56.510
a bit of delay, which is not causing a safety concern,
133
00:09:56.830 --> 00:10:00.350
then we wouldn't want the user ID to be immediately
134
00:10:00.350 --> 00:10:03.070
circulated into various dashboards and
135
00:10:03.230 --> 00:10:09.715
into the logs to... The logs, it is already there, but we are not going to move it to s three and where we do the analysis.
136
00:10:09.955 --> 00:10:10.675
So that
137
00:10:10.995 --> 00:10:16.275
there is a confirmation that this shift or this site needs some training,
138
00:10:16.595 --> 00:10:18.995
but all of the user IDs are not exposed.
139
00:10:19.970 --> 00:10:23.490
Only at the high level where there is a very safety
140
00:10:23.650 --> 00:10:27.090
targeted issue and which is already known to them and this
141
00:10:27.330 --> 00:10:30.370
associate or personnel has that issue,
142
00:10:30.610 --> 00:10:31.490
then that...
143
00:10:31.730 --> 00:10:36.545
The details of that can be surfaced using the system, but that will require
144
00:10:36.705 --> 00:10:37.665
many
145
00:10:38.785 --> 00:10:40.145
roles and many
146
00:10:40.785 --> 00:10:44.385
clearances before people get to it. So we are making sure that
147
00:10:45.339 --> 00:10:51.580
none of the warehouse personnel are adversely affected because of this analysis and data being available.
148
00:10:51.899 --> 00:10:53.100
That's the first one.
149
00:10:53.820 --> 00:11:00.825
And then the weak points of the system, let me touch base upon that. We treat the physical layer as a first class telemetry,
150
00:11:00.825 --> 00:11:09.225
not as a noise, which is to be smoothed later. So each vehicle entry point reports state, and we tag events with side clock and ingest clock.
151
00:11:09.800 --> 00:11:15.000
So if there is any difference between this, then there are major effects seen downstream.
152
00:11:15.399 --> 00:11:19.000
So any pipeline stage that assumes tight ordering,
153
00:11:19.240 --> 00:11:27.745
we do a reconciliation step on it to confirm that everything is up to that point has happened as per expectation.
154
00:11:28.225 --> 00:11:35.425
The lesson from about 1,000,000 cycles that we have completed in last year for 30 plus warehouses,
155
00:11:35.425 --> 00:11:38.705
which we were launched at that time, and now we are at 100 plus.
156
00:11:40.240 --> 00:11:48.160
On those 1,000,000 cycles, we saw that the late or duplicate data will happen at every layer. So we designed a ledger
157
00:11:48.480 --> 00:11:54.000
with the event key, and we made a summarization layer on top of it, which is explicitly aware.
158
00:11:54.665 --> 00:12:02.505
As I said, that the last thing you heard from a specific door was for a specific event at a specific vehicle,
159
00:12:03.545 --> 00:12:08.425
so was ninety seconds ago. So based on that context, you are able to dictate further.
160
00:12:08.829 --> 00:12:12.110
So as we learn more lessons, we are trying to
161
00:12:12.589 --> 00:12:14.750
add more summarizations
162
00:12:14.750 --> 00:12:28.845
and synergies into the AI layer that we are building, but there might be some cases which we are still... Have still not encountered, we are looking forward to work with it and update it as we go. So
163
00:12:28.845 --> 00:12:30.445
digging now into
164
00:12:30.685 --> 00:12:46.510
what you're building, you mentioned some of the technical components, but I'm wondering if you can talk through a bit more of the journey that the data takes and some of the types of analysis that you're doing, and then especially the role that the LLMs are playing in that overall system.
165
00:12:47.150 --> 00:12:48.990
Yeah. Absolutely. Would love to.
166
00:12:49.790 --> 00:12:50.430
So first,
167
00:12:51.404 --> 00:12:53.324
for analysis,
168
00:12:53.324 --> 00:12:58.045
we are checking all of the events from the time stream as well as,
169
00:12:58.365 --> 00:12:59.725
from the CloudWatch,
170
00:12:59.725 --> 00:13:01.725
any errors and any
171
00:13:01.964 --> 00:13:05.005
messages that we are getting on each of the events.
172
00:13:05.165 --> 00:13:09.080
We tie those together to get a full picture of
173
00:13:09.800 --> 00:13:18.840
the whole network, but specifically one site, one door at that particular moment. So we go down to that level. Though... So the Firehose basically
174
00:13:19.080 --> 00:13:28.195
takes the CloudWatch data and we ingest into s three. As you have previously pointed on one of your episodes, CloudWatch is basically expensive
175
00:13:28.355 --> 00:13:34.275
if we treat it as a primary source and start doing analysis on it. So our way
176
00:13:34.355 --> 00:13:34.755
to,
177
00:13:36.170 --> 00:13:39.529
what do you call, mitigate that risk is basically
178
00:13:39.850 --> 00:13:44.329
moving it directly to s three, which is a low cost storage.
179
00:13:44.889 --> 00:13:46.170
And then,
180
00:13:46.490 --> 00:13:48.250
we basically analyze
181
00:13:48.250 --> 00:13:51.370
on it, with the Amazon AI framework.
182
00:13:51.785 --> 00:13:53.625
So once it is in s three,
183
00:13:53.865 --> 00:13:58.505
we have a way to aggregate it or analyze it further
184
00:13:58.665 --> 00:14:13.460
based on what operations team or org needs or our product needs to make decisions on the road map or where we are going next. Also, the leadership team to check the entire state of the network
185
00:14:13.620 --> 00:14:15.380
with new launches.
186
00:14:15.540 --> 00:14:17.060
We have that ability.
187
00:14:17.380 --> 00:14:20.180
So let me dig a little bit deeper into this.
188
00:14:21.515 --> 00:14:25.835
So we also move this data into Redshift for
189
00:14:25.915 --> 00:14:29.755
consumption into the various dashboards that are pretty static,
190
00:14:29.915 --> 00:14:31.035
but that's another
191
00:14:31.675 --> 00:14:32.475
consumer
192
00:14:32.955 --> 00:14:34.235
from our system.
193
00:14:34.970 --> 00:14:36.410
So Orkfit.
194
00:14:36.410 --> 00:14:37.450
Amazon, basically,
195
00:14:38.090 --> 00:14:38.570
for
196
00:14:39.610 --> 00:14:42.730
CloudWatch data to migrate into s three and Firehose,
197
00:14:42.810 --> 00:14:48.890
we can also purge it from CloudWatch as per the retention policy. So we are controlling the cost there.
198
00:14:49.605 --> 00:15:05.280
Then I will talk about the AI summarization layer. So now AI summarization layer is basically AI automation along with a chatbot conversational agent specifically targeted for the operational team. So if they see that there is an issue docking
199
00:15:05.360 --> 00:15:07.360
a vehicle at a door,
200
00:15:07.760 --> 00:15:19.505
then they only get from the associates who are pretty much at the job and doing their work. They do not have much time to explain what's happening, but they are just like the automated workflow is failing.
201
00:15:19.745 --> 00:15:44.850
So we have to, based on that, we have to make sure that we are giving, analyzing the right cause and giving them a way to fix it. Otherwise they are interested in basically completing the work with or without the automated workflow. So we have a a risk of our usage dropping as well. So for that, we have given the operations team a way to just ask questions to the agent
202
00:15:45.695 --> 00:15:52.895
in a simple language saying, hey. Tell me the issue happening at, say, Door 42 at this site, in this warehouse,
203
00:15:53.215 --> 00:15:55.935
and at around this time between eleven
204
00:15:56.175 --> 00:16:05.760
and noon on this particular day. So all of this three, the logs which were transferred in terms of the messages are scanned
205
00:16:05.920 --> 00:16:12.495
and specifically a history on that door or on that site is also tied back to the question.
206
00:16:12.894 --> 00:16:16.894
And it said that, hey. This issue is happening because
207
00:16:17.214 --> 00:16:18.495
an unauthorized
208
00:16:18.735 --> 00:16:19.535
user
209
00:16:19.615 --> 00:16:23.135
who doesn't have access to canceling the workflow is trying to do it.
210
00:16:23.774 --> 00:16:35.110
And this is similar to the same user accessing at various inbound doors. So they get a little bit more context. Also, can reach out to that user and say, you are not authorized. So please get authorization
211
00:16:35.430 --> 00:16:44.455
from this person. We also give them the links on how that authorization is requested. So in case that user is not supposed to have the authorization,
212
00:16:44.615 --> 00:16:50.615
somebody else will do and come come and complete the workflow. But if they are just missing that authorization
213
00:16:50.615 --> 00:16:55.255
by a mistake, then they will be able to get that authorization and continue that workflow
214
00:16:55.590 --> 00:17:15.034
from that point on. So currently, it is about an hour of a delay in getting those CloudWatch and putting it into s three because our jobs run every hour. But we are trying to move it down to 10 so that we have close to real time data and we are... That will be more...
215
00:17:15.675 --> 00:17:16.715
The turnaround
216
00:17:16.795 --> 00:17:21.835
time is reduced for the people on the job and working on the doors.
217
00:17:22.875 --> 00:17:23.515
In terms of
218
00:17:24.220 --> 00:17:28.700
where the system was when you started, I'm wondering what are some of the
219
00:17:29.100 --> 00:17:29.980
evolutionary
220
00:17:29.980 --> 00:17:33.179
points that you used as checkpoints
221
00:17:33.179 --> 00:17:44.004
to where you are today to make sure that you're headed in the right direction and some of the types of feedback and some of the change management given the number of people who are involved.
222
00:17:46.565 --> 00:17:52.725
Yeah. That's a great point. So adding a conversational AI agent versus what was there initially
223
00:17:53.220 --> 00:17:58.659
versus just using push notifications or dashboards made the platform more accessible
224
00:17:58.820 --> 00:18:17.364
for the operations teams, who are the people on the ground, and they are working with the actual associates, so who are giving real time feedback for our system. So making this most useful for them was one of the key design choices that we made. So if the, conversational AI agent we had not
225
00:18:17.605 --> 00:18:28.430
added to this mix, then we would have basically push notifications and we would have dashboards, but we would not have something real time which can solve their problems. So that's one
226
00:18:29.150 --> 00:18:32.190
design choice we made that served as a data plumbing.
227
00:18:32.605 --> 00:18:37.565
That lets physical operations run themselves. And we did it basically in the
228
00:18:38.205 --> 00:18:41.325
crossroads or sims where telemetry warehouses
229
00:18:41.325 --> 00:18:51.739
and now the AI have to actually cooperate. So we don't have to get out and build something specific, but we managed to build it as an integrated part of this whole solution.
230
00:18:51.899 --> 00:18:53.580
With the AI
231
00:18:53.820 --> 00:18:55.740
usage at Amazon skyrocketing,
232
00:18:55.740 --> 00:18:56.539
this was a natural
233
00:18:57.235 --> 00:18:58.914
design choice that we made.
234
00:18:59.315 --> 00:19:00.354
Also,
235
00:19:00.355 --> 00:19:01.955
as I was telling earlier,
236
00:19:02.674 --> 00:19:03.234
the
237
00:19:04.195 --> 00:19:05.075
CloudWatch
238
00:19:05.075 --> 00:19:20.739
only could have... And Firehose could have been also a choice, but we decided to use S3 and put a retention policy on CloudWatch because we are anyways having all of the logs for this particular analysis in s three. In that particular
239
00:19:21.539 --> 00:19:25.620
effort, we also managed to reduce the overall cost by
240
00:19:25.945 --> 00:19:27.945
putting a more, what do you call,
241
00:19:28.425 --> 00:19:35.705
strict poll... Data retention policy on CloudWatch. So we saved a lot of cost for the, teams.
242
00:19:36.425 --> 00:19:38.905
So those were the two design choices I'm pretty proud of.
243
00:19:40.070 --> 00:19:42.070
Adding an LLM to
244
00:19:42.150 --> 00:19:43.590
a high volume
245
00:19:43.670 --> 00:19:50.390
data system can add latency if you're not careful about where it's being applied and what responsibilities
246
00:19:50.390 --> 00:19:55.590
you're giving to it. So I'm wondering what was your process for figuring out what were the
247
00:19:56.044 --> 00:20:03.244
constraints that you wanted to place on how and where the LLMs were used and some of the ways that you work on
248
00:20:03.565 --> 00:20:11.245
doing the context engineering around what specific details are useful without just overwhelming the LLM and having it
249
00:20:11.940 --> 00:20:16.739
produce a a bunch of useless or irrelevant details in its responses?
250
00:20:18.260 --> 00:20:19.059
So,
251
00:20:19.220 --> 00:20:23.299
frankly speaking, we are still working on that. We are still experimenting
252
00:20:23.700 --> 00:20:26.019
which of the asks
253
00:20:26.019 --> 00:20:29.065
caused the AI agent to either
254
00:20:29.625 --> 00:20:32.825
cause a significant delay or hallucinate.
255
00:20:33.145 --> 00:20:38.504
So we do not have the option to cache because every request is coming from a different
256
00:20:38.745 --> 00:20:40.985
site and a different door
257
00:20:41.170 --> 00:20:47.330
and means either a hardware failure or something in the automated workflow, which is related to software.
258
00:20:47.810 --> 00:20:51.730
So we are still experimenting it, but a large...
259
00:20:51.890 --> 00:20:53.570
From our
260
00:20:53.650 --> 00:20:54.370
knowledge,
261
00:20:54.610 --> 00:20:56.050
we have seen that
262
00:20:56.305 --> 00:20:58.945
analysis network wide or
263
00:20:59.025 --> 00:21:04.465
something related to the site with multiple dog doors saying, hey, what is the pattern I'm observing
264
00:21:04.865 --> 00:21:05.745
for
265
00:21:05.985 --> 00:21:09.745
this particular shift when there is a high inbound traffic
266
00:21:10.140 --> 00:21:11.660
and multiple
267
00:21:11.820 --> 00:21:15.740
dog doors having, like, authorization issues or
268
00:21:15.820 --> 00:21:18.540
something where the workflows are not synced.
269
00:21:18.940 --> 00:21:23.500
Those are the, some examples where we have seen a high degree of latency,
270
00:21:24.235 --> 00:21:30.634
and we are looking at ways to reduce it into a threshold which is manageable.
271
00:21:30.635 --> 00:21:33.674
Like, if there are times when
272
00:21:34.315 --> 00:21:35.835
my leadership wanted
273
00:21:35.835 --> 00:21:37.674
basically all of the network
274
00:21:40.040 --> 00:21:43.080
health status. And then it has taken about
275
00:21:43.240 --> 00:21:51.800
thirty minutes to forty five minutes to pull that out over a period of time. And that is a lot of delay as per my
276
00:21:53.035 --> 00:21:55.914
understanding. So what we are doing is trying to
277
00:21:56.235 --> 00:22:01.274
get those into batch jobs versus real time asks and
278
00:22:01.435 --> 00:22:05.434
run those pretty frequently before business review
279
00:22:05.515 --> 00:22:09.909
or a specific period of time and then send it proactively
280
00:22:10.149 --> 00:22:12.229
to not get the ask versus
281
00:22:12.309 --> 00:22:17.909
making it a batch process that runs in the background. But that said, there are some other also
282
00:22:18.309 --> 00:22:27.494
asks which are causing a little bit of delay because if I am from here asking for a Europe data center or AWS
283
00:22:27.495 --> 00:22:33.254
setup to run a query on the doors in, say, Germany or somewhere,
284
00:22:33.654 --> 00:22:35.255
then that does cause
285
00:22:36.080 --> 00:22:39.120
more delay than I'm running something in,
286
00:22:39.360 --> 00:22:40.320
The US
287
00:22:40.720 --> 00:22:42.000
data centers. So
288
00:22:42.960 --> 00:22:49.274
those particular things we are trying to figure out whether there should be a backup of it locally
289
00:22:49.595 --> 00:22:52.154
or at multiple nodes,
290
00:22:52.315 --> 00:22:57.354
and that's all on the road map. So we have still working through it.
291
00:22:58.154 --> 00:23:01.034
And a lot of what we've been discussing so far
292
00:23:02.070 --> 00:23:04.549
seems like it's focused predominantly
293
00:23:04.549 --> 00:23:05.269
on
294
00:23:05.350 --> 00:23:11.350
current state and recent state where you've mentioned some ability to do more longitudinal
295
00:23:11.350 --> 00:23:12.309
analysis.
296
00:23:12.309 --> 00:23:25.604
But I'm curious what are some of the potential capabilities that you see for being able to do some historical analysis of issues that have recurred and maybe doing some time series
297
00:23:25.684 --> 00:23:29.284
forward looking projections on being able to anticipate
298
00:23:29.720 --> 00:23:31.080
potential problems,
299
00:23:31.160 --> 00:23:33.799
whether that's for maintenance requirements or
300
00:23:33.960 --> 00:23:45.240
being able to adjust some of the delivery scheduling because of... You know, maybe we're... Because we're approaching the holiday season, you know that there's going to be a lot more volume churning through and some of the ways that that factors into
301
00:23:45.455 --> 00:23:48.575
the design of how you're processing the telemetry,
302
00:23:48.655 --> 00:23:49.935
both mechanically
303
00:23:49.935 --> 00:23:51.855
and using the LLMs.
304
00:23:52.655 --> 00:23:54.815
Yeah, absolutely. This is a great
305
00:23:55.615 --> 00:23:57.855
segue into the future plan
306
00:23:58.120 --> 00:24:08.679
because this agent... We have other agents also built by my team where we have sort of an agentic framework over the physical logistics automation, and we are trying to
307
00:24:09.080 --> 00:24:11.559
integrate that whole framework and
308
00:24:11.835 --> 00:24:13.274
create basically
309
00:24:13.674 --> 00:24:15.595
a portal sorts
310
00:24:15.835 --> 00:24:18.075
for the operations team
311
00:24:18.235 --> 00:24:19.034
where
312
00:24:19.115 --> 00:24:22.475
they can actually look at any door and
313
00:24:22.635 --> 00:24:24.395
the health status on that door.
314
00:24:25.390 --> 00:24:28.030
So it can tell you if the hardware
315
00:24:28.030 --> 00:24:32.989
that gives you the automated workflow is basically optimal at that point.
316
00:24:33.390 --> 00:24:35.149
And if there are any
317
00:24:35.230 --> 00:24:36.909
previous occurrences
318
00:24:36.990 --> 00:24:40.510
which are possible to happen again, previous incidents,
319
00:24:40.855 --> 00:24:43.174
which have either caused a
320
00:24:43.815 --> 00:24:47.734
high severity issue on that particular door, an outage,
321
00:24:48.054 --> 00:24:49.334
a Wi Fi issue,
322
00:24:49.894 --> 00:24:53.894
or, some sensors failing because of which this automated
323
00:24:54.070 --> 00:24:59.270
workflow itself could not be accessed and the vehicle had to be moved to a different door.
324
00:24:59.430 --> 00:25:02.230
So we are trying to exactly do that, that check
325
00:25:02.710 --> 00:25:08.390
it from the door, which is basically the object which we are tracking for the automated workflow
326
00:25:08.804 --> 00:25:16.324
and seeing if the failures in the past give us some context and pointers to see
327
00:25:16.725 --> 00:25:18.804
if a sensor is
328
00:25:19.044 --> 00:25:19.845
this
329
00:25:19.845 --> 00:25:21.524
particular is causing
330
00:25:21.950 --> 00:25:22.990
some concern,
331
00:25:23.150 --> 00:25:25.070
then how long do we have before
332
00:25:25.150 --> 00:25:29.629
that door starts giving an alarm or creates the...
333
00:25:29.870 --> 00:25:30.830
An incident?
334
00:25:30.990 --> 00:25:38.184
In addition, we also have connectivity issues going on at the site because there are various access points, which are
335
00:25:38.424 --> 00:25:47.544
sensors and other equipment connects to. And if there is a central IT team, which manages all of that. So if there is any access point signals,
336
00:25:47.705 --> 00:25:48.904
which are fluctuating,
337
00:25:49.065 --> 00:25:55.630
then our doors don't get signals and then our devices are not able to really work in full capacity.
338
00:25:55.870 --> 00:25:59.230
So that too, we are trying to see if automated
339
00:25:59.870 --> 00:26:04.030
tickets or automated incidents can be cut to the central
340
00:26:04.485 --> 00:26:07.125
support team where they can actually
341
00:26:07.365 --> 00:26:08.325
proactively
342
00:26:08.485 --> 00:26:09.285
fix
343
00:26:09.285 --> 00:26:11.605
any such issues where we are
344
00:26:12.325 --> 00:26:13.924
going to see problems.
345
00:26:13.925 --> 00:26:18.565
And we are currently figuring out what are the patterns that we see,
346
00:26:18.885 --> 00:26:20.725
which cause
347
00:26:19.909 --> 00:26:30.629
what issues exactly. So we are building an error framework with actual messages and the actions to be taken and the person or the role
348
00:26:30.630 --> 00:26:32.149
which will be taking that action.
349
00:26:32.935 --> 00:26:37.175
So based on that, we are creating an automated framework. So that's in
350
00:26:37.655 --> 00:26:38.615
our roadmap.
351
00:26:39.335 --> 00:26:44.295
One of the perennial challenges of doing anything at scale and across
352
00:26:44.620 --> 00:26:50.460
a broad set of... And especially physical locations is that you can't guarantee
353
00:26:50.460 --> 00:26:51.580
consistency,
354
00:26:52.140 --> 00:26:58.965
particularly when you're talking about any physical elements where different sensors might be running different versions of firmware,
355
00:26:58.965 --> 00:27:12.940
or you might have different generations of sensors across the different locations based on when and how they were built, or different locations might have different physical or geological constraints that prevent you from having the exact same
356
00:27:13.100 --> 00:27:14.620
estate everywhere.
357
00:27:14.860 --> 00:27:23.545
And then also because of the local differences between whether it's the plant management or just the seasonality
358
00:27:23.545 --> 00:27:29.545
of that location. People are going to be asking different questions of the system. So I'm wondering how you manage the
359
00:27:29.705 --> 00:27:30.585
scalability
360
00:27:30.585 --> 00:27:32.025
of the core
361
00:27:32.185 --> 00:27:33.065
platform
362
00:27:33.065 --> 00:27:35.625
while allowing for some of that
363
00:27:36.090 --> 00:27:37.690
more personalized
364
00:27:37.690 --> 00:27:41.289
access or personalized treatment for the different locations,
365
00:27:41.289 --> 00:27:42.009
given the
366
00:27:42.410 --> 00:27:46.009
degree of variance and the different axes along which they can vary?
367
00:27:47.450 --> 00:27:50.890
Yeah. So I would like to go back a little to answer that question.
368
00:27:51.775 --> 00:27:54.735
With the certification that we have for this
369
00:27:54.895 --> 00:27:55.774
setup,
370
00:27:56.335 --> 00:27:58.735
it was tested on all of the
371
00:27:59.455 --> 00:28:00.415
variants,
372
00:28:00.415 --> 00:28:02.654
I would say, whether it's temperature
373
00:28:02.655 --> 00:28:03.295
or
374
00:28:03.870 --> 00:28:06.909
location specific in terms of camera, if
375
00:28:07.150 --> 00:28:11.309
some locations are not getting enough light to actually
376
00:28:11.390 --> 00:28:18.430
see a vehicle backing up properly or seeing if somebody else is in the field in the yard, which is causing...
377
00:28:19.345 --> 00:28:23.665
Will be causing any harm or damage or fatality to that person.
378
00:28:24.385 --> 00:28:29.185
So this hole causes a little bit of difference, but when we were certified,
379
00:28:29.585 --> 00:28:31.665
the OSHA basically
380
00:28:33.140 --> 00:28:46.020
came to different locations where we are planning to install these devices and did a threshold analysis of what is the range we should be considering for each of our components in this hardware ecosystem.
381
00:28:46.905 --> 00:28:52.025
And we were able to either create the enclosures or add additional lenses
382
00:28:52.345 --> 00:28:53.145
where
383
00:28:53.225 --> 00:28:54.985
we are supporting the
384
00:28:55.705 --> 00:28:56.825
extraneous
385
00:28:57.865 --> 00:28:59.865
ends of our spectrum.
386
00:29:00.740 --> 00:29:07.540
And we are still very much within that. So we have not seen anything which will
387
00:29:07.700 --> 00:29:11.700
be pushing the boundaries either ways. So that's one. But
388
00:29:13.225 --> 00:29:14.584
in usage,
389
00:29:14.664 --> 00:29:17.224
there is variance in the usage. So
390
00:29:17.945 --> 00:29:21.624
also, as you correctly said, in the holiday season,
391
00:29:22.504 --> 00:29:31.690
all of the shifts are working extremely hard. So in some usages, we see people being a little bit too busy to take care of something
392
00:29:31.770 --> 00:29:43.850
which needs to be moved before a trailer gets out. So if there is like a fan component which has to be moved, then if they are forgetting to move that and it gets hit multiple times,
393
00:29:44.170 --> 00:29:48.384
then our camera basically won't capture the needed image.
394
00:29:48.545 --> 00:29:53.825
So to avoid that, we are making changes in the training plans to make sure
395
00:29:54.145 --> 00:29:57.425
that they have a very good understanding of
396
00:29:58.210 --> 00:30:07.570
exactly this needs to be done. Otherwise, the automated workflow will be failing or will be compromised the next time. So we try to make that
397
00:30:08.130 --> 00:30:10.770
into the newsletter of Amazon Logistics
398
00:30:10.770 --> 00:30:22.985
or making sure the sites which are non compliant or we see more problems in those hardware at those sites have these training inputs sent to them proactively with our data collection and analysis.
399
00:30:23.385 --> 00:30:28.000
And that's one of the usages I had not seen my agent
400
00:30:28.080 --> 00:30:35.679
being capable of doing, but this is a surprise to me as well that this insight was surfaced that we
401
00:30:36.240 --> 00:30:40.800
can do this training changes based on site level patterns
402
00:30:40.800 --> 00:30:53.495
that we see. So we are trying to address that as they come and send some proactive input to the learning teams, which sees... Takes over all of the Amazon logistics learning tasks.
403
00:30:53.735 --> 00:30:58.210
And that is one of the usage that are... We are looking to improve in the future.
404
00:30:58.610 --> 00:31:07.170
One of the other interesting aspects of running a platform that other people rely on is that sometimes you have to be responsible
405
00:31:07.170 --> 00:31:12.855
for identifying who are the heaviest users as well as whether or not it's being used at all.
406
00:31:13.335 --> 00:31:16.294
And so there's one aspect of that from the
407
00:31:16.615 --> 00:31:18.054
training and
408
00:31:18.375 --> 00:31:19.175
evangelizing
409
00:31:19.175 --> 00:31:23.255
perspective of just getting people to want to use the system. And then
410
00:31:23.650 --> 00:31:48.424
because of the fact that nothing in this world is free, you need to sometimes do cost attribution of who is causing the heaviest load on the system so that we can make sure that everybody is aware of the actual impacts of increased load or increased usage. And I'm wondering what are some of the elements of that training and onboarding as well as the cost attribution that needs to be done.
411
00:31:49.950 --> 00:31:51.070
Yeah.
412
00:31:51.870 --> 00:31:54.110
So Amazon engineering,
413
00:31:54.910 --> 00:31:57.870
from the various teams doing this workflow,
414
00:31:57.870 --> 00:32:00.669
as well as the operations leadership,
415
00:32:00.990 --> 00:32:05.684
they wanted to see economics per site and rightly so, as you said.
416
00:32:06.725 --> 00:32:11.525
So the option was to take the fleet functionality in AWS,
417
00:32:11.525 --> 00:32:15.044
which rolls up the costs by teams or orgs, but that's what,
418
00:32:15.684 --> 00:32:17.604
excuse me, that was not
419
00:32:18.565 --> 00:32:20.485
giving us the enough granularity,
420
00:32:21.030 --> 00:32:22.710
which was needed to see
421
00:32:23.190 --> 00:32:27.750
which were the power users or which is causing the most cost on the
422
00:32:28.150 --> 00:32:29.590
network wide usage.
423
00:32:29.910 --> 00:32:36.390
So we were looking as the objective to attribute every dollar of pipeline cost
424
00:32:35.785 --> 00:32:38.024
to sites specifically.
425
00:32:38.345 --> 00:32:39.625
So that was the task.
426
00:32:40.025 --> 00:32:40.904
And then
427
00:32:41.305 --> 00:32:51.950
what we did, to accommodate this ask was basically tag the Firehose streams by sites. So when the data comes in, we are tagging it by site. So every
428
00:32:52.190 --> 00:32:55.950
query on the s three that we are generating
429
00:32:56.030 --> 00:33:02.990
with a conversational ask from our org team or some analysis request from other teams,
430
00:33:03.695 --> 00:33:10.335
then we roll up those costs daily into a cost ledger sort of for the sites.
431
00:33:11.055 --> 00:33:11.695
So
432
00:33:12.175 --> 00:33:14.575
we create alerts on
433
00:33:14.735 --> 00:33:18.495
every site cost per assignment, which is above threshold.
434
00:33:19.310 --> 00:33:26.590
So whether it's a day, whether it's a week, whether it's a month, we take care of all the periods and try to generate these thresholds
435
00:33:27.150 --> 00:33:29.950
with which we can send out these alerts to
436
00:33:30.110 --> 00:33:31.950
whoever is the
437
00:33:31.950 --> 00:33:37.195
POC for that site, as well as the leadership for operations at
438
00:33:37.515 --> 00:33:38.475
logistics
439
00:33:38.475 --> 00:33:39.835
as well as our
440
00:33:40.315 --> 00:33:41.595
product leadership.
441
00:33:41.835 --> 00:33:45.755
With this, we make sure that the leaders see the cost
442
00:33:46.570 --> 00:33:49.850
per site and also the anomalies that we are seeing.
443
00:33:50.090 --> 00:33:55.450
So whether the, workflow is not functioning properly, that's why we are seeing this high cost,
444
00:33:55.850 --> 00:34:02.124
or there is some additional stuff going on there. So the anomalies are detected pretty easily.
445
00:34:02.205 --> 00:34:11.805
Then we have to do some root cause analysis with the help of our data gathered. But the anomalies, I would say, are caught within a day almost because of our
446
00:34:12.365 --> 00:34:15.430
tagging the site to the rolled up cost.
447
00:34:16.070 --> 00:34:25.670
And this helps, tremendously figure out the power users, figure out any anomalies, figure out any issues in training or, in the hardware and so on.
448
00:34:26.545 --> 00:34:27.185
And
449
00:34:27.345 --> 00:34:34.145
going back to the LLM question, you mentioned that you're still working on dialing in the accuracy
450
00:34:34.225 --> 00:34:40.785
and some of the specific use cases and investing in making it more of an agentic capability.
451
00:34:41.600 --> 00:34:48.240
And agents in particular can be challenging because of the fact that they're so open ended in terms of their capabilities.
452
00:34:48.400 --> 00:34:50.720
And so I'm wondering what your infrastructure
453
00:34:50.720 --> 00:34:54.000
is for managing the experimentation
454
00:34:54.000 --> 00:34:56.925
and being able to track and evaluate
455
00:34:56.925 --> 00:35:06.365
the accuracy as far as when and how it improves or degrades so that you can continue to maintain and evolve the system with confidence.
456
00:35:08.540 --> 00:35:11.500
As you correctly said, LLM outputs are probabilistic.
457
00:35:11.500 --> 00:35:12.060
So
458
00:35:13.100 --> 00:35:13.820
the
459
00:35:14.380 --> 00:35:17.980
operations humans will trust this summary if it looks confident,
460
00:35:17.980 --> 00:35:20.620
but there could be hallucinations.
461
00:35:21.020 --> 00:35:22.140
So
462
00:35:22.460 --> 00:35:25.180
we
463
00:35:23.925 --> 00:35:25.525
have, like, a structure
464
00:35:25.525 --> 00:35:31.685
created for the output, which are based on their ledger events and event IDs.
465
00:35:31.925 --> 00:35:35.765
So we attach that to every summary at the bottom. So even if
466
00:35:36.540 --> 00:35:42.620
that particular operations user are not aware what that really means, but any...
467
00:35:42.780 --> 00:35:47.500
If we see their feedback and we collect regular feedback from every
468
00:35:48.905 --> 00:35:50.345
question asked.
469
00:35:50.425 --> 00:35:51.865
The operations
470
00:35:52.185 --> 00:35:53.545
personnel have
471
00:35:53.865 --> 00:35:57.945
a way to, at the end, rate it if the issue was resolved
472
00:35:58.105 --> 00:35:58.905
or
473
00:35:59.145 --> 00:36:01.785
it was basically not the correct root cause.
474
00:36:02.450 --> 00:36:06.850
And based on the last twelve months of the summarizer
475
00:36:06.850 --> 00:36:08.130
that I have created,
476
00:36:08.770 --> 00:36:12.130
there was no production incident traceable to the hallucination.
477
00:36:12.530 --> 00:36:14.690
So since this infra setup was completed.
478
00:36:15.155 --> 00:36:20.755
But saying this, it's not true that we won't be facing this. So we are trying to
479
00:36:21.395 --> 00:36:26.835
basically limit the summary length. We are trying to, as I said, attach the ledger event and event IDs
480
00:36:27.190 --> 00:36:28.390
and the vocabulary.
481
00:36:28.390 --> 00:36:30.870
We are having very strict vocabulary
482
00:36:30.950 --> 00:36:31.750
control
483
00:36:31.750 --> 00:36:32.710
on the,
484
00:36:33.590 --> 00:36:35.750
agent take a framework as well.
485
00:36:36.310 --> 00:36:42.150
So this along with the human in the loop, like, enable the users to share feedback on the resolution recommendation,
486
00:36:43.015 --> 00:36:45.975
that actually has a power to
487
00:36:46.055 --> 00:36:49.735
do a quick check before actually letting the associates know.
488
00:36:50.215 --> 00:36:51.735
So that is a multiple layer.
489
00:36:51.975 --> 00:36:54.695
First of all, the summary is pretty well structured.
490
00:36:55.700 --> 00:37:02.099
And then there is a human in the loop, which is our operations team who's communicating the root cause to the
491
00:37:02.900 --> 00:37:03.779
associate
492
00:37:03.779 --> 00:37:11.695
saying, do this. It will solve the problem. So if they think think something is seriously wrong, they will give us the feedback and stop
493
00:37:12.175 --> 00:37:17.055
giving that feedback ahead or result ahead to the actual associate.
494
00:37:17.135 --> 00:37:19.295
So that's our current method.
495
00:37:19.295 --> 00:37:21.455
And once we get enough feedback
496
00:37:21.935 --> 00:37:24.655
and saying, okay, this is good enough
497
00:37:24.974 --> 00:37:26.095
as per the threshold,
498
00:37:26.450 --> 00:37:31.730
and we should be just creating an automated way for associates to ask without
499
00:37:31.810 --> 00:37:34.930
our operations team involved in every ask
500
00:37:34.930 --> 00:37:36.130
at every point.
501
00:37:36.450 --> 00:37:40.930
We will be automating this in terms of tickets coming in, answers going out,
502
00:37:41.330 --> 00:37:41.890
and so on.
503
00:37:42.565 --> 00:37:43.605
So,
504
00:37:43.605 --> 00:37:46.005
yeah, that's our plan. Another
505
00:37:46.005 --> 00:37:53.845
interesting element of the system that you're working with is because you're dealing with such a high volume and velocity of data,
506
00:37:54.085 --> 00:37:57.525
it opens the question as far as what is
507
00:37:57.290 --> 00:37:58.730
the utility
508
00:37:58.730 --> 00:37:59.370
of
509
00:37:59.690 --> 00:38:06.330
using embeddings or if embeddings are useful at all given the scale and shape of the data that you're dealing with.
510
00:38:06.890 --> 00:38:14.224
And if and when you do decide to bring some form of embeddings and vector search into that agentic harness,
511
00:38:14.704 --> 00:38:25.265
whether and how you're evaluating the viability of s three vectors as that storage layer since you've invested so heavily in s three as the core storage layer for the rest of the system?
512
00:38:27.690 --> 00:38:30.410
Yeah. So we have not... That's a
513
00:38:31.210 --> 00:38:32.170
awesome
514
00:38:32.730 --> 00:38:33.770
next step,
515
00:38:33.930 --> 00:38:34.490
but
516
00:38:34.890 --> 00:38:39.850
I have not had enough deep dive into what could be the next
517
00:38:41.415 --> 00:38:56.330
better option for us to invest in. But as we... I was telling you, we are trying to create this agentic framework. And at that point, we will be looking at our infrastructure to see what we can make better in terms of search and
518
00:38:56.570 --> 00:38:59.530
the capability of reducing the delay or,
519
00:38:59.930 --> 00:39:06.010
improving the cost element. So this is one of the topics we will be looking into. But currently,
520
00:39:06.890 --> 00:39:07.930
I do not have any
521
00:39:09.525 --> 00:39:14.725
great knowledge for that. I have been pretty busy making this agent useful,
522
00:39:14.805 --> 00:39:17.445
but that would definitely be the logical next step
523
00:39:17.845 --> 00:39:18.645
for us.
524
00:39:19.525 --> 00:39:23.125
And as you have been building this system and
525
00:39:23.600 --> 00:39:29.440
training up the various stakeholders and end users on its capabilities,
526
00:39:29.600 --> 00:39:40.065
what are some of the most interesting or innovative or unexpected ways that you have either seen it used or seen it change some of outcomes that it is designed to feed into.
527
00:39:41.905 --> 00:39:44.145
So let me think about this a little.
528
00:39:46.785 --> 00:39:50.545
Yeah. So our operations team is basically...
529
00:39:50.865 --> 00:39:53.185
Has a specific way of
530
00:39:53.930 --> 00:39:54.730
working
531
00:39:54.730 --> 00:39:56.250
with the associates.
532
00:39:56.490 --> 00:40:01.369
So sometimes they are on-site, sometimes they are at a different location, and they are still
533
00:40:01.690 --> 00:40:05.049
looking at their phones, seeing what is the issue
534
00:40:05.595 --> 00:40:09.995
on a video call with the associates saying, okay. This is what is happening at the
535
00:40:10.315 --> 00:40:12.635
door and what the issue could be.
536
00:40:13.115 --> 00:40:14.155
So in that,
537
00:40:14.715 --> 00:40:21.275
they are unable to basically log on to a proper system and start any
538
00:40:22.450 --> 00:40:23.410
Amazon
539
00:40:23.410 --> 00:40:26.850
agentic framework on anything like that. So
540
00:40:27.410 --> 00:40:28.530
my system
541
00:40:28.610 --> 00:40:29.330
is
542
00:40:30.050 --> 00:40:32.530
basically compliant enough that
543
00:40:32.770 --> 00:40:34.690
operations teams have
544
00:40:35.315 --> 00:40:36.994
their own wrapper
545
00:40:36.994 --> 00:40:40.035
on top of it, whether it is
546
00:40:41.154 --> 00:40:43.714
a very, very conversational chatbot
547
00:40:43.714 --> 00:40:45.714
made in Claude or
548
00:40:45.795 --> 00:40:54.760
there is a different kind of interface that they are currently using and this is an API call that coming to my agent or my framework,
549
00:40:55.480 --> 00:40:57.400
I have made those available
550
00:40:57.480 --> 00:40:58.920
to integrate with.
551
00:40:59.240 --> 00:41:10.105
So this request I got after I thought of a conversational AI chatbot agent, and they use it when they are at a site or Amazon corporate sites and working with the associates.
552
00:41:10.425 --> 00:41:21.800
But this was something I had not initially planned for. But based on their feedback, we were able to make this integration touch points available to them with which they find their best way to create that
553
00:41:22.200 --> 00:41:25.000
for their wrapper to integrate with my framework.
554
00:41:25.914 --> 00:41:28.075
So that makes it more usable,
555
00:41:28.474 --> 00:41:29.994
to multiple contractors
556
00:41:29.994 --> 00:41:30.955
as well as
557
00:41:31.835 --> 00:41:36.795
in future associates as well at the site when the operation dependencies
558
00:41:36.795 --> 00:41:37.675
go away.
559
00:41:38.075 --> 00:41:39.914
So this is something I had not
560
00:41:40.480 --> 00:41:41.520
anticipated
561
00:41:41.520 --> 00:41:42.560
supporting.
562
00:41:43.120 --> 00:41:46.240
Also, the other thing as I was saying was training needs.
563
00:41:46.480 --> 00:41:50.160
So depending on the shift, we can actually trace the
564
00:41:50.640 --> 00:41:53.680
callouts or different behavior patterns
565
00:41:53.920 --> 00:41:55.760
down to a shift site.
566
00:41:56.295 --> 00:42:01.815
We can also go to a user ID, but we try to avoid doing that. So up to a shift, if we do that,
567
00:42:02.135 --> 00:42:04.375
then the training teams get
568
00:42:04.535 --> 00:42:05.415
valuable
569
00:42:05.415 --> 00:42:06.695
feedback saying,
570
00:42:07.255 --> 00:42:13.470
this training seems to be missing because the workflow goes up to this point without a hitch. And here,
571
00:42:13.870 --> 00:42:17.550
many people are going back to doing the old way.
572
00:42:17.950 --> 00:42:19.790
In that case, they know exactly
573
00:42:20.190 --> 00:42:21.470
where the training
574
00:42:21.630 --> 00:42:24.030
doesn't seem to be working for those folks
575
00:42:24.365 --> 00:42:29.565
Because based on cultural differences or any... Even regional differences, there could be
576
00:42:29.885 --> 00:42:33.005
different needs that the people have in those areas
577
00:42:33.085 --> 00:42:38.684
in terms of training. So they will be able to modify the training based on these patterns.
578
00:42:38.684 --> 00:42:39.724
So I'm giving
579
00:42:40.059 --> 00:43:02.375
right now training teams access directly to my framework instead of me sending out the summaries to them. This is one of the things I'm currently working on so that they get the insights by themselves as per the format they want, and they can make those decisions on training updates and stuff. So these are two of the things that I had not initially thought of. But after
580
00:43:02.375 --> 00:43:02.935
the
581
00:43:03.175 --> 00:43:10.990
framework was used, these came to the notice. So I'm looking forward to even supporting more cross teams and more functionality.
582
00:43:11.550 --> 00:43:19.710
And as you have been working in this space and building this system, what are some of the most interesting or unexpected or challenging lessons that you've learned personally?
583
00:43:21.625 --> 00:43:28.425
So I would say the culture wise, right? Amazon's culture, as you had also hinted a little bit earlier,
584
00:43:28.825 --> 00:43:30.904
makes it easier to ship these,
585
00:43:31.144 --> 00:43:31.944
more than
586
00:43:32.710 --> 00:43:36.550
at a startup. But then, again, at Amazon scale,
587
00:43:36.950 --> 00:43:39.990
you are actually launching these tools
588
00:43:40.310 --> 00:43:41.510
to the network.
589
00:43:41.910 --> 00:43:44.310
So it does not give us the organizational
590
00:43:44.310 --> 00:43:46.310
patience, say, for a slow rollout.
591
00:43:47.484 --> 00:43:49.325
So we have been
592
00:43:49.965 --> 00:43:51.805
walking this thin line
593
00:43:51.965 --> 00:43:55.325
with boring kind of small blast radius
594
00:43:55.484 --> 00:43:56.365
method
595
00:43:56.365 --> 00:44:00.925
where we are actually launching it to test doors at the sites where we
596
00:44:01.490 --> 00:44:04.450
have the test doors and making sure we
597
00:44:04.770 --> 00:44:08.610
test out the full functionality there and then go network wide.
598
00:44:08.930 --> 00:44:15.170
But we also have something planned for next year where we would be able to select the sites
599
00:44:15.250 --> 00:44:23.865
and the doors in advance and go at at least at a few doors per site in the areas which seem pretty diverse
600
00:44:23.865 --> 00:44:24.585
and
601
00:44:24.745 --> 00:44:26.505
test it out without,
602
00:44:27.065 --> 00:44:28.265
like, compromising
603
00:44:28.265 --> 00:44:29.545
on basically
604
00:44:29.545 --> 00:44:34.640
just the test doors that we have and so on. So we are building this intrinsically
605
00:44:34.640 --> 00:44:37.359
into the software layer
606
00:44:37.680 --> 00:44:38.320
of
607
00:44:38.960 --> 00:44:42.240
slow rollout and targeted rollout for testing,
608
00:44:42.559 --> 00:44:44.240
and then we can go network wide.
609
00:44:44.865 --> 00:44:51.985
So that's one of the unexpected challenges that we saw, and we are still working on fixing it as
610
00:44:52.145 --> 00:44:53.425
what our need.
611
00:44:55.265 --> 00:45:05.950
And you mentioned a little bit on some of the future work that you have planned for this system. I'm just curious, what are some of the ways that you anticipate
612
00:45:05.950 --> 00:45:08.430
the broader ecosystem
613
00:45:08.910 --> 00:45:10.510
encompassing and integrating
614
00:45:10.510 --> 00:45:11.150
with
615
00:45:11.310 --> 00:45:14.285
this physical logistics automation platform,
616
00:45:14.285 --> 00:45:20.125
particularly as you make it more agentic and as multi agent systems become
617
00:45:20.925 --> 00:45:22.605
part of the general
618
00:45:22.605 --> 00:45:24.525
substrate of how we build technical systems?
619
00:45:25.610 --> 00:45:27.530
That's a great question. So I would
620
00:45:27.850 --> 00:45:30.090
up level this a little bit. What
621
00:45:30.810 --> 00:45:52.435
everybody is doing is building AI agent tech systems, and they are trying to build agents which actually do the work and then connect with some other agent. But modular agentic AI systems is something which I see a need at least for such physical logistics automation. So whatever we are doing in terms of connecting the vehicles to the doors,
622
00:45:53.349 --> 00:45:55.430
there are other players,
623
00:45:55.510 --> 00:45:56.950
other Amazon teams
624
00:45:57.109 --> 00:46:01.430
doing a similar effort but for different part of the logistics
625
00:46:01.430 --> 00:46:02.390
workflows.
626
00:46:02.710 --> 00:46:04.550
So some part of the modular
627
00:46:04.710 --> 00:46:05.589
framework
628
00:46:05.589 --> 00:46:09.635
could be shared between these teams, like, specifically
629
00:46:10.915 --> 00:46:12.275
for authorization,
630
00:46:12.595 --> 00:46:14.595
authentication and authorization
631
00:46:15.395 --> 00:46:16.195
parts
632
00:46:16.195 --> 00:46:16.915
or
633
00:46:17.075 --> 00:46:18.515
basically rule based
634
00:46:18.869 --> 00:46:19.510
checks
635
00:46:19.750 --> 00:46:23.509
where there is an integration with a very specific hierarchy
636
00:46:23.509 --> 00:46:24.710
in logistics.
637
00:46:24.950 --> 00:46:29.589
So that that part could be a modular, and then we could have a cross team
638
00:46:29.750 --> 00:46:30.630
integration,
639
00:46:30.630 --> 00:46:34.865
basically. And that would be very easy to even integrate across agents
640
00:46:34.865 --> 00:46:35.905
in that case.
641
00:46:36.225 --> 00:46:40.385
So trying to build a cross org council at the logistics level
642
00:46:40.625 --> 00:46:43.345
where it's more focused on automation
643
00:46:43.425 --> 00:46:49.570
and various teams in Amazon, which are trying to automate that workflow, whether it's the robots loading
644
00:46:49.810 --> 00:46:50.530
or,
645
00:46:51.250 --> 00:46:53.890
like, something like docking that I'm working on
646
00:46:54.290 --> 00:46:56.130
or even loading and loading
647
00:46:56.450 --> 00:46:59.650
the belts, automated belts. Those all work together.
648
00:47:00.174 --> 00:47:05.855
So we are in... Very in the initial stages to figure out that what the mechanism will be and
649
00:47:06.095 --> 00:47:08.415
how the cross org council could
650
00:47:08.575 --> 00:47:11.454
make it more modular so that ongoing,
651
00:47:11.454 --> 00:47:12.815
there is less of a
652
00:47:13.055 --> 00:47:15.320
disposable work and more like,
653
00:47:15.560 --> 00:47:17.560
on top of each other we are building.
654
00:47:18.200 --> 00:47:19.480
So that's one of the,
655
00:47:19.960 --> 00:47:21.240
items I'm
656
00:47:21.240 --> 00:47:25.400
seriously looking to pursue along with other POCs from other teams.
657
00:47:27.095 --> 00:47:31.415
Are there any other aspects of this overall space of Amazon's
658
00:47:31.415 --> 00:47:39.015
physical logistics automation or the systems that you're building to support that that we didn't discuss yet that you'd like to cover before we close out the show?
659
00:47:39.710 --> 00:47:41.950
I think we covered most of it.
660
00:47:42.430 --> 00:47:57.905
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for building data systems today.
661
00:47:59.665 --> 00:48:01.105
I think governance
662
00:48:01.184 --> 00:48:02.305
for the
663
00:48:02.625 --> 00:48:03.825
probabilistic
664
00:48:03.825 --> 00:48:04.785
writers
665
00:48:05.425 --> 00:48:11.280
is the biggest gap I see. So every governance framework I have used assumes
666
00:48:11.280 --> 00:48:13.440
basically deterministic producers.
667
00:48:13.599 --> 00:48:19.360
But AI agents that write to your operational store are probabilistic producers, and none
668
00:48:19.440 --> 00:48:20.640
are the mature
669
00:48:20.720 --> 00:48:21.119
tooling.
670
00:48:22.704 --> 00:48:29.745
So they don't know how to draw a boundary around them or track their provenance or roll back their effects cleanly.
671
00:48:29.825 --> 00:48:33.265
So specifically the last one, rolling back the effects cleanly.
672
00:48:33.585 --> 00:48:38.700
So you have touched this in a previous episode as well, I think the Omnigraph episode.
673
00:48:38.859 --> 00:48:44.940
And I think that is the frontier for the next at least few years that we will have to concentrate on.
674
00:48:45.579 --> 00:48:49.660
Yeah. It's definitely a interesting time to be working in technology.
675
00:48:49.660 --> 00:48:57.285
So I appreciate you taking the time today to join me and share the work that you've been doing and some of the ways that you're integrating the
676
00:48:57.445 --> 00:48:59.125
physical requirements
677
00:48:59.125 --> 00:49:07.400
into these technical systems and bringing LLMs to bear to make the overall system easier to interact with and more powerful for
678
00:49:07.560 --> 00:49:18.200
the people who rely on it. So I appreciate all the time and energy you're putting into that, and I hope you enjoy the rest of your day. Thank you so much, Tobias. It was exciting to speak with you and your audience about this.
679
00:49:26.085 --> 00:49:30.405
Thank you for listening, and don't forget to check out our other shows. Podcast.net
680
00:49:30.405 --> 00:49:39.560
covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems.
681
00:49:40.119 --> 00:49:56.275
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about Email hosts at data engineering podcast dot com with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.