Data Corpus: Why Smallholder AI Starts With What We Don't Collect

WEBVTT
00:00:00.000 --> 00:00:02.439
When we talk about large language models, we
00:00:02.439 --> 00:00:05.719
really need data to drive their development and
00:00:05.719 --> 00:00:08.099
delivery of valued services. AI in agriculture
00:00:08.099 --> 00:00:12.279
is only as good as the data it is built on. Data
00:00:12.279 --> 00:00:15.400
exists, but it sits in silos. We don't have a
00:00:15.400 --> 00:00:18.140
data scarcity problem. We have a data fragmentation
00:00:18.140 --> 00:00:20.980
problem. When you are trying to drill down into
00:00:20.980 --> 00:00:23.100
who should make these decisions, how should they
00:00:23.100 --> 00:00:25.379
be made, it's got to be really contextualized
00:00:25.379 --> 00:00:27.719
in the realities in that country or subnational
00:00:27.719 --> 00:00:31.019
context in order to fit the right kind of data
00:00:31.019 --> 00:00:34.679
governance and data sharing. How do we generate
00:00:34.679 --> 00:00:38.340
the local data set so that we... are not actually
00:00:38.340 --> 00:00:41.100
delivering harmful information in the context
00:00:41.100 --> 00:00:43.039
of the environment we're trying to serve. So
00:00:43.039 --> 00:00:47.020
I think the real value from a researcher's perspective
00:00:47.020 --> 00:00:50.119
is more strategic decisions and taking calculated
00:00:50.119 --> 00:00:53.219
risks so that we can improve farmers' livelihood
00:00:53.219 --> 00:00:56.500
and benefit and income at the end of the day
00:00:56.500 --> 00:01:08.439
and the end of the season. Hello, my name is
00:01:08.439 --> 00:01:11.000
David Bergvinson and welcome to Grounded Intelligence.
00:01:11.099 --> 00:01:12.980
This is a series that looks at the responsible
00:01:12.980 --> 00:01:16.040
use of AI by engaging the community on their
00:01:16.040 --> 00:01:20.480
perspectives on the use and application and some
00:01:20.480 --> 00:01:22.819
of the concerns around artificial intelligence
00:01:22.819 --> 00:01:25.540
to empower farmers to realize their full economic
00:01:25.540 --> 00:01:29.000
potential and to manage the challenges of small
00:01:29.000 --> 00:01:33.430
-scale farming in emerging markets. uh we'll
00:01:33.430 --> 00:01:36.250
be going through a whole series on this topic
00:01:36.250 --> 00:01:39.890
starting with the data corpus or data and i'm
00:01:39.890 --> 00:01:44.069
joined by three very insightful guests who are
00:01:44.069 --> 00:01:48.049
dealing with fragmented data systems and transposing
00:01:48.049 --> 00:01:51.329
those into valuable integrated data sets to drive
00:01:51.329 --> 00:01:54.650
models to generate agricultural insights to help
00:01:54.650 --> 00:01:57.750
farmers make better decisions on the ground some
00:01:57.750 --> 00:01:59.530
of the terms you're going to hear are like frontier
00:01:59.530 --> 00:02:02.909
models these are large language models that have
00:02:02.909 --> 00:02:05.230
been developed by large companies like microsoft
00:02:05.230 --> 00:02:09.370
or open ai and you may be familiar with some
00:02:09.370 --> 00:02:13.169
of their names like chat gpt or claude but we're
00:02:13.169 --> 00:02:14.830
also going to talk about small language models
00:02:14.830 --> 00:02:17.530
and the data that is needed to drive those small
00:02:17.530 --> 00:02:19.969
language models which is a critical gap so we're
00:02:19.969 --> 00:02:22.430
going to explore these in the discussion ahead
00:02:22.430 --> 00:02:26.729
so we welcome your suggestions please leave those
00:02:26.729 --> 00:02:30.469
below on questions that have been raised through
00:02:30.469 --> 00:02:33.610
this conversation and suggestions of future conversations
00:02:33.610 --> 00:02:36.710
as we work together towards unlocking the power
00:02:36.710 --> 00:02:39.889
and potential of artificial intelligence to enable
00:02:39.889 --> 00:02:41.949
farmers to realize their full economic potential
00:02:41.949 --> 00:02:50.770
well so welcome everyone today we're going to
00:02:50.770 --> 00:02:54.969
be looking at the topic of data and so in the
00:02:54.969 --> 00:02:56.849
series of grounded intelligence there's many
00:02:56.849 --> 00:02:59.849
aspects of the responsible use of ai And data
00:02:59.849 --> 00:03:02.449
is a core component, because without data, we
00:03:02.449 --> 00:03:05.250
can't really drive the whole system. So this
00:03:05.250 --> 00:03:07.949
is a foundational topic, and we're privileged
00:03:07.949 --> 00:03:14.169
to be joined by three experts, Jawu from IFPRI,
00:03:14.310 --> 00:03:20.449
and Somya and Mike from Athena Infonomics. And
00:03:20.449 --> 00:03:22.169
so I'm going to let each of them introduce themselves,
00:03:22.229 --> 00:03:24.349
just so that you can understand the experts that
00:03:24.349 --> 00:03:27.050
we heard. we have to facilitate this discussion
00:03:27.050 --> 00:03:30.050
so jao would you mind a brief introduction yeah
00:03:30.050 --> 00:03:33.030
sure yeah thanks yeah hi everyone my name is
00:03:33.030 --> 00:03:36.210
jao ku i'm a senior research fellow at international
00:03:36.210 --> 00:03:38.969
food policy research institute based in washington
00:03:38.969 --> 00:03:42.349
dc my background is crop modeling in agricultural
00:03:42.349 --> 00:03:46.030
engineering field so i work on many different
00:03:46.030 --> 00:03:50.009
types of data that goes into the crop model and
00:03:50.009 --> 00:03:53.270
also trying to understand you know what models
00:03:53.819 --> 00:03:56.240
rather generate. So, yeah, it's very interdisciplinary
00:03:56.240 --> 00:04:00.139
research I've been doing at EPRIN for over almost
00:04:00.139 --> 00:04:04.860
two decades now. Thank you, Joe. Mike, over to
00:04:04.860 --> 00:04:10.699
you. Sure. Thank you, David. So, Mike Minkoff,
00:04:10.699 --> 00:04:15.479
I've been overseeing, operating as the director
00:04:15.479 --> 00:04:17.839
of artificial intelligence and digital public
00:04:17.839 --> 00:04:24.379
infrastructure services at Athena. Been particularly
00:04:24.379 --> 00:04:28.699
focused in this kind of ag XAI community of practice
00:04:28.699 --> 00:04:33.199
and working on issues around especially data
00:04:33.199 --> 00:04:36.139
corpus and benchmarking deeply in that community
00:04:36.139 --> 00:04:38.519
of practice for the last year on the heels of
00:04:38.519 --> 00:04:43.199
a 15 year career in global development that span
00:04:43.199 --> 00:04:46.319
working in environment and sustainability, climate
00:04:46.319 --> 00:04:49.759
risk, and more recently kind of data systems,
00:04:49.920 --> 00:04:54.300
data governance and emerging technology. Thank
00:04:54.300 --> 00:04:59.339
you. And Soumya, please. Hi, I'm Soumya Alamuru.
00:04:59.819 --> 00:05:03.579
I am a senior consultant in Athena Infonomics.
00:05:03.600 --> 00:05:07.420
I did a PhD in economics. Now, working closely
00:05:07.420 --> 00:05:10.139
with AP government, we are providing technical
00:05:10.139 --> 00:05:13.740
assistance to Andhra Pradesh, which is a state
00:05:13.740 --> 00:05:17.680
in India, on AI advisory. I'm leading the project
00:05:17.680 --> 00:05:22.350
from Andhra Pradesh. Wonderful. So we have some
00:05:22.350 --> 00:05:25.170
practical field experience. So we're well represented
00:05:25.170 --> 00:05:29.189
here. So our opening topic, as I said, you know,
00:05:29.189 --> 00:05:32.930
around data, when we talk about large language
00:05:32.930 --> 00:05:35.930
models, we really need data to, you know, drive
00:05:35.930 --> 00:05:38.209
their development and delivery of valued services.
00:05:38.430 --> 00:05:41.310
And so let's talk about why data comes first.
00:05:41.449 --> 00:05:44.389
So Jao, would you, you know, give us a little
00:05:44.389 --> 00:05:46.790
bit of a tour around the AI models and tools
00:05:46.790 --> 00:05:50.120
that we'll be exploring? And from your perspective,
00:05:50.180 --> 00:05:53.420
why does the data layer matter so much in realizing
00:05:53.420 --> 00:05:56.600
the benefits of AI in agriculture? Yeah, sure.
00:05:56.800 --> 00:05:59.519
So, yeah, again, because my background is crop
00:05:59.519 --> 00:06:03.379
modeling, I always have that tendency to apply
00:06:03.379 --> 00:06:06.620
all this data topic and similar discussion in
00:06:06.620 --> 00:06:09.379
the context of crop modeling. So bear with me.
00:06:09.939 --> 00:06:13.040
So crop models, for example, it's not exactly
00:06:13.040 --> 00:06:17.899
AI model, but it has the same constraint on data.
00:06:18.810 --> 00:06:22.850
Any model, including AI models, are only as good
00:06:22.850 --> 00:06:27.370
as the data we provide to develop the models
00:06:27.370 --> 00:06:31.389
for. So, for example, if the models are only
00:06:31.389 --> 00:06:35.029
trained or developed based on specific types
00:06:35.029 --> 00:06:37.730
of environments, say, like North American agricultural
00:06:37.730 --> 00:06:41.730
systems and maize farming in large -scale farms,
00:06:43.410 --> 00:06:48.379
it may not have all... a good enough understanding
00:06:48.379 --> 00:06:50.980
of what are the challenges that farmers are facing,
00:06:51.120 --> 00:06:53.939
how to address the risk, and what kind of advice
00:06:53.939 --> 00:06:57.420
it needs to provide under certain climate conditions
00:06:57.420 --> 00:07:00.660
for small -scale farmers in developing countries
00:07:00.660 --> 00:07:04.970
where we are focusing on. So that will introduce
00:07:04.970 --> 00:07:09.529
inherent biases. The models might start giving
00:07:09.529 --> 00:07:12.910
advisories that are not relevant for smallholder
00:07:12.910 --> 00:07:15.329
farmers, not really actionable. It might provide
00:07:15.329 --> 00:07:19.430
advisories to access tools and use chemicals
00:07:19.430 --> 00:07:22.290
that are not even available in smallholder farming
00:07:22.290 --> 00:07:25.009
systems in the countries where farmers are part
00:07:25.009 --> 00:07:30.410
of. So that kind of limitations all start stemming
00:07:30.410 --> 00:07:34.279
from the data. data we provide to the mother
00:07:34.279 --> 00:07:36.600
and the mothers are trained for really need to
00:07:36.600 --> 00:07:42.120
be extensively comprehensive for the environment
00:07:42.120 --> 00:07:48.139
that where our target users are, you know, cultivating
00:07:48.139 --> 00:07:51.060
farms, cultivating crops for. So that really
00:07:51.060 --> 00:07:55.860
worries me. So all these large scale hyperscalers,
00:07:55.860 --> 00:08:01.540
hyperscaling models are using data from English
00:08:01.540 --> 00:08:06.259
language in well -defined farming systems and
00:08:06.259 --> 00:08:09.259
well -documented systems. So it may not capture
00:08:09.259 --> 00:08:13.939
all these nuances or really detailed information
00:08:13.939 --> 00:08:18.779
on smallholder farmer system. So yeah, it might
00:08:18.779 --> 00:08:22.620
miss all these details or these challenges of
00:08:22.620 --> 00:08:24.579
smallholder farmers. Yeah, the context of that
00:08:24.579 --> 00:08:27.889
data is very important, as you pointed out. So
00:08:27.889 --> 00:08:30.949
as we dig deeper into understanding the problem
00:08:30.949 --> 00:08:33.909
of data, I'm going to turn to Mike and Soumya
00:08:33.909 --> 00:08:38.549
on this, because a lot of the current AI efforts
00:08:38.549 --> 00:08:41.409
in emerging markets relies on existing data sets.
00:08:41.690 --> 00:08:44.750
So, Soumya, first to you, drawing on your experience
00:08:44.750 --> 00:08:47.450
in Andhra Pradesh, India, in scaling AI advisory,
00:08:47.730 --> 00:08:50.090
what do you see as the biggest challenges around
00:08:50.090 --> 00:08:54.190
data access to drive AI -enabled advisory services?
00:08:55.340 --> 00:08:58.779
Well, as Jahu correctly said, AI in agriculture
00:08:58.779 --> 00:09:03.100
is only as good as the data it is built on. And
00:09:03.100 --> 00:09:05.460
in regions like Andhra Pradesh, the biggest challenge
00:09:05.460 --> 00:09:09.139
is not lack of AI, but the lack of reliable,
00:09:09.240 --> 00:09:14.799
representative and usable data. The data is fragmented
00:09:14.799 --> 00:09:17.720
across departments. Data exists, but it sits
00:09:17.720 --> 00:09:20.639
in silos. We don't have a data scarcity problem.
00:09:20.820 --> 00:09:24.080
We have a data fragmentation problem. For example,
00:09:24.240 --> 00:09:26.899
agriculture department holds crop data, farmer
00:09:26.899 --> 00:09:31.220
level data, AP planning department manages procurement
00:09:31.220 --> 00:09:36.720
data, AP space application holds satellite imagery,
00:09:37.039 --> 00:09:41.159
and banks hold credit data. None of these were
00:09:41.159 --> 00:09:44.519
designed to talk to each other. Moreover, institutional
00:09:44.519 --> 00:09:46.980
silos lead to duplicated data collection and
00:09:46.980 --> 00:09:52.559
incompatible formats. challenge is last mile
00:09:52.559 --> 00:09:56.279
data gaps. The most granular actionable data,
00:09:56.460 --> 00:10:01.019
what a former example, what a former in one of
00:10:01.019 --> 00:10:04.980
the districts actually planted when they irrigated,
00:10:05.080 --> 00:10:08.700
what inputs they used rarely gets captured digitally.
00:10:09.139 --> 00:10:12.700
Village level agriculture assistants still work
00:10:12.700 --> 00:10:19.600
on paper in many corners of Andhra Pradesh. The
00:10:19.600 --> 00:10:24.159
third, perhaps most critical and deeply structural
00:10:24.159 --> 00:10:27.100
challenge is the mismatch between farmer identity
00:10:27.100 --> 00:10:32.000
and land records. In AB, more than 80 % of cultivable
00:10:32.000 --> 00:10:35.159
land is operated by tenant farmers. Among those,
00:10:35.220 --> 00:10:39.659
a substantial portion is informal tenant farmers.
00:10:40.700 --> 00:10:43.539
They often don't appear in official datasets.
00:10:47.080 --> 00:10:50.179
major challenges that are there in Andhra Pradesh.
00:10:50.399 --> 00:10:54.039
So what you get is AI system trained on skewed
00:10:54.039 --> 00:10:56.460
data if you don't consider these challenges.
00:10:57.120 --> 00:11:00.480
Yeah, good point. Mike, you know, so what is
00:11:00.480 --> 00:11:03.220
the big risk then given what Ramya just said
00:11:03.220 --> 00:11:07.159
around building these systems that are using
00:11:07.159 --> 00:11:09.740
data that really isn't designed to serve small
00:11:09.740 --> 00:11:12.419
-scale producers in mind? Ramya, in your experience,
00:11:12.600 --> 00:11:15.399
what... are the farmer realities that are most
00:11:15.399 --> 00:11:19.279
frequently uh not considered or excluded or distorted
00:11:19.279 --> 00:11:22.059
in the current data ecosystem drawing again on
00:11:22.059 --> 00:11:26.940
your ap work as i said earlier tenant farmers
00:11:26.940 --> 00:11:31.659
they are out of the data structure ap has a significant
00:11:31.659 --> 00:11:34.179
portion of cultivated land it is operated by
00:11:34.179 --> 00:11:36.759
tenant farmers who are invisible to government
00:11:36.759 --> 00:11:39.740
data systems they don't appear in land records
00:11:39.740 --> 00:11:44.700
don't receive crop insurance or schemes uh their
00:11:44.700 --> 00:11:48.320
use of inputs yields are not captured anywhere
00:11:48.320 --> 00:11:52.360
this is one of the uh biggest issues farmers
00:11:52.360 --> 00:11:56.840
are tenant farmers sorry come what percentage
00:11:56.840 --> 00:12:00.580
of the farmers in ap are tenant farmers uh percentages
00:12:00.580 --> 00:12:04.159
wise it is like around 30 to 40 percent but 80
00:12:04.159 --> 00:12:07.679
percent of cultivable land is operated by tenant
00:12:07.679 --> 00:12:14.850
person tenant farmers so the but those farms
00:12:14.850 --> 00:12:16.830
would have data though it's just that they don't
00:12:16.830 --> 00:12:19.029
have data on the tenant farmer who's cultivating
00:12:19.029 --> 00:12:22.710
that one yeah the the like the kind of crops
00:12:22.710 --> 00:12:26.190
they saw or the inputs they use or the market
00:12:26.190 --> 00:12:31.370
or the yields they they arrive at are not captured
00:12:31.370 --> 00:12:35.590
anywhere and most of these farmers hold most
00:12:35.590 --> 00:12:38.470
of these farmers would be cultivating in less
00:12:38.470 --> 00:12:43.620
than two acres of land And these government data
00:12:43.620 --> 00:12:49.440
collection systems have thresholds like less
00:12:49.440 --> 00:12:53.019
than one acre or two acres are not captured in
00:12:53.019 --> 00:12:56.039
the data systems. So that's a big challenge.
00:12:56.240 --> 00:13:02.399
Yeah, it is. Second is land is disproportionately
00:13:02.399 --> 00:13:07.159
registered in men's name. In Andhra Pradesh or
00:13:07.159 --> 00:13:12.519
in India, farming is a family. It's never one
00:13:12.519 --> 00:13:16.100
person's job. The whole family, husband and wife
00:13:16.100 --> 00:13:18.600
and their children gets involved in the farming.
00:13:19.679 --> 00:13:24.059
In most of the cases, probably women take farming
00:13:24.059 --> 00:13:27.779
decisions, but the lands are not their names.
00:13:27.940 --> 00:13:31.720
We don't consider women farmers as farmers and
00:13:31.720 --> 00:13:35.440
the data doesn't capture that part of it. Yeah,
00:13:35.460 --> 00:13:38.039
you've raised some two very good examples around
00:13:38.039 --> 00:13:42.230
inclusion. And we'll circle back to those later.
00:13:42.490 --> 00:13:44.990
You know, one of our other big challenges, as
00:13:44.990 --> 00:13:47.889
you said, Soumya, is the fragmented data. And
00:13:47.889 --> 00:13:51.809
so, you know, going to this topic of shared data,
00:13:51.990 --> 00:13:56.850
we call it a corpus or a body of data. Jau, I'd
00:13:56.850 --> 00:14:00.230
like to get your perspectives on the discussion
00:14:00.230 --> 00:14:03.950
papers that AgXAI is working on, especially as
00:14:03.950 --> 00:14:06.250
we talk about data corpus and the sharing of
00:14:06.250 --> 00:14:11.259
data. Why is this concept so important in addressing
00:14:11.259 --> 00:14:14.539
this isolated data ecosystem that agriculture
00:14:14.539 --> 00:14:19.080
faces? Right. So there are two things for this
00:14:19.080 --> 00:14:22.240
question. So one is to really understand what
00:14:22.240 --> 00:14:24.679
kind of language the farmers are speaking, even
00:14:24.679 --> 00:14:27.500
in just agriculture in general, which may or
00:14:27.500 --> 00:14:31.620
may not be similar to a general purpose. like
00:14:31.620 --> 00:14:34.480
large language models are trained on. Agricultural
00:14:34.480 --> 00:14:38.080
languages and norms and specific terminology,
00:14:38.340 --> 00:14:41.720
some are really colloquial terms that only kind
00:14:41.720 --> 00:14:44.899
of used in, you know, farming context, again,
00:14:45.039 --> 00:14:47.960
like may or may not be so central to the large
00:14:47.960 --> 00:14:51.179
language model. So the providing and managing
00:14:51.179 --> 00:14:54.600
and consolidating this agriculture data corpus
00:14:54.600 --> 00:14:58.779
is really important if we are serious about providing
00:14:58.779 --> 00:15:02.120
really actionable and, you know, meaningful advisory
00:15:02.120 --> 00:15:06.120
for farmers. They wouldn't take... advisory seriously
00:15:06.120 --> 00:15:10.039
if we use very generalized language not really
00:15:10.039 --> 00:15:14.960
speaking the same tone or same kind of the terminology.
00:15:15.379 --> 00:15:17.379
So I think that's really important to communicate
00:15:17.379 --> 00:15:20.759
better with farmers and understand better what
00:15:20.759 --> 00:15:23.519
kind of language they are speaking. Also in generally
00:15:23.519 --> 00:15:26.639
this kind of corpus sharing shared data corpus
00:15:26.639 --> 00:15:31.759
provides again the more broader context to the
00:15:31.759 --> 00:15:34.379
types of information and other values that we
00:15:34.379 --> 00:15:38.419
are providing. It's not just agriculture on the
00:15:38.419 --> 00:15:41.419
ground and field level expertise and field level
00:15:41.419 --> 00:15:46.080
practices. It also needs to touch on labor, agricultural
00:15:46.080 --> 00:15:48.919
labor and financial access and everything in
00:15:48.919 --> 00:15:50.720
rural communities also needs to be captured.
00:15:51.019 --> 00:15:53.480
And again, this just cannot be done, cannot be
00:15:53.480 --> 00:15:56.419
captured in one single or limited number of data
00:15:56.419 --> 00:15:58.940
sources. So this shared data corpus is really
00:15:58.940 --> 00:16:02.389
important for for reaching out to farmers and
00:16:02.389 --> 00:16:04.909
providing meaningful advisories. Yeah, so you
00:16:04.909 --> 00:16:07.950
lead to an interesting point here, really around
00:16:07.950 --> 00:16:10.129
the balance between scale and specificity of
00:16:10.129 --> 00:16:12.629
these data sets. Sami has already mentioned that
00:16:12.629 --> 00:16:16.590
large segments of society are missing in these
00:16:16.590 --> 00:16:20.350
representative data sets. So how do we find the
00:16:20.350 --> 00:16:23.309
right balance? Because going into that very granular
00:16:23.309 --> 00:16:26.789
data, that costs quite a bit of money to collect.
00:16:27.090 --> 00:16:30.049
So how do we find this balance between the specificity
00:16:30.509 --> 00:16:34.470
and scaling these geographically large data sets,
00:16:34.509 --> 00:16:38.149
at least from the CG perspective. Yeah, I know.
00:16:38.210 --> 00:16:41.730
So this is really challenging in a way. As a
00:16:41.730 --> 00:16:44.070
research institution, things are rapidly changing
00:16:44.070 --> 00:16:48.850
so fast. So, yeah, it's even just hard to just
00:16:48.850 --> 00:16:53.600
keep. keep track of what's going on, where we
00:16:53.600 --> 00:16:56.399
should focus. So getting the right balance is
00:16:56.399 --> 00:16:59.299
absolutely critical. I think the general agricultural
00:16:59.299 --> 00:17:03.139
knowledge and to some extent, the country level,
00:17:03.200 --> 00:17:05.839
large scale, large economy level information
00:17:05.839 --> 00:17:08.480
and data and knowledge are very well captured
00:17:08.480 --> 00:17:13.619
in hyperscalers and global scale corpus kind
00:17:13.619 --> 00:17:16.380
of exercise already. So I think what we can do.
00:17:16.890 --> 00:17:20.630
again, as an institution like CGIR, is to provide
00:17:20.630 --> 00:17:25.349
more detailed, focused, really thoughtful capturing
00:17:25.349 --> 00:17:28.630
of the processes that farmers are, you know,
00:17:28.650 --> 00:17:32.109
challenges that farmers are dealing with. Like,
00:17:32.150 --> 00:17:36.849
for example, if we are to provide advisories
00:17:36.849 --> 00:17:40.289
for, like, say, chili farmers in India, the general
00:17:40.289 --> 00:17:42.950
farming practices or the best practice for chili
00:17:42.950 --> 00:17:46.480
farming in India. must be already captured very
00:17:46.480 --> 00:17:49.460
well in the larger corpus or the existing corpus
00:17:49.460 --> 00:17:51.839
already out there. But what we can do is, what
00:17:51.839 --> 00:17:55.319
is the latest trend of the managing charities?
00:17:55.480 --> 00:17:59.299
What is the market demand, emerging kind of consumer
00:17:59.299 --> 00:18:03.960
demand, and something more actionable, more dynamic?
00:18:05.430 --> 00:18:07.490
types of data, I think that will be probably
00:18:07.490 --> 00:18:10.869
more meaningful effort that we can provide so
00:18:10.869 --> 00:18:13.670
that we can be complementary and to reach the
00:18:13.670 --> 00:18:17.349
balance of the specificity and scaling. So that's
00:18:17.349 --> 00:18:19.450
a great segue into some broader issues around
00:18:19.450 --> 00:18:22.410
governance incentives and trust, which is so
00:18:22.410 --> 00:18:25.650
important. If you betray the trust of the consumer
00:18:25.650 --> 00:18:28.009
in this, it'll take a long time for them to come
00:18:28.009 --> 00:18:30.450
back to this technology. So Sami, I'd like to
00:18:30.450 --> 00:18:34.980
ask you around data governance. it's a term that
00:18:34.980 --> 00:18:37.440
feels abstract so why don't you give it some
00:18:37.440 --> 00:18:40.599
context as it relates to the reality on the ground
00:18:40.599 --> 00:18:43.700
in ap around data sharing access and responsible
00:18:43.700 --> 00:18:47.720
use of personal information so i can talk about
00:18:47.720 --> 00:18:54.180
um ap ai advisory is still growing in in the
00:18:54.180 --> 00:18:57.480
sense like we really don't have advisory systems
00:18:57.480 --> 00:19:01.420
that uh actually considers granular level data
00:19:01.420 --> 00:19:07.339
or use a data corpus which is which can which
00:19:07.339 --> 00:19:10.279
can includes granular level to aggregate data
00:19:10.279 --> 00:19:16.059
but what a representative data would be looking
00:19:16.059 --> 00:19:22.880
like is to cover all agro economic zones in ap
00:19:22.880 --> 00:19:28.430
and major eight to ten cropping systems and irrigation
00:19:28.430 --> 00:19:32.869
systems like rainfed, canal or borewell or tank
00:19:32.869 --> 00:19:36.769
irrigated context and full spectrum of farm size
00:19:36.769 --> 00:19:42.369
from 0 .5 acres to 20 plus acres. If a data set
00:19:42.369 --> 00:19:45.730
covers all of these aspects, we can say that
00:19:45.730 --> 00:19:50.309
is a representative data. And if I have to give
00:19:50.309 --> 00:19:55.529
a recent example of a good AI model from Andhra
00:19:55.529 --> 00:20:02.319
Pradesh, A model that uses Godavari Delta region,
00:20:02.579 --> 00:20:05.500
it's a real -time Godavari, Godavari is a river
00:20:05.500 --> 00:20:07.920
in Andhra Pradesh. It is real -time Godavari
00:20:07.920 --> 00:20:11.900
flood advisory is a game changer. When the state
00:20:11.900 --> 00:20:14.500
integrated IMD weather feeds with canal flow
00:20:14.500 --> 00:20:18.200
data and farmer location data, it became possible
00:20:18.200 --> 00:20:21.140
to push geo -targeted text -based flood warnings
00:20:21.140 --> 00:20:24.789
to farmers in specific regions. These flood alerts
00:20:24.789 --> 00:20:27.930
help farmers in many ways, not just the safety
00:20:27.930 --> 00:20:32.109
point, but to make a decision on sowing or harvesting.
00:20:35.150 --> 00:20:38.950
And recently, like last year, like a few, another
00:20:38.950 --> 00:20:43.549
example I can give is ahead of Cyclone Month
00:20:43.549 --> 00:20:50.750
into 2025. AP Ames, which is a data repository
00:20:50.750 --> 00:20:54.220
system that Recently, the Andhra Pradesh government
00:20:54.220 --> 00:20:58.779
has started building and delivered 3D flood simulations
00:20:58.779 --> 00:21:02.019
and early warnings via SMS and IVR to millions
00:21:02.019 --> 00:21:06.900
of farmers, which helped in protecting 800 ,000
00:21:06.900 --> 00:21:14.480
hectares from flood damage. So, I mean, that's
00:21:14.480 --> 00:21:16.960
really important, especially against the backdrop
00:21:16.960 --> 00:21:20.599
of climate change and extreme rainfall events.
00:21:21.950 --> 00:21:26.410
i i i turn to you mike around okay at a higher
00:21:26.410 --> 00:21:28.990
level who's making the decisions and on what
00:21:28.990 --> 00:21:32.150
data is made publicly available and how is it
00:21:32.150 --> 00:21:34.950
regulated maybe just drawing on your experience
00:21:34.950 --> 00:21:38.849
again with ap or perhaps african nations that
00:21:38.849 --> 00:21:42.940
you've worked with Yeah, I mean, I think that
00:21:42.940 --> 00:21:45.839
the question of who is making those decisions
00:21:45.839 --> 00:21:48.839
or who should make those decisions is an interesting
00:21:48.839 --> 00:21:51.759
one and a challenging one. And I think what we've
00:21:51.759 --> 00:21:56.019
seen, and I know, you know, in other fora, Jiawu,
00:21:56.119 --> 00:22:00.339
we've kind of touched on this, right, where there's
00:22:00.339 --> 00:22:04.519
definitely not one size fits all. And, you know,
00:22:04.559 --> 00:22:06.359
the models that are successful, I mean, even
00:22:06.359 --> 00:22:09.759
if you look at India, India's got a variety of
00:22:09.759 --> 00:22:12.519
models in the way that they are approaching kind
00:22:12.519 --> 00:22:14.759
of the challenge around building data corpus
00:22:14.759 --> 00:22:19.079
or corpora. There's a kind of federated structure
00:22:19.079 --> 00:22:21.700
that's been built at the federal level through
00:22:21.700 --> 00:22:26.259
Open Agronet. AgriStack and Vista, you know,
00:22:26.259 --> 00:22:29.299
now Bharat Vistar. And then there's kind of state
00:22:29.299 --> 00:22:32.599
level deployments or adjacent deployments. Andhra
00:22:32.599 --> 00:22:35.460
Pradesh is kind of an adjacent deployment to
00:22:35.460 --> 00:22:38.539
that. Maharashtra has a state level. And all
00:22:38.539 --> 00:22:40.079
of these have kind of different functions, you
00:22:40.079 --> 00:22:42.099
know, different features and different roles
00:22:42.099 --> 00:22:45.240
of government. And then if you look at, you know,
00:22:45.259 --> 00:22:47.339
context in certain countries in sub -Saharan
00:22:47.339 --> 00:22:50.180
Africa. It's very different as far as what is
00:22:50.180 --> 00:22:52.140
the existing digital infrastructure capacity
00:22:52.140 --> 00:22:54.859
and capability? What is the existing government
00:22:54.859 --> 00:22:57.380
kind of capacity and levels of trust in government
00:22:57.380 --> 00:23:00.700
to fund and monitor and manage something like
00:23:00.700 --> 00:23:05.180
this effectively? So that's all a kind of long
00:23:05.180 --> 00:23:09.180
preface to say, I think when you are trying to
00:23:09.180 --> 00:23:11.180
drill down into who should make these decisions,
00:23:11.339 --> 00:23:14.279
how should they be made? It really has to be.
00:23:16.069 --> 00:23:18.210
contextualized to the political economic reality,
00:23:18.549 --> 00:23:23.910
the techno -institutional realities, the sociocultural
00:23:23.910 --> 00:23:28.230
realities, choose your hyphenate terminology,
00:23:28.670 --> 00:23:30.670
but it's got to be really contextualized to the
00:23:30.670 --> 00:23:33.349
realities in that country or subnational context
00:23:33.349 --> 00:23:38.430
in order to fit the right kind of data governance
00:23:38.430 --> 00:23:43.279
and data sharing processes and protocols. uh
00:23:43.279 --> 00:23:47.339
that that can really kind of sustain um and and
00:23:47.339 --> 00:23:51.720
those need to be designed with real clear -eyed
00:23:51.720 --> 00:23:54.299
on kind of the what's in it for them for all
00:23:54.299 --> 00:23:57.200
of the parties involved right and like how how
00:23:57.200 --> 00:23:59.559
are they gonna you know what are the meaningful
00:23:59.559 --> 00:24:02.220
incentives to sustain contribution and collaboration
00:24:02.220 --> 00:24:04.940
through this data sharing mechanism and partnership
00:24:04.940 --> 00:24:08.180
and so whether that's a government entity that's
00:24:08.180 --> 00:24:11.019
contributing whether it's a research institute
00:24:11.019 --> 00:24:14.000
or like network like cgi are whether it's private
00:24:14.000 --> 00:24:16.640
sector actors that you that have great data but
00:24:16.640 --> 00:24:18.839
have competitive interests and maybe not sharing
00:24:18.839 --> 00:24:22.640
all of it uh you know all all of them have their
00:24:22.640 --> 00:24:26.079
different sure realities yeah well let's let's
00:24:26.079 --> 00:24:29.140
pivot to joe then from the research context perspective
00:24:29.140 --> 00:24:31.940
around the data sharing uh and the incentives
00:24:31.940 --> 00:24:35.890
for them joe are there you know incentives or
00:24:35.890 --> 00:24:38.349
policies in place that facilitate data sharing
00:24:38.349 --> 00:24:41.589
across the cg and make it accessible to a broad
00:24:41.589 --> 00:24:44.750
range of stakeholders in country absolutely so
00:24:44.750 --> 00:24:48.920
cgir As the largest publicly funded agriculture
00:24:48.920 --> 00:24:51.400
research institution in the world, we do have
00:24:51.400 --> 00:24:56.839
a policy since 2021 on fair and open data access
00:24:56.839 --> 00:25:00.279
across all the CGR centers, including my center,
00:25:00.480 --> 00:25:03.319
IPRI. So all the research output, research data,
00:25:03.519 --> 00:25:05.480
research publications should be in the public
00:25:05.480 --> 00:25:09.710
domain. And eventually, we are given a little
00:25:09.710 --> 00:25:12.450
bit of embargo period while we are developing
00:25:12.450 --> 00:25:15.589
and finalizing publication. But the idea is that
00:25:15.589 --> 00:25:19.230
everything publicly funded project that produced
00:25:19.230 --> 00:25:21.849
should be in the public domain. But having said
00:25:21.849 --> 00:25:24.289
that, we do have a challenge. We have a little
00:25:24.289 --> 00:25:27.430
bit of issue or say mismatch between the purpose
00:25:27.430 --> 00:25:30.589
of this policy and what's really useful purposeful.
00:25:31.289 --> 00:25:35.490
like say training AI and LLMs and AI models for
00:25:35.490 --> 00:25:38.650
artificial services. The nature of our research
00:25:38.650 --> 00:25:43.490
is to develop and generate knowledge around how
00:25:43.490 --> 00:25:46.710
we improve farming and how we improve productivity,
00:25:47.109 --> 00:25:51.549
etc. All this towards good outcomes and better
00:25:51.549 --> 00:25:55.920
outcomes in food systems. This may or may not
00:25:55.920 --> 00:25:59.480
be the reality in the farmer's field. So we often
00:25:59.480 --> 00:26:03.039
have, like say, agronomic trials, very controlled
00:26:03.039 --> 00:26:06.019
environment where we know what's going in and
00:26:06.019 --> 00:26:08.200
what's coming out. We can measure perfectly.
00:26:08.539 --> 00:26:12.519
We can observe perfectly so that we have better
00:26:12.519 --> 00:26:14.819
understanding of what works and what doesn't.
00:26:15.500 --> 00:26:18.960
So again, this may or may not be really relevant
00:26:18.960 --> 00:26:22.180
for smallholder farming systems. So we always
00:26:22.180 --> 00:26:27.420
have a little bit of the issue. or trying to
00:26:27.420 --> 00:26:31.039
fully understand and digest the research output,
00:26:31.140 --> 00:26:33.720
research findings, and turn that into agronomic
00:26:33.720 --> 00:26:37.900
or farming advisory is always a little bit of
00:26:37.900 --> 00:26:41.220
kind of, there is a human really need to be in
00:26:41.220 --> 00:26:44.099
the loop so that interpret that research finding
00:26:44.099 --> 00:26:46.700
into the right context for farmers. So that's
00:26:46.700 --> 00:26:49.680
a little bit of issue. So having said that, research
00:26:49.680 --> 00:26:53.200
data is, again, still fully open. We have a policy.
00:26:53.559 --> 00:26:56.619
We also have a governance framework we are developing
00:26:56.619 --> 00:27:00.519
that defines four specific roles in this data
00:27:00.519 --> 00:27:04.059
ecosystem. Data owner, data lead, data producer,
00:27:04.279 --> 00:27:07.599
and data users. And we are also articulating
00:27:07.599 --> 00:27:10.740
whose responsibility, at what stage, at what
00:27:10.740 --> 00:27:14.059
level. So who will make such decisions on data
00:27:14.059 --> 00:27:19.109
sharing and opening data. yeah in different different
00:27:19.109 --> 00:27:22.369
ways so yeah it's a lot of very complex issues
00:27:22.369 --> 00:27:24.890
and we are dealing with again a research institution
00:27:24.890 --> 00:27:27.990
can do so much so we really need partners on
00:27:27.990 --> 00:27:30.130
the ground to make sure what we are generating
00:27:30.130 --> 00:27:33.109
is a fully useful and meaningful for farmers
00:27:33.109 --> 00:27:35.470
too and all this is happening in the context
00:27:35.470 --> 00:27:38.269
of things moving so fast in the ai world you
00:27:38.269 --> 00:27:41.460
mentioned frontier models and for for those not
00:27:41.460 --> 00:27:43.599
familiar these are you know the models that have
00:27:43.599 --> 00:27:47.460
been generated uh by anthropic called claude
00:27:47.460 --> 00:27:52.500
or through open ai chat gpt gemini in case of
00:27:52.500 --> 00:27:57.099
google meta etc and they're moving at such a
00:27:57.099 --> 00:27:59.720
pace and yet the clients that we're trying to
00:27:59.720 --> 00:28:01.559
serve namely smallholder farmers and emerging
00:28:01.559 --> 00:28:05.519
markets that's a very different ecosystems and
00:28:05.519 --> 00:28:07.480
so there's some hard choices and trade -offs
00:28:07.480 --> 00:28:13.500
to to manage here So in light of that comment
00:28:13.500 --> 00:28:18.000
and these rapidly moving frontier models, how
00:28:18.000 --> 00:28:21.900
do we generate the local data set so that we
00:28:21.900 --> 00:28:25.019
are not actually delivering harmful information
00:28:25.019 --> 00:28:27.140
that isn't in the context of the environment
00:28:27.140 --> 00:28:29.519
we're trying to serve because it's coming from
00:28:29.519 --> 00:28:33.900
an advanced economy data set? How do we sort
00:28:33.900 --> 00:28:35.660
of round that off? And Mike, maybe just give
00:28:35.660 --> 00:28:37.500
some of your high -level perspectives on this,
00:28:37.599 --> 00:28:43.180
given your India experience. Yeah, I mean, what's
00:28:43.180 --> 00:28:46.279
really one of the interesting parts of this,
00:28:46.440 --> 00:28:49.299
right? A, there's kind of a juxtaposition or
00:28:49.299 --> 00:28:51.420
tradeoff between the use of the frontier models
00:28:51.420 --> 00:28:55.339
that you described and do they have the good
00:28:55.339 --> 00:28:58.440
data in them, period. But even if they do, there's
00:28:58.440 --> 00:29:02.279
a different question of... Do we want to be using
00:29:02.279 --> 00:29:04.579
frontier models in these contexts all the time?
00:29:04.640 --> 00:29:07.720
Or, I mean, increasingly we're seeing how powerful
00:29:07.720 --> 00:29:11.380
and how accurate small language models, localized
00:29:11.380 --> 00:29:14.539
small language models can be if they have the
00:29:14.539 --> 00:29:17.779
right data in hand. And there's also these kind
00:29:17.779 --> 00:29:21.789
of ongoing subtext questions around... sovereignty
00:29:21.789 --> 00:29:25.710
of models and, you know, on -premise solutions
00:29:25.710 --> 00:29:29.009
and on -site data storage and management. And
00:29:29.009 --> 00:29:32.269
so I think there's a couple layers to it. And
00:29:32.269 --> 00:29:35.509
I think, you know, to collect that data, right,
00:29:35.589 --> 00:29:40.599
as small language models get more powerful. you
00:29:40.599 --> 00:29:43.799
actually, it starts lowering the burden of how
00:29:43.799 --> 00:29:46.900
much good data you need to get to be effective,
00:29:47.299 --> 00:29:50.799
to be able to develop effective localized models
00:29:50.799 --> 00:29:54.140
that can... fill in some of these gaps that these
00:29:54.140 --> 00:29:57.359
frontier models don't have. And that also then
00:29:57.359 --> 00:29:59.779
helps you solve for these sovereignty challenge
00:29:59.779 --> 00:30:02.500
and on premise data management challenges and
00:30:02.500 --> 00:30:05.220
compute cost and infrastructure cost challenges
00:30:05.220 --> 00:30:08.019
that are, that are real, real limiting factors
00:30:08.019 --> 00:30:10.180
in a lot of low and middle income country context.
00:30:10.240 --> 00:30:16.619
So, so I would say, you know, Continuing through
00:30:16.619 --> 00:30:20.160
programs like what we're doing in Andhra Pradesh
00:30:20.160 --> 00:30:22.220
right now or other programs, that gives you really
00:30:22.220 --> 00:30:25.359
an opportunity to start collecting this localized
00:30:25.359 --> 00:30:30.299
data or work that CGIR can do and different approaches
00:30:30.299 --> 00:30:35.720
to complementary data collection. If I was capturing
00:30:35.720 --> 00:30:40.220
what you were saying correctly, those present
00:30:40.220 --> 00:30:42.880
opportunities to start building these complementary...
00:30:43.680 --> 00:30:47.640
more localized data sets that can feed into whether
00:30:47.640 --> 00:30:50.420
it becomes a frontier model at a large language
00:30:50.420 --> 00:30:52.500
model scale or it becomes small language model
00:30:52.500 --> 00:30:57.960
deployments. Either way, I think, and from what
00:30:57.960 --> 00:31:00.400
I'm seeing, it seems like small language models
00:31:00.400 --> 00:31:03.019
will increasingly be the way to go. I think that's
00:31:03.019 --> 00:31:07.180
kind of where the opportunity continues to emerge.
00:31:07.460 --> 00:31:09.039
Well, especially around the data sovereignty
00:31:09.039 --> 00:31:11.940
and model sovereignty that you mentioned, that's
00:31:11.940 --> 00:31:15.130
important. you know i and you brought this up
00:31:15.130 --> 00:31:17.950
earlier on jawu is you know we're using that
00:31:17.950 --> 00:31:21.710
data to make decisions through modeling and so
00:31:21.710 --> 00:31:24.829
you know you also pointed out that we need quality
00:31:24.829 --> 00:31:27.710
data to drive a quality model that delivers relevant
00:31:27.710 --> 00:31:32.009
results so what does good data look like to you
00:31:32.009 --> 00:31:36.789
that enables farmers advisories or system level
00:31:36.789 --> 00:31:40.349
interventions to be effective using this new
00:31:40.349 --> 00:31:44.460
technology Right. So through our partnership
00:31:44.460 --> 00:31:49.200
with Digital Green, who are already scaling this
00:31:49.200 --> 00:31:53.099
agricultural advisory, the AI -powered agricultural
00:31:53.099 --> 00:31:56.759
advisory to many, many farmers. So we actually
00:31:56.759 --> 00:31:59.279
had a chance to look into what kind of questions
00:31:59.279 --> 00:32:02.400
farmers ask. And there are just two groups of,
00:32:02.420 --> 00:32:05.750
you know, two. I guess a bucket of questions.
00:32:05.849 --> 00:32:07.970
One are very strategic, one are very tactical.
00:32:08.230 --> 00:32:11.410
So tactical questions like what should I do right
00:32:11.410 --> 00:32:14.690
now for addressing my problem? What should I
00:32:14.690 --> 00:32:17.329
do today or tomorrow? And then there are more
00:32:17.329 --> 00:32:22.109
strategic questions. Okay, how do I change my
00:32:22.109 --> 00:32:24.890
farming to be more profitable, more resilient
00:32:24.890 --> 00:32:27.140
to climate change and things like that? What
00:32:27.140 --> 00:32:30.119
we saw was that AI models are currently used
00:32:30.119 --> 00:32:32.920
for very tactical questions only. And sometimes
00:32:32.920 --> 00:32:34.859
farmers even ask questions they already have
00:32:34.859 --> 00:32:37.420
no answer to. They are just testing how good
00:32:37.420 --> 00:32:41.500
this model is. So I think the real value from
00:32:41.500 --> 00:32:44.339
a researcher's perspective is more strategic
00:32:44.339 --> 00:32:49.160
decisions and taking calculated risks so that
00:32:49.160 --> 00:32:51.839
we can improve farmers' livelihood and benefit
00:32:51.839 --> 00:32:54.680
and income at the end of the day and the end
00:32:54.680 --> 00:32:58.430
of the season. the more sophisticated system
00:32:58.430 --> 00:33:00.910
we are developing and more comprehensive data
00:33:00.910 --> 00:33:03.430
we are providing, I think we can enable a farmer
00:33:03.430 --> 00:33:08.450
to think more medium to long -term kind of goals
00:33:08.450 --> 00:33:11.710
and developing vision and plan for that so that
00:33:11.710 --> 00:33:14.269
they can really improve their livelihood over
00:33:14.269 --> 00:33:18.250
longer term and more thriving in farming. Yeah.
00:33:18.369 --> 00:33:21.609
And I think, you know, in my experience, the
00:33:21.609 --> 00:33:24.009
market integration is a really important component
00:33:24.009 --> 00:33:26.289
of this so it's the advisory to produce the crop
00:33:26.289 --> 00:33:28.910
but as you point out thinking even beyond that
00:33:28.910 --> 00:33:31.809
strategically what market is that crop going
00:33:31.809 --> 00:33:34.529
to go into and planning accordingly so very good
00:33:34.529 --> 00:33:38.670
point sami i turn to you around examples where
00:33:38.670 --> 00:33:44.829
data design and collection has materially changed
00:33:44.829 --> 00:33:47.690
the outcome of a project so you started with
00:33:47.690 --> 00:33:50.779
a protocol for either pulling in existing data
00:33:50.779 --> 00:33:52.880
or even surveying for new data but found that
00:33:52.880 --> 00:33:56.259
they weren't meeting the needs or delivering
00:33:56.259 --> 00:34:00.279
the kind of benefits you anticipated. So how
00:34:00.279 --> 00:34:04.059
has that played out in advisory services in AP
00:34:04.059 --> 00:34:06.619
where you've seen this kind of situation where
00:34:06.619 --> 00:34:09.960
data quality or access has really limited the
00:34:09.960 --> 00:34:13.260
ability to deliver the impact desired by the
00:34:13.260 --> 00:34:17.139
government, the key stakeholders and farmers
00:34:17.139 --> 00:34:21.820
themselves? Yeah, as I mentioned earlier, like
00:34:21.820 --> 00:34:26.079
I just want to add one point to the when we are
00:34:26.079 --> 00:34:28.400
talking about what kind of data do we have to
00:34:28.400 --> 00:34:34.360
collect and how we how do we on what basis we
00:34:34.360 --> 00:34:38.960
build AI systems, the data collection should
00:34:38.960 --> 00:34:42.139
take a bottom up approach. Usually what happens
00:34:42.139 --> 00:34:45.059
is like we have district level state level data
00:34:45.059 --> 00:34:50.199
and then we just. use averages to proportionally
00:34:50.199 --> 00:34:53.000
distribute to lower levels village level or farmer
00:34:53.000 --> 00:34:56.199
level but if we start from the farmer level or
00:34:56.199 --> 00:34:58.780
village level that what data exists and then
00:34:58.780 --> 00:35:02.079
add up to you know a mental level or district
00:35:02.079 --> 00:35:07.280
level and or state level your data would be much
00:35:07.280 --> 00:35:10.059
more close to the ground and it would be reliable
00:35:10.059 --> 00:35:17.670
so recently in Andhra Pradesh some real efforts
00:35:17.670 --> 00:35:23.230
have been put in place to make this data corpus
00:35:23.230 --> 00:35:26.070
the government is working on building large integrated
00:35:26.070 --> 00:35:30.329
data repositories like ap aims which is andhra
00:35:30.329 --> 00:35:32.829
pradesh agriculture information management system
00:35:32.829 --> 00:35:36.250
that aims to bring various data points including
00:35:36.250 --> 00:35:39.050
cropping patterns soil health pest management
00:35:39.050 --> 00:35:44.150
weather data all into a single platform So using
00:35:44.150 --> 00:35:48.530
this platform recently, they have started delivering
00:35:48.530 --> 00:35:53.750
weather alerts. It could be like cyclone alerts
00:35:53.750 --> 00:35:58.849
or whirlwind alerts. These are really having
00:35:58.849 --> 00:36:04.750
an impact on farmer life. These have a positive
00:36:04.750 --> 00:36:08.710
impact on their farming decisions, also on their
00:36:08.710 --> 00:36:13.019
safety nets. that is one example i could give
00:36:13.019 --> 00:36:19.900
in and then if i have to talk about poor quality
00:36:19.900 --> 00:36:23.960
ai advice which are like there are many ai agri
00:36:23.960 --> 00:36:28.400
fintech companies are releasing a lot of ai advisories
00:36:28.400 --> 00:36:32.019
which depend on standard university or you know
00:36:32.019 --> 00:36:38.059
textbook guidance which are like useless to particularly
00:36:38.059 --> 00:36:45.139
regional smallholder or tribal area farmers.
00:36:46.099 --> 00:36:52.699
The advisory has to be totally related on ground
00:36:52.699 --> 00:36:59.079
-rooting or close to reality data. I'm happy
00:36:59.079 --> 00:37:02.260
to say that in Andhra Pradesh, these efforts
00:37:02.260 --> 00:37:06.280
have been initiated and they started giving good
00:37:06.280 --> 00:37:12.659
results yeah so and and i guess you're mapping
00:37:12.659 --> 00:37:15.139
these data gaps in the case of under pradesh
00:37:15.139 --> 00:37:17.860
and addressing them and making sure they're relevant
00:37:17.860 --> 00:37:19.639
which is really encouraging to see especially
00:37:19.639 --> 00:37:24.039
when you point out the issue of um you know farmers
00:37:24.039 --> 00:37:26.840
that don't hold title or women who are often
00:37:26.840 --> 00:37:30.300
out of the loop when it comes to formal extension
00:37:30.300 --> 00:37:32.840
services and so this really opens up a huge opportunity
00:37:33.639 --> 00:37:37.139
for inclusion if it's done properly so um that
00:37:37.139 --> 00:37:40.820
leads to my could i just add one little bit on
00:37:40.820 --> 00:37:44.139
this on the andhra pradesh sure yeah just very
00:37:44.139 --> 00:37:46.980
quickly i mean one thing that's very cool about
00:37:46.980 --> 00:37:50.079
the what the government of andhra pradesh is
00:37:50.079 --> 00:37:52.099
aiming to do and some of the work that we're
00:37:52.099 --> 00:37:55.780
doing it extends beyond you know agricultural
00:37:55.780 --> 00:37:58.679
data as we're talking about it and includes and
00:37:58.679 --> 00:38:01.920
a big focus of of the program uh we're involved
00:38:01.920 --> 00:38:04.219
in is looking at kind of the horticultural data
00:38:04.219 --> 00:38:07.800
the livestock data the the fisheries on aquaculture
00:38:07.800 --> 00:38:10.380
data as well and it the the vision is to bring
00:38:10.380 --> 00:38:13.519
it all into an integrated space where all of
00:38:13.519 --> 00:38:15.949
this data can kind of talk to to each other in
00:38:15.949 --> 00:38:18.829
addition to kind of market and, and kind of financial
00:38:18.829 --> 00:38:22.070
access information and data and climate and weather
00:38:22.070 --> 00:38:24.449
data. So you can get this really, you know, kind
00:38:24.449 --> 00:38:26.610
of job. Like you were talking about this, this
00:38:26.610 --> 00:38:29.909
more sophisticated strategic inferential advisory.
00:38:31.289 --> 00:38:33.510
And, and so that's kind of the direction it's
00:38:33.510 --> 00:38:35.389
heading. And right now there's a lot of data
00:38:35.389 --> 00:38:38.949
ecosystem gap mapping and, and whole filling
00:38:38.949 --> 00:38:42.050
and, and kind of integration that the team is
00:38:42.050 --> 00:38:45.690
doing. But I, I just, I think that's an important
00:38:45.690 --> 00:38:48.190
kind of dimension to highlight as well as the
00:38:48.190 --> 00:38:49.869
possibilities that are there. That's leading
00:38:49.869 --> 00:38:52.250
into our last question. Thanks, Mike, around
00:38:52.250 --> 00:38:54.110
our last two questions around looking forward.
00:38:54.269 --> 00:38:56.750
So what do you see in the future? I'm going to
00:38:56.750 --> 00:38:59.650
ask you to frame both questions and we'll do
00:38:59.650 --> 00:39:01.730
a roundtable on your responses. The first one
00:39:01.730 --> 00:39:04.989
is, if we get the data corpus right, it's federated,
00:39:05.130 --> 00:39:08.090
responsibly assembled, contextually relevant
00:39:08.090 --> 00:39:11.690
data set. What becomes possible for AI and agriculture
00:39:11.690 --> 00:39:15.590
that isn't possible today? and the second question
00:39:15.590 --> 00:39:20.090
is what worries you most if data remains fragmented
00:39:20.090 --> 00:39:24.070
so those two questions as we close off on this
00:39:24.070 --> 00:39:27.869
topic of data corpus so let's turn to jao first
00:39:27.869 --> 00:39:34.150
yeah sure so um it's not totally correct to say
00:39:34.150 --> 00:39:36.710
it's not possible today because something's already
00:39:36.710 --> 00:39:40.150
happening but it's hyper personalized assistant
00:39:41.610 --> 00:39:45.349
almost like persistently with you to make suggestions,
00:39:45.630 --> 00:39:48.369
to make recommendations just for you. I think
00:39:48.369 --> 00:39:52.559
that will be really, you know... transformative
00:39:52.559 --> 00:39:56.159
to some extent for smallholder farmers who understand
00:39:56.159 --> 00:39:58.800
what kind of resources you have, what kind of
00:39:58.800 --> 00:40:00.739
challenges you have, what's the best course of
00:40:00.739 --> 00:40:04.119
action, the minimal risk, et cetera. Right now,
00:40:04.179 --> 00:40:06.780
I think when you start chatting session with
00:40:06.780 --> 00:40:09.820
any chatbot, it kind of starts from scratch or
00:40:09.820 --> 00:40:13.440
have only very limited session information, only
00:40:13.440 --> 00:40:17.000
focus on what you provided for the chatbot. But
00:40:17.000 --> 00:40:21.329
if all this data... sources are connected as
00:40:21.329 --> 00:40:24.110
a more digital public infrastructure. It can
00:40:24.110 --> 00:40:26.989
pull many different types of data from different
00:40:26.989 --> 00:40:30.289
sources. So it can provide more customized and
00:40:30.289 --> 00:40:33.809
really personalized advice. Personalized advisor
00:40:33.809 --> 00:40:38.690
for farmers. Yeah, exactly. And I think it can
00:40:38.690 --> 00:40:41.809
eventually lead to something more physical. I
00:40:41.809 --> 00:40:44.429
think there are a lot of innovations happening
00:40:44.429 --> 00:40:49.429
in robotics. Again, But in a way, LLM, yeah,
00:40:49.550 --> 00:40:54.170
so we have been hearing LLM is going beyond language,
00:40:54.230 --> 00:40:56.329
already past the language barrier. It's just
00:40:56.329 --> 00:41:00.170
becoming multimodal, becoming operating system
00:41:00.170 --> 00:41:03.449
for robotics as well. So I think you can really
00:41:03.449 --> 00:41:08.590
pave ways for more physical and different types
00:41:08.590 --> 00:41:11.389
of AI transformation that we can lead to. Well,
00:41:11.389 --> 00:41:13.489
I'm already seeing, like, you know, drone integration
00:41:13.489 --> 00:41:17.130
of AI and analytics, for example, today. It's
00:41:17.130 --> 00:41:20.449
a hint of what's to come. Mike, over to you,
00:41:20.469 --> 00:41:24.489
your, your thoughts on those two questions. Yeah.
00:41:24.590 --> 00:41:27.369
I mean, I largely agree with Joe Wu as far as
00:41:27.369 --> 00:41:29.869
like, if we get it right, I think it presents
00:41:29.869 --> 00:41:31.909
this opportunity, you know, like at this point.
00:41:32.360 --> 00:41:36.079
I've integrated frontier model tools like Cloud
00:41:36.079 --> 00:41:39.179
or ChatGPT into my daily work stream because
00:41:39.179 --> 00:41:42.360
it's a super useful personalized advisor on the
00:41:42.360 --> 00:41:46.599
types of things that I use it for. It has this
00:41:46.599 --> 00:41:50.840
global library of knowledge that it can infer
00:41:50.840 --> 00:41:53.360
from and then that gets complemented by the personalized
00:41:53.360 --> 00:41:56.739
exchanges that I have with it. If you get that
00:41:56.739 --> 00:42:00.039
at the farm level for these And it's something
00:42:00.039 --> 00:42:01.880
we haven't explicitly stated, I don't think,
00:42:01.880 --> 00:42:04.579
in this conversation, but is generally well known.
00:42:04.679 --> 00:42:09.239
There's a huge extension gap for the smallholder
00:42:09.239 --> 00:42:12.639
and tenant farmers. You're talking 1 to 6 ,000,
00:42:12.699 --> 00:42:17.480
1 to 10 ,000 in some places. one extension officer
00:42:17.480 --> 00:42:20.719
for 10 ,000 small -scale producers or tenant
00:42:20.719 --> 00:42:24.119
farmers. And so a lot of these farmers don't
00:42:24.119 --> 00:42:27.019
necessarily have the advisory access, generally
00:42:27.019 --> 00:42:28.940
speaking. And then if suddenly they can have
00:42:28.940 --> 00:42:31.179
a personalized assistant that can tell them not
00:42:31.179 --> 00:42:33.900
just, like you said earlier, David, how to plant
00:42:33.900 --> 00:42:37.460
rice more effectively, but actually why they
00:42:37.460 --> 00:42:40.239
should be focusing more on their livestock than...
00:42:40.679 --> 00:42:43.119
you know, farm crops this season because of market
00:42:43.119 --> 00:42:45.880
conditions, anticipated weather, and who knows,
00:42:45.960 --> 00:42:48.559
geopolitical realities and global, you know,
00:42:48.579 --> 00:42:51.119
whatever factors become integrated into this,
00:42:51.179 --> 00:42:54.059
that becomes a super powerful and potentially
00:42:54.059 --> 00:42:56.820
very meaningful livelihood resource. And another
00:42:56.820 --> 00:42:59.840
thing we haven't quite said in this is, right,
00:42:59.940 --> 00:43:02.440
agriculture operates in seasons, right? We're
00:43:02.440 --> 00:43:05.599
not talking like a week or a month. We're talking
00:43:05.599 --> 00:43:08.590
about... six months, three months, a year of
00:43:08.590 --> 00:43:11.710
someone living off in subsistence or near subsistence
00:43:11.710 --> 00:43:16.489
conditions being derailed, that's a huge vulnerability
00:43:16.489 --> 00:43:20.889
and kind of moral hazard risk. So there's real
00:43:20.889 --> 00:43:23.489
powerful potential and excitement there. On the
00:43:23.489 --> 00:43:28.750
worry side, building these systems is very hard,
00:43:28.869 --> 00:43:31.610
especially in the context we're talking about
00:43:31.610 --> 00:43:35.880
with the level of... integration, that kind of,
00:43:35.880 --> 00:43:39.280
you know, multi -state, multinational, you know,
00:43:39.280 --> 00:43:42.840
global levels that gets sustained, especially
00:43:42.840 --> 00:43:46.880
under kind of very dynamic geopolitical times
00:43:46.880 --> 00:43:50.199
and circumstances. So, you know, I worry about
00:43:50.199 --> 00:43:56.179
if the data remains fragmented and biased, then
00:43:56.179 --> 00:43:59.829
we start putting out digital tools. which agriculture
00:43:59.829 --> 00:44:02.469
has done many times in the past. And farmers
00:44:02.469 --> 00:44:04.510
are like, this doesn't actually help me. And
00:44:04.510 --> 00:44:06.670
then it just becomes a lot of investment that
00:44:06.670 --> 00:44:08.769
doesn't lead to... And you betray the trust of
00:44:08.769 --> 00:44:10.590
the farmer. And you betray the trust of the farmers
00:44:10.590 --> 00:44:13.170
yet again with your newest, latest, and greatest
00:44:13.170 --> 00:44:15.710
digital attempt, attempted digital intervention.
00:44:15.989 --> 00:44:19.650
So, yeah. Good points. And Sami, final word,
00:44:19.829 --> 00:44:23.289
boots on the ground perspective. Love to hear
00:44:23.289 --> 00:44:26.360
your perspective on these two questions. i totally
00:44:26.360 --> 00:44:29.900
agree with jahu and mike uh if we get the data
00:44:29.900 --> 00:44:34.059
corpus right we can give truly personalized advisories
00:44:34.059 --> 00:44:37.840
not district level not village level recommendations
00:44:37.840 --> 00:44:40.980
but plot level advice that accounts for your
00:44:40.980 --> 00:44:44.440
soil type your variety your irrigation access
00:44:44.440 --> 00:44:47.400
your historical pest patterns and your market
00:44:47.400 --> 00:44:51.159
access point i want to give one example that
00:44:51.159 --> 00:44:56.969
we did recently in andhra pradesh we are trying
00:44:56.969 --> 00:45:02.750
to get plot level granule granular data in in
00:45:02.750 --> 00:45:06.130
one of the regions which is a banana rich region
00:45:06.130 --> 00:45:12.309
most of the farmers are have banana plantations
00:45:12.309 --> 00:45:17.329
we try to collect the data on which farmers are
00:45:17.329 --> 00:45:20.170
using fruit care activities like bunch covering
00:45:20.170 --> 00:45:23.510
or injection giving injections for to get it
00:45:24.110 --> 00:45:27.789
uh fruit quality to our surprise we came to know
00:45:27.789 --> 00:45:31.570
that out of 600 farmers there are only 12 farmers
00:45:31.570 --> 00:45:34.829
who are following these measures rest of the
00:45:34.829 --> 00:45:37.989
farmers don't even know are they're not aware
00:45:37.989 --> 00:45:42.730
of or they don't know how to do it now the if
00:45:42.730 --> 00:45:46.289
if we get this data that out of 600 12 farmers
00:45:46.289 --> 00:45:49.769
are only following the fruit care activity that
00:45:49.769 --> 00:45:55.070
granular data can truly target those rest of
00:45:55.070 --> 00:45:57.170
the farmers who are not following fruit care
00:45:57.170 --> 00:46:00.429
activities. And then the guidance can go to them
00:46:00.429 --> 00:46:03.889
how to improve the fruit quality and get better
00:46:03.889 --> 00:46:09.849
price in the market. So the real data corpus,
00:46:09.889 --> 00:46:15.030
like real ground level data corpus can give such
00:46:15.030 --> 00:46:18.239
advisory. And second point on tenant farmers
00:46:18.239 --> 00:46:20.539
and women farmers or tribal farmers who have
00:46:20.539 --> 00:46:25.340
been excluded from land records can be visible,
00:46:25.400 --> 00:46:29.659
may get visible through alternative data like
00:46:29.659 --> 00:46:32.659
input purchase records or market transaction
00:46:32.659 --> 00:46:36.579
history. So we can bring them into system by
00:46:36.579 --> 00:46:40.940
covering these data sources. Third point I want
00:46:40.940 --> 00:46:43.900
to... uh talk from the government perspective
00:46:43.900 --> 00:46:47.500
governments can have real -time granular data
00:46:47.500 --> 00:46:51.199
on what farmers are actually doing not what they
00:46:51.199 --> 00:46:53.840
are supposed to be doing policy can be adjusted
00:46:53.840 --> 00:46:56.639
mid -season rather than waiting for annual assessments
00:46:56.639 --> 00:47:00.380
estimating losses and gains and then taking a
00:47:00.380 --> 00:47:09.730
decision what worries is If the data models or
00:47:09.730 --> 00:47:14.869
AI systems are not built on representative data,
00:47:15.230 --> 00:47:19.230
AI will deliver high -quality personalized advice
00:47:19.230 --> 00:47:24.070
to large connected progressive farmers only and
00:47:24.070 --> 00:47:26.590
generate potentially harmful advice to smart
00:47:26.590 --> 00:47:32.570
holders. That widens productivity gap. That widens
00:47:32.570 --> 00:47:36.210
wealth gap. it will increase the inequalities
00:47:36.210 --> 00:47:39.690
which are already existing in the agriculture
00:47:39.690 --> 00:47:44.889
field. AI becomes another mechanism for agricultural
00:47:44.889 --> 00:47:49.769
polarization. Yeah, we don't want to create a
00:47:49.769 --> 00:47:52.630
digital divide 2 .0. So, you know, responsible
00:47:52.630 --> 00:47:55.210
collection, use and delivery of data insights
00:47:55.210 --> 00:47:58.969
to farmers, policymakers and value chain actors
00:47:58.969 --> 00:48:01.329
is critically important to unlock the value of
00:48:01.329 --> 00:48:04.010
this. Well, thank you to all three of you for
00:48:04.010 --> 00:48:06.429
a very insightful conversation. Sonamia, thank
00:48:06.429 --> 00:48:08.110
you for your boots on the ground perspective
00:48:08.110 --> 00:48:10.849
from Andhra Pradesh. Mike, from your broader
00:48:10.849 --> 00:48:13.829
perspective of translating this into projects,
00:48:13.889 --> 00:48:19.289
policies, and impact. And Jau for wrangling data
00:48:19.289 --> 00:48:22.269
across a wide range of international organizations
00:48:22.269 --> 00:48:24.429
and delivering value to national stakeholders
00:48:24.429 --> 00:48:28.190
through data models and a wide range of tools.
00:48:28.719 --> 00:48:30.559
Thanks to all of you. You've raised some actually
00:48:30.559 --> 00:48:32.780
interesting questions for a follow -up conversation,
00:48:33.059 --> 00:48:35.679
especially around personal identification information
00:48:35.679 --> 00:48:39.219
and other topics that we'll follow up in Grounded
00:48:39.219 --> 00:48:44.219
Intelligence. So thank you all and a very exciting
00:48:44.219 --> 00:48:46.340
journey that we're on. And thank you for your
00:48:46.340 --> 00:48:49.539
contributions towards making it impactful and
00:48:49.539 --> 00:48:51.179
real for smallholder farmers around the world.
00:48:56.039 --> 00:48:58.469
Today, we're going to be talking about apples
00:48:58.469 --> 00:49:01.130
and not what you think. We're going to be talking
00:49:01.130 --> 00:49:04.510
about the diversity that exists in an orchard
00:49:04.510 --> 00:49:08.170
of apples and that no two apples in that orchard
00:49:08.170 --> 00:49:11.369
are exactly the same. That variation depends
00:49:11.369 --> 00:49:15.670
on the variety, depends on the micro environment
00:49:15.670 --> 00:49:18.590
in which that apple was produced, depends on
00:49:18.590 --> 00:49:22.610
the soil, the level of water, how it was cultivated
00:49:22.610 --> 00:49:26.250
or pruned by the farmer. So when we talk about
00:49:26.639 --> 00:49:29.860
variation not just in apples but variation in
00:49:29.860 --> 00:49:32.480
our data systems to support modern agriculture
00:49:32.480 --> 00:49:35.559
we're talking about data that contextually is
00:49:35.559 --> 00:49:37.599
very different and we need to make sure that
00:49:37.599 --> 00:49:40.900
that variation is captured in the data that's
00:49:40.900 --> 00:49:43.000
going into training models especially models
00:49:43.000 --> 00:49:46.639
to support advisory services for farmers so that
00:49:46.639 --> 00:49:49.059
means making sure that we capture the diversity
00:49:49.059 --> 00:49:53.429
of production systems Farmer profiles, especially
00:49:53.429 --> 00:49:55.750
women that are often underrepresented in these
00:49:55.750 --> 00:49:58.690
data assets for a variety of reasons. That we
00:49:58.690 --> 00:50:03.010
capture variation in the soil type in which these
00:50:03.010 --> 00:50:06.170
crops are grown. And also making sure that that
00:50:06.170 --> 00:50:09.250
data that is being captured has associated with
00:50:09.250 --> 00:50:11.809
it what we call metadata, which is sort of smaller
00:50:11.809 --> 00:50:14.369
bits of data that actually describe the context
00:50:14.369 --> 00:50:17.659
of how that data was collected. the parameters
00:50:17.659 --> 00:50:20.139
that would help you understand how to best use
00:50:20.139 --> 00:50:23.079
that data in order to deliver targeted services
00:50:23.079 --> 00:50:26.199
to farmers. So when we talk about data systems,
00:50:26.320 --> 00:50:28.719
let's talk about the diversity they represent,
00:50:28.980 --> 00:50:31.860
making sure that diversity captures the diversity
00:50:31.860 --> 00:50:34.199
of farmers and production systems that we want
00:50:34.199 --> 00:50:37.519
to deliver advisory services to, and to make
00:50:37.519 --> 00:50:39.519
sure that we're getting the feedback so that
00:50:39.519 --> 00:50:43.739
outputs from those advisory services are constantly
00:50:43.739 --> 00:50:46.940
improved over time. And that in itself is a different
00:50:46.940 --> 00:50:50.380
and diverse data set. So that's it for today.
00:50:50.659 --> 00:50:52.440
We're going to dive into this in more detail.
00:50:53.260 --> 00:50:54.760
See you on the next episode.