Cleaning And Curating Open Data For Archaeology

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 1:11:29 PM
Duration: 3655.612
Channels: 1
1
00:00:12.775 --> 00:00:26.150
Hello. Welcome to the Data Engineering podcast, the show about modern data management. When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy them. So check out Linode. With 200 gigabit private networking,
2
00:00:26.505 --> 00:00:33.405
scalable shared block storage, and a 40 gigabit public network, you've got everything you need to run a fast, reliable, and bulletproof data platform.
3
00:00:34.000 --> 00:00:39.940
If you need global distribution, they've got that covered too with worldwide data centers, including new ones in Toronto and Mumbai.
4
00:00:40.320 --> 00:00:42.375
Go to data engineering podcast dotcom/linode
5
00:00:43.155 --> 00:00:48.135
today to get a $20 credit and launch a new server in under a minute. And go to data engineering podcast.com
6
00:00:48.594 --> 00:00:54.860
to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. And don't forget to go to data engineering podcast.com/chat
7
00:00:55.960 --> 00:01:14.719
to join the community and keep the conversation going. Your host is Tobias Macy. And today I'm interviewing Eric Kansa about Open Context, a platform for publishing, managing, and sharing research data. So, Eric, could you start by introducing yourself? Hi. Yeah. My name's Eric. I, direct the Open Context Project and a nonprofit
8
00:01:15.100 --> 00:01:16.035
publishing service
9
00:01:16.435 --> 00:01:17.095
for archaeology,
10
00:01:17.875 --> 00:01:20.295
archaeological research data. And it
11
00:01:20.755 --> 00:01:23.735
is now let's see. Started out in 2006.
12
00:01:24.590 --> 00:01:31.410
It's gone through several different iterations over over the past several years, and we currently have about 1, 500, 000
13
00:01:32.430 --> 00:01:36.085
entities that we've published. To talk about more what that means later,
14
00:01:36.945 --> 00:01:37.445
from
15
00:01:37.825 --> 00:01:40.545
roughly about a 1000 different researchers and,
16
00:01:41.025 --> 00:01:41.525
institutions
17
00:01:41.905 --> 00:01:44.340
around the world. And it is
18
00:01:44.800 --> 00:02:07.640
really intended to be the, a way of sharing a lot of the structured data that comes out of the world of archaeological excavations and archaeological surveys. And do you remember how you first got involved in the area of data management? Yeah. Mostly, it was born out of frustration. I was doing my graduate work, my dissertation work in the late 19 nineties in, archaeology.
19
00:02:08.180 --> 00:02:12.355
My background was focused on studying the early bronze age and,
20
00:02:12.995 --> 00:02:16.055
the impact of the create the formation of
21
00:02:16.355 --> 00:02:21.950
the Egyptian state and civilization on its neighboring regions. And that work involved
22
00:02:22.329 --> 00:02:23.310
building some databases
23
00:02:23.930 --> 00:02:24.909
and some GIS.
24
00:02:25.450 --> 00:02:26.510
And I was
25
00:02:26.935 --> 00:02:27.595
really interested
26
00:02:28.215 --> 00:02:28.795
in seeing,
27
00:02:29.095 --> 00:02:43.120
you know, how the data that I was creating and look and exploring and how it would relate to the data that other researchers had also developed. And it's super frustrating because there was really no access to their data. So we would get these very,
28
00:02:44.055 --> 00:02:45.595
in some ways frustratingly,
29
00:02:46.935 --> 00:02:50.955
superficial publications where they would summarize a few results from excavations
30
00:02:51.255 --> 00:02:56.640
that were related to what I was looking at, but I was never able to see, like, the structured databases.
31
00:02:56.940 --> 00:03:18.275
What was in the what are the actual counts of the different things that they were finding and how did that relate to what I had? So that was 1 of the real reasons why I wanted to get into this area is because it was a there was a real niche there that there's really a strong need to be able to compare these different datasets and try to see the bigger picture. And decades later, we're still working at it,
32
00:03:18.834 --> 00:03:26.775
and it's a and it turned out to be a much harder problem than I initially naively thought it would be. And so along that journey, you ended up
33
00:03:27.150 --> 00:03:29.490
working on building the Open Context
34
00:03:29.870 --> 00:03:31.170
platform and organization.
35
00:03:31.550 --> 00:03:35.970
So can you describe a bit about what it is that you're doing at Open Context
36
00:03:36.335 --> 00:03:41.475
and the mission that you're trying to drive towards and how the overall project got started?
37
00:03:41.855 --> 00:03:46.310
Yeah. So we recognized early on that there are conventional
38
00:03:46.849 --> 00:03:47.909
modes of publishing
39
00:03:48.209 --> 00:03:55.510
in in the research world, in academia, were just not sufficient for managing structured data. So conventional articles, books and reports,
40
00:03:55.815 --> 00:03:57.995
those are the sort of bread and butter of
41
00:03:58.295 --> 00:04:00.315
scholarly communication between researchers
42
00:04:00.935 --> 00:04:05.630
and that's that world of publish and perish. Right? So people are publishing
43
00:04:05.930 --> 00:04:14.995
at a feverish pace, but those, publications are very difficult to use in any sort of for for any sort of quantified analysis or difficult to aggregate
44
00:04:15.295 --> 00:04:17.935
all sorts of different difficult issues with them,
45
00:04:18.815 --> 00:04:22.220
in actually reusing the information that they're presenting
46
00:04:22.520 --> 00:04:27.020
in these conventional publications. And the other issue is that there's a big issue
47
00:04:27.795 --> 00:04:35.735
with archaeology in that it is often relying on destructive methodologies. So when you dig a site, you're actually destroying that site.
48
00:04:36.189 --> 00:04:38.129
And in the process of excavation,
49
00:04:38.990 --> 00:04:48.815
you have a really strong professional and ethical imperative to do very detailed recording of what it is that you're encountering when you do that excavation. You have to
50
00:04:49.115 --> 00:04:52.080
know where you're finding everything. You have to know stratigraphic
51
00:04:52.460 --> 00:05:04.875
relationships between different deposits. You have to know where different architectural features are. There's a huge amount of recording that has to take place, and that recording is actually quite complicated. And it's typically
52
00:05:05.335 --> 00:05:08.475
done using databases of 1 form or another
53
00:05:08.950 --> 00:05:12.090
to, actually record this excavation process.
54
00:05:12.470 --> 00:05:17.450
And unless we come up with ways of keeping that information, keeping those records
55
00:05:17.925 --> 00:05:19.145
in in those databases,
56
00:05:19.845 --> 00:05:23.065
then that process of excavation destroying sites,
57
00:05:23.605 --> 00:05:24.585
all the information
58
00:05:25.045 --> 00:05:50.759
that comes out of that will be lost unless we do something to preserve an archive and share these digital data. So in addition to, you know, opening up new research opportunities, which is 1 of the things that got me excited about this issue about, you know, managing data, we have this really important ethical imperative that actually, you know, this is the way that, we're going to pass an archaeological record down to future generations
59
00:05:51.539 --> 00:06:07.360
if we are successful in the management of this research data. So those are the, sort of driving forces behind this, and then open context is our attempt to try to provide, you know, practical real world services in order to try to meet those sort of larger needs
60
00:06:07.979 --> 00:06:08.560
of dissemination,
61
00:06:09.020 --> 00:06:10.560
opening new research opportunities,
62
00:06:11.195 --> 00:06:11.695
and
63
00:06:12.235 --> 00:06:14.815
also putting these data into formats,
64
00:06:15.595 --> 00:06:22.550
into a larger public context and into digital repositories where they could be preserved. And so the datasets that you're
65
00:06:22.930 --> 00:06:23.430
managing,
66
00:06:23.810 --> 00:06:26.550
are they solely dedicated to archaeological
67
00:06:26.850 --> 00:06:32.295
research, or do you have other scientific domains represented as well? The vast majority are archaeological.
68
00:06:32.835 --> 00:06:40.449
We have 1 test dataset in public health, but for the most part, we're busy enough to with the archaeologists
69
00:06:40.750 --> 00:06:44.930
that we're really focused on that need. And the other issue is that our
70
00:06:45.425 --> 00:06:47.365
publishing model, the way that we
71
00:06:47.665 --> 00:06:57.450
curate the data, really requires some domain expertise. So just because of who we are in terms of our own background and staffing in LVAD, we do focus on archaeology,
72
00:06:58.390 --> 00:07:00.490
not other other outside domains
73
00:07:00.974 --> 00:07:04.835
where there could be also very different kinds of data modeling and metadata
74
00:07:05.854 --> 00:07:14.100
and ontology kinds of concerns that beyond our expertise and comfort level. And in terms of determining which datasets
75
00:07:14.400 --> 00:07:25.355
you are willing and able to work with, do you have a particular set of guidelines or protocols for when you're first interacting with somebody who comes to you with a particular,
76
00:07:26.350 --> 00:07:29.330
set of records that they want to publish on your platform?
77
00:07:29.710 --> 00:07:35.330
Yeah. Absolutely. It's important for us that the research that we publish is coming from projects that
78
00:07:35.784 --> 00:07:36.845
are meeting the,
79
00:07:37.384 --> 00:07:41.164
normal standards of professional conduct that are in the discipline of archaeology.
80
00:07:41.544 --> 00:07:47.920
And these are professional societies in the field have, you know, ethic codes, professional norms,
81
00:07:48.380 --> 00:07:56.694
and there's also laws that govern how archaeology is conducted in different jurisdictions around the world. And so all of those different kinds of standards
82
00:07:57.155 --> 00:07:59.235
and norms have to be met for us to,
83
00:07:59.875 --> 00:08:02.055
engage with the researcher to publish a dataset.
84
00:08:02.400 --> 00:08:04.900
So this is an important thing because
85
00:08:05.520 --> 00:08:06.500
with those sorts
86
00:08:07.280 --> 00:08:09.620
of conduct professional conduct frameworks
87
00:08:10.005 --> 00:08:10.745
help establish,
88
00:08:11.125 --> 00:08:11.685
you know, kind
89
00:08:12.245 --> 00:08:23.009
that we are working for the archaeological research community, and we don't wanna provide a platform for basically for for for people who are doing treasure hunting, you know. So there there's a
90
00:08:23.310 --> 00:08:29.615
a world out there of people who are doing things like, illicit metal detecting and whatnot and treasure hunting
91
00:08:30.155 --> 00:08:40.920
that, would actually if if we were to publish that kind of information, may endanger sites. They might get vandalized, they might get looted, and we wanna make sure that we're not working with not facilitating
92
00:08:41.380 --> 00:08:43.639
the destruction of cultural heritage.
93
00:08:44.065 --> 00:08:49.205
So this is, so working with a professional community of people who agree to a common set of ethics,
94
00:08:49.665 --> 00:08:50.965
that's something that is,
95
00:08:51.570 --> 00:08:59.990
an important aspect of, who we work with. And in terms of the actual data that you receive from these different research projects
96
00:09:00.375 --> 00:09:01.115
and archaeological
97
00:09:01.575 --> 00:09:02.075
excavations,
98
00:09:02.855 --> 00:09:08.155
what sorts of data formats are you dealing with and some of the unique challenges
99
00:09:08.535 --> 00:09:09.870
that come along with
100
00:09:10.650 --> 00:09:16.270
the nature of the data that you're dealing with in terms of how it's obtained, how it's recorded, and how it's structured?
101
00:09:16.685 --> 00:09:19.265
So most archaeologists are not necessarily
102
00:09:19.725 --> 00:09:22.945
experts in databases, and, and the
103
00:09:23.485 --> 00:09:24.465
domain typically
104
00:09:25.005 --> 00:09:25.505
involves
105
00:09:26.180 --> 00:09:27.800
a lot of use of
106
00:09:28.420 --> 00:09:32.120
pretty normal sort of office suite kind of products. So,
107
00:09:32.579 --> 00:09:33.480
the more sophisticated
108
00:09:33.860 --> 00:09:36.215
archaeologists are using relational databases,
109
00:09:36.755 --> 00:09:38.055
databases like FileMaker
110
00:09:38.435 --> 00:09:39.175
or Access,
111
00:09:39.555 --> 00:09:42.375
especially. A lot of people use Excel for
112
00:09:42.889 --> 00:09:51.870
recording structured data and there's a lot of, variability in the kinds of consistency that of the data that people record. So some people do
113
00:09:52.295 --> 00:10:04.040
have different kinds of protocols in place for data validation so that the datasets are more consistent and a lot of people don't. And so there's this kind of an issue that some data need a lot of work and,
114
00:10:04.840 --> 00:10:08.220
after the fact cleanup and that could be pretty labor intensive.
115
00:10:08.555 --> 00:10:10.735
1 of the other issues is that archaeologists
116
00:10:11.435 --> 00:10:23.395
collaborate when they build their datasets typically. So an a single excavation may have different individuals who are documenting, describing archaeological context, which are different kinds of deposits of dirt that they
117
00:10:23.955 --> 00:10:27.495
dig, and they would have different specialists who would be studying different
118
00:10:28.195 --> 00:10:29.735
the different classes of materials.
119
00:10:30.115 --> 00:10:31.655
So typically, there'd be,
120
00:10:32.080 --> 00:10:36.980
say, animal bones that are recovered from an archaeological site and those are described by a zooarchaeologist,
121
00:10:37.440 --> 00:10:39.795
somebody who has training in zoology
122
00:10:40.095 --> 00:10:40.915
and anatomy.
123
00:10:41.295 --> 00:10:45.795
There could be seeds, charred seed remains that are being described by a botanist.
124
00:10:46.310 --> 00:10:49.210
There and there could be different experts who will be studying,
125
00:10:49.670 --> 00:10:53.450
different parts aspects of material culture. So pottery, coins,
126
00:10:54.325 --> 00:10:55.145
stone tools,
127
00:10:55.525 --> 00:10:56.585
metal implements.
128
00:10:57.205 --> 00:10:58.745
Some other people might be studying
129
00:10:59.285 --> 00:10:59.785
different
130
00:11:00.165 --> 00:11:02.345
kinds of, artwork or sculptures
131
00:11:02.880 --> 00:11:12.260
or all sorts of different kinds of materials, sometimes even textiles. So there's a huge number of people with very different kinds of expertise and they typically create their own datasets.
132
00:11:12.945 --> 00:11:23.430
And the main way that these different datasets can be related to 1 another is through a shared context of where where is it that an object or a bone or a seed was
133
00:11:23.810 --> 00:11:25.990
found, which archaeological context,
134
00:11:26.610 --> 00:11:28.070
was the source of that
135
00:11:28.529 --> 00:11:30.630
material. And 1 of the
136
00:11:31.375 --> 00:11:37.795
interesting and challenging bits is that because different researchers are basically managing their own datasets
137
00:11:38.630 --> 00:11:39.130
individually,
138
00:11:39.829 --> 00:11:45.130
bringing them all together can actually be a lot harder than you think it should be because, you know, if somebody
139
00:11:45.589 --> 00:11:53.945
writes, in their Excel spreadsheet that a certain bone comes from locus, which is a common term for an archeological deposit,
140
00:11:54.645 --> 00:11:55.785
locus 10,
141
00:11:56.490 --> 00:12:04.110
And, they write l dot 10, and then somebody else in a different database is looking at stone tools from that same locust.
142
00:12:04.410 --> 00:12:07.735
They might write locust 10 or just 10.
143
00:12:08.275 --> 00:12:11.335
So there's this issue of inconsistent identifiers
144
00:12:11.795 --> 00:12:13.815
in order that are used to reference
145
00:12:14.270 --> 00:12:20.210
archaeological context. And that could actually be a very big headache in order to relate these different kinds of materials altogether
146
00:12:20.590 --> 00:12:23.915
for just 1 archaeological site. And so this is why,
147
00:12:24.855 --> 00:12:28.075
there needs to be a lot of investment in
148
00:12:28.695 --> 00:12:32.795
trying to go through and understand the different ways that some of these
149
00:12:33.240 --> 00:12:35.500
identifiers are expressed and just reconcile
150
00:12:35.880 --> 00:12:37.579
them so that you can actually
151
00:12:37.959 --> 00:12:40.940
bring the materials together in the way that they should be.
152
00:12:41.305 --> 00:12:41.805
And
153
00:12:42.185 --> 00:12:49.485
in the process of onboarding these different datasets from various research activities or dig sites,
154
00:12:49.940 --> 00:13:02.365
As you were saying, there's a lot of domain expertise that's required to be able to make an effective use of that source data and convert it into the schema that you've established for OpenContacts.
155
00:13:02.905 --> 00:13:06.845
So can you talk through some of your strategies and tactics
156
00:13:07.160 --> 00:13:09.500
for ensuring that you are able to
157
00:13:09.959 --> 00:13:11.899
make an appropriate set of transformations
158
00:13:12.200 --> 00:13:13.420
for these different datasets
159
00:13:13.800 --> 00:13:14.940
and try to
160
00:13:15.485 --> 00:13:16.545
extract patterns
161
00:13:16.925 --> 00:13:22.225
to make the process more repeatable and also any issues that you encounter
162
00:13:22.879 --> 00:13:25.060
during that activity as far
163
00:13:25.519 --> 00:13:26.180
as mitigating
164
00:13:26.560 --> 00:13:29.060
data loss because of those conversion efforts?
165
00:13:29.439 --> 00:13:32.259
Yeah. I mean, on the aspect of data loss, whatever
166
00:13:32.625 --> 00:13:35.685
the open context is a very, very abstract to generalized
167
00:13:36.305 --> 00:13:37.205
global schema.
168
00:13:37.665 --> 00:13:54.945
There's not much more to it than sort of almost like a triple store or a key value pair type of thing describing every entity. And we have some common metadata requirements and some common rules of inference around all that that help make things a bit more discoverable. But, really, a researcher
169
00:13:56.040 --> 00:14:00.220
who submits their data dataset to us can describe that those data with
170
00:14:00.680 --> 00:14:09.035
any sorts of attributes that they want. They often have their own control vocabulary. Sometimes they're gonna be referencing a shared control vocabulary
171
00:14:09.495 --> 00:14:14.150
that might be professionally curated. So, know, especially the people work with animal bones,
172
00:14:14.690 --> 00:14:15.750
pretty much they're,
173
00:14:16.370 --> 00:14:16.870
classifying
174
00:14:17.170 --> 00:14:24.205
the species, the biological of the bone in very similar kinds of ways, and so that becomes an easier issue in terms of sharing
175
00:14:24.905 --> 00:14:26.205
linking across different
176
00:14:27.010 --> 00:14:28.230
classification systems.
177
00:14:28.930 --> 00:14:39.735
But for the most part, people come up with their own idiosyncratic ways of describing stuff. And Open Context's main assumption is that stuff is related via contextual relationships,
178
00:14:40.514 --> 00:14:46.730
that a certain record will have relationships to other records that we can describe. So a lot of the
179
00:14:47.270 --> 00:14:53.130
you know, we're very sort of flexible, I guess, in terms of the different kinds of schemas that we accept
180
00:14:55.225 --> 00:14:57.565
in terms of especially with descriptive attributes.
181
00:14:57.945 --> 00:15:01.725
The the hard thing really centers in making sure that we're understanding
182
00:15:02.025 --> 00:15:04.525
the identifiers correctly in the source dataset.
183
00:15:04.930 --> 00:15:07.650
So context identifiers are the thing that,
184
00:15:08.370 --> 00:15:21.305
we focus a lot of attention on. And the other thing is just, you know, what is it that is being described is 1 of the things that we would care a lot about. So sometimes, depending on the way that somebody structures the dataset,
185
00:15:21.910 --> 00:15:23.690
there could be multiple records,
186
00:15:24.070 --> 00:15:27.850
say, they're just using a flat table like an Excel spreadsheet. 1,
187
00:15:28.390 --> 00:15:28.890
multiple,
188
00:15:29.510 --> 00:15:30.010
records
189
00:15:30.865 --> 00:15:37.365
are describing the same thing. It's just that they wanna provide add multiple attributes to that 1 thing. So
190
00:15:37.800 --> 00:15:38.840
like a a,
191
00:15:39.400 --> 00:15:41.660
a coin might have multiple,
192
00:15:42.280 --> 00:15:46.700
motifs that are on it, and they would have multiple rows in the spreadsheet to to add
193
00:15:47.075 --> 00:15:50.935
that there's multiple motifs on a certain coin or a certain pot shirt or whatever.
194
00:15:51.395 --> 00:15:51.795
And,
195
00:15:52.675 --> 00:16:01.450
we need to understand that, okay, this is the same coin, it just has multiple attributes. It's not the a mistake that this thing is repeated over and over again.
196
00:16:01.830 --> 00:16:14.330
So that's so there are these those are the sorts of, questions that we have to look at when we look at, mapping the, schemas from a source dataset and going through our extract transform load process
197
00:16:14.710 --> 00:16:20.410
and moving things into open context. Yeah. Essentially, it's just that, you know, we have to understand what is being described
198
00:16:20.790 --> 00:16:29.695
and, you know, are they being described by attributes that could take on multiple values or not, and then what are the relationships between the things that are being described.
199
00:16:30.040 --> 00:16:34.459
And for the data that ends up in the open context system,
200
00:16:34.839 --> 00:16:44.345
you encourage the use of Creative Commons licensing. So I'm wondering if you've had any issues in terms of needing to educate researchers
201
00:16:44.805 --> 00:16:56.210
as far as the implications of that licensing or if you've had any pushback from people who initially approach you once you mention those licensing requirements and just your overall
202
00:16:56.955 --> 00:16:57.455
considerations
203
00:16:58.315 --> 00:17:04.895
of using Creative Commons as the license category for the data that you are hosting on your platform.
204
00:17:05.299 --> 00:17:13.559
Yeah. That's, I mean, the whole issue of intellectual property in this space is a huge area of research, and it touches on a bunch of different
205
00:17:14.325 --> 00:17:15.445
considerations. So there's,
206
00:17:16.165 --> 00:17:17.465
there's a sort of practical
207
00:17:18.165 --> 00:17:18.665
interoperability
208
00:17:19.205 --> 00:17:27.059
consideration that, you know, if we all use common standard licenses, then the content that we have, would be legally interoperable
209
00:17:27.440 --> 00:17:35.495
and, which is nice and that's 1 of the sort of wonderful things about Creative Commons, you know, it's a standard licenses. You can have interoperability,
210
00:17:36.034 --> 00:17:49.010
and it's all expressed with standard metadata too so that, you know, in a machine readable way, you can know what the copyright status is of some content. So that's great. But 1 of the big complications is that, you know, it's not just interoperability
211
00:17:49.710 --> 00:17:51.005
that we need to optimize,
212
00:17:52.184 --> 00:17:56.285
especially in the field like archaeology, but in archaeology, we're dealing with
213
00:17:56.665 --> 00:17:59.965
lots of different communities around the world with different sets of values
214
00:18:00.409 --> 00:18:06.350
and different assumptions about what the archaeological past means to them and who owns that archaeological
215
00:18:06.650 --> 00:18:07.150
past,
216
00:18:07.529 --> 00:18:12.265
you know, who who can speak for it. And those kinds of concerns
217
00:18:12.965 --> 00:18:27.240
mean that the ethical landscape around using creative commons licenses is a lot more complicated. And so we actually say in open context, you know, open access, open licensing, these are wonderful and powerful tools, but they're not universally
218
00:18:27.605 --> 00:18:31.145
appropriate. And so there are definitely going to be circumstances
219
00:18:31.525 --> 00:18:47.315
where open context is not a good platform for sharing data. And so if the data have specific kinds of sensitivities, especially, say, if there's indigenous people who regard this information as, important for their own heritage, you know, especially in situations where there's maybe
220
00:18:47.775 --> 00:18:48.915
a history of colonialism,
221
00:18:49.375 --> 00:19:00.220
then you have to be very careful about, license choices and whether or not a platform like Open Context is an appropriate way of disseminating the data. You might have to find some more restrictive mechanisms
222
00:19:00.674 --> 00:19:03.735
in order to take a sort of more judicious
223
00:19:04.115 --> 00:19:08.809
and sort of situationally aware approach about all that. So it's,
224
00:19:09.370 --> 00:19:13.950
it's an interesting kind of an issue. 1 of the other issues about copyright licenses,
225
00:19:14.410 --> 00:19:19.205
like with Creative Commons is that they're copyright licenses. And in the United States,
226
00:19:19.665 --> 00:19:23.365
there's this distinction between facts and expressions. So factual
227
00:19:23.950 --> 00:19:39.225
data, kinds of things like measurements and, you know, the height of Mount Everest or something like that. Factual data is typically not seen as something that copyright actually covers. So there's a sort of an ambiguity also about what aspects of a dataset are
228
00:19:39.605 --> 00:19:41.625
expressive and that are sort of
229
00:19:42.030 --> 00:19:45.250
the domain of copyright and what aspects of an archaeological dataset
230
00:19:45.790 --> 00:19:46.690
are more factual,
231
00:19:47.310 --> 00:19:54.155
where copyright probably doesn't apply anyway. And so no matter what license you apply to it, the copyright is probably
232
00:19:54.535 --> 00:20:11.765
not legally applicable anyway. So there's a those are the kinds of issues that we have to sort of walk through with the the research researcher community and also essentially the professional community to try to deal with some of these questions because we wanna make sure that we're trying to use these tools like,
233
00:20:12.245 --> 00:20:12.745
interoperability,
234
00:20:14.245 --> 00:20:15.225
license interoperability,
235
00:20:16.085 --> 00:20:17.620
open data. We wanna
236
00:20:18.240 --> 00:20:19.940
have that to lead
237
00:20:20.240 --> 00:20:30.995
to good outcomes and not and not harmful outcomes. And so that's why we want people to be, thoughtful about how they're applying these kinds of tools. And further along the topic
238
00:20:31.534 --> 00:20:32.274
of expressiveness
239
00:20:32.815 --> 00:20:35.280
versus just factual information
240
00:20:35.740 --> 00:20:36.240
is
241
00:20:36.540 --> 00:20:41.120
any research articles that might either accompany or
242
00:20:41.500 --> 00:20:42.000
reference
243
00:20:42.585 --> 00:20:47.005
the data that is stored in OpenContacts or that's being submitted to OpenContacts.
244
00:20:47.385 --> 00:20:48.285
So I'm wondering
245
00:20:48.905 --> 00:20:52.970
in terms of the metadata that you track for a given dataset,
246
00:20:53.270 --> 00:20:56.809
if there is any either reference or content
247
00:20:57.190 --> 00:21:01.210
of any research articles that might be associated with those datasets
248
00:21:01.955 --> 00:21:02.935
and how
249
00:21:03.635 --> 00:21:06.775
any sort of industry journals or publications
250
00:21:07.235 --> 00:21:08.135
for archaeology,
251
00:21:08.960 --> 00:21:12.580
any relationship that you might have with them, whether it's positive
252
00:21:13.600 --> 00:21:14.260
or ambivalence?
253
00:21:14.880 --> 00:21:19.365
Yeah. Actually, it's, mostly been very positive in terms of, the sort
254
00:21:19.765 --> 00:21:21.225
the conventional publishers
255
00:21:21.845 --> 00:21:26.600
that we work with. And there's a couple different issues. 1 is that citation
256
00:21:26.900 --> 00:21:29.640
is the main kind of currency in the academic
257
00:21:30.420 --> 00:21:33.860
research and promotion, tenure, that type of thing. So the you want
258
00:21:34.904 --> 00:21:42.764
we definitely want people to cite datasets because people put a lot of work in creating those datasets, and then they put a lot of work in
259
00:21:43.130 --> 00:21:53.105
annotating, describing. And, you know, we put a lot of work working with researchers and cleaning them up and making them hopefully more usable by publishing them with open context. So
260
00:21:53.485 --> 00:21:54.385
that effort,
261
00:21:55.005 --> 00:21:55.985
we want to
262
00:21:56.285 --> 00:22:01.025
recognize a reward because it helps motivate people to do the right thing in sharing data.
263
00:22:01.630 --> 00:22:02.610
And if
264
00:22:02.990 --> 00:22:04.290
people get citations
265
00:22:04.830 --> 00:22:14.185
and people cite cite published datasets, then that should start nice positive feedback loop of people getting that kind of recognition and then, you know,
266
00:22:14.565 --> 00:22:16.105
and hope the ideal scenario
267
00:22:16.405 --> 00:22:30.395
is that researchers will do well by doing right, you know, that by sharing data, then they will advance their own careers. So in that sense, we participate with a variety of mechanisms in order to try to make citation
268
00:22:30.775 --> 00:22:31.275
easier
269
00:22:31.735 --> 00:22:33.355
and, and more meaningful.
270
00:22:34.250 --> 00:22:35.150
We OpenContacts
271
00:22:35.450 --> 00:22:37.310
uses a service called EasyID,
272
00:22:37.770 --> 00:22:41.070
which is hosted by the California Digital Library.
273
00:22:41.815 --> 00:22:44.635
California Digital Library is the main sort of institutional
274
00:22:45.815 --> 00:22:46.635
digital service,
275
00:22:47.175 --> 00:22:48.235
institutional repository
276
00:22:49.360 --> 00:22:53.620
for the University of California system. So all the different campuses of the UC system
277
00:22:54.080 --> 00:22:59.765
are involved with it, and EZ ID is a service to mint persistent
278
00:23:00.385 --> 00:23:07.285
identifiers. And in our case, the persistent identifiers that we meant with EZ ID are we use DOIs
279
00:23:07.665 --> 00:23:15.050
for large aggregations of data. In open context, those are called projects. We also use DOIs to identify
280
00:23:15.350 --> 00:23:21.725
tables. So another large aggregation of data, which would like a table that would be a dump a CSV dump of
281
00:23:22.025 --> 00:23:24.685
maybe thousands of records that we would express.
282
00:23:25.305 --> 00:23:32.690
And then we also use something called arcs, which are called arch archival resource keys, and that's another persistent identifier.
283
00:23:32.990 --> 00:23:41.785
And we use that for more granular kinds of content, in order to facilitate citation or something maybe very specific. And this is 1 of the advantages of open context versus,
284
00:23:42.265 --> 00:23:48.169
sort of more conventional data repository is that, we can make very specific entities
285
00:23:48.870 --> 00:23:50.250
of interest to an archaeologist
286
00:23:50.950 --> 00:24:02.220
directly citable. So, say, an example could be a coin. So somebody might discover a coin. It might have some interesting you know, it might be minted maybe in Rome and discovered
287
00:24:02.520 --> 00:24:09.955
clear across the empire all the way over in Turkey, and that would be kind of interesting to be able to trace while that really traveled far in antiquity.
288
00:24:10.255 --> 00:24:14.995
And you might wanna reference that specific object to that coin. If we only published,
289
00:24:15.670 --> 00:24:16.809
or only curated
290
00:24:17.110 --> 00:24:19.770
datasets as they sort of came to us and big
291
00:24:20.390 --> 00:24:27.804
tables of spreadsheets and big relational databases, that coin would not be something that you could actually cite because it would be
292
00:24:28.105 --> 00:24:28.765
a record,
293
00:24:29.465 --> 00:24:36.720
that might be in a giant data table, or it actually might be information that might be scattered on several different data tables. Right? So
294
00:24:37.100 --> 00:24:37.600
1
295
00:24:37.980 --> 00:24:41.555
Excel spreadsheet might describe the coin as described by the numismatist.
296
00:24:42.575 --> 00:24:48.835
Another relational database might describe a coin that was just that, you know, as it was recovered by
297
00:24:49.140 --> 00:24:54.280
somebody creating an object inventory. And another relational database might describe the contextual
298
00:24:54.980 --> 00:24:55.480
relationship,
299
00:24:56.420 --> 00:24:59.205
the context where that coin was actually discovered.
300
00:24:59.825 --> 00:25:06.804
And, when we publish things without the context, we're bringing all of that information from all those different source data files together
301
00:25:07.450 --> 00:25:09.230
to create a more cohesive
302
00:25:09.850 --> 00:25:25.530
picture of what the excavation results are. That means that that information about that 1 entity, that 1 coin, pulling that information out from several different sources and actually making it very convenient to be able to cite and discuss
303
00:25:26.070 --> 00:25:26.570
in,
304
00:25:27.750 --> 00:25:28.250
publications.
305
00:25:28.630 --> 00:25:31.130
And that's that's 1 of the sort of
306
00:25:31.430 --> 00:25:33.210
reasons why a lot of researchers
307
00:25:33.635 --> 00:25:38.615
find our approach actually kind of interesting and is that, yeah, they wanna be able to talk about
308
00:25:39.075 --> 00:25:42.535
a coin. They don't necessarily only wanna cite a spreadsheet,
309
00:25:43.260 --> 00:25:46.480
right, that could have thousands of other things in it too.
310
00:25:46.780 --> 00:25:49.920
So that's that's 1 of the key aspects of,
311
00:25:50.460 --> 00:25:52.995
all of this is that it's not just the sort of,
312
00:25:53.554 --> 00:25:58.695
fitting into that academic cycle of rewards of, you know, the citation attribution
313
00:26:00.220 --> 00:26:02.080
is is how you advance your career,
314
00:26:02.460 --> 00:26:09.440
but citation is actually a really important aspect where people make sense of things. You know, you want to be able to talk about a specific
315
00:26:10.175 --> 00:26:18.675
object or context or cite some other, some other entity of interest. And in order to talk about it, you have to be able to point to it. You have to reference it.
316
00:26:19.059 --> 00:26:22.279
And citation bit plays an important aspect in that as well.
317
00:26:22.659 --> 00:26:26.519
And in terms of the technical architecture
318
00:26:26.820 --> 00:26:27.799
that underpins
319
00:26:28.215 --> 00:26:29.755
the open context platform
320
00:26:30.215 --> 00:26:33.755
and the ability to manage these
321
00:26:34.135 --> 00:26:35.275
entity aggregations
322
00:26:35.975 --> 00:26:36.475
and
323
00:26:36.929 --> 00:26:43.429
expose the data for these direct citations and for being able to discover interrelations between the data.
324
00:26:43.730 --> 00:26:48.715
Can you just discuss the overall technical platform that you've built and,
325
00:26:49.495 --> 00:26:51.115
some of the challenges
326
00:26:51.630 --> 00:26:54.610
and architectural evolutions that have occurred along the way?
327
00:26:55.070 --> 00:26:55.890
Well, currently,
328
00:26:57.230 --> 00:26:58.270
well, first of all,
329
00:26:58.750 --> 00:27:02.544
the entire stack, everything that we use is fully open source. And,
330
00:27:03.164 --> 00:27:07.024
currently, the current iteration of open context is, it's a
331
00:27:07.580 --> 00:27:10.480
Python 3 application using the Django framework,
332
00:27:10.860 --> 00:27:16.565
and we have a Postgres relational databases or primary data store in the back end, and
333
00:27:17.185 --> 00:27:19.205
we use an Apache Solar index.
334
00:27:19.825 --> 00:27:24.720
It's basically a big document, NoSQL database for things like faceted search and
335
00:27:25.100 --> 00:27:30.559
not a lot of Open Context is really meant for sort of itself or data analysis or visualization.
336
00:27:31.274 --> 00:27:35.934
What we're trying mainly to do is to make the data that we curate
337
00:27:36.235 --> 00:27:36.735
browsable,
338
00:27:37.195 --> 00:27:37.695
discoverable.
339
00:27:37.995 --> 00:27:43.050
You can, actually look at it, and hopefully, it's somewhat aesthetically pleasing at least.
340
00:27:43.430 --> 00:27:47.770
That also matters because it's publishing and there's the aesthetic element is an issue.
341
00:27:48.070 --> 00:28:07.630
And, then once it's browsable and discoverable and all that, we wanna try to make it relatively easy to be able to grab the data that you want, and then you can use your own tools that you're comfortable with to, actually do your own analysis and visualization. So mostly for the community of people that we serve, that would be like
342
00:28:07.955 --> 00:28:08.775
browsing around,
343
00:28:09.315 --> 00:28:10.455
seeing, here's
344
00:28:10.755 --> 00:28:34.554
a set of pottery from this 1 archaeological site and then you narrow it down. Okay. So I'm just selecting the the certain time period of my interest, and then you can just export a CSV table of that and then walk away and play with it in Excel, something like that. More sophisticated users can use our API to do, more sophisticated kinds of things with it. The API that we provide
345
00:28:35.015 --> 00:28:35.515
provides
346
00:28:36.054 --> 00:28:37.419
most all of the data
347
00:28:38.279 --> 00:28:49.055
in, GeoJSON, which is a very popular geospatial data format. And you could also usually interpret the data that we have as JSON LD and then convert have that converted into
348
00:28:50.075 --> 00:28:52.095
a a graph of RDF triples.
349
00:28:52.690 --> 00:28:53.750
So there's
350
00:28:54.690 --> 00:29:09.425
different options for for using it, but but, basically, we're mainly aiming to try to make the data easier to discover so that you can grab and use in the tools that you're interested in. When we initially started, we were working at PHP and had a MySQL background,
351
00:29:11.165 --> 00:29:15.950
back end. And I guess I started working in Python probably 2013.
352
00:29:17.210 --> 00:29:17.929
It's when we,
353
00:29:18.570 --> 00:29:20.669
made the switch, 2013, 2014.
354
00:29:22.154 --> 00:29:31.455
And in the process of that switch going from MySQL to Postgres and PHP to Django, was there a high level of difficulty in terms of being able to translate
355
00:29:31.850 --> 00:29:35.710
the data that you had present on the platform at the time to the new system,
356
00:29:36.410 --> 00:29:38.670
or was it a fairly clean mapping
357
00:29:39.049 --> 00:29:48.545
where because it was simply just different relational databases, you were able to largely do an export and restore just as a set of SQL statements? Yeah. Actually,
358
00:29:49.750 --> 00:29:55.370
weirdly, we went through XML as an intermediary. So the the PHP by SQL
359
00:29:55.670 --> 00:30:05.315
version of open context, instead of having mainly a sort of GeoJSON structured data format that you could use as a view for all the different items, it was a XML
360
00:30:06.350 --> 00:30:10.530
structured data format. And, this goes back to some of the sort of intellectual
361
00:30:10.910 --> 00:30:20.045
pedigree of open context. There's a system at the University of Chicago called OCCR, Online Cultural Heritage Research Environment, and the initial incarnation
362
00:30:20.825 --> 00:30:21.645
of OCCR,
363
00:30:22.665 --> 00:30:27.090
was I mean, it still is an XML database, a native XML database,
364
00:30:27.390 --> 00:30:29.250
and they had a schema that they,
365
00:30:29.790 --> 00:30:30.850
publicized called
366
00:30:32.275 --> 00:30:35.815
archaeological markup language. Essentially, using rqml as
367
00:30:36.195 --> 00:30:38.775
our main kind of schema in our own back end,
368
00:30:39.235 --> 00:30:42.980
a simplified set of the archaML schema actually.
369
00:30:43.600 --> 00:30:45.700
And so, when we did the export
370
00:30:46.080 --> 00:30:46.559
from,
371
00:30:46.880 --> 00:30:50.815
migration from the PHP MySQL system in to our current
372
00:30:51.274 --> 00:30:55.135
iteration, it was using those XML documents as the source data.
373
00:30:55.755 --> 00:30:59.294
And as far as the ontology that you maintain
374
00:30:59.850 --> 00:31:01.070
and the taxonomical
375
00:31:01.370 --> 00:31:02.910
details of the datasets,
376
00:31:03.610 --> 00:31:05.230
is there a lot of difficulty
377
00:31:05.610 --> 00:31:07.390
as far as being able to
378
00:31:07.770 --> 00:31:09.230
map the different entities
379
00:31:09.585 --> 00:31:13.445
to those sets of attributes in order to make them more easily
380
00:31:14.065 --> 00:31:21.270
linkable so that you can traverse any relations between them. And then in terms of the metadata, I'm curious
381
00:31:21.570 --> 00:31:23.430
if you track the
382
00:31:23.810 --> 00:31:32.035
dig or the specific research project that the different entities are derived from so that you can have sort of a meta level of
383
00:31:32.495 --> 00:31:40.660
linkage. So where you track the different digs in relation to each other, and then from there, dig into the separate entities that are discovered?
384
00:31:40.960 --> 00:31:41.460
Yeah.
385
00:31:41.840 --> 00:31:42.340
So
386
00:31:43.014 --> 00:31:43.514
every,
387
00:31:43.975 --> 00:31:46.235
every single project, as I mentioned, has
388
00:31:47.174 --> 00:31:58.220
and even within a project, there'd be a lot of diversity. Different researchers who are working on different kinds of classes of materials, so like the pottery specialists, the bone specialists, the stone tool specialists,
389
00:31:58.760 --> 00:32:02.060
would come up with their own set of descriptive attributes to describe
390
00:32:02.774 --> 00:32:06.955
the materials that they're looking at, and, they could have their own control vocabularies
391
00:32:07.335 --> 00:32:16.140
and they in archaeology, those are called, typologies usually, sets of measurements that they're interested in, etcetera, etcetera, etcetera. And that is,
392
00:32:16.840 --> 00:32:24.635
unfortunately, the sort of state of discipline is that there's not a huge amount of consensus between researchers describing similar kinds of stuff.
393
00:32:24.935 --> 00:32:26.795
So that that diversity
394
00:32:27.255 --> 00:32:27.755
is
395
00:32:28.160 --> 00:32:28.900
1 of
396
00:32:29.360 --> 00:32:35.060
the reasons why we have the architecture that we have and then in that we have to
397
00:32:35.405 --> 00:32:42.545
publish the ways in which archaeologists are describing their materials. We have to publish all of these different kinds of attributes
398
00:32:42.845 --> 00:32:44.305
as data also,
399
00:32:44.730 --> 00:32:54.505
and that's part of the sort of intellectual contribution that these researchers are creating because, you know, they're usually encountering stuff that very few other people have encountered, and
400
00:32:54.805 --> 00:33:04.000
they are making it up as they go along. There's this is an active area of of of of research where people are trying to find
401
00:33:04.700 --> 00:33:06.799
new and better ways of describing
402
00:33:07.625 --> 00:33:09.565
archaeological deposits or archaeological
403
00:33:10.585 --> 00:33:11.485
material culture,
404
00:33:12.025 --> 00:33:17.799
that sort of thing. So this is like you know, they have a lot of, intellectual investment, basically,
405
00:33:18.660 --> 00:33:21.640
in creating and defining their own descriptive attributes.
406
00:33:22.715 --> 00:33:25.055
So when we publish datasets, we're also publishing
407
00:33:25.434 --> 00:33:28.255
a custom set of attributes associated with those datasets.
408
00:33:28.555 --> 00:33:30.495
And in order to achieve some
409
00:33:31.300 --> 00:33:31.800
interoperability
410
00:33:32.420 --> 00:33:32.920
where
411
00:33:33.460 --> 00:33:34.920
we can, we
412
00:33:35.380 --> 00:33:44.585
use the SCAS, a simple knowledge organization system, to say that, this 1 term in this 1 researcher's data set, you know, let's say, control vocabulary
413
00:33:45.285 --> 00:33:50.960
term, might be a close match to this concept in the Getty Art and Architecture Thesaurus.
414
00:33:51.340 --> 00:34:00.775
And so that would be a common standard that would be widely applicable in cultural heritage. So a good example might be a type of Etruscan pottery called bukero.
415
00:34:01.395 --> 00:34:03.495
So it's a the bukero is
416
00:34:04.059 --> 00:34:05.519
this really pretty,
417
00:34:06.139 --> 00:34:07.679
sort of dark gray shiny,
418
00:34:09.180 --> 00:34:11.599
heavily burnished kind of pot a pottery.
419
00:34:12.305 --> 00:34:13.685
And it is,
420
00:34:14.145 --> 00:34:18.065
it might be described in a couple different ways and different researchers' datasets, but,
421
00:34:18.785 --> 00:34:19.605
there are,
422
00:34:20.000 --> 00:34:24.900
some museums and the Getty Art and Architecture Thesaurus that have their own control vocabulary
423
00:34:25.520 --> 00:34:30.444
that describe that. And Getty Art and Architecture Thesaurus is a nice openly licensed
424
00:34:30.744 --> 00:34:33.065
link data kind of thesaurus, and then we can say,
425
00:34:33.944 --> 00:34:39.319
this term bucurro is a close match for what Getty said how Getty defines it as bucur.
426
00:34:40.500 --> 00:34:44.359
And then we could essentially make those those common linkages
427
00:34:44.660 --> 00:34:46.119
across these idiosyncratic
428
00:34:47.454 --> 00:34:48.435
kinds of descriptions
429
00:34:49.055 --> 00:34:52.115
to point to some common standards. And so that is
430
00:34:52.655 --> 00:34:53.555
more or less
431
00:34:53.855 --> 00:34:56.194
the sort of level of semantic
432
00:34:56.750 --> 00:34:57.250
interoperability,
433
00:34:57.550 --> 00:34:59.330
I guess, that you would get in open context
434
00:34:59.710 --> 00:35:01.010
where we'd we basically
435
00:35:01.550 --> 00:35:04.290
start drawing relationships between different
436
00:35:04.830 --> 00:35:05.330
vocabulary
437
00:35:05.630 --> 00:35:07.734
terms. So it's not that sophisticated.
438
00:35:08.515 --> 00:35:11.414
We're not in the sort of business of supporting sort
439
00:35:11.714 --> 00:35:16.310
of inferences on owl ontologies or anything like that. That would be an awesome,
440
00:35:16.850 --> 00:35:19.190
thing to do more in the future, especially,
441
00:35:19.970 --> 00:35:23.670
if we get to collaborate with some people with those sorts of interest and expertise.
442
00:35:24.535 --> 00:35:33.674
But we we do get is the ability to, you know, combine different datasets, and we've been mostly most successful with zooarchaeology, with animal bones,
443
00:35:34.160 --> 00:35:44.395
where people have, we've basically seen that kind of technique. You can actually do searches now and discover all the cattle bones and, you know, all the tibia of cattles from
444
00:35:44.775 --> 00:35:46.075
dozens of different archaeological
445
00:35:46.375 --> 00:35:55.900
projects. And that is a pretty big accomplishment in a field like archaeology where everybody's doing their own thing. And once the data is written into
446
00:35:56.520 --> 00:35:57.340
open context,
447
00:35:57.960 --> 00:36:03.805
is it largely static at that point, or do you have people who will periodically
448
00:36:04.105 --> 00:36:07.965
go back through and add additional information or metadata
449
00:36:08.870 --> 00:36:09.770
or linkages
450
00:36:10.470 --> 00:36:15.610
after the fact? And is that something that users of the platform are able to contribute
451
00:36:15.910 --> 00:36:18.755
as suggestions or enhancements to different datasets?
452
00:36:19.055 --> 00:36:26.250
That would be really interesting to be able to support that. No. We haven't built out that sort of functionality of sort of, sort of user contributed
453
00:36:26.710 --> 00:36:31.450
semantic enhancement or or even just, you know, hey. There's a bug here, an error.
454
00:36:31.885 --> 00:36:50.435
1 of the things that we've, been experimenting with and and and and are gonna be moving ahead with in the near term is, also just version control of data. For the most part, most of the data that we publish with OpenContacts is once it's published, it's pretty static. The main changes are usually going to be additions
455
00:36:50.974 --> 00:36:58.559
where, you know, we might publish a set of context first, and then the pottery person is ready with their pottery data and we add to that context
456
00:36:58.940 --> 00:37:04.799
dataset pottery and etcetera etcetera. So most of it's additions, but sometimes there's gonna be error corrections
457
00:37:05.099 --> 00:37:06.640
and and that type of thing.
458
00:37:07.105 --> 00:37:15.605
And we have essentially, because we generate all these JSON LD documents, we have used Git actually as a a version control
459
00:37:16.100 --> 00:37:24.280
mechanism for all of that. We're doing more of that internally because it was starting to get unwieldy with Git. We're just building these giant bloated,
460
00:37:24.580 --> 00:37:25.080
Git
461
00:37:34.240 --> 00:37:47.065
internally, and, we're trying to figure out a good way of of of expressing that publicly of what those changes are. So that's a actually an area of active development, but we do have a sort of an audit trail of how things are changing internally.
462
00:37:47.365 --> 00:37:48.984
And that brings me into
463
00:37:49.380 --> 00:37:52.519
a couple of my next questions. 1 of which is
464
00:37:53.140 --> 00:37:54.680
the average size
465
00:37:55.140 --> 00:37:58.359
of the typical dataset that you would be processing,
466
00:37:59.455 --> 00:38:05.235
just in terms of relative volumes? Are they generally in the megabyte, gigabyte, terabyte scale?
467
00:38:05.695 --> 00:38:11.680
And then I know that part of your overall endeavor is also to serve as an archive
468
00:38:12.140 --> 00:38:12.640
repository
469
00:38:12.940 --> 00:38:14.240
for a lot of this information.
470
00:38:14.540 --> 00:38:22.305
So I'd be interested in exploring some of the complications that arise from trying to maintain a long term
471
00:38:22.605 --> 00:38:24.150
record of this information,
472
00:38:24.450 --> 00:38:24.950
particularly
473
00:38:25.250 --> 00:38:25.750
as
474
00:38:26.210 --> 00:38:27.270
storage formats
475
00:38:27.730 --> 00:38:37.545
and physical medium changes over time? Well, I'll just start with the archival issue first. We're not a preservation repository because that involves a whole host of
476
00:38:38.165 --> 00:38:41.125
institutional requirements and workflows and processes that,
477
00:38:41.700 --> 00:38:43.000
we just don't have the
478
00:38:43.460 --> 00:38:52.035
capability, we don't have the scale to be able to achieve ourselves. But what we do do is we when when we publish data, we put them we put the data that we publish
479
00:38:52.575 --> 00:38:54.195
into preservation repositories.
480
00:38:54.655 --> 00:38:56.994
So, but those are managed by other institutions.
481
00:38:57.470 --> 00:39:03.250
So, historically, the main institution that we've worked with and we continue to work with is the California Digital Library,
482
00:39:03.710 --> 00:39:07.010
which is, again, the main repository of the University of California system.
483
00:39:07.775 --> 00:39:09.395
We're also using Zenodo,
484
00:39:09.855 --> 00:39:11.235
which is another repository
485
00:39:11.535 --> 00:39:14.755
that's in basic it's at CERN, the, you know, the the particle
486
00:39:15.130 --> 00:39:17.869
physics research lab in in Switzerland. And
487
00:39:18.170 --> 00:39:25.375
the rationale for using Zenodo is that they have a very nice convenient API, and they support a lot of metadata
488
00:39:25.675 --> 00:39:33.410
using the linked data entities that find useful. And there's already a lot of sort of convergence on using them by other,
489
00:39:33.950 --> 00:39:37.070
archaeologists. So it's it's the the material is there with,
490
00:39:37.869 --> 00:39:38.590
there's a good,
491
00:39:39.195 --> 00:39:43.195
sort of a community of people already engaged with Zenodo. So that's,
492
00:39:43.675 --> 00:39:46.255
that's for the for for long term data preservation,
493
00:39:46.555 --> 00:39:47.195
we don't,
494
00:39:47.675 --> 00:39:48.175
actually,
495
00:39:48.670 --> 00:39:54.850
you know, we're not gonna be the end place for it. But what we can do on our end is to try to make that data preservation
496
00:39:55.310 --> 00:39:57.345
more feasible, more likely to
497
00:39:57.744 --> 00:40:02.484
actually work out in the sense that, when by publishing data with open context,
498
00:40:02.865 --> 00:40:05.825
then we're extracting data from a bunch of, different,
499
00:40:06.145 --> 00:40:06.645
source
500
00:40:07.000 --> 00:40:15.340
datasets with very different kinds of file formats and whatnot and expressing it as structured data and, you know, adjacent JSON LD for the most part,
501
00:40:15.704 --> 00:40:20.365
CSV, so these nice simple text formats that are lend themselves to preservation.
502
00:40:20.744 --> 00:40:40.484
And then the other thing is that, oftentimes, we often get a lot of media associated with the structured data that we publish. So, you know, talk about coins or pots or whatever, 1 of the nice things is about having that sort of level of granularity and citation that we have is that we could also do things like relate specific pictures to specific objects
503
00:40:40.865 --> 00:40:42.164
or specific pictures
504
00:40:42.670 --> 00:40:52.635
or even 3 d models to objects or in context, and we do do that also. And when we publish the material in Open Context, we're also depositing a lot of media files, binary media files
505
00:40:53.015 --> 00:40:54.475
of 1 form or another,
506
00:40:54.775 --> 00:40:59.915
into digital repositories also. So those are the ways in which we we work for,
507
00:41:00.375 --> 00:41:00.875
archival
508
00:41:01.230 --> 00:41:06.190
in the archival sense and providing standard metadata and everything that hopefully makes the,
509
00:41:06.670 --> 00:41:07.730
job of the digital,
510
00:41:08.270 --> 00:41:08.770
archives
511
00:41:09.455 --> 00:41:16.035
easier in order to maintain the data. Right. So, I mean, it it varies all over the place. So, some datasets are quite small,
512
00:41:16.450 --> 00:41:22.310
say, a megabyte spreadsheet or something like that, and and they and then they go up from there.
513
00:41:23.090 --> 00:41:29.655
What really bloat size or expands size greatly is if somebody has a lot of images or the digital media.
514
00:41:30.115 --> 00:41:33.970
So, 3 d files are take up a lot of space, obviously.
515
00:41:34.270 --> 00:41:34.770
But,
516
00:41:35.310 --> 00:41:36.770
in in certain circumstances,
517
00:41:37.390 --> 00:41:38.770
we recently published
518
00:41:39.070 --> 00:41:39.550
a set of,
519
00:41:40.565 --> 00:41:44.585
some archival materials for the describing the great sphinx
520
00:41:45.045 --> 00:41:46.585
at Giza in Egypt.
521
00:41:47.365 --> 00:41:49.790
And, you know, even 1 file
522
00:41:50.410 --> 00:41:51.950
1 1 scan image,
523
00:41:52.410 --> 00:41:53.710
could be a gigabyte
524
00:41:54.010 --> 00:41:55.850
itself. So, and that was
525
00:41:56.335 --> 00:41:59.855
that that's and we would have, something like 30, 000. So,
526
00:42:00.734 --> 00:42:05.315
that there's sometimes some large space and storage concerns. And, again,
527
00:42:06.530 --> 00:42:14.310
we really couldn't do what we do without having that, fundamental infrastructure, the services being provided by the digital repositories, the digital libraries,
528
00:42:14.695 --> 00:42:20.955
hosting all these things over the long term because hosting all these things on our own actually starts getting expensive too,
529
00:42:21.415 --> 00:42:22.635
with all those files.
530
00:42:23.130 --> 00:42:24.590
And as you mentioned,
531
00:42:25.210 --> 00:42:26.670
you don't do
532
00:42:26.970 --> 00:42:28.350
any in house,
533
00:42:28.890 --> 00:42:31.310
complicated analyses or visualizations
534
00:42:32.035 --> 00:42:38.295
of the data that you are hosting. So I'm wondering what some of the most interesting or
535
00:42:38.835 --> 00:42:40.055
unexpected uses
536
00:42:40.380 --> 00:42:49.200
of your data you have seen out in the wild? Yeah. Probably the most significant use of data for us has been the digital index of North American archeology,
537
00:42:50.615 --> 00:42:53.035
and it's the DENA project is this acronym.
538
00:42:53.575 --> 00:42:55.195
So what DENA is is
539
00:42:55.805 --> 00:42:56.555
a aggregation
540
00:42:57.175 --> 00:43:03.839
of data that's a little bit different for us in Open Context is that most of the data in Open Context come to us from academic researchers.
541
00:43:04.460 --> 00:43:09.815
DINA, the datasets coming to us in that project are coming to us from state government officials.
542
00:43:10.675 --> 00:43:13.494
And so in the United States, there are historical
543
00:43:14.115 --> 00:43:15.335
preservation laws
544
00:43:15.790 --> 00:43:18.130
that are, at the federal level,
545
00:43:18.910 --> 00:43:28.295
and the federal government has delegated the enforcement of those laws to different state governments. And so, each state has officials in charge of,
546
00:43:28.695 --> 00:43:32.620
managing cultural properties, historical resources within the state borders.
547
00:43:33.100 --> 00:43:34.400
And each state develops,
548
00:43:35.580 --> 00:43:39.760
information systems, so databases basically to help in that administration.
549
00:43:40.415 --> 00:43:42.655
And you can imagine there are about 50 different,
550
00:43:43.455 --> 00:43:43.955
schemas,
551
00:43:44.655 --> 00:43:45.155
datasets.
552
00:43:45.935 --> 00:43:47.075
50 different ways
553
00:43:47.375 --> 00:43:50.500
that states have sort of tried to figure out how to manage
554
00:43:52.320 --> 00:43:54.820
historical and archaeological sites. And
555
00:43:55.965 --> 00:43:57.965
what's interesting about that is that,
556
00:43:58.525 --> 00:44:00.865
the state governments have the best window
557
00:44:01.565 --> 00:44:02.065
onto
558
00:44:02.445 --> 00:44:05.025
the sort of what is known about
559
00:44:06.270 --> 00:44:09.970
the past in terms of what is known about the settlement history of
560
00:44:10.349 --> 00:44:11.970
the North American continent,
561
00:44:12.430 --> 00:44:17.125
since the Pleistocene, because the states have aggregate all of this data that is being
562
00:44:17.825 --> 00:44:24.570
required by law to be able to to to to create. And, what's the neat thing about Dina is that we're gathering,
563
00:44:25.190 --> 00:44:25.850
by bringing
564
00:44:26.470 --> 00:44:37.295
these datasets together from several different states. For the first time, researchers are able to see a much bigger picture about what human settlement is like across the North American context over very deep
565
00:44:37.675 --> 00:44:46.500
long periods of time. So you can see what things look like 10000 years ago in terms of what's known about where there are archaeological sites
566
00:44:46.800 --> 00:44:51.315
and compare that to what what the situation was like 500 years ago,
567
00:44:52.035 --> 00:44:53.575
the beginning of European colonization.
568
00:44:54.435 --> 00:44:55.714
So that is a,
569
00:44:56.195 --> 00:44:58.375
a really powerful kind of a resource.
570
00:44:59.000 --> 00:45:04.540
There's all sorts of different kinds of really interesting issues in terms of, like, these data sets are also
571
00:45:04.840 --> 00:45:16.090
not created with any sort of systematic sampling at all. There's all sorts of biases in them, and that's something that I think is a really fascinating area to explore and it's gonna be a huge challenge for interpretation.
572
00:45:16.470 --> 00:45:17.210
But nevertheless,
573
00:45:17.750 --> 00:45:19.530
this is this dataset is,
574
00:45:19.910 --> 00:45:22.330
inserted as big as it gets in archaeology,
575
00:45:23.365 --> 00:45:25.944
for looking at big picture kinds of questions.
576
00:45:26.885 --> 00:45:29.385
And the the big research outcome actually,
577
00:45:29.845 --> 00:45:30.345
hasn't
578
00:45:31.530 --> 00:45:38.905
focused on so much of, like, you know, where people are living in the past, but the big research outcome we've had with the DNID data set has been
579
00:45:39.305 --> 00:45:41.805
modeling what's gonna happen to these sites in the future.
580
00:45:42.185 --> 00:45:53.630
And we have most of the coverage of DNIS focuses on the American Midwest and the Southeast, but we have a lot of the, Gulf Coast and the, Atlantic Seaboard in the DNIS dataset. And,
581
00:45:54.030 --> 00:46:02.235
this paper that was published in PLOS 1 described what the impact would be with, say, a 1 meter, 2 meter, 3 meter rise in sea level,
582
00:46:02.795 --> 00:46:04.175
driven by climate change.
583
00:46:04.660 --> 00:46:06.839
And, just with a very,
584
00:46:07.300 --> 00:46:12.920
modest 1 meter rise in sea level, which a lot of projections are saying is gonna be happening within,
585
00:46:13.305 --> 00:46:14.685
you know, next few decades,
586
00:46:16.025 --> 00:46:18.925
we're gonna lose 13, 000 known archeological
587
00:46:19.225 --> 00:46:32.924
sites. And, you know, that had a huge amount of press coverage and media impact because a lot of people didn't even know that there were 13, 000 archaeological sites in North America, let alone just along the coast that we'd be losing to, sea level rise.
588
00:46:33.385 --> 00:46:36.204
So that 1 was a big that made a big, impact,
589
00:46:36.585 --> 00:46:38.525
pardon the pun. It made a big splash,
590
00:46:39.370 --> 00:46:42.830
and it was, recently cited in the 2018,
591
00:46:43.450 --> 00:46:44.830
national climate assessment,
592
00:46:45.130 --> 00:46:46.090
which was a major,
593
00:46:46.490 --> 00:46:53.825
US government report about the challenges of climate change. So that that's probably 1 of the key research outcomes that we've had,
594
00:46:54.285 --> 00:46:57.184
with Open Context, and then there have been, several,
595
00:46:57.960 --> 00:47:02.059
smaller but still interesting and significant ones. Again, with the zooarchaeological
596
00:47:02.520 --> 00:47:04.859
data, people be able to see
597
00:47:05.275 --> 00:47:07.695
patterns in the way that animal domestication
598
00:47:08.155 --> 00:47:12.895
that which happened first in the, near east in in places like
599
00:47:13.210 --> 00:47:15.470
Iraq and Syria and Southeast Turkey
600
00:47:16.010 --> 00:47:16.970
and how those,
601
00:47:17.290 --> 00:47:21.150
domestic animals and the and the economies around animal husbandry,
602
00:47:22.224 --> 00:47:27.285
spread, towards Europe. That's something that was facilitated with Open Context and there's been,
603
00:47:27.825 --> 00:47:31.430
publication with that. And then the data that we published originally with that,
604
00:47:31.910 --> 00:47:40.815
has been reused in other publications, which is also interesting. So there's actually reuse of this published data for more than 1 research purpose. And then,
605
00:47:41.115 --> 00:47:41.615
there's,
606
00:47:42.395 --> 00:47:42.895
other
607
00:47:43.195 --> 00:47:52.110
applications which are kind of more fun. People using the API to do things like pull in the data for virtual reality or for augmented reality kinds of applications.
608
00:47:52.570 --> 00:48:05.320
Those are smaller scale, but what's fun about that and what's also useful about that is that we really need to build up the technical skills in archaeology. People who have the data skills to be able to engage with the system like Open Context
609
00:48:05.860 --> 00:48:11.800
and develop the sort of understanding about how to use a web API and what can you do with it. And so
610
00:48:12.115 --> 00:48:17.095
those are the types of things that are maybe in the longer term really important because
611
00:48:17.395 --> 00:48:21.710
that starts getting people more engaged with this whole idea of
612
00:48:22.190 --> 00:48:23.410
why shared data
613
00:48:23.710 --> 00:48:34.855
is important and, you know, what can you do with APIs, or whether it's an open context API. Oh, look, there's data from a completely different source. Maybe you combine it with geo names or combine it with some other,
614
00:48:35.255 --> 00:48:38.555
Wikidata or something else. And then and then that starts to maybe,
615
00:48:39.170 --> 00:48:41.670
enable some really interesting kinds of applications.
616
00:48:42.290 --> 00:48:43.430
And in the process
617
00:48:43.730 --> 00:48:44.230
of
618
00:48:44.610 --> 00:48:47.590
building and maintaining the open context platform
619
00:48:47.965 --> 00:48:50.545
and working with researchers and archaeologists,
620
00:48:51.325 --> 00:48:58.830
what have been some of the most interesting or useful or challenging lessons that you've learned? 1 of the biggest things is that it is
621
00:49:00.330 --> 00:49:11.835
incredibly slow to change a disciplinary culture, and and that is that, you know, most academic researchers in the world of publish or perish, and so they have a lot of professional pressures to
622
00:49:12.455 --> 00:49:13.275
make publications
623
00:49:13.575 --> 00:49:18.600
that they're sure that their peers will recognize as something is valuable. And so that,
624
00:49:19.000 --> 00:49:27.565
sometimes leads to some risk aversion, like publishing data is something that's relatively new and it's not a sort of tried and true path to tenure.
625
00:49:27.944 --> 00:49:30.265
And so that's 1 of the reasons why it's,
626
00:49:30.665 --> 00:49:42.484
adoption has not been as fast as we would want. Fortunately, that situation is changing now that we're starting to see some interesting reuses of data and people are starting to cite it, and it's appearing in bibliographies
627
00:49:42.945 --> 00:49:44.565
of research papers and whatnot.
628
00:49:45.105 --> 00:49:52.599
So things are getting are definitely getting better, but it but it has taken, you know, more than 10 years to get to this point, and that and that and,
629
00:49:53.140 --> 00:50:17.265
you would think that that would be something that would have been more of a no brainer. Other issues that are also really fascinating is just that, again, that what gets people excited is the the there are a bunch of things that get people excited. So some some research really like seeing their data online. They like to they want to show it off. They want to see it's a great way to have sort of an exhibition. So, you know, there are museums and,
630
00:50:17.725 --> 00:50:18.545
other institutions
631
00:50:19.165 --> 00:50:22.390
have beautiful websites that show off their collections, and and,
632
00:50:22.789 --> 00:50:23.289
researchers
633
00:50:23.589 --> 00:50:32.375
are happy to use a service like OpenContacts sometimes so they can participate in that. Show off, you know, what is it that they've discovered and and also show off
634
00:50:32.935 --> 00:50:45.339
in a in in a lot of ways that, look, I do such really awesome and rigorous recording. Look at all this wonderful data that I've got. So that's a so that's a a a positive incentive, and so that's something that I think has been underappreciated
635
00:50:46.040 --> 00:50:51.494
in the world of research data management, and that a lot of times, there's a sort of discussion of carrots and sticks
636
00:50:51.875 --> 00:50:53.175
about this. And and,
637
00:50:54.115 --> 00:50:57.950
there's a sort of efforts to make sharing data mandatory
638
00:50:58.410 --> 00:50:59.790
and and that, you know,
639
00:51:00.090 --> 00:51:03.230
if you want a grant, you have to share your data. And
640
00:51:03.635 --> 00:51:22.685
in a lot of ways, I think that's I'm very sympathetic to that. I think that's, you know, definitely the right thing to do, but we also need the positive incentives. So, you know, citation is obviously a positive incentive, people doing coming up with good research outcomes from shared data, that's a good positive incentive. But the other side is just that we want people to feel
641
00:51:23.145 --> 00:51:27.244
that publishing and sharing data is something that's recognized rewarding.
642
00:51:27.625 --> 00:51:31.039
It helps make their data look more attractive. It's more usable.
643
00:51:31.339 --> 00:51:42.525
Their own dataset is more usable to them. There's a lot of positive incentives that I think is good that we try to emphasize with open context because then we wanna try to get past the sort of notion that data sharing is just, like,
644
00:51:42.825 --> 00:51:52.410
minimal checkbox compliance thing, you know, where, I I threw a couple ugly spreadsheets into a repository and I'm done. Right? We that's not very meaningful.
645
00:51:53.015 --> 00:51:56.235
The data are probably not that useful if they're just chucked,
646
00:51:56.615 --> 00:52:09.650
left as is in Excel with different kinds of color codes and random typos and all sorts of problems in them. It's a lot better if people put in that sort of investment and that intellectual investment in actually trying to
647
00:52:10.325 --> 00:52:34.435
understand how to make their data usable and intelligible by our wider community. And that's where I think the things needs to be much more emphasis on that, and that's where I think it's why this field is actually really interesting. It's that it's hard to communicate meaning with structured data and, especially in a field like this where we have practitioners who are working very particular kinds of things. There's not a tradition of this already established,
648
00:52:34.735 --> 00:52:37.960
and we're trying to build that tradition kind of from scratch. And
649
00:52:38.740 --> 00:52:41.480
what do you have in mind as far
650
00:52:41.780 --> 00:52:42.440
as future
651
00:52:42.740 --> 00:52:43.240
improvements
652
00:52:43.540 --> 00:52:46.440
or future additions or overall goals
653
00:52:46.755 --> 00:52:55.174
for the future of Open Context? Well, I mean, 1 of the big things right now is that we're really excited because we're working with another developer who's
654
00:52:55.560 --> 00:53:00.780
deep, in the guts of the system right now, Raymond Yee. And Raymond has a great background
655
00:53:01.160 --> 00:53:02.780
in, research computing.
656
00:53:03.240 --> 00:53:08.035
He also has he's worked with Unglue It. He's he's done some really wonderful work,
657
00:53:08.575 --> 00:53:11.955
and, he's a he's he's a fantastic Python developer.
658
00:53:12.339 --> 00:53:23.885
And so 1 of the first things that we're working on is making it easier to, deploy open context actually so that, it's so much more of a sort of a 1 line command line argument type of a thing where you just,
659
00:53:24.265 --> 00:53:29.085
you know, can set the whole thing up. So using technologies like Ansible and Docker,
660
00:53:29.900 --> 00:53:32.720
to be able to do that where you can get an instance going,
661
00:53:33.020 --> 00:53:34.160
maybe with some data,
662
00:53:34.859 --> 00:53:35.359
and,
663
00:53:36.220 --> 00:53:37.635
work with it on your own,
664
00:53:38.115 --> 00:53:42.615
play with it, and and hopefully, that'll encourage maybe, more experimentation
665
00:53:42.915 --> 00:53:45.495
and maybe even more people contributing to this,
666
00:53:46.069 --> 00:54:17.925
to the open source project, which would be cool. So there's that. Then we have a lot of needs in terms of, we're bumping up into some scaling issues. Things are starting to slow down as, as as our index has grown. And so just sharding it is is gonna be an important next step. Lots of optimization side things to do in the back end to try to improve speed. And once we've got some of that worked out, then we're gonna be in a much better position to start tackling some really challenging user experience kinds of issues. So 1 of the hard things about archaeology
667
00:54:18.225 --> 00:54:20.805
and especially the approach that we have is that
668
00:54:21.265 --> 00:54:21.765
because
669
00:54:22.390 --> 00:54:24.890
there's so many different ways of describing
670
00:54:25.190 --> 00:54:26.010
these materials,
671
00:54:26.310 --> 00:54:26.710
then,
672
00:54:27.270 --> 00:54:27.770
presenting
673
00:54:28.070 --> 00:54:31.210
search results and making information easy to
674
00:54:31.615 --> 00:54:34.355
retrieve and find is is hard. And
675
00:54:34.655 --> 00:54:36.095
so we need to,
676
00:54:36.495 --> 00:54:38.595
put a lot more investment in in,
677
00:54:39.695 --> 00:54:41.155
trying to make
678
00:54:41.680 --> 00:54:42.420
the exploration
679
00:54:43.280 --> 00:54:44.819
of Open Context
680
00:54:45.200 --> 00:54:48.579
much easier than it is right now because it's it's it it is challenging
681
00:54:49.295 --> 00:55:15.830
because of the diversity of the materials in there. We need to come up with better ways of guiding users to be able to find what they're looking for and then, making sure that they once they found what they're looking for, making sure that they can leave with the the dataset that they're they expect it to have in a format that is going to be useful for for them without too many technical hurdles in the way. And are there any other aspects of Open Context
682
00:55:16.130 --> 00:55:18.710
or research data curation and publication
683
00:55:19.010 --> 00:55:23.035
that we didn't discuss yet that you'd like to cover before we close out the show?
684
00:55:23.494 --> 00:55:24.454
Well, I think that,
685
00:55:25.175 --> 00:55:36.110
again, 1 of the weird things about this and the and the the the harder aspects of this too is that this is a really new area of of research in a lot of ways. And what it really requires
686
00:55:37.115 --> 00:55:46.950
are, you know, people with, the sort of technical programming kinds of skills, data skills, so this would, you know, data science, I guess, and also
687
00:55:47.650 --> 00:55:51.269
library skills, thinking about issues of metadata or preservation,
688
00:55:52.130 --> 00:55:53.875
copyright and intellectual property,
689
00:55:54.355 --> 00:55:56.375
and then and then there's the domain skills,
690
00:55:56.995 --> 00:56:06.109
understanding the domain of archaeology and its specific requirements and why is it that we have a 100 different researchers with and 200 different classification
691
00:56:06.410 --> 00:56:06.910
systems
692
00:56:07.289 --> 00:56:10.755
for the same set of materials. So that's a so there's a lot of these
693
00:56:11.234 --> 00:56:16.535
kinds of areas of expertise that we have to bring together, and we don't really have right now,
694
00:56:17.154 --> 00:56:18.454
institutions that are
695
00:56:19.380 --> 00:56:34.665
well set up to be able to cope with these new needs. We don't really have really good clear career paths and and, ways of sustaining all of this. And so that's so that's 1 of the big challenges that, you know, things are very siloed in universities
696
00:56:35.045 --> 00:56:37.145
where, say, a university librarian
697
00:56:37.850 --> 00:56:38.990
does library things,
698
00:56:39.370 --> 00:56:39.870
academic
699
00:56:40.250 --> 00:56:47.150
researchers publish in conventional journals. We're asking people to come in together and bridge several different kinds of domains
700
00:56:47.464 --> 00:56:57.005
that traditionally haven't really worked closely together. And that's that's that's a real challenge, and that's something that is not just in archaeology, but it's 1 of the real fundamental challenges
701
00:56:57.440 --> 00:56:57.920
of,
702
00:56:58.240 --> 00:57:01.859
managing research data across all sorts of different disciplines.
703
00:57:02.240 --> 00:57:22.900
So we definitely need institutional structures that can help organize that, facilitate that, sustain that, and and and also the people that are involved in that. There's a lot of expertise that has to go into this that doesn't really fit well within conventional structures. So, you know, the, big challenge moving forward is that I think a lot of people real realize and
704
00:57:23.280 --> 00:57:37.110
that research data is important, that we have to curate it well, we wanna preserve it so they can be accessible for people in the future. People can say new things with it, so there's there's all sorts of interest in this, but we still haven't nailed down
705
00:57:37.730 --> 00:57:38.950
exactly how
706
00:57:39.490 --> 00:57:39.990
institutions
707
00:57:40.450 --> 00:57:41.490
are going to,
708
00:57:41.890 --> 00:57:42.950
make this information
709
00:57:43.744 --> 00:57:46.085
broadly accessible and usable for the future,
710
00:57:46.385 --> 00:57:54.610
right now, it it feels still pretty rickety in terms of the sort of sustaining supports. Like, we wanna make sure that this type of thing has becomes
711
00:57:55.310 --> 00:58:00.905
less weird and much more of a regular part of the way research is conducted in the 21st century.
712
00:58:01.365 --> 00:58:04.185
Alright. Well, for anybody who wants to
713
00:58:04.565 --> 00:58:21.275
get in touch with you or follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today. Oh, alright. Well, I mean,
714
00:58:22.934 --> 00:58:28.599
that's a really good question. From our own specific work, we spend a huge amount of time cleaning up datasets.
715
00:58:29.220 --> 00:58:30.200
We use OpenRefine
716
00:58:32.625 --> 00:58:36.325
a lot, and we spend more time with using OpenRefine than actually
717
00:58:36.625 --> 00:58:38.085
munging data within OpenContacts
718
00:58:38.385 --> 00:58:42.880
itself. I think so much of it is that so many of the problems that we encounter
719
00:58:43.339 --> 00:58:46.640
work well with OpenRefine, but there's still that 20%
720
00:58:47.099 --> 00:58:49.760
where, essentially, you have to write custom scripts,
721
00:58:50.315 --> 00:58:52.575
you know, a little Python to be able to
722
00:58:53.115 --> 00:58:58.975
manage data in as in a certain way. And where I think that 1 of the challenges
723
00:58:59.800 --> 00:59:02.520
around all that is, you know, those custom scripts we have,
724
00:59:03.400 --> 00:59:08.700
checked into the source control and everything, but I I I wanna have that as something that it's a lot more
725
00:59:09.725 --> 00:59:10.225
reproducible
726
00:59:10.605 --> 00:59:11.665
and something that,
727
00:59:12.685 --> 00:59:16.465
and people have been working on this with the whole area of reproducible research.
728
00:59:16.780 --> 00:59:25.075
You know, there's been a lot of work, interesting work using things like, Jupyter Notebooks and all that type of thing, but it needs to be a much
729
00:59:25.714 --> 00:59:27.255
it'd be really good to be
730
00:59:27.795 --> 00:59:29.734
able to have these kinds
731
00:59:30.195 --> 00:59:30.855
of preprocessing,
732
00:59:31.234 --> 00:59:33.095
data processing kinds of pipelines
733
00:59:33.859 --> 00:59:35.160
much easier to,
734
00:59:36.180 --> 00:59:45.305
curate and to be able to sort of, I guess, audit, to be able to see exactly what the inputs are, what you did, and what the outputs came out to be so that you can
735
00:59:45.605 --> 00:59:49.385
make sure that if you have a problem, you can trace it, and you can justify
736
00:59:49.685 --> 00:59:52.025
what you did along every step of the way
737
00:59:52.450 --> 01:00:02.815
and do that in a way that is not overly onerous in terms of you know, you also have to get stuff done. Right? So there's this balance between trying to be trying to document everything being reproducible
738
01:00:03.355 --> 01:00:04.815
and also trying to,
739
01:00:06.395 --> 01:00:23.540
actually make sure that, you know, you you can make a deadline, get the data actually out without, spending too much time and resources and then doing that. So I think that that's an interesting area that could use some tooling. I I think it's a really challenging area, but that's something that I'm really, interested in.
740
01:00:23.940 --> 01:00:37.305
Anybody has any suggestions too. Well, I want to thank you very much for taking the time today to join me and discuss the work that you're doing at Open Context. It's definitely a very interesting project, and it's filling a very
741
01:00:37.684 --> 01:00:38.184
useful
742
01:00:38.565 --> 01:00:39.464
and necessary
743
01:00:39.765 --> 01:00:47.210
need. So thank you for all of your work on that and for taking the time tonight, and I hope you enjoy the rest of your evening. Alright. Thanks, Tobias.