UDSKRIFT 🔗
1
00:00:05,401 --> 00:00:08,931
Jennifer Reif: You are listening to the
Breaktime Tech Talks podcast, a bite-sized
2
00:00:08,952 --> 00:00:13,495
tech podcast for busy developers, where
we'll briefly cover technical topics, news
3
00:00:13,495 --> 00:00:15,777
snippets, and more in short time blocks.
4
00:00:16,236 --> 00:00:19,942
I'm your host, Jennifer Reif, an
avid developer and problem solver
5
00:00:20,152 --> 00:00:24,114
with special interest in data,
learning, and all things technology.
6
00:00:24,607 --> 00:00:27,927
This week, I will give you a whirlwind
tour of everything I have coming
7
00:00:27,927 --> 00:00:32,100
up, from virtual events to in-person
appearances, and what I've been
8
00:00:32,100 --> 00:00:33,648
working on to present for those.
9
00:00:34,120 --> 00:00:37,259
There will be a few sneak peeks
and learnings with AI, as well
10
00:00:37,260 --> 00:00:39,899
as a couple of failures that
put me back on the right track.
11
00:00:40,265 --> 00:00:43,981
Then, in preparation for a benchmarking
webinar in a couple of weeks, I
12
00:00:43,982 --> 00:00:47,998
read the published academic paper
on RAG results, so I will give
13
00:00:47,999 --> 00:00:49,132
my perspective on that as well.
14
00:00:49,687 --> 00:00:54,373
Also, has anyone noticed that many
websites now don't let you copy/paste?
15
00:00:54,922 --> 00:00:59,794
I'm wondering if that's an agent or a
bot blocker, but it is super frustrating
16
00:00:59,795 --> 00:01:04,408
when I'm trying to copy over exact blog
titles or little snippets of code in
17
00:01:04,409 --> 00:01:06,727
order to include in various places.
18
00:01:07,145 --> 00:01:09,930
Anyway, without further delay,
let's get into all the goodies.
19
00:01:10,781 --> 00:01:16,306
First off, coming up is NODES 2026,
and therefore leading up to that
20
00:01:16,340 --> 00:01:18,398
are the Road to NODES workshops.
21
00:01:18,697 --> 00:01:23,992
Now, NODES 2026 is Neo4j's big
virtual event that happens every year.
22
00:01:24,227 --> 00:01:27,066
It is very developer
and technical centric.
23
00:01:27,277 --> 00:01:31,019
It is entirely global, entirely
free, entirely virtual.
24
00:01:31,202 --> 00:01:33,182
It's one of my favorite events every year.
25
00:01:33,772 --> 00:01:37,479
It's just so interesting and cool
to see all of the different things,
26
00:01:37,545 --> 00:01:44,008
whether customers or community members
and anyone else out there just doing
27
00:01:44,009 --> 00:01:47,889
some really amazing and cool things
with technology, specifically graphs.
28
00:01:48,121 --> 00:01:51,453
I learn a ton throughout the
entire event and also find it very
29
00:01:51,462 --> 00:01:53,104
inspiring and motivating as well.
30
00:01:53,329 --> 00:01:55,497
It's just a really fun
event to get involved with.
31
00:01:56,142 --> 00:02:00,344
So if you are not part of that, I
highly recommend you register for that.
32
00:02:00,546 --> 00:02:04,389
And then if you're interested,
there are some workshops that
33
00:02:04,436 --> 00:02:07,135
lead up to the main event as well.
34
00:02:07,355 --> 00:02:12,559
Each of the six weeks prior to
the event, there are two-hour-long
35
00:02:12,573 --> 00:02:15,618
workshops that are presented by experts.
36
00:02:15,816 --> 00:02:20,069
And this year, they're on stopping
agent hallucinations, building the
37
00:02:20,069 --> 00:02:24,429
knowledge layer, graph reasoning,
building AI that learns, and then
38
00:02:24,431 --> 00:02:26,281
context graphs and agent memory.
39
00:02:26,389 --> 00:02:30,914
So all really fabulous and relevant
topics for current things going on today.
40
00:02:31,215 --> 00:02:31,509
All right.
41
00:02:32,598 --> 00:02:35,205
On now to some of the next
things that I'm working on.
42
00:02:35,479 --> 00:02:39,166
I'm working on some new demo
applications for upcoming content.
43
00:02:39,180 --> 00:02:42,986
I have several events going on
where I'm presenting or creating
44
00:02:42,987 --> 00:02:46,258
and putting together new content,
and so I have some apps that I'm
45
00:02:46,264 --> 00:02:47,716
building in order to prep for those.
46
00:02:48,035 --> 00:02:50,754
The first is an agentic Java application.
47
00:02:51,504 --> 00:02:54,706
I do reading challenges through
my local library, and it's kind of
48
00:02:54,707 --> 00:02:58,672
a pain to get ahold of a book for
that particular month's challenge.
49
00:02:58,931 --> 00:03:03,551
So for instance, late at night, I'm on my
phone checking the suggestions that they
50
00:03:03,553 --> 00:03:08,072
have that fit that topic for the month,
whether it's, pick a certain genre or
51
00:03:08,072 --> 00:03:12,605
pick something with a blue cover on it
or whatever it might be for that month.
52
00:03:12,935 --> 00:03:15,690
Then I have to pick out from
the suggestions a few books that
53
00:03:15,690 --> 00:03:16,912
I think might be interesting.
54
00:03:17,596 --> 00:03:21,703
And then I start searching across
multiple platforms to see who has
55
00:03:21,706 --> 00:03:25,779
what copy available, whether that's
my local library has a physical copy
56
00:03:25,811 --> 00:03:31,215
or I have three or four different,
digital reading applications that
57
00:03:31,226 --> 00:03:36,802
have e-books or audiobooks that I
can borrow and listen to or read.
58
00:03:37,511 --> 00:03:40,223
And so I check all those
applications to see, hey, does
59
00:03:40,225 --> 00:03:41,398
this have what I'm looking for?
60
00:03:41,424 --> 00:03:42,416
Does this have this book?
61
00:03:42,419 --> 00:03:45,370
Oh, this has a hold on it, so
I can't get that for another
62
00:03:45,372 --> 00:03:47,009
four weeks, and this and that.
63
00:03:47,039 --> 00:03:49,691
And it just ends up
taking a really long time.
64
00:03:49,874 --> 00:03:52,734
I'm hopping back and forth between
lots of different apps, and I
65
00:03:52,734 --> 00:03:54,480
spend a lot of time doing that.
66
00:03:54,870 --> 00:03:58,286
So I wanted to build a better
workflow, for
Load full transcript
lack of a
67
00:03:58,286 --> 00:03:59,851
better term, for this process.
68
00:04:00,577 --> 00:04:03,189
I'm really excited about how this
application is coming together
69
00:04:03,210 --> 00:04:06,025
and hopefully what it will look
like in the next weeks and months.
70
00:04:06,895 --> 00:04:12,515
Now, as an aside, AI has surfaced
more gaps in the search to make
71
00:04:12,516 --> 00:04:17,646
things easier, but I also feel like
it has expanded what I feel like
72
00:04:17,646 --> 00:04:19,040
is possible in a short time block.
73
00:04:19,234 --> 00:04:24,408
I would have never taken on an application
this large to showcase for a demo with
74
00:04:24,775 --> 00:04:29,390
even just several weeks before an event or
even a couple of months before an event.
75
00:04:29,886 --> 00:04:33,648
Now, on the downside to that, one
easy thing that I found is that
76
00:04:33,751 --> 00:04:38,281
it's very easy to over-optimize
your solution where the scope or the
77
00:04:38,281 --> 00:04:40,101
complexity kind of gets away from you.
78
00:04:40,553 --> 00:04:44,653
As an example, I'm doing a keyword
search in the library application.
79
00:04:45,340 --> 00:04:48,678
And I sat there and debated, do
I want to do a, a vector search?
80
00:04:48,679 --> 00:04:51,337
Do I want to do something
really highly custom?
81
00:04:51,348 --> 00:04:54,929
If I over-index and cut out filler
words, what does that do when you
82
00:04:54,934 --> 00:04:58,666
have a title that has a lot of filler
words in a book title and so on.
83
00:04:58,928 --> 00:05:01,741
So again, you have to optimize and
pick and choose what things you
84
00:05:01,741 --> 00:05:04,671
really want to focus on, just like you
always have in traditional software.
85
00:05:05,049 --> 00:05:07,320
But you have to pick an
optimization approach.
86
00:05:07,523 --> 00:05:11,326
You still can't optimize for everything
without sacrificing something.
87
00:05:11,337 --> 00:05:15,888
In this case, I'm trying to optimize
for a good conference and live
88
00:05:15,889 --> 00:05:19,136
demonstration approach and perspective.
89
00:05:19,611 --> 00:05:23,640
Well, then that means I have to sacrifice
some of the complexity and fancy features
90
00:05:23,883 --> 00:05:27,328
in order to get those points across
and show someone how it's working.
91
00:05:28,322 --> 00:05:34,052
Another thing is that I had previously
said that AI is great at outlining, and I
92
00:05:34,054 --> 00:05:38,205
still think that's true, but there are a
few caveats I'm going to add to this now.
93
00:05:38,851 --> 00:05:43,825
First, I tested a few use cases of
AI generating presentation outlines.
94
00:05:44,102 --> 00:05:47,010
Now again, not the content, I still
did most of the content, but I would
95
00:05:47,020 --> 00:05:50,911
have it generate the outline for me
on my technical developer sessions.
96
00:05:51,110 --> 00:05:53,181
And I tried a few different
approaches to this.
97
00:05:53,916 --> 00:05:58,227
I tried a few different tactics,
and what resulted was good content.
98
00:05:58,895 --> 00:06:02,818
But honestly, it left me
feeling off or dissatisfied.
99
00:06:03,266 --> 00:06:07,895
Now, I always go back to, I love the
TV show NCIS, and so I always think of
100
00:06:07,906 --> 00:06:12,734
the NCIS style Gibbs' gut, where when
something feels off, your body tells you
101
00:06:12,734 --> 00:06:16,732
that, and the particular character in the
show is teaching others and colleagues
102
00:06:16,732 --> 00:06:18,393
how to pay attention to their gut.
103
00:06:18,445 --> 00:06:21,681
And that's kind of the vibe that
I was getting when I was working
104
00:06:21,697 --> 00:06:26,108
with the AI developing the
technical presentation outlines.
105
00:06:26,663 --> 00:06:31,538
Again, it was good content, but
it was just missing something.
106
00:06:32,269 --> 00:06:36,570
What I recently realized is that I
was handing the storytelling over to
107
00:06:36,571 --> 00:06:40,113
the large language model, which is why
the material didn't connect with me.
108
00:06:40,439 --> 00:06:41,613
It was good content.
109
00:06:41,631 --> 00:06:46,494
It was accurate, it was correct, it flowed
decently well, but it was just missing
110
00:06:46,495 --> 00:06:51,135
that relatable storytelling that I really
enjoy when I get up and present something.
111
00:06:51,826 --> 00:06:56,826
So I designed a completely new session
where I drafted the slide story and
112
00:06:56,867 --> 00:07:00,958
everything from scratch, and then I
went back to the session that I didn't
113
00:07:00,964 --> 00:07:06,006
feel like was aligning super well,
constructed the outline myself, and
114
00:07:06,562 --> 00:07:08,358
already just felt better about it.
115
00:07:09,067 --> 00:07:11,829
So this is something that I think
all of us should keep in mind.
116
00:07:12,092 --> 00:07:18,288
What elements are you comfortably able and
want to outsource to AI, and what feels
117
00:07:18,289 --> 00:07:21,006
valuable for you to keep and maintain?
118
00:07:21,769 --> 00:07:26,051
It goes back to during my school
studies, we talked a lot about IT
119
00:07:26,063 --> 00:07:28,283
companies outsourcing their talent.
120
00:07:28,478 --> 00:07:31,630
When you'd look at IT strategy
and things like that, it would
121
00:07:31,632 --> 00:07:32,822
talk a lot about outsourcing.
122
00:07:33,630 --> 00:07:36,841
And it goes back to all of the
textbooks said, what are your
123
00:07:36,847 --> 00:07:41,134
company's strengths that need to remain
in-house in order for you to stay
124
00:07:41,136 --> 00:07:42,745
competitive and valuable in the market?
125
00:07:43,156 --> 00:07:48,075
And then what are the projects
or tasks or particular weaknesses
126
00:07:48,317 --> 00:07:51,527
that you can delegate or outsource
to either other companies or
127
00:07:51,527 --> 00:07:53,241
other people or other resources?
128
00:07:54,508 --> 00:07:58,073
And I think AI falls into
that pattern as well.
129
00:07:58,786 -->
00:08:03,772
We need to look at for us to optimize
our workflow, what are the pieces that we
130
00:08:03,773 --> 00:08:07,661
can and should delegate to our AI tools?
131
00:08:07,903 --> 00:08:11,864
And then what are the pieces for our
personal workflows that we can and
132
00:08:11,874 --> 00:08:14,405
should keep in-house and do ourselves.
133
00:08:14,482 --> 00:08:18,750
And this, for me, was one of those
that I didn't feel like I could
134
00:08:18,760 --> 00:08:22,946
outsource and still feel good about
getting up and presenting that content.
135
00:08:23,726 --> 00:08:25,409
That line is gonna be
different for everyone.
136
00:08:25,619 --> 00:08:29,264
If you're someone who's optimizing,
not for the story, but for something
137
00:08:29,264 --> 00:08:32,556
else entirely, then you might be
okay delegating that story out.
138
00:08:32,599 --> 00:08:34,050
But it wasn't working for me.
139
00:08:34,918 --> 00:08:35,391
All right.
140
00:08:35,411 --> 00:08:38,462
So all of that to say, here are some
of the upcoming events where I'm
141
00:08:38,463 --> 00:08:39,739
going to be in the next few weeks.
142
00:08:40,190 --> 00:08:43,320
So starting next week, I will
be at All Things Open, which
143
00:08:43,321 --> 00:08:44,814
is in Raleigh, North Carolina.
144
00:08:45,025 --> 00:08:48,787
And then the following week, I will
be at JCon USA in Atlanta, Georgia.
145
00:08:49,138 --> 00:08:53,435
And then also I have a Florida Java
User Group tour, which I've mentioned
146
00:08:53,435 --> 00:08:54,873
on previous episodes of this podcast.
147
00:08:55,073 --> 00:08:57,881
That is all settled and will
be the first week of November.
148
00:08:58,117 --> 00:09:00,907
I'll be at a few different cities
throughout Florida, stopping at different
149
00:09:00,915 --> 00:09:04,118
meetup groups and talking some really
cool tech with some phenomenal people.
150
00:09:04,513 --> 00:09:08,811
And then the following week after
that is our NODES 2026 event, which
151
00:09:08,811 --> 00:09:10,289
I'm super excited to be a part of.
152
00:09:10,412 --> 00:09:13,817
I also have a few smaller and/or
virtual commitments that are
153
00:09:13,817 --> 00:09:16,964
sprinkled in and around those
things, but those are the big ones.
154
00:09:17,672 --> 00:09:21,279
Now, for the content highlight
this week, I read the NICD
155
00:09:21,290 --> 00:09:26,372
report, which is from the National
Innovation Center for Data, N-I-C-D.
156
00:09:26,418 --> 00:09:28,762
NICD is how we've shortened it, I think.
157
00:09:28,778 --> 00:09:32,485
And then this is a paper that is
talking about RAG benchmarking.
158
00:09:32,495 --> 00:09:33,653
It's an academic paper.
159
00:09:33,918 --> 00:09:37,199
It was funded by Neo4j, but all
the research and the paperwork and
160
00:09:37,200 --> 00:09:38,905
everything was done by a third party.
161
00:09:39,482 --> 00:09:42,090
Now, this paper walks through,
does a really nice job.
162
00:09:42,249 --> 00:09:46,186
It is an academic paper, so however you
feel about that is totally up to you.
163
00:09:46,469 --> 00:09:49,368
It was really interesting, and I
thought all of the results were really
164
00:09:49,369 --> 00:09:53,103
good, and everything walked through
very structurally and sensibly.
165
00:09:53,754 --> 00:09:57,626
Now, the first thing that the paper
goes through is talking about how
166
00:09:57,642 --> 00:10:01,928
false positives and false negatives
are an issue for large language models.
167
00:10:01,947 --> 00:10:04,117
It's hard for them to
discern those things.
168
00:10:04,622 --> 00:10:09,345
And then you have some questions
that require a multi-hop or
169
00:10:09,348 --> 00:10:12,129
multi-document or cross-entity answers.
170
00:10:12,339 --> 00:10:17,815
In other words, when you have inputs
or tasks or questions, those answers
171
00:10:18,236 --> 00:10:22,747
require more than just a single shot,
"Here's the bit of data that I need
172
00:10:22,748 --> 00:10:23,922
in order to answer this question."
173
00:10:24,160 --> 00:10:28,259
Sometimes it takes two or three
different hops across sources or
174
00:10:28,271 --> 00:10:29,973
connecting different types of entities.
175
00:10:29,974 --> 00:10:33,347
"Okay, I have this document, but who
are the people or the companies that are
176
00:10:33,349 --> 00:10:35,067
connected with that document?" And so on.
177
00:10:35,787 --> 00:10:39,924
And that also can cause some problems
for large language models because they're
178
00:10:39,924 --> 00:10:44,219
typically a very limited, somewhat narrow
set of information, especially when you're
179
00:10:44,220 --> 00:10:48,016
looking at vector RAG, you get chunks
of data that are relevant, semantically
180
00:10:48,017 --> 00:10:51,718
similar, and then those come back, and
you don't really have additional context
181
00:10:51,719 --> 00:10:53,615
that might also be relevant to that.
182
00:10:54,020 --> 00:10:57,175
And then some questions also
have some safe refusal answers.
183
00:10:57,187 --> 00:11:02,932
In other words, if the RAG system
does not have information to solve
184
00:11:02,949 --> 00:11:07,044
that question or that problem, then
the large language model should say,
185
00:11:07,053 --> 00:11:10,894
"Hey, I don't have that data. I cannot
answer that question." There should be
186
00:11:10,895 --> 00:11:14,309
an unknown state that doesn't surface
content from the knowledge source.
187
00:11:14,466 --> 00:11:16,394
There's all these different
types of questions.
188
00:11:17,076 --> 00:11:21,501
You have the single shot,
basic concise definitions.
189
00:11:21,745 --> 00:11:25,059
You have some of these multi-hop,
multi-document, cross-entity answers,
190
00:11:25,079 --> 00:11:28,019
and then you should have some answers
as well that are safe refusals, where
191
00:11:28,019 --> 00:11:30,914
the large language model says, "I
don't have that data. I can't respond."
192
00:11:31,634 --> 00:11:36,334
Now, as the paper talks, there are
four main
aspects of a RAG solution.
193
00:11:36,675 --> 00:11:40,296
First is question
understanding or decomposition.
194
00:11:40,776 --> 00:11:44,738
The second is information retrieval,
which is the focus of this paper.
195
00:11:45,187 --> 00:11:49,179
Then the third one is reasoning,
and the fourth one is response
196
00:11:49,246 --> 00:11:50,804
synthesis and grounding.
197
00:11:51,082 --> 00:11:55,482
Now again, the paper can't cover all four
of those in depth, so it chose to focus
198
00:11:55,482 --> 00:12:00,209
in on that information retrieval piece,
that piece of RAG that retrieves data to
199
00:12:00,219 --> 00:12:04,317
add to the large language model, and so
that's what this entire paper focuses on.
200
00:12:04,743 --> 00:12:07,105
There's some criteria that
go along with this as well.
201
00:12:07,955 --> 00:12:12,980
There's some end-to-end question answer,
which just basically means they're
202
00:12:12,982 --> 00:12:15,319
not focusing in just on the retrieval.
203
00:12:15,329 --> 00:12:18,826
How good were the results
that got retrieved to provide
204
00:12:18,826 --> 00:12:19,661
to the large language model?
205
00:12:19,994 --> 00:12:24,772
There's also an answer only eval,
so only evaluating the answer that
206
00:12:24,773 --> 00:12:26,182
the large language model provides.
207
00:12:26,400 --> 00:12:31,272
This particular criteria for this
paper is doing the end-to-end question
208
00:12:31,278 --> 00:12:36,720
answer, and how good, how high quality
is that end-to-end process, both for
209
00:12:36,721 --> 00:12:41,047
the information retrieval and for the
answer and response piece of that.
210
00:12:41,786 --> 00:12:46,346
Another criteria is using both
unstructured and structured data in this.
211
00:12:46,364 --> 00:12:48,259
Again, you could focus
on one or the other.
212
00:12:48,496 --> 00:12:51,674
This paper is choosing to look at
where you have sources for both.
213
00:12:51,991 --> 00:12:55,385
And then the last bit of criteria
for this paper and the study is
214
00:12:55,617 --> 00:12:57,766
multi-hop and multi-entity reasoning.
215
00:12:57,766 --> 00:13:02,393
They were really evaluating when you
go across multiple sources and pull
216
00:13:02,395 --> 00:13:08,221
together connected entities, i.e.
graph data in this case, then how good
217
00:13:08,229 --> 00:13:09,940
are those responses that come back?
218
00:13:09,940 --> 00:13:12,047
How high quality is your RAG solution?
219
00:13:12,625 --> 00:13:15,900
There are some common metrics too that
the paper talks a little bit about.
220
00:13:16,329 --> 00:13:21,114
There's answer relevancy, where you have
no golden answer, which generates the
221
00:13:21,125 --> 00:13:26,163
similarity to the original question by
generating three more similar queries,
222
00:13:26,245 --> 00:13:30,679
and then measuring the average cosine
similarity between those three new
223
00:13:30,679 --> 00:13:31,987
generated queries and the original.
224
00:13:32,845 --> 00:13:36,387
Then another metric is faithfulness
and factual correctness.
225
00:13:36,850 --> 00:13:40,252
Both of those break down claims
in the response to check for
226
00:13:40,262 --> 00:13:42,172
the support in the context.
227
00:13:42,900 --> 00:13:48,254
And then finally, all of these things
together, notice, require additional
228
00:13:48,254 --> 00:13:50,056
large language model calls to do this.
229
00:13:50,286 --> 00:13:53,449
For instance, the answer
relevancy requires you to generate
230
00:13:53,457 --> 00:13:55,035
three more similar queries.
231
00:13:55,044 --> 00:13:58,320
A large language model will do that
generation of the similar queries,
232
00:13:58,342 --> 00:14:01,979
and then check how similar those
new queries are to the original.
233
00:14:02,635 --> 00:14:06,776
And then the faithfulness and factual
correctness will break down individual
234
00:14:06,789 --> 00:14:12,371
claims within the response and check
to see does each of these claims have
235
00:14:12,373 --> 00:14:15,799
support in the context that was retrieved,
the information that was retrieved?
236
00:14:16,088 --> 00:14:20,109
So it has to break down the response,
and see how close the facts relate
237
00:14:20,123 --> 00:14:22,606
to those claims in the response.
238
00:14:23,280 --> 00:14:27,812
All of that requires multiple LLM calls
in order to break those different pieces
239
00:14:27,814 --> 00:14:29,851
down, and then to do those comparisons.
240
00:14:30,039 --> 00:14:35,650
What this study did is they took some
of those metrics, but they modified them
241
00:14:35,652 --> 00:14:41,067
so that you wouldn't have the expensive
measurement that traditionally goes on.
242
00:14:41,243 --> 00:14:44,059
And the study goes through exactly
what they did and how they did it.
243
00:14:44,268 --> 00:14:45,439
I thought it was really well done.
244
00:14:45,495 --> 00:14:50,186
I thought it was valuable information
to have, while also they talked about
245
00:14:50,187 --> 00:14:54,163
some of the caveats and things that
they weren't able to research or
246
00:14:54,164 --> 00:14:58,182
focus in on, of course, opening up
some more doors for further study.
247
00:14:59,195 --> 00:15:02,958
Temperatures might be cooling off for
fall, but daily life certainly isn't.
248
00:15:03,275 --> 00:15:06,824
This week, we covered everything
I'm tackling over the next few weeks
249
00:15:06,845 --> 00:15:08,246
and what I'm learning along the way.
250
00:15:08,596 --> 00:15:12,662
From improving my AI pairing process
to brushing up on benchmarking with
251
00:15:12,662 --> 00:15:15,996
an ap-academic paper, there's too
little to do and too much time.
252
00:15:16,455 --> 00:15:17,732
Wait, reverse that.
253
00:15:18,395 --> 00:15:20,909
As always, thanks for
listening, and happy coding.