00:00:00.640 --> 00:00:06.959
Today, I want to talk about a question that every communications leader has likely asked themselves by now.
00:00:07.360 --> 00:00:10.640
Can't I just monitor the media myself using AI?
00:00:11.519 --> 00:00:13.839
Maybe you're just breaching this idea.
00:00:14.000 --> 00:00:22.160
Maybe it feels a bit overwhelming, or maybe you've already tried it and you can't figure out why your LLM isn't giving you the results you want.
00:00:22.480 --> 00:00:25.039
Either way, we want to bring you the facts.
00:00:25.280 --> 00:00:36.799
Because our experts tested it with several clients, and we identified the five most common pitfalls of using AI for media intelligence so that your team doesn't have to learn the hard way.
00:01:05.200 --> 00:01:07.680
So the team asked it what happened.
00:01:07.920 --> 00:01:13.599
And the AI told them directly, I wrote it before my reasoning was complete.
00:01:14.480 --> 00:01:16.640
That is not a science fiction problem.
00:01:16.799 --> 00:01:18.079
It's an everyday one.
00:01:18.239 --> 00:01:23.599
And if you're using AI on your own coverage, there's a good chance you will never see it.
00:01:23.760 --> 00:01:26.560
You will just feel it in the results.
00:01:27.040 --> 00:01:30.159
Welcome to the Relay, a podcast from Public Relay.
00:01:30.319 --> 00:01:38.959
We're the communications team running intelligence for some of the world's biggest brands, which mostly means we read everything so that you don't have to guess.
00:01:39.280 --> 00:01:41.519
This is the Relay Special Edition.
00:01:41.680 --> 00:01:44.560
It's an extra episode between our monthly deep dives.
00:01:44.719 --> 00:01:50.319
The Public Relay Intelligence team walked us through real errors from real client work.
00:01:50.480 --> 00:01:56.640
I've removed the client names and details so that I wouldn't identify anyone, but the mistakes are real.
00:01:57.040 --> 00:01:58.640
Let me say this up front.
00:02:01.359 --> 00:02:02.959
It's part of our process.
00:02:03.280 --> 00:02:06.159
This isn't an episode about AI being bad.
00:02:06.319 --> 00:02:13.680
It's about what happens when AI works alone without any context and without an expert checking or guiding.
00:02:14.000 --> 00:02:18.719
Today, the five pitfalls of using AI for media intelligence.
00:02:19.039 --> 00:02:24.240
In that conversation, one phrase kept coming up: the lazy thinker.
00:02:24.479 --> 00:02:32.000
Someone who's fast and confident, who always has an answer, but always takes the shortest path to get to that answer.
00:02:32.240 --> 00:02:34.879
Not the deep thinker you thought you were hiring.
00:02:35.120 --> 00:02:41.439
Every pitfall I'm about to walk you through is a different way that that colleague shows up when you use AI.
00:02:42.159 --> 00:02:45.199
The most important idea might be the simplest one.
00:02:45.520 --> 00:02:47.680
Discernment creates direction.
00:02:48.080 --> 00:02:49.680
I'll come back to that.
00:02:50.560 --> 00:02:53.759
Pitfall one, it reads half the story.
00:02:54.080 --> 00:03:00.800
Take an article from a major business outlet covering an auto show with quotes from several consulting firms.
00:03:01.120 --> 00:03:07.599
The AI's analysis missed four of those quotes, plus several more from one particular firm.
00:03:07.919 --> 00:03:09.199
The explanation?
00:03:09.439 --> 00:03:10.800
It was almost boring.
00:03:11.039 --> 00:03:13.840
The AI never read the whole article.
00:03:14.319 --> 00:03:17.199
AI models can only take in so much at once.
00:03:17.439 --> 00:03:19.680
This limit is called a context window.
00:03:19.840 --> 00:03:23.520
So when the material runs past it, the model just cuts it off.
00:03:23.759 --> 00:03:27.280
It presented what it had read as if it was everything.
00:03:27.520 --> 00:03:32.000
So the team wrote a rule into the process: don't truncate.
00:03:32.319 --> 00:03:34.240
Here's why that matters to you.
00:03:34.479 --> 00:03:42.000
As my colleague Allison pointed out, someone using AI on their own, without experience, might never catch it.
00:03:42.879 --> 00:03:44.240
Pitball 2.
00:03:44.479 --> 00:03:46.719
It tells you what you want to hear.
00:03:47.039 --> 00:03:56.479
The team was running an analysis for a global consulting firm, and one of the questions was, in this article, which firm has the most authoritative voice?
00:03:56.719 --> 00:03:59.039
The AI's answer, the client.
00:03:59.360 --> 00:04:03.360
But the client had contributed a single indirect quote.
00:04:03.520 --> 00:04:06.000
A different firm had the right quotes.
00:04:06.159 --> 00:04:11.280
The AI knew who it was working for, and it tried to please that person.
00:04:11.759 --> 00:04:13.840
This happens with LLMs.
00:04:14.080 --> 00:04:16.160
It happened again in another article.
00:04:16.480 --> 00:04:24.399
The AI named a rival the client had asked us to track, which had one quote instead of an untracked firm that had four.
00:04:24.560 --> 00:04:28.160
Nobody had told it to look only among the tracked firms.
00:04:28.399 --> 00:04:30.240
I put it to the team this way.
00:04:30.560 --> 00:04:35.439
AI tends to match the user's sentiment the same way it does in personal conversations.
00:04:35.839 --> 00:04:39.439
In a lot of ways, it gives you what it thinks you want to hear.
00:04:39.600 --> 00:04:44.879
And if you're a communications leader, your own name is the one usually in question.
00:04:45.279 --> 00:04:46.639
Pitfall 3.
00:04:46.879 --> 00:04:48.800
It inflates what's there.
00:04:49.040 --> 00:04:55.600
Picture an article that mentions a consulting firm once, in a single phrase, as a source of some analysis.
00:04:55.839 --> 00:04:59.680
No data point, no named expert, no claim of any kind.
00:04:59.839 --> 00:05:07.279
The AI filled in the claim type, the attribution type, the firm's role, as if there were real substance assort.
00:05:07.839 --> 00:05:14.160
A brand name used as a credibility signal got treated like an analytic contribution.
00:05:14.399 --> 00:05:17.519
The team calls this substance inflation.
00:05:17.759 --> 00:05:25.120
Two sentences in an article treated like the biggest part of the story, and the conclusion you draw from it is wrong.
00:05:25.439 --> 00:05:26.879
Pitfall 4.
00:05:27.120 --> 00:05:30.639
It reads the words, but it doesn't get the meaning.
00:05:30.959 --> 00:05:33.839
A utility company's equipment was vandalized.
00:05:34.079 --> 00:05:38.800
The article was straight reporting, a theft, some vandalism, the facts.
00:05:39.120 --> 00:05:43.040
The utility company was the victim, and no customers were hurt.
00:05:43.199 --> 00:05:46.959
The AI tagged it as a negative for the utility company.
00:05:47.120 --> 00:05:51.199
It read as vandalism and theft, and it reacted quickly.
00:05:51.439 --> 00:05:53.920
We call this reactionary analysis.
00:05:54.079 --> 00:06:00.399
The AI hears a trigger word and it reacts instead of reading what the article actually says about the company.
00:06:00.639 --> 00:06:05.040
Then we landed on a better name, analysis by association.
00:06:05.279 --> 00:06:06.639
There's a second version.
00:06:06.800 --> 00:06:12.480
An article was strongly positive about the utility company co-founding a safety tech startup.
00:06:12.639 --> 00:06:23.199
That glow spilled onto the legal and regulatory tag, which came back positive, even though the regulator appeared only as neutral in a procedural step.
00:06:23.439 --> 00:06:28.160
That's why our team at Public Relay tongues every topic on its own.
00:06:28.399 --> 00:06:33.759
A company can be negative on safety, positive on sustainability, and neutral overall.
00:06:34.079 --> 00:06:37.600
The team's definition of lazy thinking covers it all.
00:06:37.839 --> 00:06:43.920
Generalizing, losing the specificity, inflating things, and sometimes hallucinating.
00:06:44.240 --> 00:06:45.439
Pitfall 5.
00:06:45.680 --> 00:06:48.079
The AI makes up its own rules.
00:06:48.399 --> 00:06:50.879
A bank client gave clear instructions.
00:06:51.120 --> 00:06:58.160
If a spokesperson's comments had no regional focus, not the US, not Canada, apply no region at all.
00:06:58.399 --> 00:07:00.560
The comments were about global supply.
00:07:00.720 --> 00:07:03.920
The AI tagged Canada and then explained itself.
00:07:04.240 --> 00:07:07.920
Because there was no regional focus, it must default to Canada.
00:07:08.160 --> 00:07:13.519
When we heard that, we said, mmm, sometimes AI makes up its own rules.
00:07:13.759 --> 00:07:16.399
The team's word for it was stronger.
00:07:16.639 --> 00:07:18.079
It hallucinated.
00:07:18.480 --> 00:07:21.759
In another case, it added a condition that no one had set.
00:07:22.000 --> 00:07:31.920
Commentary on North American inflation with real US content got tagged Canada because the AI decided that Canada was the more prominent one.
00:07:32.240 --> 00:07:34.639
And then there's the one we opened with.
00:07:34.879 --> 00:07:40.480
The AI asked for a yes or no on each tag with the explanation in the next column.
00:07:40.720 --> 00:07:45.759
The AI gave the correct explanation and the wrong yes or no repeatedly.
00:07:46.000 --> 00:07:49.839
Asked why, it said it wrote the answer before its reasoning was complete.
00:07:50.079 --> 00:07:52.480
The team called that incomplete work.
00:07:52.800 --> 00:07:55.519
My colleague's explanation was quite simple.
00:07:55.759 --> 00:07:58.160
AI failed to check its own work.
00:07:58.560 --> 00:08:00.319
So what do you do about it?
00:08:00.560 --> 00:08:04.800
Our answer isn't to stop using AI, it's to build controls.
00:08:05.040 --> 00:08:15.759
Public relay trains against lazy thinking, generalization, quick associations, and cutting down data just to fit some context window, or substance inflation.
00:08:15.920 --> 00:08:27.759
And behind every control is a person who understands what the client is actually trying to learn, tests everything down to the nuances, and validates constantly, specifically to the client.
00:08:28.000 --> 00:08:30.240
Is there a place to use AI on its own?
00:08:30.399 --> 00:08:31.680
We're fair about that.
00:08:31.839 --> 00:08:35.519
For something very high level or very directional, it may be enough.
00:08:35.759 --> 00:08:40.000
But accuracy changes from article to article and you just never know.
00:08:40.240 --> 00:08:41.440
That's the difference.
00:08:41.679 --> 00:08:48.559
We use AI too, but someone is checking what it does, someone understands the answers that you're looking for.
00:08:48.879 --> 00:08:52.399
This brings me back to the line we wrote down in the room.
00:08:52.720 --> 00:08:54.879
Discernment creates direction.
00:08:55.120 --> 00:09:00.480
Without discernment, you don't get clear direction on what the communications leader should do next.
00:09:00.720 --> 00:09:03.120
With it, you get the winning advantage.
00:09:03.360 --> 00:09:05.919
The team's image for this is hard to beat.
00:09:06.080 --> 00:09:11.120
Ask a general purpose AI where to go, and you get go to the North Pole.
00:09:11.279 --> 00:09:12.960
That's a direction, technically.
00:09:13.200 --> 00:09:14.879
It's just not one you can act on.
00:09:15.039 --> 00:09:19.120
What you actually need to know is whether you're going to Norway, Greenland, or Iceland.
00:09:19.360 --> 00:09:21.039
So here's something you can use.
00:09:21.279 --> 00:09:27.120
Whoever is doing your media intelligence, including us, five questions, one for each pitfall.
00:09:27.360 --> 00:09:30.080
One, did it actually read the whole article?
00:09:30.159 --> 00:09:31.440
And how do you know?
00:09:31.759 --> 00:09:36.639
Two, does the answer change depending on who you say you're working for?
00:09:37.039 --> 00:09:41.600
Three, can you show me a quote or data point behind every claim?
00:09:41.919 --> 00:09:46.000
Four, is the tone assigned per topic or per article?
00:09:46.159 --> 00:09:49.759
Or does a neutral story about a bad event come back as neutral?
00:09:50.240 --> 00:09:53.840
Five, when the rule says to do nothing, does it do nothing?
00:09:53.919 --> 00:09:56.159
Or does it make up its own rules?
00:09:56.559 --> 00:10:00.000
If the answers are vague, you found your lazy thinker.
00:10:00.879 --> 00:10:04.000
One more because it happened while we were making this episode.
00:10:04.159 --> 00:10:08.399
We ran the transcript of that conversation through an AI model to clean it up.
00:10:08.559 --> 00:10:14.000
It dropped parts of the conversation and some of the interjections that we hadn't asked it to touch.
00:10:14.159 --> 00:10:15.360
No one told it to.
00:10:15.519 --> 00:10:17.039
It just decided.
00:10:17.679 --> 00:10:19.360
That's the special edition.
00:10:19.519 --> 00:10:25.279
The deep dive is back in October with a fresh data set on a topic that's already got our PR teams nervous.
00:10:25.440 --> 00:10:31.840
If you want to see how these pitfalls get caught before they reach your dashboard, come talk to us at publicrelay.com.
00:10:32.000 --> 00:10:35.679
I'm NatLennox, and this has been the special edition from Public Relay.
00:10:35.840 --> 00:10:36.879
Thanks for listening.
00:10:37.120 --> 00:10:43.440
To receive our topical monthly sentiment briefs completely free, you can sign up for the relay at publicrelay.com.