What 99% Accurate Really Means on a Body Composition Scale

WEBVTT

00:00.000 --> 00:06.160
Hey everyone, it's Mikki here. You're listening to Mini-Mikkipedia on a Monday.

00:06.520 --> 00:13.960
Before I go on, if you like these episodes, or just Mikkipedia in general, can you subscribe

00:13.960 --> 00:19.040
on your favourite podcast listening platform, share it with a friend, because that then

00:19.040 --> 00:24.200
increases the visibility of Mikkipedia in amongst literally thousands of other podcasts out there.

00:24.520 --> 00:29.560
And if you feel so inclined, give me a five-star review. That would be amazing.

00:30.000 --> 00:35.200
All right, what are we talking about this week on Mini-Mikkipedia? I got sent an email from one

00:35.200 --> 00:41.500
of my clients, and she had a link to a very flash Withings scale. And I'm like, oh, this is

00:41.500 --> 00:49.180
interesting. And oh my goodness, it's 819 New Zealand dollars. JB high five. And it is a,

00:49.660 --> 00:57.220
essentially, it's a bioelectrical impedance scale, but apparently it's like 99% accurate.

00:57.220 --> 00:59.740
And I'm like, really? This sounds super interesting.

00:59.740 --> 01:04.140
And basically, my client was like, Mikki, is this worth the money? Because she's in the market

01:04.140 --> 01:09.980
for something that's actually worth its salt. And she says, I know that my usual scales just track

01:09.980 --> 01:15.540
kilograms, but I get a DEXA every year, but I'm still curious as to whether anything on the market

01:15.540 --> 01:22.680
right now is actually going to give me more information than that. And so that 99% as

01:22.680 --> 01:29.500
accurate as a DEXA as marketed was like, oh, that's super interesting. And with that price tag, I'm like,

01:29.740 --> 01:36.080
really? So I understand why that is appealing. Clearly, if you've ever paid for a DEXA scan,

01:36.520 --> 01:40.900
you know, it's not cheap. Well, you know, a few hundred bucks, at least you have to book it,

01:41.140 --> 01:46.480
you have to drive there, and you get one number on one day, whereas a scale in your bathroom,

01:47.100 --> 01:51.920
if that gave you that same information every morning for the price of like three scans or

01:51.920 --> 01:58.000
whatever, pretty appealing. So I went to read the research that the company links to. So I want to

01:58.000 --> 01:59.720
walk through what I found. And I'm like, oh, this is really, really, really, really, really, really,

01:59.740 --> 01:59.940
really, really, really, really, really, really, really, really, really, really, really, really,

01:59.940 --> 02:06.000
because the problem here isn't really about just one scale. It's about a statistical issue that

02:06.000 --> 02:11.000
shows up constantly in health tech marketing. And it's worth you knowing if you're at all

02:11.000 --> 02:15.660
interested in this. And for what it's worth, people always ask me about these BIA machines.

02:15.660 --> 02:24.440
So the claim that the Withings scales was 99% as accurate as a DEXA rests on this validation

02:24.440 --> 02:29.640
document that the company had produced. There were 80 people, each of them had one DEXA

02:29.640 --> 02:35.660
scan and three measurements on the scale. And the headline result was a correlation of 0.99 for

02:35.660 --> 02:42.180
total fat mass. Now 0.99 clearly is a very high correlation. And that part is clearly legit.

02:42.600 --> 02:48.760
But what isn't true is that correlation of 0.99 means that the scale is 99% accurate.

02:49.420 --> 02:54.780
A correlation tells you whether two things move together in the same direction. It tells you that

02:54.780 --> 02:59.200
people who have more fat mass on DEXA also have more fat mass on the scale.

02:59.640 --> 03:06.120
That's it. That is all that it tells you. So imagine three people. DEXA says they have 10,

03:06.120 --> 03:15.700
20, and 30 kilos of fat. The scale says 5, 10, and 15 kilos. Every single reading is wrong by one

03:15.700 --> 03:21.580
half. The scale has failed completely. But the correlation between these two sets of numbers

03:21.580 --> 03:29.380
is a perfect 1.0. It's exactly the same, yet doesn't really give you great data. And what you

03:29.640 --> 03:35.340
want when you're comparing two measurement methods is agreement, not correlation. You want

03:35.340 --> 03:40.400
to know the mean difference between the two. And you want the limits of agreement, which tell you

03:40.400 --> 03:45.480
how far off an individual person's reading is likely to be. And that's something called a

03:45.480 --> 03:52.560
bland Altman analysis. And it's been the standard way to do this since the mid-1980s. It actually

03:52.560 --> 03:57.360
isn't obscure. I mean, I'm just going to say right up front, I know next to nothing about this. I

03:57.360 --> 03:59.620
believe I used a bland Altman in one of my papers. I don't know if you've seen it. I don't know if

03:59.640 --> 04:04.120
it's actually from my thesis, but I cannot recall. But it's certainly not a niche statistical

04:04.120 --> 04:10.000
preference. It's the thing that you do. And they didn't do it. So I think this is always something

04:10.000 --> 04:15.040
interesting when you look at research and you know, or you learn that there is a particular

04:15.040 --> 04:21.780
statistical analysis that really suits a question. And it's completely ignored by the people doing

04:21.780 --> 04:27.800
the analysis. So that's worth noting. What I will say, however, I don't think the technology is

04:27.800 --> 04:29.600
rubbish at all, right? With that withings.

04:29.640 --> 04:30.100
So I think that's a good thing. And I think that's a good thing. And I think that's a good thing.

04:30.100 --> 04:30.260
So I think that's a good thing. And I think that's a good thing. And I think that's a good thing.

04:30.260 --> 04:36.140
This particular one uses eight electrodes, including a handheld bar, and it does run multiple

04:36.140 --> 04:41.300
frequencies. So that means the current passes through your arms, your trunk, and your legs,

04:41.580 --> 04:46.880
rather than just up one leg and down the other, which is what a cheap bathroom scale that I got

04:46.880 --> 04:52.380
at home does. So it is actually better hardware. But the evidence is a separate question from the

04:52.380 --> 04:57.920
hardware. So what I couldn't find anywhere in that material was the mean difference from the

04:57.920 --> 04:59.620
DEXA. The limits of a green screen are not as good as the green screen. So I think that's a good thing.

04:59.620 --> 05:05.100
That individual prediction error, the age and sex breakdown of the participants,

05:05.800 --> 05:11.540
the body fat range, whether hydration and fasting were standardized, as I've talked about with

05:11.540 --> 05:17.600
Professor Grant Tinsley on this podcast, who is a body composition expert, whether the validation

05:17.600 --> 05:22.400
sample was independent of the data used to build the prediction algorithm in the first place.

05:22.540 --> 05:27.080
And that does actually matter. And I couldn't actually find any peer reviewed publication

05:27.080 --> 05:29.440
of that study. Something else that's worth

05:29.440 --> 05:34.040
sort of chatting about is that the results are presented as though all four measures were

05:34.040 --> 05:40.000
validated against DEXA, but total body water, which is something else you can glean from

05:40.000 --> 05:45.900
these scales, was compared against an entirely different device. And I mean, of course, that

05:45.900 --> 05:52.200
makes sense because DEXA doesn't measure total body water. But it's worth noting that because

05:52.200 --> 05:56.240
it does tell you something about how carefully that document was put together. Basically,

05:56.240 --> 05:59.420
it's all marketing speak, and they're going to do as much as they can,

05:59.440 --> 06:06.800
to convince you of the accuracy and the utility of that scale. And then there was a line in it that

06:06.800 --> 06:12.500
says in the marketing that scale does not merely provide an estimate. I mean, it absolutely provides

06:12.500 --> 06:17.100
an estimate. That's all it's providing. This is what that technology does. The device measures

06:17.100 --> 06:23.320
your weight, and it measures electrical impedance. Everything else, your fat mass, your muscle mass,

06:23.320 --> 06:28.940
your visceral fat, your metabolic rate is calculated from those two inputs,

06:29.440 --> 06:36.360
plus your age, your sex, your height. It is run through this propriety algorithm. And the

06:36.360 --> 06:42.940
company's own technical explanation of the method tells you all of this, so that the entire thing

06:42.940 --> 06:48.620
is an absolute estimate. And then, you know, it is worth noting that there is good independent

06:48.620 --> 06:55.800
research on how accurate bioelectrical impedance actually is. And in January this year, there was

06:55.800 --> 06:59.420
a systematic review published in the Journal of Functional Morphology, which is a journal that

06:59.440 --> 07:07.020
looked at bio-impedance devices compared against a four-compartment model, which is stronger

07:07.020 --> 07:12.660
reference than a DEXA alone, because it accounts for body volume, total body water, bone mineral,

07:12.940 --> 07:19.400
and body mass separately. There were 12 studies, and across those studies, the mean bias for body

07:19.400 --> 07:26.000
fat percentage ranged from about three and a half percentage points, too low, to four and a half

07:26.000 --> 07:29.420
percentage points, too high. But the number I want you to look at is the number of studies that

07:29.420 --> 07:36.000
really think about is the limits of agreement, which typically spanned 15 to 20 percentage points.

07:36.180 --> 07:42.540
That's 15 to 20 percent. Wow. And the authors concluded that the BIA estimates of body fat

07:42.540 --> 07:47.780
percentage and fat-free mass were not equivalent to the reference model. And I mean, that does

07:47.780 --> 07:54.940
sound damning, and at the individual level it is, but there is a second finding that does make me

07:54.940 --> 07:59.100
think, you know, that may change how you should use these devices.

07:59.420 --> 08:07.560
So in a 2023 study in the British Journal of Nutrition, Seidler and colleagues put 73 adults

08:07.560 --> 08:14.820
through 15 different BIA devices and compared them all against a four-compartment model. 37 of those

08:14.820 --> 08:19.240
people came back 12 to 16 weeks later, so the researchers could look at whether the devices

08:19.240 --> 08:25.180
detected change accurately. The reliability was excellent. Precision error across the devices

08:25.180 --> 08:29.220
ranged from zero to about half a percentage point, which means that these

08:29.420 --> 08:36.620
devices are very consistent with themselves. So that is a key distinction. Reliable does not

08:36.620 --> 08:42.420
mean accurate. A device can be reliably wrong. If your scale reads three percentage points

08:42.420 --> 08:48.280
high every single time, your absolute number is wrong, but the direction and the size of the change

08:48.280 --> 08:53.580
might still be informative. The longitudinal results in that study I just mentioned were

08:53.580 --> 08:58.640
more mixed though. Constant error for detecting change ranged from about half a percentage point

08:59.420 --> 09:04.880
to just over one percentage point too high. So really it's telling you that some devices

09:04.880 --> 09:10.620
tracked change quite reasonably, but others really didn't. And in June of this year,

09:10.620 --> 09:15.580
there was a study published in the Journal of Primary Care and Community Health

09:15.580 --> 09:21.960
looking at the InBody H20N, which is a consumer-grade portable device

09:21.960 --> 09:28.940
against DEXA in 55 adults with obesity. Their average age was 62 and two-thirds of them were

09:28.940 --> 09:29.300
women.

09:29.420 --> 09:34.800
Participants fasted for eight hours, stayed hydrated, and avoided exercise and alcohol for

09:34.800 --> 09:39.660
24 hours beforehand, which is, you know, sort of gold standard practice, standardizing that stuff.

09:39.940 --> 09:45.580
For body fat percentage, the agreement with DEXA was excellent by the usual statistical standards.

09:46.120 --> 09:51.720
And they reported actual numbers. Mean difference of 1.64 percentage points,

09:51.720 --> 09:59.180
limits of agreement, minus 3.4 to plus 6.7 percentage points. So on average across

09:59.420 --> 10:05.360
the group, it's quite good. For an individual standing on it, the true value could plausibly be

10:05.360 --> 10:10.520
three and a half points below or nearly seven points above what the screen actually says.

10:10.880 --> 10:16.540
And for skeletal muscle mass, it was worse actually. The device overestimated by an average

10:16.540 --> 10:23.200
of 5.66 kilos. That's pretty huge compared to the DEXA with the limits of agreement spanning about

10:23.200 --> 10:29.340
10 kilos. So I mean, that's what proper reporting looks like. It's way less impressive than the

10:29.340 --> 10:29.400
90-day-old. But it's a good thing. It's a good thing. It's a good thing. It's a good thing. It's a good thing.

10:29.420 --> 10:36.240
But it's infinitely more useful because now you know how much you can actually trust that number.

10:36.900 --> 10:42.080
So two other things about that study. It was the first to compare that particular device,

10:42.200 --> 10:50.080
the InBody H2O-N against DEXA. And the authors did say that that longitudinal agreement,

10:50.360 --> 10:55.460
so whether it actually tracks change accurately over time, that hasn't been established.

10:55.720 --> 10:58.680
So that is a critical next research question.

10:58.680 --> 11:04.880
They do actually sell that machine, the H2O-N. As I said, it's a direct-to-consumer

11:04.880 --> 11:14.780
device. But InBody actually have a range of different devices, like the Dial H20,

11:15.300 --> 11:21.740
and then it's an H20B, and an H30, and an H40 in the same family. So these different models

11:21.740 --> 11:28.660
haven't been tested the way that the H, I called it the H20N, it's probably the H20B,

11:28.660 --> 11:35.800
has. So just note that, I think. And then if you look at how much it costs,

11:36.060 --> 11:40.640
like I started off this episode by telling you that the Wything scale was about 819 New Zealand

11:40.640 --> 11:47.820
dollars. The H20N InBody is 529 Australian dollars, but I couldn't find a New Zealand

11:47.820 --> 11:53.640
retailer actually stocking it. So it's cheaper than $800, but it's still not cheap.

11:53.640 --> 11:58.500
So here's something I just want you to bear in mind, regardless of what scale you,

11:58.660 --> 12:04.700
have. These are the metrics that generate pretty much a lot of attention, but are not worth paying

12:04.700 --> 12:11.120
attention to. The first one is visceral fat. On most of these devices, this is a propriety index

12:11.120 --> 12:15.880
on an arbitrary scale, meaning it's basically made up by the manufacturers, calculated from

12:15.880 --> 12:21.400
impedance, plus your age, sex, height, and weight. It is not a measurement of visceral adipose tissue

12:21.400 --> 12:26.840
in the way that a CT or MRI would give you. So don't treat a number going from nine to eight

12:26.840 --> 12:28.640
like it's something worth celebrating.

12:28.660 --> 12:34.180
Because really, it doesn't tell you anything. It's almost made up. Bone mass. This is not bone

12:34.180 --> 12:39.380
mineral density. Bioimpedance cannot tell you anything meaningful about your skeletal health

12:39.380 --> 12:46.260
or your osteoporosis risk. So if this is a concern of yours, then a DEXA scan is absolutely your best

12:46.260 --> 12:51.460
bet, because that is gold standard. Basal metabolic rate. This is calculated from an equation. It is

12:51.460 --> 12:56.980
not measured. Measuring it requires indirect colometry, which involves a mask and a metabolic

12:56.980 --> 12:58.480
card. And, you know, I don't want to go into too much detail about this, but I do want to

12:58.660 --> 12:59.200
know what you think about this. And I'm going to go into a little bit more detail about this.

12:59.200 --> 13:04.080
It basically will be derived, again, from an algorithm. And depending on the population that

13:04.080 --> 13:11.060
they derived it from, it's as accurate or otherwise as you would expect. So I just would

13:11.060 --> 13:17.120
not hang my hat on it. And body water. This is better informed on that multi-frequency device,

13:17.240 --> 13:21.780
like those eight different levers, than a cheap one. But it's still derived from an electrical

13:21.780 --> 13:27.780
model of your body fluids. So do I think you should throw your scale out? I mean, clearly not.

13:27.780 --> 13:28.640
I think you should throw it out. I think you should throw it out. I think you should throw it out.

13:28.660 --> 13:33.300
You should use it for what it's good at. So here's what I'd do. You want to standardize

13:33.300 --> 13:38.220
everything you can control. Same time of day, first thing in the morning, after you've been

13:38.220 --> 13:44.460
to the toilet, before food, fluid, or training. Hydration, glycogen, and even where you are in

13:44.460 --> 13:49.500
your cycle will move these numbers around. And they'll move them around more than a real change

13:49.500 --> 13:54.560
in body composition will over a short window. So there's a lot of noise. So you want to measure

13:54.560 --> 13:58.640
often. Three or four times a week is fine. Then average it across a week.

13:58.660 --> 14:06.300
Look at four-week trends, not daily readings. Your body fat estimate drops from 27.2 to 26.6.

14:06.600 --> 14:11.840
That's probably noise. But if it drifts consistently downward across two months of weekly averages,

14:12.080 --> 14:18.160
that's a signal. So that's a good thing. And then you want to triangulate. Don't just look at what

14:18.160 --> 14:22.420
your in-body tells you. Look at body weight, waist circumference, how your clothes fit,

14:22.820 --> 14:27.200
what's happening with your training numbers, photos if you find them useful. One measurement

14:27.200 --> 14:32.220
is a data point, but several measurements pointing in the same way. That's information.

14:32.820 --> 14:39.220
And then change that sentence you say to yourself of, you know, my body fat is 23.7%.

14:39.220 --> 14:46.220
Instead, think, my body fat estimate has trended down over the last 10 weeks. So that is much more

14:46.220 --> 14:51.880
accurate than that first sentence that I described. What I do want to be clear on is that, I mean,

14:51.880 --> 14:56.920
the Wything scale is as good as the in-body H20N. You know, like,

14:56.920 --> 15:01.900
I went in and read the in-body's own marketing after I read their study. And on their product

15:01.900 --> 15:07.300
page, they claim their technology has shown over 98% agreement with DEXA, which is pretty much

15:07.300 --> 15:15.060
exactly what the Wything scale said. So the entire category market correlation is accuracy because

15:15.060 --> 15:20.260
correlation coefficients are, they're pretty flattering, aren't they? But agreement statistics

15:20.260 --> 15:25.600
are not. And those are the ones that we want to be looking at. The only reason we know anything

15:25.600 --> 15:26.740
about that.

15:26.920 --> 15:30.400
In-body scale is that a research group in Alabama went and did the work themselves,

15:30.800 --> 15:35.260
published the limits of agreement and told us plainly that the device overestimates muscle

15:35.260 --> 15:39.540
by five and a half kilos. Obviously, a manufacturer is never going to put that on the box, right?

15:40.040 --> 15:45.980
And I guess, really, when it comes down to it, the point of this episode might not even be about

15:45.980 --> 15:51.860
those scales at all, right? When someone tells you a number is 99% accurate, ask what they mean.

15:52.320 --> 15:56.800
Ask what it was compared against, who paid for the comparison, and how far off an

15:56.800 --> 16:02.440
individual reading can be. Because those three questions will get you through most of this

16:02.440 --> 16:07.800
health tech that you'll be sold on in the coming years. And isn't there a lot of health tech these

16:07.800 --> 16:14.960
days? So I will say that, you know, a well-built consumer bioimpedance device can definitely be

16:14.960 --> 16:20.400
useful for tracking direction over months, but it's certainly not giving you 99% accuracy. But

16:20.400 --> 16:26.600
as long as you're aware about that, you triangulate that data, then I think you're good. If that is,

16:26.800 --> 16:32.260
you actually want to pay that $800 that these are selling for. Anyway, I'd love to hear whether

16:32.260 --> 16:37.200
you've got one of these machines, and you find it useful, or if you found this useful,

16:37.660 --> 16:42.340
hit me up. I'm over on Instagram, Threads and X, at Mikki Williden. I'm at Facebook,

16:42.340 --> 16:46.340
at Mikki Williden Nutrition, or head to my website, mikkiwilliden.com,

16:46.720 --> 16:52.720
and scroll right down to the bottom, pop your name in the box to jump on my email list.

16:52.960 --> 16:54.880
All right, guys, you have the best week. See you later.

16:56.800 --> 16:59.320
Bye-bye.