Seeing beyond BMI: Estimating cardiometabolic risk with smartphone imagery

Skip to main content

Google

Research

Search

Seeing beyond BMI: Estimating cardiometabolic risk with smartphone imagery

August 17, 2026<br>Cassie Zhou, Research Scientist, and Ahmed Metwally, Staff Research Scientist, Google Research

We demonstrate the feasibility of PhotoScan, a deep learning approach estimating body composition from smartphone photos, to predict insulin resistance with accuracy comparable to DXA scans in a clinical research setting.

Quick links

Paper

Share

Copy link

Insulin resistance is one of the most critical yet underdiagnosed drivers of modern metabolic disease. Predating the clinical onset of type 2 diabetes by years, impaired insulin sensitivity stealthily impairs vascular health, liver function, and energy metabolism long before fasting blood sugar rises into diagnostic ranges. Homeostasis Model Assessment for Insulin Resistance (HOMA-IR) models the feedback loop between liver glucose production and insulin secretion under steady-state fasting conditions, and a HOMA-IR score greater than 2.9 is considered insulin resistant based on epidemiological reviews. Recent studies demonstrate that multimodal machine learning frameworks integrating wearable sensor data with routine lab tests can accurately predict HOMA-IR to flag early metabolic risk. Integrating objective measures of body composition offers a vital complement to wearable technology; while wearables track daily physiological behaviors, body composition provides a distinct structural assessment of adiposity to form a complete picture of metabolic risk.<br>While knowing your total body fat percentage is a good baseline to measure adiposity versus lean mass, additional body composition biomarkers provide much deeper clinical insights. For instance, the Android-to-Gynoid fat ratio (A/G ratio) compares the fat stored in your trunk (an "apple" shape) versus your hips and thighs (a "pear" shape); the Visceral-to-Subcutaneous fat area ratio (V/S ratio) distinguishes between the highly metabolic internal fat surrounding your organs and the subcutaneous fat stored just beneath your skin. Elevated A/G ratios and higher visceral fat mass strongly correlate with insulin resistance prevalence. Currently, the gold standard for measuring true body composition is Dual-Energy X-Ray Absorptiometry (DXA) scans. These scans are incredibly precise, but aren't built for everyday screening because they are expensive, require specialized clinical infrastructure, and expose patients to low doses of radiation.<br>Building on the growing capability of smartphones to passively monitoring user health during daily use, such as continuous heart-rate monitoring, we introduce PhotoScan: an investigational deep learning framework that estimates three-dimensional body composition metrics including body fat percentage (BF%), A/G ratio and V/S ratio, directly from standard 2D smartphone photos. To build this, we pre-trained a deep neural network on over 35,000 participant records from the UK Biobank and fine-tuned it with a diverse new cohort of 677 adults. Validated across clinical cohorts, PhotoScan demonstrates higher body fat percentage accuracy than smartwatch-based bioelectrical impedance analysis (BIA) sensors while unlocking A/G and V/S ratios beyond BIA's capabilities, offering a scalable, non-invasive framework to predict insulin resistance with near-DXA accuracy.

Overview of the body composition pipeline and insulin resistance classification. Body composition metrics are first estimated from the pretrained PhotoScan model, and the body composition features, combined with user demographics are used for insulin resistance classification.

How PhotoScan works<br>PhotoScan bypasses the clinical measurements by extracting geometric body information directly from smartphone images. We built and evaluated this framework in three key phases:<br>Pre-training (UK Biobank, N = 35,323)[73abd3]: With a subset from the UK BioBank dataset, which contains both MRI images and body composition ground truth from DXA, we trained a ResNet-50 backbone (initialized with ImageNet weights) to predict body composition metrics from 2D frontal and lateral projection images generated from 3D MRI scans, with DXA scans serving as ground truth. The model fused image features with participant sex, height, weight, and internal BMI through a final dense layer to output probability density functions for target metrics.<br>Fine-tuning (PhotoBIA cohort, N = 677): With the PhotoBIA dataset[d7d94d], we fine-tuned the model using real-world smartphone photos paired with DXA ground truth with 5-fold cross-validation. To augment training data, an automated landmark detection pipeline selected optimal frontal and lateral pose frames directly from 360-degree participant videos.<br>Validation (MetabolicMosaic cohort, N = 132): Evaluated on an independent cohort[294b87], PhotoScan achieved strong DXA agreement across BF%, A/G...