Virtual try-on lets shoppers preview makeup before applying it. Now, Adeia is using AI to model how that makeup will look hours later under real-world conditions.
Adeia’s CTO, Serhad Doken, sees the company’s research taking beauty tech beyond the initial look. The goal is to move from “Does this look good on me?” to “How do I make this look work in real life?”
He described a system that can recommend products or touch-ups based on a user’s schedule and the cosmetics they already own. Adeia’s smart mirror technology also uses computer vision to mark application zones on a user’s image and guide them through complex makeup looks.
How It Works
According to Doken, the system estimates the time between makeup application and each scheduled event, then models wear based on product type, coverage, humidity, and temperature.
It also predicts how the makeup will appear under different lighting, including direct sunlight and fluorescent office lights. Weather forecasts, cloud cover, solar position, and time of day help it estimate outdoor lighting. Reference images or a live camera feed provide information about brightness, color temperature, and light direction. Location and humidity data can also inform product recommendations, such as oil-free, lightweight formulas for humid climates.
The system identifies products through optical character recognition or barcode scans, retrieving information about expected wear, including fading and color changes.
By analyzing images after application or tracking where a hand or applicator touches the face, it maps makeup placement. It then overlays predicted wear and lighting effects for upcoming events.
He also described an approach intended to capture appearance updates during routine device use:
“Rather than requiring users to check their makeup throughout the day deliberately, we have additional technology that captures face photos passively by piggybacking on routine Face ID unlock events, using the moment of successful authentication as the anchor frame for a live photo.”
Predicting Wear Around Daily Schedules
Doken said the system can prompt users through lock-screen notifications to capture a new image at set intervals. It can also detect lighting changes since earlier images were taken.
Timestamped images tagged with lighting conditions could help refine the wear model by comparing its predictions with observed fading, settling, and color changes.
The system predicts upcoming environments from calendar data, messages, and location history. It can infer likely activities during gaps in the calendar, such as a recurring gym visit, using historical behavior or reservation confirmations.
The software evaluates makeup options against expected lighting, giving greater weight to longer or more important events and those likely to involve photos or video. It can prioritize a look for the main event and recommend touch-ups for others.
Smart Mirrors Guide Makeup Application
Keeping application guides aligned with a user’s face requires precise 3D facial tracking. Computer vision models and edge-processing techniques reduce latency and maintain alignment as users turn their heads or press on their skin.
The team built a prototype using an M2 iPad Pro to handle the smart mirror interface and computer vision processing, Doken said.
A custom-trained You Only Look Once (YOLO) object detection model runs at roughly 30 frames per second, tracking the user’s face and a custom-designed applicator through the iPad’s cameras.
“The applicator detects pressure when it contacts the user’s face, determining how much and where makeup should be applied, and shares that data back to the iPad Pro acting as the virtual mirror. The applicator also delivers haptic feedback based on the location and type of makeup being applied,” he explained.
Adjusting for Ambient Lighting
Lighting conditions range from dimly lit bedrooms to bathrooms with harsh glare. The software adjusts for ambient light, shadows, and mirror reflections to help maintain accurate application guides and skin-tone rendering.
The software samples the live camera feed, then constructs a 3-by-3 color correction matrix in a device-independent color space. That mathematical adjustment maps the current lighting to a target profile.
Where controllable lighting is available, a camera or embedded sensors measure the ambient light. The system dims those fixtures to establish a baseline, then calculates how much additional light they need to provide.
An additional model uses samples from specific facial landmarks to estimate how the user’s skin and makeup reflect light.
“So that accounts for how skin and makeup interact with light rather than assuming neutral reflectance, which matters most in mixed or extreme lighting,” he said.
The technology primarily focuses on evaluating makeup and recommending adjustments. By analyzing shine, texture, and coverage during application, it can suggest blending, powdering, changing shades, or adjusting the amount of product.
“Additionally, the system can describe gesture-level correction during application itself: detecting a rubbing motion and warning the user, or substituting a patting motion to avoid smudging and using haptic pulses on a wrist bracelet or stylus to guide correct placement or pressure in real time,” he said.
Potential Uses Beyond Cosmetics
Doken sees potential applications in dermatology, telehealth, post-procedure recovery, fatigue monitoring, and wellness assessment. The approach would track features such as redness, texture, and swelling against a user’s personal baseline to identify changes over time.
“The system’s bracelet sensors, which can report skin hydration, UV exposure, and skin temperature, would be a natural complementary data source for this direction,” he offered.
