Toward Fully Onboard Vision-Language Models for Smart Glasses

This project aims to enable energy-efficient vision–language models (VLMs) to run entirely on smart glasses. Our goal is to provide reliable, real-time understanding of the user’s surroundings within the resource constraints of wearable devices. To achieve this, we will develop compact VLMs and deploy them on energy-efficient systems-on-chip (SoCs).

Learning Stable Nonlinear Reduced Dynamics for Real-Time Deformable Simulation

Real-time simulation of deformable objects is essential for XR, AR, and VR applications, yet existing methods remain too expensive for interactive use. We will develop a data-driven framework that learns both a nonlinear deformation manifold and its dynamics from simulation data. The resulting method will enable fast, stable, and physically faithful simulation of deformable objects in interactive XR environments.

Generative Upper-Body Avatar Motion from Minimal AI Glasses Sensing

This project aims to generate plausible upper-body avatar motion from the limited sensing available on lightweight AI glasses. We will develop a generative motion model that infers natural torso, shoulder, and arm behavior from head motion, speech, and occasional hand observations. The goal is to enable expressive avatar motions without additional body-worn cameras or motion-capture hardware.

EgoScene-FM: Persistent Object-State Memory for Context-Aware XR Assistants

Spatial State Memory and Continuous Consistency for Fast Generative 3D Scene Evolution

Speech- and Gesture-Driven Direction of Intelligent 3D Characters in Augmented Reality

This project explores how a user can create and direct intelligent 3D characters in augmented reality (AR) the way a film director directs actors: through natural speech and gestures rather than menus and controllers. We combine large language models with gesture and scene understanding so that characters interpret high-level instructions, reason about their physical surroundings, and act accordingly. The goal is an intuitive AR authoring workflow for interactive storytelling, education, and entertainment.

Neural Online Calibration for XR Devices

This project will develop a novel online calibration framework that leverages deep learning to intelligently fuse data from multiple sensors (IMU, cameras, eye-trackers). Remove user involvement and enable a truly seamless XR online calibration experience.

Neural Simulation of Contact-Dominated Materials

From living tissues to sand and snow, materials made of discrete elements exhibit a rich space of macroscopic behaviors including jamming, segregation, and shear banding. These global effects are driven by frictional and adhesive contacts between individual particles which, due their nonlinear and non-smooth nature, make large-scale simulations extremely challenging. This project aims to develop neural simulation methods that learn complex contact mechanics from data to efficiently and robustly predict the collective dynamics of granular materials.

Efficient and Accurate Facial Expression Tracking on Glasses

We aim to reconstruct and track the wearer's facial expressions from headset-native sensors with the goal to capture both verbal and non-verbal expressions. The project comprises a comprehensive data collection and the implementation of baselines for facial reconstruction.

Scalable Contextualized Scene Representation for Robotics

We are building a framework for large-scale, semantically-aware Gaussian splats to be used in robotic planning. By leveraging scene graphs and shared Gaussian embeddings, our system allows for dynamic, task-driven Level-of-Detail filtering and efficient querying by Vision-Language Models. This approach will be deployed on an ANYmal legged robot in challenging outdoor environments, with a XR pilot interacting with the semantic scene.

Fast and Scalable Multi-View 3D Reconstruction with Semantic Shape Priors

The goal of the project is design methods for generative 3D reconstruction from a set of unposed input images. By leveraging semantic information extracted from the input, we focus completing the 3D geometry to reconstruct more than what is directly observable. This will allow for reconstructing complete scenes from significantly fewer images than what the state of the art needs.

Stochastic Interpolants for Controllable Scene Generation

Stochastic interpolants offer a powerful toolbox for generative modeling, including denoising diffusion, flow matching, and inductive moment matching. In the context of ETHAR, they hold great promise as a path towards controllable sampling and synthesis of multimodal content across the 2D and 3D domains. This potential is exemplified by scenarios where users aim to reimagine or enhance physical environments based on minimal imagery and high-level guidance. The proposed framework leverages stochastic interpolants to generate semantically consistent and geometrically plausible 3D reconstructions, including novel views that complete and enrich the scene beyond what is directly observable. We propose to develop a framework for interactive, semantically guided, and geometrically coherent content creation based on stochastic interpolants with variational regularization, in service of immersive user experiences and augmented reality.

Uncertainty in Online Conditional Motion Synthesis

In this project, we aim to study how uncertainty in motion synthesis varies over time when conditioned on sparse AR/VR tracking inputs. Unlike existing approaches that assume uniform uncertainty across frames [1,2], we hypothesize that some motion phases are more predictable than others; for example, a walking cycle is rarely interrupted mid-step. By modeling this temporally varying uncertainty, we aim to improve the efficiency and coherence of interactive virtual experiences.

Learning Human Interaction from Videos

This project aims to develop self-supervised and unsupervised methods to learn actions directly from large-scale videos. Unlike current approaches that rely on heavy supervision or predefined latent structures, our goal is to discover an implicit latent action space with broader generalization. Such a representation can transfer across domains and support robotics, human motion and interactions, customizable video generation, and planning tasks via world models.