Projects

Neural Online Calibration for XR Devices

This project will develop a novel online calibration framework that leverages deep learning to intelligently fuse data from multiple sensors (IMU, cameras, eye-trackers). Remove user involvement and enable a truly seamless XR online calibration experience.

Neural Simulation of Contact-Dominated Materials

From living tissues to sand and snow, materials made of discrete elements exhibit a rich space of macroscopic behaviors including jamming, segregation, and shear banding. These global effects are driven by frictional and adhesive contacts between individual particles which, due their nonlinear and non-smooth nature, make large-scale simulations extremely challenging. This project aims to develop neural simulation methods that learn complex contact mechanics from data to efficiently and robustly predict the collective dynamics of granular materials.

Efficient and Accurate Facial Expression Tracking on Glasses

We aim to reconstruct and track the wearer's facial expressions from headset-native sensors with the goal to capture both verbal and non-verbal expressions. The project comprises a comprehensive data collection and the implementation of baselines for facial reconstruction.

Scalable Contextualized Scene Representation for Robotics

We are building a framework for large-scale, semantically-aware Gaussian splats to be used in robotic planning. By leveraging scene graphs and shared Gaussian embeddings, our system allows for dynamic, task-driven Level-of-Detail filtering and efficient querying by Vision-Language Models. This approach will be deployed on an ANYmal legged robot in challenging outdoor environments, with a XR pilot interacting with the semantic scene.

Fast and Scalable Multi-View 3D Reconstruction with Semantic Shape Priors

The goal of the project is design methods for generative 3D reconstruction from a set of unposed input images. By leveraging semantic information extracted from the input, we focus completing the 3D geometry to reconstruct more than what is directly observable. This will allow for reconstructing complete scenes from significantly fewer images than what the state of the art needs.

Stochastic Interpolants for Controllable Scene Generation

Stochastic interpolants offer a powerful toolbox for generative modeling, including denoising diffusion, flow matching, and inductive moment matching. In the context of ETHAR, they hold great promise as a path towards controllable sampling and synthesis of multimodal content across the 2D and 3D domains. This potential is exemplified by scenarios where users aim to reimagine or enhance physical environments based on minimal imagery and high-level guidance. The proposed framework leverages stochastic interpolants to generate semantically consistent and geometrically plausible 3D reconstructions, including novel views that complete and enrich the scene beyond what is directly observable. We propose to develop a framework for interactive, semantically guided, and geometrically coherent content creation based on stochastic interpolants with variational regularization, in service of immersive user experiences and augmented reality.

Uncertainty in Online Conditional Motion Synthesis

In this project, we aim to study how uncertainty in motion synthesis varies over time when conditioned on sparse AR/VR tracking inputs. Unlike existing approaches that assume uniform uncertainty across frames [1,2], we hypothesize that some motion phases are more predictable than others; for example, a walking cycle is rarely interrupted mid-step. By modeling this temporally varying uncertainty, we aim to improve the efficiency and coherence of interactive virtual experiences.

Learning human interaction from videos

This project aims to develop self-supervised and unsupervised methods to learn actions directly from large-scale videos. Unlike current approaches that rely on heavy supervision or predefined latent structures, our goal is to discover an implicit latent action space with broader generalization. Such a representation can transfer across domains and support robotics, human motion and interactions, customizable video generation, and planning tasks via world models.