DEGAS: Detailed Expressions on Full-Body Gaussian Avatars

arXiv Paper
Zhijing Shao1,2, Duotun Wang1, Qing-Yao Tian2, Yao-Dong Yang1, Hengyu Meng1,
Zeyu Cai1, Bo Dong4, Yu Zhang2, Kang Zhang1,3, Zeyu Wang1,3*

1The Hong Kong University of Science and Technology (Guangzhou)
2Prometheus Vision Technology Co., Ltd.
3The Hong Kong University of Science and Technology
4Swinburne University of Technology
* Corresponding author @ Creative Intelligence and Synergy Lab     

Paper

Code (Coming)

Dataset (Coming)

Abstract

We present DEGAS (Detailed Expressions on Full-Body Gaussian Avatars), the first 3D Gaussian Splatting (3DGS)-based modeling of
full-body avatars with rich facial expressions.


Although neural rendering has made significant advancements in creating lifelike, animatable full-body and head avatars, incorporating detailed expressions into full-body avatars remains largely unexplored. We present DEGAS, the first 3D Gaussian Splatting (3DGS)-based modeling method for full-body avatars with rich facial expressions. Trained on multiview videos of a given subject, our method learns a conditional variational autoencoder that takes both the body motion and facial expression as driving signals to generate Gaussian maps in the UV layout. To drive the facial expressions, instead of the commonly used 3D Morphable Models (3DMMs) in 3D head avatars, we propose to adopt the expression latent space trained solely on 2D portrait images, bridging the gap between 2D talking faces and 3D avatars. Leveraging the rendering capability of 3DGS and the rich expressiveness of the expression latent space, the learned avatars can be reenacted to reproduce photorealistic rendering images with subtle and accurate facial expressions. Experiments on an existing dataset and our newly proposed dataset of full-body talking avatars demonstrate the efficacy of our method. We also propose an audio-driven extension of our method with the help of 2D talking faces, opening new possibilities to interactive AI agents.



Watch Video



DREAMS Avatar Dataset

We propose DREAMS Avatar Dataset, including multi-view captures of 6 subjects. Each subject performs two sequences. The first sequence is of standard body motions and required facial expressions including smile, laugh, angry, surprise, etc,. The second sequence is a freestyle.

Sequence 1 - render by DEGAS
Sequence 2 - render by DEGAS
Person 1 Sequence 2 Person 2 Sequence 2 Person 3 Sequence 2
Person 4 Sequence 2 Person 5 Sequence 2 Person 6 Sequence 2


Reenactment

All Reenacted by Person 1's Sequence 2

 

Audio-driven. Expressions generated by SadTalker and extracted by DPE

 



Citation


  @misc{shao2024degas,
    title={DEGAS: Detailed Expressions on Full-Body Gaussian Avatars}, 
    author={Zhijing Shao and Duotun Wang and Qing-Yao Tian and Yao-Dong Yang and Hengyu Meng and Zeyu Cai and Bo Dong and Yu Zhang and Kang Zhang and Zeyu Wang},
    year={2024},
    eprint={2408.10588},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2408.10588}, 
  }