Garbin, Stephan Joachim; (2024) Learned Reconstruction and Generation of Face Images Built with Computer Graphics Scaffolding. Doctoral thesis (Ph.D), UCL (University College London).

Preview

Text Garbin_10192916_Thesis.pdf Download (68MB) | Preview

Abstract

Generating a photorealistic, faithful, animated likeness of the human face is a challenging problem in computer vision. We explore different ways in which concepts from computer graphics can be combined with machine learning and optimisation approaches to reduce the amount of real image data required for this task. In contrast to other work idealistic about the availability of training data, labelling effort, or privacy implications, we focus on practical applications. This leads us to discover several surprising synergies between computer graphics, machine learning, and optimisation. We find that low fidelity renders of faces can be used to increase the diversity of samples from generative models, that raytracing hardware can be repurposed to accelerate Neural Radiance Fields in terms of empty-space skipping as well as animation, and that traditional morphable models of the face can be extended to the 3D volumetric case. Specific regions of the face are often challenging in isolation. We first build a dataset of the human eye, demonstrating issues of working with real data. This motivates our aim of using as little real image data as possible. Focusing on our main goal, we develop a zero-shot appearance transfer model capable of increasing the realism of low fidelity renders. This reduces the required enrolment data for creating a digital likeness to a single image at the expensive of training a generative model on a moderately-sized corpus of data. Reducing the need for training data even further, and to obtain 3D consistency at interactive framerates, we subsequently develop a real-time rendering approach inspired by optimisations in classical volume rendering such as empty space skipping and ray tracing. This approach is completed by a fast and generalisable deformation model built on the principles of finite elements. Using explicit tetrahedral geometry affords us fine-grained and interpretable control over deformation. Put together, these advances enable us to generate 3D consistent, animated scenes of human faces from limited data

Download activity - last month

Export as JSONXMLCSV

Download activity - last 12 months

Export as XMLJSONCSV

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item