eprintid: 10192916 rev_number: 12 eprint_status: archive userid: 699 dir: disk0/10/19/29/16 datestamp: 2024-07-05 10:22:12 lastmod: 2024-07-05 10:22:12 status_changed: 2024-07-05 10:22:12 type: thesis metadata_visibility: show sword_depositor: 699 creators_name: Garbin, Stephan Joachim title: Learned Reconstruction and Generation of Face Images Built with Computer Graphics Scaffolding ispublished: unpub divisions: UCL divisions: B04 divisions: C05 divisions: F48 note: Copyright © The Author 2024. Original content in this thesis is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) Licence (https://creativecommons.org/licenses/by-nc/4.0/). Any third-party copyright material present remains the property of its respective owner(s) and is licensed under its existing terms. Access may initially be restricted at the author’s request. abstract: Generating a photorealistic, faithful, animated likeness of the human face is a challenging problem in computer vision. We explore different ways in which concepts from computer graphics can be combined with machine learning and optimisation approaches to reduce the amount of real image data required for this task. In contrast to other work idealistic about the availability of training data, labelling effort, or privacy implications, we focus on practical applications. This leads us to discover several surprising synergies between computer graphics, machine learning, and optimisation. We find that low fidelity renders of faces can be used to increase the diversity of samples from generative models, that raytracing hardware can be repurposed to accelerate Neural Radiance Fields in terms of empty-space skipping as well as animation, and that traditional morphable models of the face can be extended to the 3D volumetric case. Specific regions of the face are often challenging in isolation. We first build a dataset of the human eye, demonstrating issues of working with real data. This motivates our aim of using as little real image data as possible. Focusing on our main goal, we develop a zero-shot appearance transfer model capable of increasing the realism of low fidelity renders. This reduces the required enrolment data for creating a digital likeness to a single image at the expensive of training a generative model on a moderately-sized corpus of data. Reducing the need for training data even further, and to obtain 3D consistency at interactive framerates, we subsequently develop a real-time rendering approach inspired by optimisations in classical volume rendering such as empty space skipping and ray tracing. This approach is completed by a fast and generalisable deformation model built on the principles of finite elements. Using explicit tetrahedral geometry affords us fine-grained and interpretable control over deformation. Put together, these advances enable us to generate 3D consistent, animated scenes of human faces from limited data date: 2024-05-28 date_type: published oa_status: green full_text_type: other thesis_class: doctoral_open thesis_award: Ph.D language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 2279578 lyricists_name: Garbin, Stephan lyricists_id: SJGAR17 actors_name: Garbin, Stephan actors_name: Kalinowski, Damian actors_id: SJGAR17 actors_id: DKALI47 actors_role: owner actors_role: impersonator full_text_status: public pages: 192 institution: UCL (University College London) department: Computer Science thesis_type: Doctoral citation: Garbin, Stephan Joachim; (2024) Learned Reconstruction and Generation of Face Images Built with Computer Graphics Scaffolding. Doctoral thesis (Ph.D), UCL (University College London). Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/10192916/7/Garbin_10192916_Thesis.pdf