UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Photorealistic retrieval of occluded facial information using a performance-driven face model

Berisha, F.; (2009) Photorealistic retrieval of occluded facial information using a performance-driven face model. Doctoral thesis , UCL (University College London). Green open access

[thumbnail of 14673.pdf]
Preview
PDF
14673.pdf

Download (1MB)
[thumbnail of Supplementary video from Chapter 3: First 15 PCs extracted from a natural facial motion sequence, shown here oscillating +/- 2 standard deviations from the mean] AVI video (Supplementary video from Chapter 3: First 15 PCs extracted from a natural facial motion sequence, shown here oscillating +/- 2 standard deviations from the mean)
First_15_PCs.avi

Download (43MB)
[thumbnail of Supplementary video from Chapter 3: Example demonstrating PCA-based performance driven mimicry, transferring Jason's behaviour onto Joanne's face] AVI video (Supplementary video from Chapter 3: Example demonstrating PCA-based performance driven mimicry, transferring Jason's behaviour onto Joanne's face)
jason2joanne.avi

Download (7MB)
[thumbnail of Supplementary video from Chapter 3: Example demonstrating PCA-based performance driven mimicry, transferring Joanne's behaviour onto Jason's face] AVI video (Supplementary video from Chapter 3: Example demonstrating PCA-based performance driven mimicry, transferring Joanne's behaviour onto Jason's face)
joanne2jason.avi

Download (8MB)
[thumbnail of Supplementary video from Chapter 4: Example of face-map mimicry, Glyn to Fatos] AVI video (Supplementary video from Chapter 4: Example of face-map mimicry, Glyn to Fatos)
dia_glyn2fatos.avi

Download (1MB)
[thumbnail of Supplementary video from Chapter 4: Example of "good" bubblemask mimicry, as categorised by the PC coefficient comparison with ground-truth] AVI video (Supplementary video from Chapter 4: Example of "good" bubblemask mimicry, as categorised by the PC coefficient comparison with ground-truth)
gc_mimic_02013.avi

Download (1MB)
[thumbnail of Supplementary video from Chapter 4: Example of "bad" bubblemask mimicry, as categorised by the PC coefficient comparison with ground-truth] AVI video (Supplementary video from Chapter 4: Example of "bad" bubblemask mimicry, as categorised by the PC coefficient comparison with ground-truth)
gc_mimic_02022.avi

Download (1MB)
[thumbnail of Supplementary video from Chapter 5: Mouth occlusion affected mimicry, no PC exaggeration] AVI video (Supplementary video from Chapter 5: Mouth occlusion affected mimicry, no PC exaggeration)
mho2full_sharp.avi

Download (48MB)
[thumbnail of Supplementary video from Chapter 5:  Mouth occlusion affected mimicry, PC coefficients multiplied by 1.5] AVI video (Supplementary video from Chapter 5: Mouth occlusion affected mimicry, PC coefficients multiplied by 1.5)
mho2fullx1p5_sharp.avi

Download (48MB)
[thumbnail of AVI video (Supplementary video from Chapter 5: Mouth occlusion affected mimicry, PC coefficients multiplied by 2] AVI video (AVI video (Supplementary video from Chapter 5: Mouth occlusion affected mimicry, PC coefficients multiplied by 2)
mho2fullx2p0_sharp.avi

Download (48MB)
[thumbnail of Supplementary video from Chapter 5: Mouth occlusion affected mimicry, PC coefficients multiplied by 3] AVI video (Supplementary video from Chapter 5: Mouth occlusion affected mimicry, PC coefficients multiplied by 3)
mho2fullx3p0_sharp.avi

Download (48MB)
[thumbnail of Supplementary video from Chapter 5: Mouth occlusion affected mimicry, PC coefficients multiplied by 4.5] AVI video (Supplementary video from Chapter 5: Mouth occlusion affected mimicry, PC coefficients multiplied by 4.5)
mho2fullx4p5_sharp.avi

Download (48MB)
[thumbnail of Supplementary video from Chapter 6: First 30 PCs extracted from a natural facial motion sequence (Glyn), shown here oscillating +/- 2 standard deviations from the mean] AVI video (Supplementary video from Chapter 6: First 30 PCs extracted from a natural facial motion sequence (Glyn), shown here oscillating +/- 2 standard deviations from the mean)
PCAsine.avi

Download (21MB)
[thumbnail of Supplementary video from Chapter 6: First 33 ICs extracted from a natural facial motion sequence (Glyn), shown here oscillating +/- 2 standard deviations from the mean] AVI video (Supplementary video from Chapter 6: First 33 ICs extracted from a natural facial motion sequence (Glyn), shown here oscillating +/- 2 standard deviations from the mean)
ICmovie_sine_glyn.avi

Download (23MB)
[thumbnail of Supplementary video from Chapter 6:  First 33 ICs extracted from a natural facial motion sequence (Fatos), shown here oscillating +/- 2 standard deviations from the mean] AVI video (Supplementary video from Chapter 6: First 33 ICs extracted from a natural facial motion sequence (Fatos), shown here oscillating +/- 2 standard deviations from the mean)
ICmovie_sine_fatos_noordering.avi

Download (23MB)
[thumbnail of Supplementary video from Chapter 6: Example of ICA-enhanced mimicry (Glyn to Fatos). Left-most frame is occluded driver, followed by plain PCA mimicry, ICA-enhanced mimicry and ground-truth] AVI video (Supplementary video from Chapter 6: Example of ICA-enhanced mimicry (Glyn to Fatos). Left-most frame is occluded driver, followed by plain PCA mimicry, ICA-enhanced mimicry and ground-truth)
glyn_mo2gt_momima_amp.avi

Download (3MB)
[thumbnail of Supplementary video from Chapter 6: Example of ICA-enhanced mimicry (Alison to Glyn). Left-most frame is occluded driver, followed by plain PCA mimicry, ICA-enhanced mimicry and ground-truth] AVI video (Supplementary video from Chapter 6: Example of ICA-enhanced mimicry (Alison to Glyn). Left-most frame is occluded driver, followed by plain PCA mimicry, ICA-enhanced mimicry and ground-truth)
mo_alison2glyn_4frm.avi

Download (3MB)

Abstract

Facial occlusions can cause both human observers and computer algorithms to fail in a variety of important tasks such as facial action analysis and expression classification. This is because the missing information is not reconstructed accurately enough for the purpose of the task in hand. Most current computer methods that are used to tackle this problem implement complex three-dimensional polygonal face models that are generally timeconsuming to produce and unsuitable for photorealistic reconstruction of missing facial features and behaviour. In this thesis, an image-based approach is adopted to solve the occlusion problem. A dynamic computer model of the face is used to retrieve the occluded facial information from the driver faces. The model consists of a set of orthogonal basis actions obtained by application of principal component analysis (PCA) on image changes and motion fields extracted from a sequence of natural facial motion (Cowe 2003). Examples of occlusion affected facial behaviour can then be projected onto the model to compute coefficients of the basis actions and thus produce photorealistic performance-driven animations. Visual inspection shows that the PCA face model recovers aspects of expressions in those areas occluded in the driver sequence, but the expression is generally muted. To further investigate this finding, a database of test sequences affected by a considerable set of artificial and natural occlusions is created. A number of suitable metrics is developed to measure the accuracy of the reconstructions. Regions of the face that are most important for performance-driven mimicry and that seem to carry the best information about global facial configurations are revealed using Bubbles, thus in effect identifying facial areas that are most sensitive to occlusions. Recovery of occluded facial information is enhanced by applying an appropriate scaling factor to the respective coefficients of the basis actions obtained by PCA. This method improves the reconstruction of the facial actions emanating from the occluded areas of the face. However, due to the fact that PCA produces bases that encode composite, correlated actions, such an enhancement also tends to affect actions in non-occluded areas of the face. To avoid this, more localised controls for facial actions are produced using independent component analysis (ICA). Simple projection of the data onto an ICA model is not viable due to the non-orthogonality of the extracted bases. Thus occlusion-affected mimicry is first generated using the PCA model and then enhanced by accordingly manipulating the independent components that are subsequently extracted from the mimicry. This combination of methods yields significant improvements and results in photorealistic reconstructions of occluded facial actions.

Type: Thesis (Doctoral)
Title: Photorealistic retrieval of occluded facial information using a performance-driven face model
Open access status: An open access version is available from UCL Discovery
Language: English
UCL classification: UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science > CoMPLEX: Mat&Phys in Life Sci and Exp Bio
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Clinical, Edu and Hlth Psychology
URI: https://discovery.ucl.ac.uk/id/eprint/14673
Downloads since deposit
28,032Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item