Project 3: Face Morphing

Ryan Mathew
October 5th, 2024
Website Repo

Overview

In this project, I will produce a "morph" of my own face into others' faces, find the average face in a dataset of faces, and create caricatures of myself through extrapolation.

Part 1: Defining Correspondences

In this part, I needed to manually define pairs of correspondences or points on the image of my face as well as George Clooney's. I first took a picture of myself against a plain background and cropped it to be the same size as George's. Using the correspondence labeling tool at this link, I labeled key points on both images in a consistent order. When labeling, I tried to focus on areas like the eyes, ears, mouth, nose, and hair to achieve the best morph results.

Here are the triangulations of the points on both my face and George Clooney's which I attained using scipy.spatial.Delaunay

Image 1
Ryan Triangulation
Image 2
George Triangulation

Part 2: Computing the "Mid-Way Face"

Before we can put together an entire morph sequence, we need to compute the mid-way face of me and George. First, we will compute the average shape which is found by taking the average of the labeled correspondence sets for both images. Next, we need to figure out where points in the average shape would come from in the original image for each triangle in the triangulation of the average shape. In order to perform this inverse warp, we first need to compute an affine transformation that will solve this equation:

Image 1
Now, using the affine transformation matrix A_i that we compute for each triangle T_i, we can dot product A_i with a polygon mask of the triangle T_i area matrix to get back the x and y coordinates of the original image that correspond with the coordinates in triangle T_i
Image 1
We can then interpolate the values at those coordinates in the original image to the triangle T_i area coordinates. After performing this for every triangle, we will get an output image of the original image warped to the average shape.

Finally, using our inverse warp function, we can warp each of the images to the average shape and take the average of the two warped images to average the colors which gives us the "mid-way face".

Image 1
Ryan
Image 1
Mid-Way Shape Triangulation
Image 2
George
Image 1
My Face Warped to Mid-Way Shape
Image 1
Mid-Way Face
Image 2
George's Face Warped to Mid-Way Shape

Part 3: The Morph Sequence

I created a morph sequence which takes a weighted average of the two images and produces 45 frames with weighting from 0 to 1 rather than only 0.5 from the previous part.

Image 1

Part 4: The "Mean Face" of a Population

I utilized the FEI face database of 200 aligned and labeled frontal face images to compute the average face of the population. I had to first go through the entire dataset and do some preprocessing to load all the images and compute the average shape.

Image 1
Average Face Shape

Then, I had to warp the faces into the average shape. Finally, I took the average of all the warped faces to achieve the average face of the dataset.
Here are a few images from the dataset warped to the average shape as well as the average face itself
Image 1
Image 197
Image 1
Warped to Average Shape
Image 1
Image 188
Image 1
Warped to Average Shape
Image 1
Image 116
Image 1
Warped to Average Shape
Image 1
Average Face

I also warped my face to the average face geometry and warped the average face to my face geometry. Before doing this, I had to resize my image and label the correspondences again manually. The warps came out as expected but because the average face image doesn't have hair to map correspondences with, my hair and forehead look a bit squished. Cropping/zooming into my image to exclude most of my hair would likely have fixed this.
Image 1
My Face Warped to Average Face
Image 1
Average Face Warped to my Face

Part 5: Caricatures: Extrapolating from the mean

In this part, I created caricatures of my face by extrapolating from the average face that I found in the previous part. I first calculated the differences of my face shape from the average face shape. Then, I scaled these differences by some alpha and added the product to my face shape to create a distorted, caricature like figure.

my_face_shape + alpha * (my_face_shape - average_face_shape)

Here are some of my results at varying alpha.

Laplacian [0] alpha = 0.25
Laplacian [1] alpha = 0.5
Laplacian [2] alpha = 0.75

Increasing the alpha will increase the scaled difference in features so therefore the distortion that is applied to my face image will be greater which is observed in the results.

Bells and Whistles: Changing Gender

For Bells and Whistles, I wanted to try to morph my face with an average South Asian Female face I found online. I first had to resize my face image to be the same size as the female face and manually added correspondences on key facial features for both images.

Laplacian [0] My Face
Laplacian [1] Average Face Shape
Laplacian [2] Average South Asian Female Face

To morph just the shape (warp), I warped my face onto the average female face. To morph just the appearance (cross-dissolve), I warped the average female on to my face. To morph both shape and appearance, I used my morph function (warp and cross-dissolve) with warp_frac = 0.5 and dissolve_frac = 0.5

Laplacian [0] Shape
Laplacian [1] Appearance
Laplacian [2] Both Shape + Appearance

I can notice that when I morphed both shape and appearance, it blends both very well. Increasing the warp_frac would give me more feminine features from the average female image and decreasing the warp_frac would give me more of my own features