Facial Keypoint Detection

Applications of Facial Keypoint Detection

There are several interesting applications of keypoint detection in human faces. A few of them are listed below.

Facial feature detection improves face recognition

Facial landmarks can be used to align facial images to a mean face shape, so that after alignment the location of facial landmarks in all images is approximately the same. Intuitively it makes sense that facial recognition algorithms trained with aligned images would perform much better, and this intuition has been confirmed by many research papers.

Head pose estimation

Once you know a few landmark points, you can also estimate the pose of the head. In other words you can figure out how the head is oriented in space, or where the person is looking. E.g. CLM-Framework described in this post also returns the head pose.

Face Morphing

Facial landmarks can be used to align faces that can then be morphed to produce in-between images. An example is shown in Figure 1.

Figure 1. A morph between President Bush and the The Governator!

Want me to write a post on face morphing ? Please write to me in the comments section below!

Virtual Makeover

At my company ( TAAZ.com ) we had written our own facial landmark detector. The detected landmarks were used to the calculate contours of the mouth, eyes etc. to render makeup virtually. An example is shown in Figure 2.

Figure 2. Landmark detection for virtual makeover.

Dlib ( C++ / Python )

Dlib is a collection of miscellaneous algorithms in Machine Learning, Computer Vision, Image Processing, and Linear Algebra. Most of the library is just header files that you can include in your C++ application. Oh you prefer python ? No problem, it has a python API as well.

I personally like Dlib more than any other facial feature detection & tracking library because the code is very clean, well documented, the license permits use in commercial applications, the algorithm they have chosen to implement is very fast and accurate, and you can easily integrate the library in your C++ project by just including the header files.

How to run Dlib’s facial landmark detector ?

After you have built the examples, to run the facial landmark detector using a webcam, do the following.

1 2 3 4 5

cd examples/build/ #Download the face landmark model wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 tar xvjf shape_predictor_68_face_landmarks.dat.bz2 ./webcam_face_pose_ex

If you want to run it on a single image, you can try

1	./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg

CLM-Framework (C++)

CLM-framework, also known as the Cambridge Face Tracker, is a C++ library for facial keypoint detection and head pose estimation. You can see how well it works in the included video. Compiling this library on OSX was bit of a challenge but it was not too bad. The library depends on OpenCV 3 and requires X11.

There are two important ways in which Dlib beats CLM-Framework. First, DLib is much faster than CLM-Framework. Second, Dlib’s license allows you to use it in commercial applications. If I had to pick, I would use Dlib. Interestingly, CLM-Framework depends on Dlib!

Facial Feature Detection Research

Many different approaches have been used to solve this problem and it is difficult to summarize them in a blog post. I am simply linking to some important papers ( with major bias toward recent work ) for people who want to investigate more.

1. Active Appearance Model (AAM) by T. Cootes, G. Edwards and C. J. Taylor. [1998]
2. Face Alignment through Subspace Constrained Mean-Shifts by Jason M. Saragih, Simon Lucey and Jeffrey F. Cohn. [2009]
3. Localizing Parts of Faces Using a Consensus of Exemplars by Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, Neeraj Kumar [ 2011 ]
4. Face Alignment by Explicit Shape Regression by Xudong Cao Yichen Wei Fang Wen Jian Sun [2012]
5. Supervised Descent Method and Its Applications to Face Alignment by Xuehan Xiong and Fernando De la Torre [2013]
6. Constrained Local Neural Fields for robust facial landmark detection in the wild by Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. [2013]
7. Extensive Facial Landmark Localization with Coarse-to-fine Convolutional Network Cascade by Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. [2013]
8. Face alignment at 3000 fps via regressing local binary features by S Ren, X Cao, Y Wei, J Sun. [2014]
9. Facial Landmark Detection by Deep Multi-task Learning by Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. [2014]
10. One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan. [2014]