Building a facial search in less than 2 hours

So, I’ve been pretty up front that I’ve been expanding my current skills in the data science and AI space, and its been a pretty fun process, and I wanted to point everyone to a demo I was able to build out very quickly.

Facial Searching, is a pretty common use case, so much so that we see it everywhere, Google Photos allows you to tag people in photos and indexes them for you automatically. Facebook makes suggestions for tagging people when you upload new pictures.

So I wanted to build a service that would take a set of selected images or 3 members of my family, and build a solution that would allow me to run any family photo through and it would search for the known members of my family to apply. Seems like a pretty basic use-case, but I’ve been wanting to get some hands on experience with the Azure Cognitive Services.

So I researched and read through our documentation, and decided before I started I was going to set aside 2 hours and see how far I could get to do the following:

  • Build a console app to read in images of 3 family members.
  • Build logic to upload an image and read back attributes about that person, things like age, gender, hair color, glasses, etc.
  • Build logic to search and match faces of people in a photo with the library that was previously uploaded.

The cool news is that I was able to implement all that logic. The full solution can be found here.

So to start, I focused on the first use case, and Azure Cognitive services has this concept of “PersonGroups” that can be leveraged with the SDK. For starters you need to install the sdk from nuget, and this is the required package.

Microsoft.Azure.CognitiveServices.Vision.Face

the first part is the key part is the client, which I configured in a parent class as follows:

public class BaseFaceApi
    {
        protected string _apiKey = ConfigurationManager.AppSettings["FaceApiKey"];
        protected string _apiUrl = ConfigurationManager.AppSettings["FaceApiUrl"];
        protected string _apiEndpoint = ConfigurationManager.AppSettings["FaceApiEndpoint"];
        protected FaceClient _client;

        protected void InitializeClient()
        {
            _client = new FaceClient(new ApiKeyServiceClientCredentials(_apiKey));
            _client.Endpoint = _apiEndpoint;
        }
    }

This allows for configuration to be the app.config, and this face client will be leveraged for all operations to hit the API.

The Face API leverages this concept of “PersonGroups” and “Persons” to handle the library of faces you are going to compare against. The process is broken into 4 parts.

  • Create the group
  • Create the person
  • Register Images for that person
  • Train the Model

If you review the source code you will find that I have broken these out to separate methods. The benefit of creating the groups is that you can limit your searching to specific groups, and have your application recognize the differences between groups.

Once you completed loading these images and “Persons” into the service you are ready to search through this repository by uploading an image. This is done with the following code:

public async Task<Dictionary<Guid,FacePerson>> IdentifyFaces(string filePath,string groupID)
        {
            InitializeClient();

            Dictionary<Guid,FacePerson> ret = new Dictionary<Guid, FacePerson>();

            using (Stream s = File.OpenRead(filePath))
            {
                // The list of Face attributes to return.
                IList<FaceAttributeType> faceAttributes =
                    new FaceAttributeType[]
                    {
            FaceAttributeType.Gender, FaceAttributeType.Age,
            FaceAttributeType.Smile, FaceAttributeType.Emotion,
            FaceAttributeType.Glasses, FaceAttributeType.Hair
                    };

                var facesTask = await _client.Face.DetectWithStreamWithHttpMessagesAsync(s,true,true,faceAttributes);
                var faceIds = facesTask.Body.Select(face => face.FaceId.Value).ToList();

                var identifyTask = await _client.Face.IdentifyWithHttpMessagesAsync(faceIds,groupID);
                foreach (var identifyResult in identifyTask.Body)
                {
                    Console.WriteLine("Result of face: {0}", identifyResult.FaceId);
                    if (identifyResult.Candidates.Count > 0)
                    { 
                        // Get top 1 among all candidates returned
                        var candidateId = identifyResult.Candidates[0].PersonId;
                        var person = await _client.PersonGroupPerson.GetWithHttpMessagesAsync(groupID, candidateId);

                        var fp = new FacePerson();
                        fp.PersonID = person.Body.PersonId;
                        fp.Name = person.Body.Name;
                        fp.FaceIds = person.Body.PersistedFaceIds.ToList();

                        var faceInstance = facesTask.Body.Where(f => f.FaceId.Value == identifyResult.FaceId).SingleOrDefault();
                        fp.Age = faceInstance.FaceAttributes.Age.ToString();
                        fp.EmotionAnger = faceInstance.FaceAttributes.Emotion.Anger.ToString();
                        fp.EmotionContempt = faceInstance.FaceAttributes.Emotion.Contempt.ToString();
                        fp.EmotionDisgust = faceInstance.FaceAttributes.Emotion.Disgust.ToString();
                        fp.EmotionFear = faceInstance.FaceAttributes.Emotion.Fear.ToString();
                        fp.EmotionHappiness = faceInstance.FaceAttributes.Emotion.Happiness.ToString();
                        fp.EmotionNeutral = faceInstance.FaceAttributes.Emotion.Neutral.ToString();
                        fp.EmotionSadness = faceInstance.FaceAttributes.Emotion.Sadness.ToString();
                        fp.EmotionSurprise = faceInstance.FaceAttributes.Emotion.Surprise.ToString();
                        fp.Gender = faceInstance.FaceAttributes.Gender.ToString();

                        ret.Add(person.Body.PersonId, fp);
                    }
                }
            }

            return ret;
        }

One key note above is the face attributes, this identifies the attributes you would like the service to review and discover. You can limit this list as you like.

Please feel free to review the sample code and I hope you find a great use-case. For me, a very cool project that is on my list next is to build a camera with a raspberry pi that captures people who come to the door and compares them against a known database of people.

It’s also worth mentioning that this service is fully available in Azure Government for customers that have requirements to be deployed in a sovereign cloud.

Leave a Reply

Your email address will not be published. Required fields are marked *