FACIAL LANDMARK DETECTION WITH MEDIAPIPE & CREATING ANIMATED SNAPCHAT FILTERS

MediaPipe Face Mesh is a face geometry solution that estimates 468 3D face landmarks in real-time even on mobile devices. It employs machine learning (ML) to infer the 3D surface geometry, requiring only a single camera input without the need for a dedicated depth sensor. Utilizing lightweight model architectures together with GPU acceleration throughout the pipeline, the solution delivers real-time performance critical for live experiences. There are two models and they are face detection and face landmark model.Face landmark screen coordinates are converted into the Metric 3D space coordinates; face pose transformation matrix is estimated as a rigid linear mapping from the canonical face metric landmark set into the runtime face metric landmark set in a way that minimizes a difference between the two; a face mesh is created using the runtime face metric landmarks as the vertex positions (XYZ), while both the vertex texture coordinates (UV) and the triangular topology are inherited from the canonical face model.Snapchat Filter Controlled by Facial Expressions: Currently, our face expression recognizer can check whether the eyes and mouth are open or not so to get the most out of it, we can overlay scalable eyes images on top of the eyes of the user when his eyes are open and a video of fire coming out of the mouth of the user when the mouth is open.