mediapipe-node

This project integrates MediaPipe Solutions with Node.js and Express for real-time computer vision tasks. It showcases examples of image segmentation, hand and face detection, and pose detection, with a combined example for all three types of landmark detection.

Utilizing MediaPipe's client-side JavaScript APIs, the project performs video processing directly in the browser, minimizing server-side computation. Additionally, it features logging for hand and face gestures, tracking interactions both in the console and a text file in the server logs directory.

Project Structure

mediapipe-node/
├── server.js
├── package.json
├── package-lock.json
├── public/
│   ├── css/
│   │   ├── full_detection_styles.css
│   │   ├── hand_face_detection_styles.css
│   │   ├── image_segmentation_styles.css
│   │   ├── main.css
│   │   └── pose_styles.css
│   ├── images/
│   │   ├── face_landmark.png
│   │   ├── face_landmarker_keypoints.png
│   │   ├── hand-landmarks.png
│   │   ├── hand_landmark.png
│   │   ├── pose_detector.png
│   │   └── pose_landmarks_index.png
│   ├── js/
│   │   ├── full_detection.js
│   │   ├── hand_face_detection.js
│   │   ├── image_segmentation.js
│   │   └── pose_detection.js
│   └── json/
│       ├── blendshape_map.json
│       ├── hand_map.json
│       ├── gesture_map.json
│       └── pose_map.json
├── models/
│   ├── hand_landmarker.task
│   ├── face_landmarker.task
│   ├── pose_landmarker_full.task
│   └── selfie_multiclass_256x256.tflite
└── templates/
    ├── full_detection.html
    ├── hand_face_detection.html
    ├── image_segmentation.html
    ├── index.html
    └── pose_detection.html

Setup Instructions

1. Install Node.js

   Follow the instructions on the Node.js official website to download and install Node.js. This will also install npm (Node Package Manager), which is required to manage project dependencies.

2. Clone the Repository

   git clone https://github.com/ffm5113/mediapipe-node.git
   cd mediapipe-node

3. Install Dependencies

   Install the necessary Node.js packages:

   npm install express
   npm install express-validator

4. Run the Application

   Start the server by running:

   node server.js

5. Access the Application

   Open your browser and navigate to http://localhost:3000 to view the homepage.

Project License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.

MediaPipe License

The MediaPipe models and code included in this project are licensed under the Apache License, Version 2.0. See the LICENSE_MediaPipe file for details.

Documentation

Express:

• Express Official Site

MediaPipe Documentation:

• Hand Landmark Documentation:

  • Hand Landmark Documentation
  • Hand Landmark Code Example
  • Hands Lite/Full Model Card

• Face Landmark Documentation:

  • Face Landmark Documentation
  • Face Landmark Code Example
  • Blendshape V2 Model Card (November 11, 2022)
  • BlazeFace Short Range Model Card (June 9, 2021)
  • Face Mesh V2 Model Card

• Pose Landmarker Documentation:

  • Pose Landmarker Documentation
  • Pose Landmarker Code Example
  • BlazePose GHUM 3D Landmarker Model Card (April 16, 2021)

• Image Segmentation Documentation:

  • Image Segmentation Documentation
  • Image Segmentation Code Example
  • Multiclass Segmentation Model Card (May 10, 2023)

Input Shapes:

• FaceDetector: 192 x 192
• FaceMesh-V2: 256 x 256
• Blendshape: 1 x 146 x 2
• HandLandmarker (Full): 192 x 192, 224 x 224
• Pose detector (Full): 224 x 224 x 3
• Pose landmarker (Full): 256 x 256 x 3
• SelfieMulticlass: 256 x 256