ESP32-CAM Face detection

The ESP32 chip is so powerful that it can run really capable machine learning models within a reasonable interval. The ESP32 core ships with a built-in face detection model that you can use to detect faces in your ESP32-CAM frames in (almost) real-time.

The internals of the API are a bit tricky, but thanks to the EloquentEsp32Cam3 library you won't even notice. This post will teach you how to configure and run face detection on your ESP32-CAM with zero effort.

Speed

How much does it take to run face detection on the ESP32-CAM? To answer this question, we have to make 2 distinctions: on the chip model and on the model accuracy.

**ESP32** is the "original" chip, the one that you find on the ubiquitous AiThinker camera. It is dirty cheap (the aforementioned camera costs 4 USD) and pretty capable. Many boards from other vendors (e.g. Lilygo) come with a large external PSRAM, which is essential for AI workloads
**ESP32-S3** is a newer version of the chip and comes with a lot more computational power and optimizations for AI execution. It usually comes with large external PSRAM
**one stage** face detection uses a single model to find face candidates. It is fast, but may yield inaccurate results
**two stage** face detection adds a refinement model that increases the accuracy of the detection and can also identify face landmarks (eyes, nose, mouth). It is slower that the single stage model, of course.

With these distinctions in mind, here's a summary table that highlights the execution time for the combinations of the 2 variants.

| Board    |    Mode   | Execution time |
|----------|:---------:|---------------:|
| ESP32    | One stage |          42 ms |
| ESP32    | Two stage |         129 ms |
| ESP32-S3 | One stage |           8 ms |
| ESP32-S3 | Two stage |          38 ms |

Arduino sketch

Here's a complete Arduino sketch that performs face detection. There are only a couple configurations you have to set:

**mode**: as stated earlier, you have to choose if you want a **fast()** detection or an **accurate()** one
**confidence**: when using the accurate mode, faces are assigned a confidence score from 0 (lowest) to 1 (highest). You can choose to discard results with a confidence lower than e.g. 0.7. The higher the value, the less likely that a face will be identified

/**
 * Face detection.
 * Requires PSRAM.
 *
 *  Requires the JPEGDEC library
 * (https://github.com/bitbank2/JPEGDEC)
 */
#include <JPEGDEC.h>
#include <eloquent_esp32cam3.h>
#include <eloquent_esp32cam3/modules/face_detection.h>

using eloquent::camera::Camera;
using eloquent::camera::face::Detection;
using eloquent::camera::face::dtypes::Face;

Camera camera;
Detection detection;

void setup() {
    Serial.begin(115200);
    Serial.println("Face detection: simple case");

    // see "GetStarted.ino" for more comments
    camera.hardware.brownout.disable();
    camera.hardware.clock.fast();
    camera.hardware.pinout.ask();
    camera.frame.pixformat.jpeg();
    camera.frame.resolution.vga();
    camera.frame.quality.high();

    // init camera and discard first frames
    camera.begin();
    camera.raise();
    camera.discard(2);

    // init face detection
    // fast is less accurate
    detection.config.fast();
    // accurate also detects face landmarks
    detection.config.accurate();
    detection.config.confidence(0.7);

    Serial.println("Face detection ready!");
    Serial.println("Put your face in front of the camera");
}

void loop() {
    // grab frame
    auto image = camera.grab();

    // detect faces
    detection.detect(image);

    if (detection.failed()) {
        Serial.print("Face detection error: ");
        Serial.println(detection.error);
        return;
    }

    // return if no face is detected
    if (detection.results.count == 0) {
        Serial.println("No face detected");
        return;
    }

    // how many faces were detected?
    Serial.printf(
        "Detected %d faces in %dms\n",
        detection.results.count,
        detection.results.duration.ms
    );

    // iterate over the faces
    // with a classic for loop
    for (uint8_t i = 0; i < detection.results.count; i++) {
        auto face = detection.results.at(i);
        // face.x: x coordinate of upper-left corner of face bbox
        // face.y: y coordinate of upper-left corner of face bbox
        // face.w: width of the face bbox
        // face.h: height of the face bbox
        // face.cx: x coordinate of face bbox center
        // face.cy: y coordinate of face bbox center

        Serial.printf(
                "Face #%d detected at coordinates %d, %d\n",
                i + 1,
                face.cx,
                face.cy
        );

        // if accurate(), you also have the landmarks
        if (face.landmarks) {
            auto l = face.landmarks;

            Serial.printf(" > left eye: %d, %d\n", l.eyes.left.x, l.eyes.left.y);
            Serial.printf(" > right eye: %d, %d\n", l.eyes.right.x, l.eyes.right.y);
            Serial.printf(" > left mouth: %d, %d\n", l.mouth.left.x, l.mouth.left.y);
            Serial.printf(" > right mouth: %d, %d\n", l.mouth.right.x, l.mouth.right.y);
            Serial.printf(" > nose: %d, %d\n", l.nose.x, l.nose.y);
        }
    }

    // iterate over the faces with a forEach loop,
    // this is a mode idiomatic way w.r.t. the for loop
    detection.results.forEach([](uint8_t i, Face &face) {
        Serial.printf(
                "Face #%d detected at coordinates %d, %d\n",
                i + 1,
                face.cx,
                face.cy
        );
    });

    delay(100);
}

The most relevant lines are

// capture frame and run detection
auto image = camera.grab();
detection.detect(image);

// loop through the results using a classic loop
for (uint8_t i = 0; i < detection.results.count; i++) {
    auto face = detection.results.at(i);
}

// loop through the results using forEach
detection.results.forEach([](uint8_t i, Face &face) {
    // do something with face
});

For each face you get the coordinates and size of the bounding box and, if accurate mode was enabled, the landmarks coordinates.

Streaming

So far, we were only able to debug face detection in the Serial Monitor. It would be much better if we could visually debug it by seeing the realtime streaming video from the camera at the same time.

Face detection streaming is implemented in the default CameraWebServer example from the Arduino IDE, but it also has a lot more options that may distract you. If you prefer a cleaner interface, you can run the sketch below. After the sketch is flashed, open the Serial Monitor and take note of the IP address of your board. If your router supports mDNS (most do), you can open the stream at http://esp32cam.local, otherwise you will need to use the IP address.

This is a short demo of what the result will look like.

/**
 * Face detection debug server.
 * Requires PSRAM.
 * Requires "Huge APP" partition scheme.
 *
 * Requires the JPEGDEC library
 * (https://github.com/bitbank2/JPEGDEC)
 */
#include <JPEGDEC.h>
#include <eloquent_esp32cam3.h>
#include <eloquent_esp32cam3/modules/face_detection.h>

using eloquent::camera::Camera;
using eloquent::camera::face::MjpegServer;
using eloquent::camera::helpers::net::setHostname;

Camera camera;
MjpegServer server(camera);

void setup() {
    Serial.begin(115200);
    Serial.println("Face detection: debug server");

    // see "Get started.ino" for more comments
    camera.hardware.brownout.disable();
    camera.hardware.clock.fast();
    camera.hardware.pinout.ask();
    camera.frame.pixformat.jpeg();
    camera.frame.resolution.vga();
    camera.frame.quality.high();

    // init camera and discard first frames
    camera.begin();
    camera.raise();
    camera.discard(2);

    // connect to Wi-Fi
    wifiConnect("SSID", "PASSWORD");
    setHostname("esp32cam");

    // configure face detection
    // fast is less accurate
    // but allows for more fluid streaming
    server.detection.config.fast();

    // don't run face detection on every frame
    // (more fluid streaming)
    server.random("1 out of 5");

    // init face detection debug server
    server.begin();
    server.raise();

    // print server instructions
    Serial.println(server.help());
}

void loop() {
    // web server runs in background
    delay(1000);
}

Conclusion

This post instructed you on how to leverage the processing power of the ESP32 chip to perform face detection at up to 125 FPS (ESP32-S3, fast mode) while displaying the results in the terminal or in the browser.