Simone Salerno Book a call

ESP32-CAM

ESP32-CAM motion detection

Detect moving objects from the camera frames, without any external PIR sensor

ESP32-CAM motion detection

Motion detection is the task of detecting when the scene in the ESP32 camera field of view changes all of a sudden.

This change may be caused by a lot of factors (an object moving, the camera itself moving, a light change...) and you may be interested in get notified when it happens.

For example, you may point your ESP32 camera to the door of your room and take a picture when the door opens.

In a scenario like this, you are not interested in localizing the motion (knowing where it happened in the frame), only that it happened.

Motion detection with PIR (infrared sensor)

Most tutorials on the web focus on human motion detection, so they equip the ESP32 with an external infrared sensor (a.k.a PIR, the one you find in home alarm systems) and take a photo when the PIR detects something.

If this setup works fine for you, go with it. It's easy, fast, low power and pretty accurate.

But the PIR approach has a few drawbacks:

  1. you can only detect living beings: since it is based on infrared sensing, it can only detect when something "hot" is in its field of view (humans and animals, basically). If you want to detect a car passing, it won't work
  2. it has a limited range: PIR sensors reach at most 10-15 meters. If you need to detect people walking on the street in front of your house at 30 meters, it won't work
  3. it needs a clear line-of-sight: to detect infrared light, the PIR sensor needs no obstacles in-between itself and the moving object. If you put it behind a window to detect people outside your home, it won't work
  4. it falsely triggers even when no motion happened: the PIR tecnique is actually a proxy for motion detection. The PIR sensor doesn't actually detects motion: it detects the presence of warm objects. For example, if a person comes into a room and lies down on the sofa, the PIR sensor will trigger for as long as the person doesn't leave the room

Motion detection without PIR (image based)

On the other hand, image motion detection can fulfill all the above cases because it performs motion detection on the camera frames, comparing each one with the previous looking for differences.

If a large portion of the image changed, it triggers.

Video motion detection has its drawbacks, nonetheless:

  1. power-hungry: comparing each frame with the previous frame means the camera must be always on. While with the PIR sensor you can put the camera to sleep, now you have to continuously check each frame
  2. insensitive to slow changes: to avoid false triggers, you will set a lower threshold on the image portion that need to change to detect motion (e.g. 10% of the frame). If something is moving slowly in your field of view such that it changes less than 10% of the frame, the algorithm will not pick it up.

Take some time to review the pros and cons of video motion detection now that you have a little more details.

Simple case

The simple case for motion detection is to grab a frame and run the detection inside the main loop. You can configure a few parameters for the detection:

  • **stride**: when you capture images at high resolution (e.g. **640x480**), it is a waste of time and resources to consider each and every pixel. Motion will chance large amounts of pixels at once, so it'd be a smart move to only consider one pixel every nth. The values **2, 4, 8** and multiples of **8** also allow for a more efficient JPEG decoding, further increasing speed.
  • delta: how much should the current pixel differ from the previous frame to be considered in the change count? The higher this value, the less sensitive the algorithm is.
  • smooth (from **0** to **1**): when updating the background model, the algorithm makes a weighted average between the current frame and the previous frame. The higher this value, the more importance is given to the current frame. If **1**, the previous frame is completely ignored. If **0**, no update happens at all.
  • train epochs: before the model starts the detection, you can form the background model on a given number of frames. This helps the algorithm understand what the scene looks like and what instead makes a change.
  • consider filter: by default, the entire image is considered when computing the moving pixels. But you may selectively include and exclude portions of the image by defining a custom function. This function accepts the x and y coordinates of the pixel and it's value: if it returns true, the pixel is considered, otherwise it is ignored.
  • throttle: even though this is not a core part of motion detection, it still is very relevant. If something is moving in the frame, it will trigger the detection for as long as it moves (and a few frames later, until the background model stabilizes). You may only be interested in getting a single event instead and mute the next ones for a given amount of time. This mute process is called throttling.

The result of motion detection is an object with the following properties:

  • total: the number of pixels considered
  • moving: the number of pixels that changed from the previous frame
  • ratio: the number of pixels that changed over the total pixels considered
  • centerOfMass.x: the x coordinate of the center of mass of pixels that changed
  • centerOfMass.y: the y coordinate of the center of mass of pixels that changed

You can then decide a threshold (between **0** and **1** above which you want to consider that motion happened. While it could be **0** (meaning even a single pixel change will trigger motion), a more sensible value could be **0.2** or **0.3**.

Here's a complete sketch that showcases the syntax and configurations described so far.

/**
 * Motion detection example, simple case.
 * Includes events throttling.
 *
 * All configurations in the setup() method
 * for the motion detection are optional.
 * Feel free to delete the ones you don't need.
 *
 * Requires the JPEGDEC library
 * (https://github.com/bitbank2/JPEGDEC)
 */
#include <JPEGDEC.h>
#include <eloquent_esp32cam3.h>
#include <eloquent_esp32cam3/modules/motion.h>

using eloquent::camera::Camera;
using eloquent::camera::motion::Motion;
using eloquent::camera::motion::dtypes::Result;

Camera camera;
Motion motion;

void setup() {
    Serial.begin(115200);
    delay(3000);
    Serial.println("Motion detection: simple case");

    // see "GetStarted.ino" for more comments
    camera.hardware.brownout.disable();
    camera.hardware.clock.fast();
    camera.hardware.pinout.ask();
    camera.frame.pixformat.jpeg();
    camera.frame.resolution.vga();
    camera.frame.quality.high();

    // configure motion algorithm
    motion.config.stride(16);
    motion.config.differBy(20);
    motion.config.smoothBy("20%");
    motion.config.trainFor(10);
    motion.throttle("3 seconds");

    // decide which pixels to consider
    motion.config.consider([](size_t x, size_t y, uint8_t pixel) {
        // only consider pixels whose x is <= 160 and y <= 320
        return x <= 160 && y <= 320;
    });

    // you can register a function to call when
    // moving ratio is above a certain value
    motion.callbacks.onMotion("30%", [](Result &result) {
        // ignore throttle, always print
        Serial.printf(
                "[callback] %.0f%% of the image changed (%d/%d)\n",
                result.ratio * 100,
                result.moving,
                result.total
        );
    });

    // init camera and discard first 2 frames
    camera.begin();
    camera.raise();
    camera.discard(2);

    // motion doesn't require init
}

void loop() {
    auto image = camera.grab();

    if (image.failed()) {
        Serial.print("Camera error: ");
        Serial.println(image.error);
        return;
    }

    motion.update(image);

    if (motion.failed()) {
        Serial.print("Motion error: ");
        Serial.println(motion.error);
        return;
    }

    // check how much of the image changed
    if (motion.result.ratio > 0.3 && motion.throttle) {
        Serial.printf(
                "%.0f%% of the image changed (%d/%d)\n",
                motion.result.ratio * 100,
                motion.result.moving,
                motion.result.total
        );

        Serial.printf(
                "Center of mass at x=%d, y=%d\n",
                motion.result.centerOfMass.x,
                motion.result.centerOfMass.y
        );

        // start counting time from now
        // no event will trigger for the next 3 seconds
        motion.throttle.touch();
    }

    delay(100);
}

Here's a recap of the most important portions of the code above.

Configure motion detection

// run motion detection on a 1/16th version
// of the frame to make it faster and use less resources
// must be a power of 2 or a multiple of 8
motion.config.stride(16);

// how much should two pixels differ to be included into the motion count?
motion.config.differBy(20);

// how much to smooth pixels when updating previous state.
// 0 means no smooth at all (use latest frame as is)
// 1 means no update al all (use first frame only)
// 0.5 is the mean between current and prev
// you can use percentages too!
motion.config.smoothBy("20%");

// while training, the background model is updated but no detection happens
motion.config.trainFor(10);

// don't fire event more than once every 3 seconds
motion.throttle("3 seconds");

// which pixels should be considered?
motion.config.consider([](size_t x, size_t y, uint8_t pixel) {
    // only consider pixels whose x is <= 160 and y <= 320
    return x <= 160 && y <= 320;
});

Check if motion happened

auto image = camera.grab();
motion.update(image);

if (motion.result.ratio > 0.3 && motion.throttle) {
    // motion happened
}

Lambda functions

If in the code snippets above the syntax

motion.config.consider([](size_t x, size_t y, uint8_t pixel) {}

looks obscure to you, those are C++ lambda functions. They allow to define a function inline where it is needed, without the need to declare it globally beforehand. If that looks akward to you, you can define your function globally and pass a pointer to it instead.

bool considerFilter(size_t x, size_t y, uint8_t pixel) {
    return x <= 160 && y <= 320;
}

motion.config.consider(&considerFilter);

Conclusion

This post showcases the basic workflow to deal with motion detection from ESP32-CAM frames without any external PIR sensor. It gives you a lot of flexibility to accomodate the vast majority of use cases.

If you want to dig deeper, consider reading the related posts for more advanced use cases and integrations.