Night time IR (back and white) detection

[alarmatwork]

Managed to get this system working with home assistant with RP4 pretty well - great and massive kudos for the team behind it!
During the day time it works really well - detects people very well and occasionally considers my narrow walking path as giraffe and other fun :)

But during the night time when Hikvision camera will switch to black and white with infrared light enabled.. there is zero detection even if the person is really close. Is there any extra conf required or a bit of more training for AI needs to be done? Because night time is probably most of the serious use of this camera in case of security.

[blakeblackshear]

You will need to watch the stream at night with motion/bounding boxes turned on to see if it is the motion detection that is not sensitive enough or if the object detection scores are too low.

[alarmatwork]

Sure

  "cameras": {
    "saun": {
      "best_image_timeout": 60,
      "clips": {
        "enabled": true,
        "objects": null,
        "post_capture": 5,
        "pre_capture": 5,
        "required_zones": [],
        "retain": {
          "default": 10,
          "objects": {
            "person": 15
          }
        }
      },
      "detect": {
        "enabled": true,
        "max_disappeared": 25
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel info -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp:https://xxx:[email protected]:554/Streaming/Channels/1/ -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/saun-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp:https://127.0.0.1/live/saun -r 1 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect",
            "rtmp",
            "clips",
            "record"
          ]
        }
      ],
      "fps": 1,
      "frame_shape": [
        2160,
        3840
      ],
      "height": 2160,
      "motion": {
        "contour_area": 100,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 360,
        "mask": null,
        "threshold": 25
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "required_zones": [],
        "timestamp": true
      },
      "name": "saun",
      "objects": {
        "filters": {
          "airplane": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "apple": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "backpack": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "banana": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bear": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bed": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bicycle": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bird": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "boat": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "book": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bottle": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bowl": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "broccoli": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bus": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cake": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "car": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "carrot": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cat": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "clock": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "couch": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cow": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cup": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "dog": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "donut": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "elephant": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "fork": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "frisbee": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "giraffe": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "handbag": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "horse": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "keyboard": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "kite": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "knife": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "laptop": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "microwave": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "motorcycle": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "mouse": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "orange": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "oven": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "person": {
            "mask": [],
            "max_area": 100000,
            "min_area": 1000,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "pizza": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "refrigerator": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "remote": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "sandwich": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "scissors": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "sheep": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "sink": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "skateboard": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "skis": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "snowboard": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "spoon": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "suitcase": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "surfboard": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "tie": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "toaster": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "toilet": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "toothbrush": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "train": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "tv": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "vase": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "zebra": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": null,
        "track": [
          "person",
          "bicycle",
          "car",
          "motorcycle",
          "airplane",
          "bus",
          "train",
          "car",
          "boat",
          "bird",
          "cat",
          "dog",
          "horse",
          "sheep",
          "cow",
          "elephant",
          "bear",
          "zebra",
          "giraffe",
          "backpack",
          "handbag",
          "tie",
          "suitcase",
          "frisbee",
          "skis",
          "snowboard",
          "kite",
          "skateboard",
          "surfboard",
          "bottle",
          "cup",
          "fork",
          "knife",
          "spoon",
          "bowl",
          "banana",
          "apple",
          "sandwich",
          "orange",
          "broccoli",
          "carrot",
          "pizza",
          "donut",
          "cake",
          "couch",
          "bed",
          "toilet",
          "tv",
          "laptop",
          "mouse",
          "remote",
          "keyboard",
          "microwave",
          "oven",
          "toaster",
          "sink",
          "refrigerator",
          "book",
          "clock",
          "vase",
          "scissors",
          "toothbrush"
        ]
      },
      "record": {
        "enabled": false,
        "retain_days": 30
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": true,
        "crop": false,
        "enabled": true,
        "height": 3840,
        "required_zones": [],
        "retain": {
          "default": 10,
          "objects": {
            "person": 15
          }
        },
        "timestamp": true
      },
      "width": 3840,
      "zones": {}
    }
  },
  "clips": {
    "max_seconds": 300,
    "retain": {
      "default": 10,
      "objects": {}
    },
    "tmpfs_cache_size": ""
  },
  "database": {
    "path": "/media/frigate/clips/frigate.db"
  },
  "detectors": {
    "cpu1": {
      "device": "usb",
      "num_threads": 3,
      "type": "cpu"
    },
    "cpu2": {
      "device": "usb",
      "num_threads": 3,
      "type": "cpu"
    }
  },
  "environment_vars": {},
  "logger": {
    "default": "INFO",
    "logs": {}
  },
  "model": {
    "height": 320,
    "width": 320
  },
  "mqtt": {
    "client_id": "frigate",
    "host": "core-mosquitto",
    "port": 1883,
    "stats_interval": 60,
    "topic_prefix": "frigate",
    "user": null
  },
  "snapshots": {
    "retain": {
      "default": 10,
      "objects": {}
    }
  }
}```

[alarmatwork]

No regions defined, it works perfectly during the daytime on the same spot.

[blakeblackshear]

You don't define regions in the config. They show up as green boxes when viewing the debug stream when you have the regions switch on. The scores may just be too low at night. Do you have the option to use a low resolution stream?

[blackgold9]

I ran into a similar issue when testing last night. When on IR mode, motion detection drops to almost nothing. I tested on an indoor and outdoor camera. Outdoor i couldn't trigger it at all, though the point of interest is somewhat distant from the camera. Triggers fine in the day though. Indoors, when I took up a reasonable ammount of the frame, it also didn't trigger motion, except when i walked in front of an open door with a light on.

In both cases, the issue wasn't detection, it was that there were no motion events detected (no red boxes). Any suggestions? Side note: It would be cool if we could set two sets of motion trigger thresholds, and either switch between on a schedule or via mqtt.

[blackgold9]

I'm wondering if ramping up sensitivity all the time and just paying the detection cost is a good solution (my coral is VERY underutilized as i understand it)

[LaurenceGough]

Bit of a bump but I experienced the same problem. There is a Tensorflow model here which has been trained on greyscale images, it works a little better for night time detection but unfortunately it now detects cats as people, lol. https://gitlab.com/Curid/TF-CCTV

I believe, but I am not sure at all that currently Frigate just uses the default sample for object detection?

[LaurenceGough]

Odds are it isn't the object detection but motion, you'll want to make sure motion -> improve_contrast is enabled at night so it can catch motion better

Yes, I have that on too but great suggestion and for sure is the best place to start. The night vision on my camera isn't the best so I did have to up the motion sensitivity and use that improve_contrast you suggested. I think I need to up the motion detection a bit more too but for the first time in pretty much forever I saw a few greyscale snapshots in the events listing since uploading that model. I'm still testing it anyway, I will probably switch between the standard one and this for more testing.

I am just trying to find the best way to setup a test camera as I am finding more and more that I need to be able to test footage to see what happened. I wish there was the option to see boundary boxes etc during event reviews.

[NickM-27]

Reviewing footage can be done manually for now using: https://docs.frigate.video/contributing#2-create-a-local-config-file-for-testing

[alarmatwork]

is the improve_contrast setting of Frigate or is it a camera setting?
I also understand that during the night IR mode no Motion boxes are being drawn in Debug view.
So no further analyse will happen in the Coral etc.

The same scene during the day time with colors and light in Debug mode are always full of motion boxes, while the tree branches move in the wind. In the night view I can clearly see the branches moving heavily but looks like the motion is not detected.

[NickM-27]

is the improve_contrast setting of Frigate or is it a camera setting? I also understand that during the night IR mode no Motion boxes are being drawn in Debug view. So no further analyse will happen in the Coral etc.

The same scene during the day time with colors and light in Debug mode are always full of motion boxes, while the tree branches move in the wind. In the night view I can clearly see the branches moving heavily but looks like the motion is not detected.

it is a frigate setting:

motion:
  improve_contrast: true

[alarmatwork]

Thanks, much better now.. at least some motion is being detected. Will see about the detection. Will test if humans are detected once I will have a chance to step in front of the camera again.