Let's Roll: Synthetic Dataset Analysis for Pedestrian Detection Across Different Shutter Types
Computer vision (CV) pipelines are typically evaluated on datasets processed
by image signal processing (ISP) pipelines even though, for
resource-constrained applications, an important research goal is to avoid as
many ISP steps as possible. In particular, most CV datasets consist of global
shutter (GS) images even though most cameras today use a rolling shutter (RS).
This paper studies the impact of different shutter mechanisms on machine
learning (ML) object detection models on a synthetic dataset that we generate
using the advanced simulation capabilities of Unreal Engine 5 (UE5). In
particular, we train and evaluate mainstream detection models with our
synthetically-generated paired GS and RS datasets to ascertain whether there
exists a significant difference in detection accuracy between these two shutter
modalities, especially when capturing low-speed objects (e.g., pedestrians).
The results of this emulation framework indicate the performance between them
are remarkably congruent for coarse-grained detection (mean average precision
(mAP) for IOU=0.5), but have significant differences for fine-grained measures
of detection accuracy (mAP for IOU=0.5:0.95). This implies that ML pipelines
might not need explicit correction for RS for many object detection
applications, but mitigating RS effects in ISP-less ML pipelines that target
fine-grained location of the objects may need additional research.