Dedelayed: Deleting remote inference delay via on-device correction
Remote inference allows lightweight devices to leverage powerful cloud
models. However, communication network latency makes predictions stale and
unsuitable for real-time tasks. To address this, we introduce Dedelayed, a
delay-corrective method that mitigates arbitrary remote inference delays,
allowing the local device to produce low-latency outputs in real time. Our
method employs a lightweight local model that processes the current frame and
fuses in features that a heavyweight remote model computes from past frames. On
video from the BDD100K driving dataset, Dedelayed improves semantic
segmentation accuracy over the stronger of the local-only and remote-only
baselines across all realistic communication network delays beyond 33 ms.
Without incurring additional delay, it improves accuracy by 6.4 mIoU compared
to fully local inference and 9.8 mIoU compared to remote inference, for a
round-trip delay of 100 ms. The advantage grows under longer delays and
higher-motion scenes, as delay-mitigated split inference sustains accuracy more
effectively, providing clear advantages for real-time tasks that must remain
aligned with the current world state.
68f0e78436f8b025381e1b89
Dan Jacobellis
claimed_verified
2025-10-17T04:12:44.493Z
63213080d2d45f3151837eba
https://cdn-avatars.huggingface.co/v1/production/uploads/63213080d2d45f3151837eba/aBhKfY-0gZhKGQmb_Gwi2.png
Dan Jacobellis
user
danjacobellis
68f0e78436f8b025381e1b8a
Mateen Ulhaq
68f0e78436f8b025381e1b8b
Fabien Racapé
68f0e78436f8b025381e1b8c
Hyomin Choi
68f0e78436f8b025381e1b8d
Neeraja J. Yadwadkar
Dedelayed: Deleting remote inference delay via on-device correction
Remote inference allows lightweight devices to leverage powerful cloud
models. However, communication network latency makes predictions stale and
unsuitable for real-time tasks. To address this, we introduce Dedelayed, a
delay-corrective method that mitigates arbitrary remote inference delays,
allowing the local device to produce low-latency outputs in real time. Our
method employs a lightweight local model that processes the current frame and
fuses in features that a heavyweight remote model computes from past frames. On
video from the BDD100K driving dataset, Dedelayed improves semantic
segmentation accuracy over the stronger of the local-only and remote-only
baselines across all realistic communication network delays beyond 33 ms.
Without incurring additional delay, it improves accuracy by 6.4 mIoU compared
to fully local inference and 9.8 mIoU compared to remote inference, for a
round-trip delay of 100 ms. The advantage grows under longer delays and
higher-motion scenes, as delay-mitigated split inference sustains accuracy more
effectively, providing clear advantages for real-time tasks that must remain
aligned with the current world state.
Dedelayed, a delay-corrective method, improves real-time semantic segmentation accuracy by fusing local and remote model outputs, mitigating communication network latency.
remote inference lightweight devices cloud models communication network latency delay-corrective method local model remote model BDD100K driving dataset semantic segmentation accuracy mIoU round-trip delay high-motion scenes split inference