Bytes Are All You Need: Transformers Operating Directly On File Bytes
Modern deep learning approaches usually transform inputs into a
modality-specific form. For example, the most common deep learning approach to
image classification involves decoding image file bytes into an RGB tensor
which is passed into a neural network. Instead, we investigate performing
classification directly on file bytes, without the need for decoding files at
inference time. Using file bytes as model inputs enables the development of
models which can operate on multiple input modalities. Our model,
ByteFormer, achieves an ImageNet Top-1 classification accuracy of
77.33% when training and testing directly on TIFF file bytes using a
transformer backbone with configuration similar to DeiT-Ti (72.2% accuracy
when operating on RGB images). Without modifications or hyperparameter tuning,
ByteFormer achieves 95.42% classification accuracy when operating on WAV
files from the Speech Commands v2 dataset (compared to state-of-the-art
accuracy of 98.7%). Additionally, we demonstrate that ByteFormer has
applications in privacy-preserving inference. ByteFormer is capable of
performing inference on particular obfuscated input representations with no
loss of accuracy. We also demonstrate ByteFormer's ability to perform inference
with a hypothetical privacy-preserving camera which avoids forming full images
by consistently masking 90% of pixel channels, while still achieving
71.35% accuracy on ImageNet. Our code will be made available at
https://github.com/apple/ml-cvnets/tree/main/examples/byteformer.
647977a8ae0fa9dc233837bc
Maxwell Horton
claimed_verified
2023-06-05T10:07:54.611Z
647a5afc732a0cd866eb512f
/avatars/fbd0ba2a7ba70414ac903e583ce62933.svg
Maxwell Horton
user
mchorton
647977a8ae0fa9dc233837bd
Sachin Mehta
647977a8ae0fa9dc233837be
Ali Farhadi
647977a8ae0fa9dc233837bf
Mohammad Rastegari
Bytes Are All You Need: Transformers Operating Directly On File Bytes
Modern deep learning approaches usually transform inputs into a
modality-specific form. For example, the most common deep learning approach to
image classification involves decoding image file bytes into an RGB tensor
which is passed into a neural network. Instead, we investigate performing
classification directly on file bytes, without the need for decoding files at
inference time. Using file bytes as model inputs enables the development of
models which can operate on multiple input modalities. Our model,
ByteFormer, achieves an ImageNet Top-1 classification accuracy of
77.33% when training and testing directly on TIFF file bytes using a
transformer backbone with configuration similar to DeiT-Ti (72.2% accuracy
when operating on RGB images). Without modifications or hyperparameter tuning,
ByteFormer achieves 95.42% classification accuracy when operating on WAV
files from the Speech Commands v2 dataset (compared to state-of-the-art
accuracy of 98.7%). Additionally, we demonstrate that ByteFormer has
applications in privacy-preserving inference. ByteFormer is capable of
performing inference on particular obfuscated input representations with no
loss of accuracy. We also demonstrate ByteFormer's ability to perform inference
with a hypothetical privacy-preserving camera which avoids forming full images
by consistently masking 90% of pixel channels, while still achieving
71.35% accuracy on ImageNet. Our code will be made available at
https://github.com/apple/ml-cvnets/tree/main/examples/byteformer.
ByteFormer performs classification directly on raw file bytes, supporting multiple modalities and privacy-preserving inference without significant accuracy loss.
ByteFormer transformer ImageNet WAV files Speech Commands v2 privacy-preserving inference obfuscated input representations pixel channel masking