JEDI: Joint Expert Distillation in a Semi-Supervised Multi-Dataset Student-Teacher Scenario for Video Action Recognition
We propose JEDI, a multi-dataset semi-supervised learning method, which
efficiently combines knowledge from multiple experts, learned on different
datasets, to train and improve the performance of individual, per dataset,
student models. Our approach achieves this by addressing two important problems
in current machine learning research: generalization across datasets and
limitations of supervised training due to scarcity of labeled data. We start
with an arbitrary number of experts, pretrained on their own specific dataset,
which form the initial set of student models. The teachers are immediately
derived by concatenating the feature representations from the penultimate
layers of the students. We then train all models in a student-teacher
semi-supervised learning scenario until convergence. In our efficient approach,
student-teacher training is carried out jointly and end-to-end, showing that
both students and teachers improve their generalization capacity during
training. We validate our approach on four video action recognition datasets.
By simultaneously considering all datasets within a unified semi-supervised
setting, we demonstrate significant improvements over the initial experts.