$\epsilon$-Seg: Sparsely Supervised Semantic Segmentation of  Microscopy Data
Sheida RahnamaiKordasiabi, Damian Nogare, Florian Jug
Semantic segmentation of electron microscopy (EM) images of biological samples remains a challenge in the life sciences.EM data captures details of biological structures, sometimes with such complexity that even human observers can find it overwhelming.Here we introduce $\epsilon$-Seg, a method based on hierarchical variational autoencoders (HVAEs), employing center-region masking, sparse label contrastive learning (CL), a Gaussian Mixture Model (GMM) prior, and clustering-free label prediction.Center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, even if training labels are sparse ($0.05$\% of the total image data or less).Additionally, we propose an entropy-based loss that can improve segmentation quality when fewer training labels are available (i.e. on $0.0025$\% of the data).For optimal performance, we employ CL and a GMM prior to shape the latent space of the HVAE such that encoded input patches tend to cluster w.r.t. the semantic classes we wish to distinguish. Finally, instead of clustering latent embeddings for semantic segmentation, we propose a semantic segmentation head composed of MLP and FiLM layers to directly predict class labels from latent embeddings.We show empirical results of $\epsilon$-Seg and baseline methods on $2$ dense EM datasets of biological tissues and demonstrate the applicability of our method also on fluorescence microscopy data. Our results show that $\epsilon$-Seg is capable of achieving competitive semi-supervised segmentation results on complex biological image data, even if only limited amounts of training labels are available.