leveraging swin transformer for local-to-global weakly supervised semantic segmentation

Recent advancements in weakly supervised semantic segmentation have highlighted the use of image-level class labels as a form of supervision. many methods use pseudo-labels from class activation maps to address the limited spatial information in class labels. however, class activation maps generated from convolutional neural networks often led to focus on prominent features, making it difficult to distinguish foreground objects from their backgrounds. while recent studies show that features from vision transformers are more effective in capturing the scene layout than convolutional neural networks, the use of hierarchical vision transformers has not been widely studied in weakly supervised semantic segmentation. this work introduces &swtformer& and explores the effect of swin transformer’s local-to-global view on improving the accuracy of initial seed class activation maps. swtformer-v1 produces class activation maps solely based on patch tokens as its input features. swtformer-v2 enhances this process by integrating a multi-scale feature fusion mechanism and employing a background-aware mechanism that refines the accuracy of localization maps, resulting in better differentiation between objects. experiments on the pascal voc 2012 dataset demonstrate that compared to state-of-the-art models, swtformer-v1 achieves 0.98% map higher in localization accuracy and generates initial localization maps that are 0.82% miou higher in accuracy while relying solely on the classification network. swtformer-v2 enhances the accuracy of the seed class activation maps by 5.32% miou. code available at: https://github.com/rozhanahmadi/swtformer