Early Fusion versus Late Fusion in Multimodal Hateful Meme Analysis

With the widespread growth of social media platforms, human-only moderation has become impractical due to the volume of new content continuously released. The goal of multimodal hateful meme analysis is to create computational models that can effectively tell whether memes, images with text captions that are usually intended to be humorous, are hateful and should be removed. These models can be split into two groups: early fusion, and late fusion. In this paper, we evaluate early fusion and late fusion models to see which is more practical for a real-world setting. Both of our models were made with ResNet-50 and DistilBERT as base models. In terms of performance, our early fusion model had a 0.9413 ROC-AUC, while our late fusion model had a 0.9438 ROC-AUC. However, the early fusion model outperformed the late fusion model on accuracy, precision, recall, and F1 Score, and better balanced the modalities. Our late fusion model had a high reliance on the visual modality, while our early fusion model was well-balanced but leaned towards the text modality. We conclude by stating that early fusion is more balanced across the different modalities than late fusion, and therefore is a more practical choice for hateful meme analysis.