Multimodal Emotion Detection
A multimodal emotion recognition system on the MELD dataset (dialogue clips from Friends), combining text, audio, and video cues — because any single modality misses too much of how humans actually convey emotion.
Architecture
Multimodal emotion recognition
- 01
Rebalance
Downsample common classes, augment rare ones
- 02
Extract
BERT text, HuBERT audio, OpenFace video
- 03
Score
Each modality predicts emotion logits
- 04
Fuse
Weighted late fusion: 50% text, 30% audio, 20% video
- 05
Explain
LoRA LLM explains via facial Action Units
Key decisions
DECISION 01
Weighted late fusion instead of early feature concatenation
Letting each modality vote by its own reliability beats mixing raw features up front.
→ 60% accuracy, 38% macro F1 vs 23% for early fusion
DECISION 02
HuBERT embeddings instead of OpenSMILE plus an MLP
The MLP had high train accuracy but generalised poorly.
→ Disgust F1 from 4% to 32% (with augmentation)
DECISION 03
Feed prior dialogue and speaker tags to the text model
A single utterance lacks the context needed to read the emotion.
→ +2-3 F1 points; best text model 44.8% macro F1
Data & Augmentation
MELD has ~13,000 utterances from 1,433 dialogues, but is heavily skewed — 47% neutral, with Fear and Disgust each under 3%. I rebalanced every class to 2,000 train / 400 dev samples by downsampling dominant classes and augmenting minority ones.
Augmentation applied pitch shift, speed change, or additive noise to audio, and Gaussian blur, brightness/contrast, zoom, or color tint to a random 50% subset of the resulting video — with custom verification scripts to catch missing files and keep train/dev/test splits consistent.
Per-Modality Modeling
Text: BERT and DistilBERT with three pooling strategies (CLS, max, mean+max). Adding prior-dialogue context (previous utterances + speaker tags, inspired by DialogueRNN) improved F1 by 2–3 points — the best configuration was BERT + CLS + context at 44.8% macro F1.
Audio: switched from OpenSMILE + MLP (high train accuracy, poor generalization) to HuBERT embeddings, which meaningfully improved minority-class detection — Disgust F1 climbed from 4% to 32% after augmentation.
Video: started with MediaPipe landmarks into MLP/BiLSTM models, then moved to MTCNN-cropped faces feeding EfficientNet-B3 and ViT for stronger frame-level features.
Fusion
Early fusion concatenated features from all three modalities into a single BiLSTM — this underperformed badly at 23% accuracy / 14% macro F1.
Late fusion, combining each modality's output logits, did much better, and weighted late fusion (50% text / 30% audio / 20% video, weighted by each modality's individual strength) was the clear winner at 60% accuracy / 38% macro F1 — showing that letting each modality vote according to its own reliability beats mixing raw features upfront.
LLM Explanation Layer
Fine-tuned a LoRA-augmented LLM on facial Action Unit (AU) features — extracted via OpenFace and labeled through Gemini API calls — so the system could generate natural-language explanations grounded in specific AU activations (e.g., recognizing AU04 Brow Lowerer + AU12 Lip Corner Puller + AU15 Lip Corner Depressor as indicators of surprise).
Shipped a Streamlit app that takes an uploaded video, runs face/audio/text extraction end-to-end, and returns the predicted emotion with its AU-grounded explanation.
Group project at USC with 3 teammates (Khushal Jhaveri, Anoushka Sinha, Sakshi Bharambe). My contributions: dataset balancing and augmentation, automated batch processing, and LoRA fine-tuning for the explanation model.