Multimodal Emotion Detection
A multimodal emotion recognition system on the MELD dataset (dialogue clips from Friends), combining text, audio, and video cues — because any single modality misses too much of how humans actually convey emotion.
Data & Augmentation
MELD has ~13,000 utterances from 1,433 dialogues, but is heavily skewed — 47% neutral, with Fear and Disgust each under 3%. I rebalanced every class to 2,000 train / 400 dev samples by downsampling dominant classes and augmenting minority ones.
Augmentation applied pitch shift, speed change, or additive noise to audio, and Gaussian blur, brightness/contrast, zoom, or color tint to a random 50% subset of the resulting video — with custom verification scripts to catch missing files and keep train/dev/test splits consistent.
Per-Modality Modeling
Text: BERT and DistilBERT with three pooling strategies (CLS, max, mean+max). Adding prior-dialogue context (previous utterances + speaker tags, inspired by DialogueRNN) improved F1 by 2–3 points — the best configuration was BERT + CLS + context at 44.8% macro F1.
Audio: switched from OpenSMILE + MLP (high train accuracy, poor generalization) to HuBERT embeddings, which meaningfully improved minority-class detection — Disgust F1 climbed from 4% to 32% after augmentation.
Video: started with MediaPipe landmarks into MLP/BiLSTM models, then moved to MTCNN-cropped faces feeding EfficientNet-B3 and ViT for stronger frame-level features.
Fusion
Early fusion concatenated features from all three modalities into a single BiLSTM — this underperformed badly at 23% accuracy / 14% macro F1.
Late fusion, combining each modality's output logits, did much better, and weighted late fusion (50% text / 30% audio / 20% video, weighted by each modality's individual strength) was the clear winner at 60% accuracy / 38% macro F1 — showing that letting each modality vote according to its own reliability beats mixing raw features upfront.
LLM Explanation Layer
Fine-tuned a LoRA-augmented LLM on facial Action Unit (AU) features — extracted via OpenFace and labeled through Gemini API calls — so the system could generate natural-language explanations grounded in specific AU activations (e.g., recognizing AU04 Brow Lowerer + AU12 Lip Corner Puller + AU15 Lip Corner Depressor as indicators of surprise).
Shipped a Streamlit app that takes an uploaded video, runs face/audio/text extraction end-to-end, and returns the predicted emotion with its AU-grounded explanation.
Group project at USC with 3 teammates (Khushal Jhaveri, Anoushka Sinha, Sakshi Bharambe). My contributions: dataset balancing and augmentation, automated batch processing, and LoRA fine-tuning for the explanation model.