FACTUAL — Real-Time Multimodal Fact-Checking

Audio is transcribed with OpenAI's Whisper, then a prompted LLM extracts check-worthy factual claims from the transcript, filtering out opinion and non-factual statements.

Verification runs through three escalating strategies: zero-shot prompting (fast but hallucination-prone), scratchpad-based reasoning (lists reasons for/against before a verdict, improving transparency), and retrieval-augmented generation, which pulls live evidence from Google Search and News API before judging the claim.

A self-critique loop sits on top of the RAG stage: a separate 'judge' LLM checks whether the answer is actually supported by the retrieved evidence, and if not, feeds structured feedback back into the query-generation step. This repeats for up to 10 iterations or until the judge approves the result.

Built a 400-claim benchmark specifically because existing fact-checking corpora didn't fit a speech-based, temporally-aware evaluation: 200 claims sampled from the PolitiFact fact-check corpus (published 2008–2022, so possibly within GPT-4o's training data) and 200 claims scraped directly from PolitiFact.com dated after the model's knowledge cutoff (November 2023), giving a clean test of retrieval versus memorization.

Every claim was rewritten with speech-like disfluencies, fillers, and hedges, then converted to audio — so the benchmark actually exercises the ASR step instead of starting from clean text.

Agentic configurations (structured prompting + self-critique) won across the board, especially on post-cutoff claims where the model's internal knowledge is naturally lacking: GPT-4o (Agentic) hit 84% accuracy / 0.83 F1 post-cutoff versus just 63% accuracy / 0.58 F1 for GPT-4o zero-shot on the same claims.

Gemini followed the same pattern (68% zero-shot → 81% agentic accuracy post-cutoff), confirming the gain comes from the retrieval + critique architecture, not a specific model.

The weak spot across every configuration was correctly saying 'insufficient information' — models strongly preferred guessing SUPPORT/REFUTE over admitting they didn't have enough evidence, with accuracy on that specific label sitting near 0% for the best configs.

The dataset (200–400 samples) skews political/news and English-only, so generalization to other domains and languages is untested.

Real-time verification is still computationally heavy (multiple LLM calls per claim), which caps how live 'real-time' can actually be without further latency work like streaming ASR and lightweight claim detectors.