I spent this month testing the ways people actually get an LLM to "watch" a video. One of them got a 2,181-video stress test from a single user. Here is what held up.
The three approaches on the table
1. Upload it to a hosted multimodal model (Gemini).
This is the shortest path and I want to be fair: for "tell me about this one clip," Gemini is genuinely good. You upload, you ask, you get an answer. The trade-offs are structural, not quality: your footage leaves your machine, results vary between runs, and you cannot hand the same evidence to a different model. When the answer looks wrong, there is nothing to check — you re-run and hope.
2. Run a describe-it-for-you pipeline (byjlw/video-analyzer, 1.5k stars).
A solid tool and an honest design: OpenCV pulls key frames, Whisper transcribes, a local vision model (Llama 3.2 11B) describes each frame, and you read the reconstructed description. The catch is that your LLM never sees the video — it sees another model's opinion of the video. Every frame the vision model misreads becomes a fact your LLM confidently repeats. You also pay for a second model in the loop.
3. Hand your LLM the evidence directly (claude-real-video).
This is the underrated one, and yes, it is mine — judge the argument, not the author. crv turns a video into what an LLM can actually read: scene-aware keyframes with true source timestamps, a timestamped transcript with optional speaker labels, and a manifest that tells the agent how to read the folder. No middleman model. Claude (or GPT, or a local model) looks at the actual frames and cites frame_012 @ 00:03:41. Everything runs locally.
What 2,181 videos taught me
A user ran crv over his entire photo library — 2,181 videos in four days — and mailed me the failure list, worst first. Two bugs from that report shaped my view of this whole category:
Percentage-based frame dedup is broken by design. A small object moving through a static scene barely moves the change percentage, so the frame gets dropped as a duplicate. The bird crossing the sky, the hand reaching into shot — exactly the frames a person would keep. Any tool that dedups by global frame difference has this blind spot. crv now scores localized cell changes; uniform-sampling tools avoid the bug by keeping everything, and then your context window pays for it.
Timestamps must survive the pipeline. After extract → dedup → rename, most pipelines lose the mapping back to source time. The model can describe the slide but cannot tell you when it was on screen. crv parses ffmpeg's real PTS and writesframes.json; the same user re-verified all 60 final frames of a 22-minute lecture against the source, 60/60.
Neither fix came from my test suite. They came from one user with a big real library. That is the honest state of this category: the hard bugs live where nobody's demo goes.
When to use which
- One clip, quick answer, privacy not a concern → Gemini.
- You want a finished text description and like the Ollama stack → video-analyzer.
- You want your LLM to reason over verifiable evidence, locally, reproducibly, with timestamps it can cite → claude-real-video.
pip install "claude-real-video[fast]"
npx skills add HUANGCHIHHUNGLeo/claude-real-video
MIT, runs on your machine, source: https://github.com/HUANGCHIHHUNGLeo/claude-real-video
SOCIAL SHARE CARD GENERATOR