Building ClipScape: a local AI search engine for personal video

A little while ago I started filming everything I was working on in the workshop. 3D printing, woodwork, painting, repairs etc. Whatever I was building or fixing on the weekends would get videoed. I plan on making a YouTube channel to post these (when I get the time!), but it didn't take long to have hundreds of video files sitting in date-sorted folders with useless filenames.

The problem I had wasn't storage (thanks to my Network Attached Storage, I've got terabytes of space I can fill). The problem was, that when I wanted to finally create that polished video, I couldn't easily find the clips I wanted. I knew I'd filmed something specific, but finding it in generically named folders with generically named files was not fun. Scrubbing through random files hoping to spot the right frame is not a workable system. So I built something to fix it.

ClipScape is a desktop app that processes a local video library and makes it searchable by content. It uses OpenAI's freely available Whisper model for transcription, FFmpeg for frame extraction, and then feeds everything through a RAG pipeline. Videos get broken down into segments, analysed by an LLM for context and tagging, and embedded into a vector store. When you search, it's semantic, so you can describe what you're looking for in plain language and it finds the right moments.

As much as possible runs locally (like the audio transcription, thumbnail generation, scene detection, and semantic search. But if you want it to try and understand what's in the video, you'll need to plug in an OpenRouter.ai API key. It will then use an LLM to automatically describe and tag the videos.

It started as a weekend project to scratch my own itch and it's turned into a proper product. If you've got a video library that's grown beyond what folders and filenames can handle, have a look. The base version is free to use (apart from the AI tokens if you turn that on).