Demo Project: Film Slate Detector
π¬π₯π» A computer vision (object detection) pet project π¨βπ»πΏπ€
Demo
A live demo should still be working if you’re reading this. It’s served by HuggingFace π€ Spaces. If it doesn’t work embedded below here, try clicking through the link.
Why
Back in ~2016-2018 I used to spend many, many hours in a dark room in SoHo scrubbing through video files to see where the slate claps were, so that I could fix the audio sync. It got very boring, to the point where I quit to go learn how to code.
What
Why not have AI automate the boring stuff? Given enough labeled examples, a pre-trained image model should be able to spot slates, after some fine-tuning. I achieved a COCOmetric of like 88% with a few hundred images. It’s just a fine-tuned faster-rcnn model.
The natural next step would be to train it on two classes: “Open” and “closed” slates, so that you could find the inflection point where the slate goes from open to closed. That might save some dailies technicians a lot of time.
Tools
My data science mentor pointed me to IceVision, FastAI, and various TIMM models. I got set up with LabelStudio and Weights and Biases. I wrote a long LinkedIn post about how I accomplished the fine-tuning.
In case you didn’t already know, it can be incredibly frustrating to do data-intensive machine learning stuff on an Apple Silicone laptop. Rather than swapping out my local hardware, I took this opportunity to get really dirty with Ec2 spot instances and used Terraform to make it dead simple to spin up and tear down spot GPU instances.