Getting video models to learn better, faster (linum.ai)
15 points by schopra909 3 hours ago
schopra909 2 hours ago
Hi HN, one of the authors here. Lmk if you have any questions, and I'll try my best to answer them!
E-Reverance 2 hours ago
Regarding the LAION aesthetic predictor footnote, I don't see why a modern model and nonlinear classifier won't do a good a job. Is there a fundamental technical problem with the idea?
schopra909 2 hours ago
What would you have in mind for a modern model? Like Dino-V3 or something of that ilk?
For the LAION classifier specifically, it's trained on-top of CLIP. The bottleneck for accuracy isn't the linear/non-linear readout, it's CLIP's specific knowledge of the world.
CLIP was trained with a contrastive learning objective, where all it has to learn is that images of the same category are closer to each other than images of other categories (e.g. two items are both dogs so they should be closer semantically, while say a dog and a computer are different and should be farther semantically.
As a result CLIP is really good at object definitions, but struggles at pretty much everything else. Modern multimodal LLM's perceptual modules are much stronger than CLIP; that's why we chose fine-tuning a LLM rather then training a MLP on-top of CLIP (which would be a direct iteration on the LAION aesthetic classifier).