I just listened to this AI generated audiobook and if it didn’t say it was AI, I’d have thought it was human-made. It has different voices, dramatization, sound effects… The last I’d heard about this tech was a post saying Stephen Fry’s voice was stolen and replicated by AI. But since then, nothing, even though it’s clearly advanced incredibly fast. You’d expect more buzz for something that went from detectable as AI to indistinguishable from humans so quickly. How is it that no one is talking about AI generated audiobooks and their rapid improvement? This seems like a huge deal to me.
I expect the data size to be a problem. Stable diffusion defaults to 512x512px, because it simply requires a lot of resources to generate an image. Even more so to train one. Now do that times 30 to generate even one second of video. I think we need something that scales better.
I fully expect this to work decently in a few years though, no matter how hard the challenge is, ai is moving really fast.
“Fisheye” generation seems obvious. Give the network a distorted view of an arbitrarily large image, where distant stuff scrunches inward toward a full-resolution point of focus. Predict only a small area - or even a single pixel. This would massively decrease the necessary network size, allowing faster training. (Or more likely, deeper networks). It’d also Hamburger Helper any size dataset by training on arbitrarily many spots within each image instead of swallowing the whole elephant.
Even without that, video only needs a few frames at a time. You want to predict a future frame from several past frames. You want to tween a frame in the middle of past and future frames. That’s… pretty much it. Time-lapse “past frames” by sampling one per second, and you can predict the next second instead of the next frame. Then the stuff between can be tweened.
Stable diffusion can do arbitrary sizes now, as long as you have the VRAM for it iirc
Of course, but that is precisely the problem. It gets expensive really really fast.