The Future of Animation? Using Reallusion's Cartoon Animator and Image to Video Generative AI to Animate Sequences

Animation software program is already able to auto producing tweens however will AI take it to the subsequent degree and tween whole sequences of movement?

My earlier article, Reallusion’s Cartoon Animator Versus PixVerse Image to Video and Lip Sync AI – Battle of the Talking Head Avatars, obtained me fascinated by utilizing generative AI for creating the ‘tween’ animation in between key poses.

Nonetheless PixVerse AI is outstanding at sustaining the artwork model of no matter picture you start with and also will allow you to embrace an finish body, making it a really perfect instrument for tweening, so…

What if, in a future model of Cartoon Animator (or your most popular 2D animation software program), all you needed to do was create the important thing moments of the scene, after which the software program’s AI kicked in to animate every thing in between every second? PixVerse may even generate sounds, speech, and lip syncing so it isn’t that fantastical a thought.

Picture to Video Generative AI – Is it That Good?

In a phrase ‘No’. To elaborate, ‘No, not but.’

I attempted quite a lot of AI websites with Picture to Video capabilities each with and with out including an finish body. All of them failed my take a look at, together with PixVerse.

In each case I used to be utilizing the free account for that AI web site. Some websites have picture to video behind their paywalls so these I utterly skipped these.

PixVerse’s free account with it is 60 free every day credit is so good, it is the one one, of two websites, I’ve paid cash to for a subscription (the opposite is ArtFlow – similar cause). I additionally like that PixVerse will allow you to purchase a minimal of 500 credit for USD$5.00 with out committing you to a subscription that you’ll want to keep in mind to cancel.

By the way, strive PixVerse’s free telephone app. I am having a whole lot of enjoyable with it, taking photographs of strange, on a regular basis issues, and animating them with PixVerse. For those who’re that manner inclined you’ll be able to create clips of you being hugged by Jesus… or a Polar bear… even a Tiger!

The Check – Highlights

Start and End Frames of a Cowboy mounting and riding a horse.

On the left is my begin body. Proper is my finish body that I used if the AI instrument had an finish body choice.
In each case I used the immediate “A cowboy standing subsequent to his horse, will get on the horse and rides ahead towards the left of body.”

My notes on how every faired are under. No single AI nailed it on the primary strive, and normally I might used my free credit score up so I could not retry – which is usually what you do when prompting an AI. You strive a immediate. It does not fairly work so you retain tweaking the immediate and retrying till you get near what you need, or hand over after 100+ tries in frustration.

If you wish to see each’s output I’ve made a compilation video you’ll be able to view on the finish of the article.

PixVerse AI – the one one to not change the artwork in any manner. It animated the cowboy strolling towards the horse completely, and driving the horse completely, together with animating the horse, shadows and extra. It simply utterly failed at animating the character getting on the horse. As a substitute a clone of the character appeared from nowhere on the horse, whereas the unique character nearly actually disappeared into the horses behind.

Krea AI – was the perfect at following the immediate with the cowboy strolling to the horse, mounting it, and driving away. How he obtained on the horse was not in anyway practical. He simply type of floated into place. The entire animation was let down by the cowboy’s art work being noticeably modified and distorted, and the background morphing between bushes showing and disappearing and even an entire new horizon showing. Horse was animated appropriately and walked nicely.

Vivago AI – Truly the subsequent greatest for sustaining the art work model, and even adopted the immediate fairly nicely. Clearly obtained confused with limbs crossing and morphing into one another in that manner AI tends to do very often.

Toon Crafter – Final time I attempted this I could not get it to generate. This time the free model labored however solely produced two seconds of an entire new cowboy morphing out of skinny air and had no thought how you can animate the horses legs. Bizarre as a result of ToonCrafter is meant for this very activity. Do not waste your time. One thing additionally failed and I could not obtain the outcome, so it isn’t in my compilation video.

Kling AI – Gave me two trial makes use of of its Skilled mode and it did comparatively nicely with holding the artwork model and having the cowboy mount the horse in such a manner that, initially, you assume it is appropriate till you notice his leg handed by way of the horses physique.

Pollo AI (Utilizing Kling 1.6) (referral hyperlink) – Pollo AI allows you to strive a bunch of various AI fashions within the one place so I had one other strive with Kling. This one had essentially the most fascinating cowboy mounting the horse approach the place he climbed up the horse’s tail. You’d by no means mount a horse that manner, however nonetheless plausible and the animation on the tail was impressed.

I do not know what Skyreels was pondering however
none of this was requested for in my immediate?

Hedra – identified for essentially the most practical speaking heads, Hedra not too long ago expanded their service to incorporate different AI fashions, making the location a one cease store. I took the chance to strive a bunch of fashions together with Skyreels 12V, Hunyuan 12V, and Minimax 12V, none of which produced something useable. Skyreels specifically launched an entire bunch of random new components to the scene.

Attempting PixVerse Once more with a Detailed Immediate

To be truthful my immediate was very primary however simply interpreted by a human (I hope). Simply to finish the experiment I purchased some credit score for PixVerse so I might strive a extra detailed immediate, after which tweak it till I obtained what I needed. A display screen shot of my remaining immediate is under.

Screen shot of PixVerse with my detailed prompt and start and end frames for my Cowboy riding a horse.

My remaining detailed immediate for PixVerse after seven makes an attempt:
“A cowboy, standing subsequent to his horse, walks in direction of his horse. The horse senses his strategy however stays in place. The cowboy holds onto the saddle and lifts his left leg as much as the saddle stirrup, letting the stirrup take his weight as he mounts the horse. The Cowboy, holding the entrance of the saddle, and horse then experience off at strolling tempo towards the left of body.”

I gave it seven tries, adjusting the immediate a little bit every time. The ultimate model is within the compilation video nevertheless it’s nonetheless not proper. I couldn’t get my cowboy to realistically mount his horse. Additionally, by way of a number of tries the AI determined to have reins randomly seem out of nowhere.

Truthfully, there comes a degree the place re-prompting simply will get ridiculously irritating. I can not think about spending my time crafting prompts to get the precise animation I would like.

AI is sweet if you happen to’re completely satisfied to compromise so much… an actual lot. If you want one thing very particular, notably if it entails sustaining characters, artwork model, and scene consistency, we’re not there but.

The type of tweens AI can deal with, does not actually require AI. If I gave a human animator my two frames and simply mentioned “animate the cowboy getting on the horse and driving away”. Even when the animator had by no means seen anybody get on a horse, or know the way a horse really walks, they’re sensible sufficient to search for reference and provides me precisely what I requested for the primary time.

Nonetheless, it is an intriguing concept that someday we could possibly merely create key moments of a scene, and have AI do all of the animation in between.

Under is the compilation video of output from numerous picture to video AI’s. The second final clip is the ultimate detailed immediate to PixVerse. The final clip is my authentic demonstration animation, animated by me in Cartoon Animator.