Video Generation using Python

In the vast landscape of multimedia technology, the art of video generation stands as a fascinating and innovative endeavor. It involves the dynamic synthesis of visual elements, breathing life into static images through intricate algorithms and models. Video generation has become an integral part of various domains, transcending mere entertainment and permeating fields like marketing, education, and virtual reality. In this article, we delve into the realm of video generation using Python, exploring the mechanisms that bring still images to vibrant motion. In this article, we will discuss how generate videos using Python programming language.

Video generation techniques

There are two most common video generation techniques which are listed below:

• First Order Motion Model: The First Order Motion Model stands at the forefront of video generation techniques, orchestrating a seamless blend of source images and driving videos. By leveraging key-point detection, this model dissects and maps the intricate movements from driving videos onto static source images. The result is a captivating synthesis, where inanimate images come alive with the dynamic rhythm of the accompanying video.
• Dynamic Pixel Transformation: Video generation involves a meticulous transformation of pixels, choreographed by complex algorithms that decode motion patterns. Through pixel-level transformations, the algorithms animate each frame, creating a coherent flow that mimics the motion of the driving video. By understanding this dynamic pixel transformation unveils the technical brilliance behind the creation of lifelike and fluid videos.

Video Generation Step-by-step implementation

Installing required modules

At first, we will install all required python modules to our runtime.

!pip install ffmpeg
!pip install imageio-ffmpeg
!git clone https://github.com/rkuo2000/first-order-model

Importing required libraries

Now we will import all required Python libraries like OS, NumPy, Matplotlib, ImageIO etc.

Python3

import os
os.chdir('first-order-model')
import imageio
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from skimage.transform import resize
from IPython.display import HTML
from demo import make_animation
from skimage import img_as_ubyte
from demo import load_checkpoints

Dataset loading

As per generating video we will download this data. Then we will unzip it for further use.

Python3

# Unzip the downloaded dataset
!unzip -q first-order-motion-model.zip -d /content/first-order-motion-model

Display function

As we are generating video so we will to define a display function to visualize it. We will define a small function which will take source image and driving video then plot the generated video along with them so that we can cross check that how much accurate the video was generated.

Python3

def display(source,driving,generated=None):
    fig=plt.figure(figsize=(8+4*(generated is not None),6))

    ims=[]
    for i in range(len(driving)):
        cols=
        cols.append(driving[i])
        if generated is not None:
            cols.append(generated[i])
        im = plt.imshow(np.concatenate(cols, axis=1), animated=True)
        plt.axis('off')
        ims.append([im])
    ani=animation.ArtistAnimation(fig,ims,interval=50,repeat_delay=1000)

    plt.close()
    return ani

Model training

Now we will load a pretrained first order motion model. Then we will pass any source image and driving video to generate new video. Finally, we will save it to our desired location.

Python3

generator, kp_detector = load_checkpoints(config_path='config/vox-256.yaml',
                            checkpoint_path='first-order-motion-model/vox-cpk.pth.tar')
source_image=imageio.v2.imread('Ariana.jpg')
driving_video=imageio.mimread('Leonardo.mp4')

#resize
source_image=resize(source_image,(256,256))[...,:3]
driving_video=[resize(frame, (256, 256))[..., :3] for frame in driving_video]

predictions=make_animation(source_image,driving_video,generator,kp_detector, relative=True )

imageio.mimsave('generated_video.mp4', [img_as_ubyte(frame) for frame in predictions]) # generated video saving loaction

Visualizing the generated video with input

Now we will call our display function to visualize the generated video with its inputs to understand the versability of the generated video.

Python3

# visulaizing video with inputs
HTML(display(source_image, driving_video, predictions).to_html5_video())

Output:

generated video with inputs

So, in this output we have displayed the generated video (right most) with the input image (left most) and the driving video (middle one). However, you can also visualize and download the raw-only generated video (MP4 format) from the saved path.

Applications of video generation

Some of the real-world applications of video generation discussed below:

1. Educational Content Creation: Video generation finds utility in education, where it breathes life into educational content by adding motion to static visuals. Complex concepts can be conveyed with greater clarity as images come alive, providing a more engaging learning experience.
2. Marketing and Advertisement: Dynamic advertisements come to life through video generation which offers marketers a powerful tool to engage audiences with visually compelling content.
3. Virtual Reality: In the realm of virtual reality, video generation enhances the realism of simulated environments by infusing them with dynamic motion to make virtual worlds more immersive as static images transform into vibrant, moving elements, enriching the user experience.
4. Scientific and Medical Imaging: Beyond the creative realms, video generation contributes to scientific simulations and medical imaging by facilitating the dynamic representation of data. Simulations like molecular dynamics, gain a visual dimension, aiding researchers and educators in conveying intricate concepts.