Share
Sign In
A Hitchhiker's Guide to AI
Part 5. Video Prompt: Borrowing from the Movies
[Image] Bison from Lascaux Cave Painting and Picasso's Bull
Where did our visual language begin? The buffalo depicted in the Lascaux cave paintings were vividly depicted as if they were alive, but Picasso's bulls were simplified to leave only the essentials. We can get various hints from the similar yet different works of mankind.
[Video] Picasso's Bull, Remastered Drawing by Luma
[Video] Picasso's Bull, Remastered Drawing by Gen-3
Sometimes realistically, sometimes abstractly, humans have used a variety of visual languages to express their daily lives and thoughts.
From cave paintings to paintings, photography to film and computer graphics, silent films to sound films, black and white to color, humanity's visual media and forms, and the ways in which we express and compress visual information, have constantly evolved.
If you’ve already witnessed the massive shift from television to cable, YouTube, Netflix, and then live streaming and shorts, you probably know that we’re on the threshold of a new era of video creation, with our partner in AI.
The tool called photography brought light and color, composition, and interpretation of reality from the traditional field of painting. Later, movies borrowed various framing techniques, lighting, lenses, and techniques of reproducing reality from the field of photography. In this way, the next generation moves forward with gratitude and debt to the previous generation.
And now it's time for us to borrow something from 21st century cinema.
From the wonderful film industry
The film industry is a field that most dynamically pursues technical perfection and artistic expression, with top-level experts in various fields backed by huge capital.
The AI video process we will experience is actually a small microcosm of this huge film industry. At the same time, it is also an abridged version or extract of all the visual art languages that mankind has accumulated. Now, this huge and dense industry can be used really simply and at low cost by individuals and small teams with low capital.
Although we have not directly participated in Hollywood projects, we can infer the production process to some extent. We have watched numerous movies, dramas, comedies, Netflix, and YouTube. So, the following will be easy to understand.
First, let's compress this massive process into three simple steps.
3 steps to making a movie
1.
Planning Stage (Pre-Production)
Story Development: Scenario, Script
Visual Development: Storyboarding, location scouting, set and prop design and production
Planning: Research, Budget Planning, Schedule Planning, Line Planning, Team Building, Casting
2.
Production Phase
Preparation: Set setup, lighting and prop preparation, transportation, on-site management, sound production
Acting: Script reading, rehearsals, acting and extras
Cinematography: Directing, camera work, on-site monitoring, data management
3.
Editing Stage (Post-Production)
Video Editing: Special Effects, Cut Editing, Scene Transitions
Post-production: color correction, post-recording, sound mixing, adding subtitles and graphics.
Completion: Final editing, marketing, distribution, screening
The complex process above involves a large number of specialized personnel, including:
Film production professionals
Production and Directing: Producer, Producer, Director, Assistant Director, Line Producer, Casting, Transportation Team
Cinematography and Lighting: Director of Photography, Camera Operator, Lighting Director, Key Grip, Grip
Audio Sound: Sound Director, Sound Designer, Boom Operator, Sound Mixer
Art & Costume: Art Director, Costume Designer, Prop Manager, Set Designer
Post-production and editing: Film editor, VFX artist, sound editor
[Video] the Red Bull by Gen-3
But here’s where AI really comes into play in the realm of so many experts. All the video generation prompts we’ll be using in the future are actually derived from the know-how of the film experts above.
Video prompts borrowed from the movie
For example, the keywords they used on set:
Long Shot / Medium Shot / Close-Up: Framing techniques that determine the size of the scene and the proportion of the character.
Zoom In / Zoom Out: The effect of moving closer to or further away from a subject using the camera lens.
Dolly In / Dolly Out: Move the camera itself closer to or further away from the subject.
Pan Shot / Tilt Shot: Follow the scene by rotating the camera left and right (Pan) or up and down (Tilt).
Crane Shot: Using a crane to raise or lower the camera and take a shot.
Steadicam Shot: Smooth moving shots using a stabilizer
Over the Shoulder Shot: A three-dimensional composition in which one person is shot over the shoulder of another person.
Point of View (POV) Shot: A shooting method that shows the point of view of a specific person.
High Angle / Low Angle: Adjust the mood of the scene by shooting from a high or low position of the camera.
Dutch Angle: tilting the camera to create an unstable or tense atmosphere.
High Key Lighting / Low Key Lighting: Create a bright or dark atmosphere by adjusting the intensity of the lighting.
BackLit: A silhouette effect that illuminates the subject from behind.
These expert prompts will help you communicate your intent more accurately and create the video you want.
If you know any cinematography or editing terms, try to actively use them in the AI video generation process. You will be more likely to get the results you want.
With AI partners
Our AI partners are streamlining the filmmaking process and significantly lowering the technological barriers, specifically in the following areas:
AI Partner Collaboration Areas
Image and video AI: storyboard writing, actor casting, extra recruitment, location scouting, set building, prop making, costume making, hair and makeup, transportation, acting direction from director, acting by veteran actors, lighting direction, camera work, drone shooting, on-site monitoring, VFX production, simple cut editing, post-production, etc.
Sound AI: Background music BGM, original soundtrack OST, sound effects, special sound sources, voice generation, script reading, voice synthesis, etc.
LLM AI: Scenarios, scripts, planning, research, technical advice, image and video prompt support, etc.
Almost every area.
Image and video AI offers tremendous opportunities for individuals and small teams, especially by simply eliminating on-site shooting, which can save enormous time and space costs.
The 15-minute AI musical drama I participated in, called [Mateo], was also completed entirely with AI, from visuals to sound, without any on-site filming. (Released in mid-October)
Create all the frames you want to direct in video as Midjourney images first, and build a storyboard. You can direct impressive compositions without real actors and extras, and have them act repeatedly from various angles. You can easily set up not only overseas locations, but also sets in space or virtual spaces, and you can direct unprecedented costumes, makeup, and special effects in human history with just a few prompts. You can suddenly change day and night as needed, change the actor's movements and emotions instantly, and even change the camera filter in the editing stage.
AI is not replacing existing systems entirely, but rather lowering barriers, reducing costs, and making new attempts more possible.
Now, before we move on to the next chapter in earnest, I would like to express my respect and gratitude to all the artists who have worked in the fields of film, photography, painting, and cave painting.
[Video] handmade cave painting by Gen-3
Process of making a video
When I teach generative video, I often see students’ eyes filled with anticipation that the entire production process will be completed in an instant with the help of AI. Of course, videos are easy to create, but that alone does not mean that everything is complete.
Just like the film industry production stages we looked at earlier, AI video also goes through a series of steps.
Before using a video creation tool, you need to plan the story, draw a storyboard, and create various images that you need. Then, you create a video with prompts. Even a well-created video needs to be edited at least a little. If necessary, you can adjust the sound or lip sync, and create a new video again to maintain the emotional line.
So, please be sure to check the steps below.
[1] Planning stage (Pre-Production)
Most students are in a hurry and start creating videos right away. I did the same. However, before creating videos, I hope you will carry out the planning stage systematically as much as possible.
First, we conceptualize and prepare the video concept. We clearly define the purpose and audience of the video, and refine the specific message we want to convey.
With image creation tools like Midjourney, you can create professional-level image references. You can visualize every scene as a storyboard with detailed information about the character’s appearance, clothing, expressions and movements, and even the emotional expressions.
[2] Production, Generating
Now it's time to actually use AI tools like Gen-3 to generate videos. There are two ways to do this: using text prompts to generate videos (T2V) and using pre-prepared image references to create videos (I2V). Video-to-video (V2V) technology, which transforms the style of an already prepared video, is also starting to be offered.
The T2V method is a basic method for describing a video with text, but it is difficult and has limitations in describing all elements of the video with text. This is similar to the difficulty of generating images from text (T2I).
The I2V method is a method of creating a video using an image creation tool, which allows you to clearly control the image and concept of the video. It has the advantage of being able to directly control the placement of the subject, the appearance and movement of the character, changing elements, color, and style.
V2V technology is still in its early stages, working primarily at the level of style conversion of videos. It is now available in Gen-3.
You can create richer, more detailed videos by combining multiple methods. For example, you can use text to tell the overall story and changes, and use images to set the key tone and manner.
Most AI tools these days support the function of converting images to videos (I2V), which makes it possible to use beautiful images as references. Using the image prompt writing technique learned in the previous step, you can create a beautiful image as a prompt, and then generate a video based on it to create a richer and more consistent video.
Especially if you utilize Midjourney's SREF and CREF functions, it really helps maintain a consistent tone and mood and the same character in the video. I think 80% of the visual beauty that AI videos show is already determined by the image reference.
[3] Editing Stage (Post-Production)
While the video creation process may seem like it would be complete with AI assistance, in reality it requires minimal editing of the generated video and syncing of sound as needed.
[Video] push in rough breathing by Gen-3
Audio adds emotion and depth to your videos. Appropriate background music maximizes the mood of the scene, and sound effects add realism. Using AI music generation tools such as Suno and Udio, you can create a unique soundtrack tailored to your videos.
Editing a video using professional video editing software is like refining your material to create a perfect sculpture. Use Adobe Premere Pro, FinalCut Pro, or the free tool CapCut to cut out unwanted parts, adjust speed, smooth transitions between scenes, and color correct to create a consistent overall tone and mood for your video.
Murals, paintings, photography, film, and next
James Cameron, the film industry master known for 'Terminator', 'Titanic', 'Alien', and 'Avatar', recently joined the board of directors of Stability AI, which is quite remarkable news (2024.09.24). This is because Stability AI is the developer of Stable Diffusion, the most powerful AI image generation tool, along with Midjourney.
Having already led Hollywood blockbusters with cutting-edge technology and innovative visual effects, choosing AI now may seem like a natural choice for him. However, given that sensitive issues such as AI-related strikes still remain, it is also a very bold choice compared to the fact that most of the film industry has been quite passive about AI.
Will we soon see Cameron's 'Avatar 4' produced with AI? And will AI tools pose any problem in front of the grand story he has shown? Of course, it seems only natural that film professionals will use video AI better.
This is not just a technological advancement, but a change in the way we tell stories and share experiences. We are now in an era where we can all routinely express our stories through video and communicate with the world.
[Video] The Man Facing a Bull by Gen-3
In the next installment, we'll dive deeper into video prompts, exploring how the same prompts used in movies can be used in AI video generation, and how to use them.
🍀 AI visual director who visualizes imagination and ideas, Mintbear
Gen-3 Video Prompt Book ➡️ https://slashpage.com/gen3