Gen3 KeyFraming (Prototype)
Category
- AI Video
Gen
- Gen-3
Date
2024/12/03
Summary 🍀🧸
Preview of the ability to organically create images and videos from a blank canvas
URL
https://runwayml.com/research/creativity-as-search-mapping-latent-space
URL 1
Empty
URL 2
Empty
Release
Coming Soon
Informer
Empty
Gen-3 KeyFraming (Prototype)
( X Posting Expert )
Today we share an early video keyframing prototype that treats creative exploration as a process of exploration of all potential artistic possibilities, allowing us to simultaneously explore this vast space with precise control and serendipitous nonlinear discovery.
Graph Structure: A Window into the Latent Space
The graph structure is the basis of the prototype. Images are represented as nodes, which act as waypoints in the latent space of the model. These nodes can be connected to other nodes to create edges. Edges are video transitions from the first frame to the last frame through latent space and time.
Balance of control and chance
Precise control helps to limit the vast space of possibilities, but at the same time, variation and unpredictability can lead to “happy accidents” – possibilities that would not have been considered if precise control had been given. To strike this balance, we provide two possibilities for the user to manipulate the image in a “relational” way that allows for unpredictability in a consistent dimension.
Users can transform a selected image via “Image to Image,” which changes the style via text prompts while preserving the original composition, while “Transform Image” changes the composition while maintaining the original style.
Nonlinear search support
Creative exploration rarely follows a straight line. Graph structures naturally encourage exploration by allowing users to branch off at various points, creating new forks of possible alternatives. As more exploration occurs, the graph naturally grows, tracing different paths of experimentation.
This allows users to construct non-linear timelines. We provide a sequencer that allows users to export non-linear timelines as videos with linear timelines, similar to a “choose your own adventure” experience.
Open workspace
Beyond the graph structure, we do not impose any organizational constraints on the workspace. Users have complete freedom to organize nodes and edges, cluster related explorations according to their process needs, or isolate unique creative experiments.
