AI Shot Creation – Keeping An Open Mind
This is an exploration of a VFX shot creation workflow that uses AI.
A Novel Approach
The Visual Effects Society chapter in New York City, where I'm based, did a deep dive on AI about a year ago. One of the panelists was there to showcase a workflow that involved feeding models what were essentially previs renders to then create a final shot. Leading up to the panel I had been incredibly skeptical about AI's utility in the realm of video generation, (I still am, but more on that later). However, seeing the ways in which a low polygon pass at a scene could be turned into a final shot was intriguing (mainly for that workflow's potential for granular control, something so rare in AI.)
Testing This Out Myself
The other day I saw someone online use Claude Code to generate a 3D scene with Blender. I've been told by many in the VFX world that these AI generated scenes don't hold up in a professional VFX workflow, and I take them at their word, and can imagine why (CG models are typically extremely complicated and built to be passed along a pipeline, many AI models don't have riggers in mind when they create a mesh for example.) That being said, I could see a world in which previs could be a useful output from these models. If the idea is not anything close to final pixel, it could be like VFX's version of a vibe coding to mock up a front end UI in UX design. Claude Code and Blender seemed like a good team up.
Previs being the mission, I came up with a simple prompt:
Create a Blender scene from scratch called spaceship.blend. Let’s track a spaceship as it comes in quite quickly over a mountain lake, hovers for a moment, then deploys tripod landing legs and lands in the shallow water. Its jets should cause turbulence and spray that settle after landing.
Have the camera move backwards and down as trees come into the foreground, ending with the ship framed by cedars, grass and mountains. Don’t linger at a bird’s-eye angle. Make the ship angular and understated, with small landing feet rather than floaties.
After about 10 minutes, I was surprised to be presented with this:
Image shows a screenshot from Blender of an AI generated scene.
Now this is typically where I'd suggest one should stop. I think this is about as far as the usefulness of this workflow really goes. Generating previs is a great tool for a filmmaker to share and illustrate ideas quickly.
...but this is not where I stopped! My curiosity about ref-to-video models that take in references and put out final AI videos was too much.
A Bit of a Gateway Drug
I prepared a style frame to go with my previs that I'd input into Seedance 2.5. Here's that style frame:
Image shows a photo real reference of what I wanted the scene to look like.
After a few minutes, here's what Seedance produced:
Drag the handle to wipe between the Blender previs and the Seedance 2.5 output. (comparison courtesy of Claude Code plugged into Zed :)
Here's the prompt I used for Seedance if you're curious:
Create a 14.9-second photoreal cinematic science-fiction shot using @[Video 1](video_1) as the motion, timing, staging and camera reference, and @[Image 1](image_1) as the visual style, spacecraft identity, materials, lighting and final-composition reference.
REFERENCE PRIORITY
Follow @[Video 1](video_1) for the spacecraft's flight path, speed changes, hover, landing-gear deployment, descent and the continuous camera move. Follow @[Image 1](image_1) for the finished live-action appearance of the ship, forest, mountains and lake. The still depicts the END of the action: begin with the airborne approach shown in the video, not with the ship already landed. Translate the previs into photorealism; do not retain its low-poly surfaces. Use one uninterrupted shot with no cuts.
CAMERA AND ACTION
0–4 seconds: The angular survey spacecraft approaches quickly, banks slightly and brakes into a hover above the lake. The camera smoothly retreats and descends along the previs route, tracking the ship from a low oblique angle. Preserve the rapid approach and continuous camera travel. Do not introduce a bird's-eye pause, orbit, extra zoom or slow motion.
4–7 seconds: The ship holds a brief, controlled hover. Three compact mechanical landing legs unfold from the underside in the same sequence and positions as the previs. Keep exactly three legs, with consistent, believable joints and dimensions.
7–11 seconds: The ship descends steadily. Foreground cedar trunks and branches progressively frame the view as the camera settles toward the reference image's closing composition. The three small landing feet pass beneath the surface and plant firmly on the very shallow lakebed. The spacecraft remains supported above the water; its hull does not touch the surface.
11–14.9 seconds: Hold the completed landing and the final camera composition. The ship is planted and still while residual water disturbances spread and subside.
JET WASH AND WATER PHYSICS — ESSENTIAL
The ship uses downward-facing lift jets to hover and control its descent. Their interaction with the lake must be clearly visible, physically grounded and synchronized to the ship's altitude and thrust. During the approach, a brief disturbed patch follows beneath the craft. During the hover, the downward jets strike the water beneath the underside engine positions, creating localized shallow depressions, rough boiling surface turbulence and fast outward-moving ripples. As the ship lowers, the concentrated jet wash grows stronger: overlapping choppy waves, broken reflections, small patches of aerated water and restrained low sheets of spray and fine mist pushed radially away from the impact zones. Keep the turbulence low to the water so the ship and landing legs remain readable. The disturbance is centered beneath the engines and spreads outward; it is not random storm surf across the whole lake.
At touchdown, add small displacement splashes and distinct contact ripples around each submerged leg. Once the feet carry the ship's weight, the lift jets throttle down and shut off. Active spray and surface boiling cease promptly, while expanding waves, interacting ripples and a little lingering mist gradually dissipate through the final hold. Do not make the water instantly still. Avoid perfectly drawn concentric rings, towering geysers, explosions or a rigid circular wall of foam. The lake stays calm outside the localized landing disturbance. The exhaust itself is mostly invisible, with subtle heat shimmer; emphasize its effect on the water rather than bright flames or glowing beams.
VISUAL FINISH
Match @[Image 1](image_1): convincing live-action large-format cinematography; austere, restrained and quietly imposing science fiction; weathered matte titanium, dark cockpit glazing, practical panel seams and exposed functional mechanisms. Retain the original angular hull silhouette and small submerged landing feet. Preserve the cedar-framed shore, detailed natural grasses, cold reflective lake and monumental rocky mountain arrangement. Use organic bark and foliage, physically plausible rock formations, gentle atmospheric depth, muted mineral greys and olive greens, warm low sidelight and cool ambient fill. Keep deep shadows readable, highlights natural, textures tactile and film grain subtle. Maintain consistent ship geometry, believable scale, correct reflections and coherent lighting throughout.
Sound: deep controlled engine rumble, a restrained rising jet hiss during descent, water spray and lapping waves, subtle landing-gear mechanisms. Engine sound diminishes after weight settles onto the legs, leaving quiet wind and residual water movement. No dialogue or music.
No flotation pads, pontoons, buoyant bobbing, extra legs, changing spacecraft geometry, people, bench, neon trim, cartoon styling, franchise-specific markings, titles, logos or watermarks. Preserve the original design rather than copying a recognizable film vehicle.