Video Generation

Generate, edit, and extend videos from prompts, images, and audio

After reading this guide, you will know:

  • How to generate a video from a text prompt.
  • How to start a video job and collect the result later.
  • How to animate a still image into a video.
  • How to edit a video or extend a generated clip.
  • How to select video models and pass provider-specific options.
  • How to control polling and timeouts.
  • How to access and save generated video data.
  • How to handle errors during video generation.

Basic Video Generation

Describe the scene and save the result:

video = RubyLLM.animate "A paper boat sailing down a rainy gutter"
video.save "boat.mp4"

animate submits the job, waits for rendering, and returns a RubyLLM::Video. Run it in a background job when your web request should return immediately.

Generating Without Blocking

When you don’t want to hold a thread while the provider renders, use RubyLLM.animate_later. It submits the same job and returns a RubyLLM::VideoJob immediately:

job = RubyLLM.animate_later("A hummingbird hovering in slow motion")

job.id      # => "0eb6910f-a353-4699-9d1e-6a4f7a5b39e2"
job.status  # => :pending
job.done?   # => false

Poll the job whenever it suits you, from a scheduled job or a retry loop:

job.refresh
job.done?      # => true
job.completed? # => true

video = job.video
video.save("hummingbird.mp4")

refresh updates the job state and does nothing after completion. Read job.video when job.completed? is true. It is nil while pending and raises RubyLLM::Error if rendering failed. Use job.wait when you want RubyLLM to handle polling.

Animating a Still Image

Models that support image-to-video take a reference image through with:, the same option chats and image generation use for attachments:

video = RubyLLM.animate(
  "Make the waterfall crash down and slowly pan out",
  model: "grok-imagine-video",
  with: "waterfall.png",
  provider_options: { duration: 5 }
)

with: accepts local files, URLs, and Active Storage attachments. Models that accept two images can use them as the first and last frames:

video = RubyLLM.animate(
  "A sunrise over the mountains",
  model: "luma.ray-v2:0",
  provider: :bedrock,
  with: ["first.png", "last.jpg"]
)

This Bedrock example requires a configured S3 output prefix.

Editing a Video

xAI accepts a video through with: to edit an existing clip:

video = RubyLLM.animate(
  "Change the background to blue",
  model: "grok-imagine-video",
  with: "scene.mp4"
)
video.save "edited.mp4"

xAI edits clips up to 8.7 seconds long and preserves their duration and aspect ratio.

Extending a Video

Use extend: to continue a clip. Pass the Video returned by generation, or a source supported by your provider:

video = RubyLLM.animate(
  "A paper boat floating down a stream",
  model: "grok-imagine-video"
)

longer = RubyLLM.animate(
  "The boat passes under a small bridge",
  model: "grok-imagine-video",
  extend: video,
  provider_options: { duration: 2 }
)
longer.save "longer.mp4"

duration is the added portion for xAI extensions. extend: also works with animate_later and cannot be combined with with:.

Gemini extends videos generated by Veo 3.1. Use veo-3.1-fast-generate-preview and pass the original Video, which retains the generated video’s URI. Gemini adds seven seconds per extension and requires a 720p Veo source generated within the last two days. It does not accept an arbitrary local video for this operation.

For Vertex AI, pass extend: "gs://your-bucket/source.mp4" with a Veo model and credentials that can read the source object.

Choosing Models

By default, RubyLLM uses the model in config.default_video_model. Pass model: to pick another one:

video = RubyLLM.animate(
  "A time-lapse of a city skyline from day to night",
  model: "veo-3.1-fast-generate-preview"
)

You can change the default globally:

RubyLLM.configure do |config|
  config.default_video_model = "grok-imagine-video-1.5"
end

Find video models on the Models page. Pass provider: for hosted deployments; see Model Resolution.

Vertex AI

Vertex Veo models require assume_model_exists: true because they are absent from the bundled registry:

video = RubyLLM.animate(
  "A calm ocean wave at sunset",
  model: "veo-3.1-fast-generate-001",
  provider: :vertexai,
  assume_model_exists: true
)

Configure a region that serves Veo. To store the output in Cloud Storage, pass provider_options: { parameters: { storageUri: "gs://your-bucket/videos/" } }.

Animating a Character with Audio

Creatify Aurora animates a character image using speech audio. Pass one image and one audio file, without a text prompt:

video = RubyLLM.animate(
  with: ["character.png", "speech.wav"],
  model: "creatify-aurora",
  provider: :elevenlabs,
  assume_model_exists: true
)

video.save "character.mp4"

This model requires ElevenLabs Image & Video access. Seedance models can also take image, audio, and video references, with additional model access from ElevenLabs.

For self-hosted video models, configure a GPUStack model proxy, then pass the deployed model name with provider: :gpustack.

Provider Options

Use provider_options: for duration, resolution, and other model settings:

# xAI
RubyLLM.animate(
  "A calm ocean wave at sunset",
  model: "grok-imagine-video",
  provider_options: { duration: 5, resolution: "720p" }
)

# Gemini
RubyLLM.animate(
  "A calm ocean wave at sunset",
  model: "veo-3.1-fast-generate-preview",
  provider_options: { parameters: { durationSeconds: 8, resolution: "1080p" } }
)

Polling and Timeouts

While waiting, animate polls the job on an interval and gives up after a timeout, both configurable:

RubyLLM.configure do |config|
  config.video_generation_timeout = 600      # seconds, default 600
  config.video_generation_poll_interval = 5  # seconds, default 5
end

When the timeout elapses, animate raises RubyLLM::Error. The provider keeps rendering; only the wait stops. Video jobs have no cancellation method. wait also accepts both values per call:

job = RubyLLM.animate_later("A rocket launch seen from orbit")
job.wait(timeout: 900, interval: 10)

Working with Generated Videos

RubyLLM::Video mirrors RubyLLM::Image:

  • video.url: the hosted video URL, for providers that return one. nil otherwise.
  • video.data: the raw video bytes, returned inline or downloaded with your provider’s credentials. nil for hosted URLs.
  • video.mime_type: the MIME type, such as "video/mp4".
  • video.duration: the clip length in seconds, when the provider reports it.
  • video.model: the id of the model that rendered the clip.
  • video.raw: the provider’s raw job response, for provider-specific fields such as reported cost.

Save hosted videos before their URLs expire. save and to_blob work with either URLs or inline data:

video = RubyLLM.animate("A steampunk mechanical owl taking flight")

video.save("owl.mp4")
blob = video.to_blob # => raw MP4 bytes

Rails Active Storage Integration

Use to_blob to attach generated videos to Active Storage attributes:

# app/jobs/generate_trailer_job.rb
class GenerateTrailerJob < ApplicationJob
  def perform(product, prompt)
    video = RubyLLM.animate(prompt)

    product.trailer.attach(
      io: StringIO.new(video.to_blob),
      filename: "#{product.slug}-trailer.mp4",
      content_type: video.mime_type
    )
  end
end

Error Handling

Video generation fails for the same reasons image generation does, plus one of its own: the job itself can fail after it was accepted, for example when the content filter blocks the prompt mid-render. RubyLLM surfaces both as RubyLLM::Error:

begin
  video = RubyLLM.animate("Your prompt here")
rescue RubyLLM::BadRequestError => e
  # The provider rejected the request up front
  puts "Request failed: #{e.message}"
rescue RubyLLM::Error => e
  # The job failed while rendering, or the wait timed out
  puts "Generation failed: #{e.message}"
end

See Error Handling for specific exceptions and recovery options.

Next Steps