# Batches {: .d-inline-block .no_toc } New in 2.0 {: .label .label-green } Answer thousands of chats at half price with provider-side batch processing {: .fs-6 .fw-300 } ## Table of contents {: .no_toc .text-delta } 1. TOC {:toc} --- After reading this guide, you will know: * How to stage questions with `ask_later` * How to submit chats as a batch with `RubyLLM.batch` * How to check on a batch and collect its messages, from any process * How to handle tool calls in batched conversations * How to persist batch results with ActiveRecord ## What are Batches? Providers process batched requests asynchronously on their own hardware schedule (usually within an hour; processing ends within 24, and anything still unfinished comes back expired) and charge half price for the privilege. Batches are the right tool whenever nobody is waiting on the answer: nightly classification runs, bulk summarization, evaluations, backfills. A batch in RubyLLM is an array of chats, each ending on an unanswered question. Everything in the request (model, instructions, history, schemas, temperature, attachments) rides along, so there is nothing new to learn about building requests. The one exception is `with_headers`: batch APIs have no per-request HTTP headers, so custom headers set on a chat don't apply to batched requests. ## Staging Questions `ask_later` is `ask` without the waiting: it adds your question to the conversation and returns the chat, leaving it awaiting a response. ```ruby chat = RubyLLM.chat(model: "claude-haiku-4-5").with_instructions("Be terse.").ask_later("What is 2 + 2?") chat.complete? # => false, the model still owes a response ``` ## Submitting a Batch Pass the staged chats to `RubyLLM.batch`. Submission happens immediately and returns a `RubyLLM::Batch`: ```ruby chats = documents.map do |doc| RubyLLM.chat(model: "claude-haiku-4-5") .with_instructions("Summarize the document in one paragraph.") .ask_later(doc.text) end batch = RubyLLM.batch(chats) batch.id # => "msgbatch_01EhcDuvb5XfWqcdJArbsfNX" batch.status # => "in_progress" ``` Chats in one Anthropic or xAI batch can use different models, instructions, schemas, and parameters; each request stands alone: ```ruby chats = tickets.map do |ticket| RubyLLM.chat(model: ticket.urgent? ? "claude-sonnet-4-5" : "claude-haiku-4-5") .with_instructions("You are #{ticket.team} support.") .ask_later(ticket.body) end batch = RubyLLM.batch(chats) ``` OpenAI, Azure OpenAI, Mistral, Gemini, Vertex AI, and Bedrock batch jobs are model-scoped, so those providers require one model per batch. Split mixed-model work into one batch per model. One provider per batch, though: submitting chats from different providers raises `ArgumentError`. ## Collecting the Answers Persist `batch.id` and walk away. From any process, any time later, look the batch up by id with `RubyLLM::Batch.find`: ```ruby batch = RubyLLM::Batch.find("msgbatch_01EhcDuvb5XfWqcdJArbsfNX", provider: :anthropic) batch.complete? # => true ``` `Batch.find` uses the global configuration. Pass `context:` to use an isolated [configuration context](/next/configuration-connection/) instead: ```ruby batch = RubyLLM::Batch.find(batch_id, provider: :anthropic, context: ctx) ``` `complete?` reads the batch's last known state without contacting the provider. In a long-running process, poll with `refresh`, which re-fetches the state from the provider and returns the batch: ```ruby sleep 60 until batch.refresh.complete? ``` Once processing ends, `messages` returns the responses in submission order: ```ruby batch.messages.each do |message| message.content # => "The document describes..." message.input_tokens # => 514 end ``` When you still hold the submitted chats, each message is also appended to its conversation, so the chats come back complete and ready to continue: ```ruby chats.first.messages.map(&:role) # => [:system, :user, :assistant] chats.first.ask "Shorter, please." ``` Requests can fail individually (or expire at the 24-hour deadline) without failing the batch. Failed slots are `nil` in `messages` (details go to the log), and their chats stay awaiting a response; resubmit them in a fresh batch or finish them synchronously with `complete`. Batch results arrive as JSONL rather than individual HTTP responses, so `message.raw` on a batch message is the provider result body hash, not a Faraday response with `status` or `headers`. You can stop a running batch with `batch.cancel`; already-processed requests still return results. {: .note } Token usage on batch results is billed at half the standard rates. `message.cost` doesn't know about batch pricing yet and reports standard rates. ## Tools in Batches A batch generates one model turn. When the model asks for a tool, the round ends there (providers can't call your Ruby code) and the response comes back with `tool_call?` true. You drive the rest of the [agentic loop](/next/agentic-workflows/#driving-the-loop-yourself) yourself between rounds: * `chat.complete` runs the tools and finishes the conversation synchronously, at standard prices. * `chat.run_tools` runs the tools and stops, leaving the chat ready for the model again, i.e. for the next batch. Looping `run_tools` into fresh batches runs entire agentic workloads at batch prices, one model turn per round: ```ruby chats = tickets.map { |t| support_chat(t).ask_later(t.body) } loop do chats.each(&:run_tools) pending = chats.reject(&:complete?) break if pending.empty? batch = RubyLLM.batch(pending) sleep 60 until batch.refresh.complete? # batch.messages appends each answer to its chat; drop chats whose request # failed (a nil slot) so the loop can terminate. chats -= pending.zip(batch.messages).filter_map { |chat, message| chat unless message } end ``` `run_tools` does nothing on chats without pending tool calls, and `reject(&:complete?)` keeps the chats heading into another round while finished conversations drop out. For tools that need human approval before acting, don't try to pause the loop. Ask inside the tool and return the outcome as its result, so the conversation stays valid: ```ruby def execute(id:) return "Approval not given; the user declined." unless approved?(id) Record.find(id).destroy! "Deleted record #{id}." end ``` ## Rails Integration Batch results flow through the same callbacks as synchronous responses, so `acts_as_chat` persistence works unchanged. `ask_later`, `run_tools`, and `complete?` all work on your records, so staged questions and collected answers land in the database with tokens and model attached. The one new thing a batch needs is somewhere to keep its id while the provider works. The installer generates a `Batch` model for exactly that: ```ruby class Batch < ApplicationRecord acts_as_batch end ``` It stores only the provider's batch id and the chats it's processing; the conversations themselves stay in your existing `chats` and `messages` tables. (Upgrading an app from 1.x? `bin/rails generate ruby_llm:upgrade` adds the `batches` table.) `Batch.create!` sends the staged chats to the provider and persists the record in one step: ```ruby chats = tickets.map do |ticket| Chat.create!(model: "claude-haiku-4-5").ask_later(ticket.body) end batch = Batch.create!(chats: chats) BatchPollJob.perform_later(batch.id) ``` A job in another process looks the batch up, checks on it, and collects: ```ruby class BatchPollJob < ApplicationJob def perform(batch_id) batch = Batch.find(batch_id) return self.class.set(wait: 10.minutes).perform_later(batch_id) unless batch.refresh.complete? batch.messages end end ``` `batch.messages` appends each answer to its chat and persists it, so the conversations come back complete with no bookkeeping on your side. It is idempotent: an answered chat ends on an assistant message, so re-running the job (a retry, an at-least-once queue) never appends an answer twice. Each `refresh` caches the provider's status onto the record, so the batches still in flight are just `Batch.where(completed: false)`, and a sweep is `Batch.where(completed: false).find_each(&:refresh)`. Stop a running batch with `batch.cancel`. Tools work the same way they do for plain chats. Because the records carry the whole conversation, a poll job can `run_tools` on the collected chats and submit the ones still awaiting the model as the next batch, running an agentic workload across batches at batch prices. ## Provider Notes * **Anthropic:** up to 100,000 requests or 256 MB per batch. Mixed models in one batch are supported. Request validation is asynchronous: a malformed request comes back as a failed result after the batch ends, not as a submission error. Results stay downloadable for 29 days. * **OpenAI:** uses the file-backed Batch API. RubyLLM supports Responses and Chat Completions payloads, and enforces OpenAI's one-model-per-file rule. Provider files are also available through `RubyLLM.upload` and `RubyLLM.download`. * **Azure OpenAI / Foundry:** uses the OpenAI-style file-backed batch workflow under `/openai/v1`. Your Azure deployment must be a batch-capable deployment type. Provider files are also available through `RubyLLM.upload` and `RubyLLM.download`. * **Mistral:** uses inline batch jobs for Chat Completions. One model per batch is required. Mistral provider files are available through `RubyLLM.upload` and `RubyLLM.download`. * **Gemini:** uses inline `generateContent` batches. One model per batch is required. * **Vertex AI:** uses `batchPredictionJobs` with Google Cloud Storage through the same storage-backed file protocol as `RubyLLM.upload`. Configure `vertexai_batch_gcs_uri` with a `gs://bucket/prefix`; the configured credentials need permission to create batch prediction jobs and read/write that bucket. RubyLLM supports Vertex Gemini, Anthropic Claude, and MaaS chat batches; Vertex-hosted Mistral batches are not wired yet. * **Bedrock:** uses Model Invocation Jobs with Converse payloads and S3 through the same storage-backed file protocol as `RubyLLM.upload`. Configure `bedrock_batch_s3_uri` and `bedrock_batch_role_arn`; the configured static keys or `bedrock_credential_provider` need permission to submit the job and write the input object, while the batch role must allow Bedrock to read the input prefix and write results. Bedrock batch inference does not support tools or structured output, so RubyLLM rejects those requests before submission. * **xAI:** uses native batch containers with chat completion requests. Each request carries its own model, and results are paginated and can be collected before every request has finished. xAI provider files are available through `RubyLLM.upload` and `RubyLLM.download`. * **Other providers:** not supported by RubyLLM batches yet. `RubyLLM.batch` raises `RubyLLM::Error` for providers without batch support. ## Next Steps * [Chatting with AI Models](/next/chat/) * [Using Tools](/next/tools/) * [Rails Integration](/next/rails/)

Batches

New in 2.0

Answer thousands of chats at half price with provider-side batch processing

What are Batches?
Staging Questions
Submitting a Batch
Collecting the Answers
Tools in Batches
Rails Integration
Provider Notes
Next Steps

After reading this guide, you will know:

How to stage questions with ask_later
How to submit chats as a batch with RubyLLM.batch
How to check on a batch and collect its messages, from any process
How to handle tool calls in batched conversations
How to persist batch results with ActiveRecord

What are Batches?

Providers process batched requests asynchronously on their own hardware schedule (usually within an hour; processing ends within 24, and anything still unfinished comes back expired) and charge half price for the privilege. Batches are the right tool whenever nobody is waiting on the answer: nightly classification runs, bulk summarization, evaluations, backfills.

A batch in RubyLLM is an array of chats, each ending on an unanswered question. Everything in the request (model, instructions, history, schemas, temperature, attachments) rides along, so there is nothing new to learn about building requests. The one exception is with_headers: batch APIs have no per-request HTTP headers, so custom headers set on a chat don’t apply to batched requests.

Staging Questions

ask_later is ask without the waiting: it adds your question to the conversation and returns the chat, leaving it awaiting a response.

chat = RubyLLM.chat(model: "claude-haiku-4-5").with_instructions("Be terse.").ask_later("What is 2 + 2?")
chat.complete? # => false, the model still owes a response

Submitting a Batch

Pass the staged chats to RubyLLM.batch. Submission happens immediately and returns a RubyLLM::Batch:

chats = documents.map do |doc|
  RubyLLM.chat(model: "claude-haiku-4-5")
    .with_instructions("Summarize the document in one paragraph.")
    .ask_later(doc.text)
end

batch = RubyLLM.batch(chats)
batch.id     # => "msgbatch_01EhcDuvb5XfWqcdJArbsfNX"
batch.status # => "in_progress"

Chats in one Anthropic or xAI batch can use different models, instructions, schemas, and parameters; each request stands alone:

chats = tickets.map do |ticket|
  RubyLLM.chat(model: ticket.urgent? ? "claude-sonnet-4-5" : "claude-haiku-4-5")
    .with_instructions("You are #{ticket.team} support.")
    .ask_later(ticket.body)
end

batch = RubyLLM.batch(chats)

OpenAI, Azure OpenAI, Mistral, Gemini, Vertex AI, and Bedrock batch jobs are model-scoped, so those providers require one model per batch. Split mixed-model work into one batch per model.

One provider per batch, though: submitting chats from different providers raises ArgumentError.

Collecting the Answers

Persist batch.id and walk away. From any process, any time later, look the batch up by id with RubyLLM::Batch.find:

batch = RubyLLM::Batch.find("msgbatch_01EhcDuvb5XfWqcdJArbsfNX", provider: :anthropic)
batch.complete? # => true

Batch.find uses the global configuration. Pass context: to use an isolated configuration context instead:

batch = RubyLLM::Batch.find(batch_id, provider: :anthropic, context: ctx)

complete? reads the batch’s last known state without contacting the provider. In a long-running process, poll with refresh, which re-fetches the state from the provider and returns the batch:

sleep 60 until batch.refresh.complete?

Once processing ends, messages returns the responses in submission order:

batch.messages.each do |message|
  message.content       # => "The document describes..."
  message.input_tokens  # => 514
end

When you still hold the submitted chats, each message is also appended to its conversation, so the chats come back complete and ready to continue:

chats.first.messages.map(&:role) # => [:system, :user, :assistant]
chats.first.ask "Shorter, please."

Requests can fail individually (or expire at the 24-hour deadline) without failing the batch. Failed slots are nil in messages (details go to the log), and their chats stay awaiting a response; resubmit them in a fresh batch or finish them synchronously with complete.

Batch results arrive as JSONL rather than individual HTTP responses, so message.raw on a batch message is the provider result body hash, not a Faraday response with status or headers.

You can stop a running batch with batch.cancel; already-processed requests still return results.

Token usage on batch results is billed at half the standard rates. message.cost doesn’t know about batch pricing yet and reports standard rates.

Tools in Batches

A batch generates one model turn. When the model asks for a tool, the round ends there (providers can’t call your Ruby code) and the response comes back with tool_call? true. You drive the rest of the agentic loop yourself between rounds:

chat.complete runs the tools and finishes the conversation synchronously, at standard prices.
chat.run_tools runs the tools and stops, leaving the chat ready for the model again, i.e. for the next batch.

Looping run_tools into fresh batches runs entire agentic workloads at batch prices, one model turn per round:

chats = tickets.map { |t| support_chat(t).ask_later(t.body) }

loop do
  chats.each(&:run_tools)
  pending = chats.reject(&:complete?)
  break if pending.empty?

  batch = RubyLLM.batch(pending)
  sleep 60 until batch.refresh.complete?

  # batch.messages appends each answer to its chat; drop chats whose request
  # failed (a nil slot) so the loop can terminate.
  chats -= pending.zip(batch.messages).filter_map { |chat, message| chat unless message }
end

run_tools does nothing on chats without pending tool calls, and reject(&:complete?) keeps the chats heading into another round while finished conversations drop out.

For tools that need human approval before acting, don’t try to pause the loop. Ask inside the tool and return the outcome as its result, so the conversation stays valid:

def execute(id:)
  return "Approval not given; the user declined." unless approved?(id)

  Record.find(id).destroy!
  "Deleted record #{id}."
end

Rails Integration

Batch results flow through the same callbacks as synchronous responses, so acts_as_chat persistence works unchanged. ask_later, run_tools, and complete? all work on your records, so staged questions and collected answers land in the database with tokens and model attached.

The one new thing a batch needs is somewhere to keep its id while the provider works. The installer generates a Batch model for exactly that:

class Batch < ApplicationRecord
  acts_as_batch
end

It stores only the provider’s batch id and the chats it’s processing; the conversations themselves stay in your existing chats and messages tables. (Upgrading an app from 1.x? bin/rails generate ruby_llm:upgrade adds the batches table.)

Batch.create! sends the staged chats to the provider and persists the record in one step:

chats = tickets.map do |ticket|
  Chat.create!(model: "claude-haiku-4-5").ask_later(ticket.body)
end

batch = Batch.create!(chats: chats)
BatchPollJob.perform_later(batch.id)

A job in another process looks the batch up, checks on it, and collects:

class BatchPollJob < ApplicationJob
  def perform(batch_id)
    batch = Batch.find(batch_id)
    return self.class.set(wait: 10.minutes).perform_later(batch_id) unless batch.refresh.complete?

    batch.messages
  end
end

batch.messages appends each answer to its chat and persists it, so the conversations come back complete with no bookkeeping on your side. It is idempotent: an answered chat ends on an assistant message, so re-running the job (a retry, an at-least-once queue) never appends an answer twice. Each refresh caches the provider’s status onto the record, so the batches still in flight are just Batch.where(completed: false), and a sweep is Batch.where(completed: false).find_each(&:refresh). Stop a running batch with batch.cancel.

Tools work the same way they do for plain chats. Because the records carry the whole conversation, a poll job can run_tools on the collected chats and submit the ones still awaiting the model as the next batch, running an agentic workload across batches at batch prices.

Provider Notes

Anthropic: up to 100,000 requests or 256 MB per batch. Mixed models in one batch are supported. Request validation is asynchronous: a malformed request comes back as a failed result after the batch ends, not as a submission error. Results stay downloadable for 29 days.
OpenAI: uses the file-backed Batch API. RubyLLM supports Responses and Chat Completions payloads, and enforces OpenAI’s one-model-per-file rule. Provider files are also available through RubyLLM.upload and RubyLLM.download.
Azure OpenAI / Foundry: uses the OpenAI-style file-backed batch workflow under /openai/v1. Your Azure deployment must be a batch-capable deployment type. Provider files are also available through RubyLLM.upload and RubyLLM.download.
Mistral: uses inline batch jobs for Chat Completions. One model per batch is required. Mistral provider files are available through RubyLLM.upload and RubyLLM.download.
Gemini: uses inline generateContent batches. One model per batch is required.
Vertex AI: uses batchPredictionJobs with Google Cloud Storage through the same storage-backed file protocol as RubyLLM.upload. Configure vertexai_batch_gcs_uri with a gs://bucket/prefix; the configured credentials need permission to create batch prediction jobs and read/write that bucket. RubyLLM supports Vertex Gemini, Anthropic Claude, and MaaS chat batches; Vertex-hosted Mistral batches are not wired yet.
Bedrock: uses Model Invocation Jobs with Converse payloads and S3 through the same storage-backed file protocol as RubyLLM.upload. Configure bedrock_batch_s3_uri and bedrock_batch_role_arn; the configured static keys or bedrock_credential_provider need permission to submit the job and write the input object, while the batch role must allow Bedrock to read the input prefix and write results. Bedrock batch inference does not support tools or structured output, so RubyLLM rejects those requests before submission.
xAI: uses native batch containers with chat completion requests. Each request carries its own model, and results are paginated and can be collected before every request has finished. xAI provider files are available through RubyLLM.upload and RubyLLM.download.
Other providers: not supported by RubyLLM batches yet. RubyLLM.batch raises RubyLLM::Error for providers without batch support.