Advanced Request Control
Reach provider-specific features with custom parameters, wire protocols, request hooks, and HTTP headers
After reading this guide, you will know:
- How to identify end users to a provider’s abuse tooling with
with_end_user. - How to compact long conversations automatically or when you choose.
- How to keep tool results within the model’s context window.
- How to pass options in the provider’s request vocabulary with
with_provider_options. - How to choose the wire protocol a provider speaks.
- How to modify the final request payload with
before_request. - How to add custom HTTP headers to a request.
Fluent Configuration
Chat configuration methods use a chainable with_* style:
chat = RubyLLM.chat
.with_temperature(0.2)
.with_max_output_tokens(200)
Value setters accept nil to clear a setting. Feature switches accept no arguments to enable their default behavior, options to configure it, and false to disable it:
chat.with_temperature(0.2)
chat.with_temperature(nil)
chat.with_thinking(effort: :high)
chat.with_thinking(false)
chat.with_caching
chat.with_caching(false)
The same feature-switch pattern applies to with_citations and with_compaction. These switches reject nil. with_tools(nil) clears the tool list; other tool settings are covered in Controlling Tool Execution.
Identifying End Users
Providers offer a per-user identifier so they can attribute abuse to one of your users instead of your whole account. with_end_user sets it once and RubyLLM maps it to whatever the provider calls it:
chat = RubyLLM.chat.with_end_user("user-#{current_user.id}")
chat.ask "Hello"
Use an opaque ID such as an account’s public ID or a UUID. The value is sent as given. Providers without support for end-user attribution ignore this setting.
Agents declare it with the matching end_user macro, which also takes a block:
class SupportAgent < RubyLLM::Agent
inputs :account
end_user { account.public_id }
end
Compacting Long Conversations
A conversation that runs long enough overflows the model’s context window and the provider starts rejecting requests. Several providers can handle that themselves: once the conversation crosses a token threshold, the provider condenses the earlier turns and carries on. with_compaction turns it on:
chat = RubyLLM.chat(model: "claude-sonnet-5").with_compaction
chat.ask "Let's go through the whole migration plan."
With no arguments the provider’s own defaults apply. Three provider-neutral options tune it:
chat.with_compaction(at: 50_000)
chat.with_compaction(at: 100_000, instructions: "Keep every decision and open question.")
chat.with_compaction(at: 50_000, pause_after: true)
chat.with_compaction(false)
at: sets the input-token threshold. instructions: guides the summary, and pause_after: stops the turn after compaction. Support differs:
| Provider | Automatic compaction behavior |
|---|---|
| Anthropic and Bedrock Mantle | Summarizes earlier turns; supports all three options, with a minimum threshold of 50,000 tokens |
| OpenAI and Azure Responses | Summarizes earlier turns; supports at: |
| OpenRouter | Drops messages from the middle at the model’s context limit; ignores these options |
| GPUStack (vLLM) | Keeps the last at: tokens of the prompt. Without at:, keeps as many as the model allows when an output limit is set, and otherwise does nothing. Summarizes nothing, can cut the system prompt, and ignores the other options |
Use a model that supports compaction. Unsupported providers ignore the setting. Summarization can incur additional usage and charges, which are included in the response’s tokens and cost.
Agents declare it with compaction:
class ResearchAgent < RubyLLM::Agent
model "claude-sonnet-5"
compaction at: 50_000
end
Compacting Now
Call compact to reduce the context sent on subsequent requests while keeping every message in your transcript:
chat = RubyLLM.chat(model: "grok-4.3", provider: :xai)
chat.ask "The project codename is Thimble. We write it in Ruby."
summary = chat.compact
chat.ask "What is the project codename?"
summary.tokens.input
summary.cost.total
chat.messages # includes the original conversation and the compaction message
Manual compaction works with xAI, OpenAI, Azure, and a local Ollama server through their Responses APIs. For Azure deployments using Chat Completions by default, and always on Ollama, pass protocol: :responses when creating the chat. Other providers, Anthropic included, raise RubyLLM::Error; use with_compaction there. It is separate from with_compaction, which asks the provider to compact automatically during generation.
compact returns a Message whose text can be empty. Later requests use its compacted context while your transcript keeps the original messages. You can still change instructions with with_instructions. The returned message and chat.tokens include compaction usage.
The same call works on an agent or a Rails chat. Rails persists the compacted context and usage, and Agent.find or a reloaded chat can continue from it without deleting earlier messages. Finish any pending tool calls, including approval decisions, before compacting.
Compaction uses the current HTTP headers and before_request hooks. Generation options such as tools, temperature, and provider_options are not forwarded to the compaction endpoint. Keep the conversation on a provider and model that can read its compacted context.
Keeping Tool Results in Budget
Tools return as much as they find: a page of campaigns with their HTML, a whole spreadsheet, a log. One such result can fill the model’s context window, and a few smaller ones add up over a long turn until requests fail. A context budget keeps tool results within shares of the model’s context window. Set it once in your configuration, and every chat and agent uses it:
RubyLLM.configure do |config|
config.context_budget = { per_result: 0.25, per_request: 0.7 }
end
The Rails install generator writes this line into config/initializers/ruby_llm.rb. Without it, chats have no budget. A context can set its own.
With per_result, a tool result longer than that share is cut when it is added, ending with a note telling the model how much it saw and to ask for a narrower query or the next page. The transcript keeps the cut result, so a persisted chat stores it too.
With per_request, a request that would take more than that share leaves out the oldest tool results, each replaced by a one-line note telling the model to call the tool again if it needs it. Only the request changes: chat.messages keeps every result. The results of the latest round go out whenever they fit, since the model hasn’t seen them yet. When that round alone is over the share, its oldest results are left out too, with a note telling the model to ask for less, so the chat can’t get stuck failing every request. The cut point moves in steps of about 15% of the budget rather than one result at a time, so the start of the request stays the same for many rounds and the provider’s prompt cache keeps working.
Shares are of the context window the model registry lists for the model, and sizes are estimated from characters, at two a token, which is how dense JSON and HTML run. Set either share alone.
A chat or an agent with a budget of its own uses that instead. with_context_budget takes the same options, and nil turns the budget off for that chat:
chat = RubyLLM.chat.with_context_budget(per_result: 0.1, per_request: 0.5)
chat = RubyLLM.chat.with_context_budget(nil)
Agents declare it with a macro, which takes nil too:
class ResearchAgent < RubyLLM::Agent
context_budget per_result: 0.1, per_request: 0.5
end
When the window depends on the chat, such as an agent that runs on a model the registry doesn’t know, pass a block. It runs for each chat with the agent’s inputs and chat in scope, and returns the options, or nil for no budget:
class UtilityAgent < RubyLLM::Agent
inputs :model_window
context_budget { { window: model_window, per_request: 0.7 } if model_window }
end
The budget errs on the side of leaving results out. It estimates at two characters a token, so with per_request: 0.7 a request starts losing results at 140% of the window in characters. Prose runs closer to four characters a token, so that’s about 35% of the real window, and a round that would have fit can lose results. Raise per_request if your tools return mostly prose.
It only cuts the text of tool results, and it only measures text and tool call arguments. A request can still overflow from what it can’t cut or doesn’t count: attachments, including those on tool results, long tool call arguments, a long question, and your tool definitions. A provider that refuses such a request raises RubyLLM::ContextLengthExceededError.
For a model the registry has no window for, such as one behind your own endpoint, say what it is with window:, in tokens. It wins over the registry’s window, so set it for one model at a time rather than across fallbacks with very different windows:
chat.with_context_budget(window: 128_000, per_request: 0.7)
Without a known window the budget can’t work out its shares, so it does nothing, and the chat logs a warning once.
With a file store, long results are saved as files first, and the model gets their start and the path, which already fits. The budget still cuts results of tools that opt out with inline_limit false.
Provider Options
Different providers offer unique features and request fields. The with_provider_options method takes options written in the provider’s own request vocabulary and merges them into the request as-is, overriding any defaults set by RubyLLM. It is the escape hatch for anything RubyLLM does not model as a first-class option.
# JSON object mode on the Responses API (OpenAI's default protocol)
chat = RubyLLM.chat.with_provider_options(text: { format: { type: "json_object" } })
response = chat.ask "What is the square root of 64? Answer with a JSON object with the key `result`."
puts JSON.parse(response.content)
# The same option on Chat Completions providers like :ollama and :deepseek
chat = RubyLLM.chat(model: "qwen3", provider: :ollama)
.with_provider_options(response_format: { type: "json_object" })
with_provider_options can override any part of the request payload, including the model, token limits, and tools, and RubyLLM passes it through without validation. Available options vary by provider and model, so consult the provider’s documentation before overriding anything. To see the exact request being sent, set RUBYLLM_DEBUG=true.
Choosing the Wire Protocol
RubyLLM normally selects the protocol for your model and operation. Choose one explicitly when a guide requires a different endpoint, such as Responses for a provider’s hosted tools.
Override it per chat with the protocol: model option, or app-wide with configuration:
chat = RubyLLM.chat(model: "gpt-6.1-sol", protocol: :chat_completions)
chat = RubyLLM.chat(model: "gpt-6.1-sol").with_model("gpt-6.1-sol", protocol: :chat_completions)
RubyLLM.configure do |config|
config.openai_protocol = :chat_completions
end
The protocol: option sits alongside provider: in model selection: a model is identified by its name, its provider, and its protocol. Unknown protocol names raise when the request is rendered or sent, listing the protocols the provider supports. A bare with_model returns the chat to the provider’s default protocol, just as it re-resolves the provider.
Perplexity Presets
Perplexity chat runs on its Agent API. Name a preset, which bundles a model, web search, and instructions, or name a model in provider/model form:
chat = RubyLLM.chat(model: "fast", provider: :perplexity)
chat.ask "What changed in Ruby this week?"
chat = RubyLLM.chat(model: "openai/gpt-5.6-luna", provider: :perplexity)
.with_provider_tools(:web_search)
The presets are fast, low, medium, high, xhigh, and wide-research. They search the web on their own and return citations. A model searches when you add the web_search provider tool. response.cost is the total Perplexity bills, search fees included.
The Agent API accepts images and text files, but not PDFs or other documents.
Perplexity Router
Perplexity’s Router requires separate account access and an explicit protocol. Use it for function tools, tool-choice controls, and cache boundaries:
chat = RubyLLM.chat(model: "perplexity/kimi-k3", provider: :perplexity,
protocol: :router_chat_completions)
.with_tools(SearchDocs)
.with_tool_options(choice: :required, calls: :one)
chat.ask "Find the installation guide."
Router requires tool descriptions and strict JSON schemas. For hosted web search, use a preset or the web_search provider tool on the default Agent API.
Request Hooks
Most of the time you can rely on RubyLLM to format messages for each provider. When a provider ships a block type RubyLLM has not wrapped yet, use before_request to see the fully rendered payload and adjust it before it is sent. The hook runs on every request, after all RubyLLM formatting and with_provider_options merging.
chat = RubyLLM.chat
chat.before_request do |payload|
payload[:messages].last[:content] << { type: "custom_context", data: provider_specific_payload }
end
chat.ask "Analyze this request using the provider-native block above."
Hooks mutate the payload in place; return values are ignored (use payload.replace(new_payload) to swap it wholesale). Use hooks sparingly: they operate on the provider’s wire format, so it is your responsibility to match what the provider expects, and switching providers means revisiting the hook. Nothing a hook adds is persisted; it is applied fresh on each request. For prompt reuse, prefer Prompt Caching.
Chat#render returns the request the chat would send, with hooks applied, which makes hook output easy to inspect and test.
The same idea applies to tool definitions:
class ChangelogTool < RubyLLM::Tool
description "Formats commits into human-readable changelog entries."
parameter :commits, type: :array, description: "List of commits to summarize"
provider_options cache_control: { type: "ephemeral" }
def execute(commits:)
# ...
end
end
Providers that do not understand these extra fields silently ignore them, so you can reuse the same tools across models. See the Tool Provider Parameters section for more detail.
Inspecting the Payload
To see the exact request a chat would send, without sending it, call render. It returns the payload with all formatting, with_provider_options merging, and before_request hooks applied, which makes it handy in tests:
payload = chat.with_provider_options(service_tier: "flex").render
payload[:service_tier] # => "flex"
Custom HTTP Headers
Some providers offer beta features or special capabilities through custom HTTP headers. Use with_headers to add them to your requests.
chat = RubyLLM.chat(model: "claude-sonnet-5")
.with_headers("anthropic-beta" => "fine-grained-tool-streaming-2025-05-14")
response = chat.ask "Tell me about the weather"
Headers are merged with the provider’s own, and the provider’s win. with_headers cannot replace the authentication headers, such as Authorization or x-api-key, or the API version headers, such as anthropic-version, that the provider sets. Use it to add headers for optional features.
Next Steps
- Chat - the core conversation interface.
- Tool Parameters - pass provider-specific options to tool definitions.
- Structured Output - get schema-validated responses instead of raw JSON mode.
- Configuration - set provider protocols and defaults app-wide.