Connection, Logging and Contexts

Timeouts, retries, proxies, connection reuse, logging, the model registry file, and isolated multi-tenant contexts.

After reading this guide, you will know:

  • Where RubyLLM reads the model registry and how to relocate it.
  • How to tune timeouts, retries, and HTTP proxies.
  • How to reuse connections between calls with a keep-alive adapter.
  • How to configure logging and debug streaming responses.
  • How to create isolated configurations with contexts for multi-tenancy.

Model Registry File

RubyLLM ships with a registry snapshot, so a new installation works without a network request. Plain Ruby applications use the registry in the operating system’s user cache when one exists:

  • Linux: $XDG_CACHE_HOME/ruby_llm/models.json, or ~/.cache/ruby_llm/models.json
  • macOS: ~/Library/Caches/RubyLLM/models.json
  • Windows: %LOCALAPPDATA%\RubyLLM\Cache\models.json

RubyLLM.models.refresh fetches the latest published catalog, merges models discovered from your configured providers, and atomically updates this cache. If the cache cannot be written, it raises RubyLLM::ModelRegistryError and leaves the in-memory registry unchanged.

Set model_registry_file only when your application needs a specific persistent location:

RubyLLM.configure do |config|
  config.model_registry_file = '/var/app/models.json'
end

RubyLLM.models.refresh

RubyLLM falls back to its bundled snapshot until the configured file exists. The first successful refresh creates the file, and later refreshes replace it. Use RubyLLM.models.save_to_json('/another/path/models.json') only when you want to export the currently loaded registry elsewhere.

With the Active Record integration, RubyLLM’s internal ruby_llm_models table is the registry. RubyLLM.models.refresh updates it automatically. While the table is empty, RubyLLM falls back to the registry file, then to the bundled snapshot.

Connection Settings

Timeouts & Retries

Fine-tune how RubyLLM handles network connections:

RubyLLM.configure do |config|
  # Basic settings
  config.request_timeout = 120        # Seconds to wait for response (default: 300)
  config.max_retries = 3              # Retry attempts on failure (default: 3)

  # Advanced retry behavior
  config.retry_interval = 0.1         # Initial retry delay in seconds (default: 0.1)
  config.retry_backoff_factor = 2     # Exponential backoff multiplier (default: 2)
  config.retry_interval_randomness = 0.5  # Jitter to prevent thundering herd (default: 0.5)
  config.retry_max_interval = 30      # Longest delay to honor between retries (default: 30)
end

When a provider reports how long to wait through rate-limit headers (the standard Retry-After, or OpenAI’s x-ratelimit-reset-requests and x-ratelimit-reset-tokens), retries wait that long instead of the backoff interval. If the requested wait exceeds retry_max_interval, the request fails immediately with RubyLLM::RateLimitError rather than sleeping.

These settings cover requests you can safely repeat. Requests that create something on the provider’s side, such as submitting a batch, starting a video generation job, uploading a file, or creating a content cache, are sent once and raise on failure, because a retry after a lost response would create a second job.

Example for high-latency connections:

RubyLLM.configure do |config|
  config.request_timeout = 300        # 5 minutes for complex tasks
  config.max_retries = 5              # More retry attempts
  config.retry_interval = 1.0         # Start with 1 second delay
  config.retry_backoff_factor = 1.5   # Less aggressive backoff
end

HTTP Proxy Support

Route requests through a proxy:

RubyLLM.configure do |config|
  # Basic proxy
  config.http_proxy = "http://proxy.company.com:8080"

  # Authenticated proxy
  config.http_proxy = "http://user:pass@proxy.company.com:8080"

  # SOCKS5 proxy
  config.http_proxy = "socks5://proxy.company.com:1080"
end

Connection Reuse

A new connection costs a TCP and TLS handshake, which can add tens of milliseconds to a fast call. RubyLLM keeps one HTTP client for each provider endpoint and set of connection settings, shared by every call, thread, and fiber in the process. Choose an adapter that keeps connections open, and consecutive calls skip the handshake:

# Gemfile
gem "faraday-net_http_persistent"

require "faraday/net_http_persistent"

RubyLLM.configure do |config|
  config.faraday_adapter = :net_http_persistent
end

The default :net_http adapter opens a new connection for every request. :net_http_persistent keeps a pool that every thread draws from, which suits Puma and Sidekiq. Twenty calls to a local HTTPS server take 2.0 ms each through :net_http, and 0.4 ms through :net_http_persistent, which connects once. On Falcon or Solid Queue fiber workers, use :async_http from the async-http-faraday gem instead. It keeps connections open for as long as the Async reactor that opened them runs, so calls made outside a reactor still connect every time.

Contexts share connections when their connection settings match. Credentials travel with each request, so tenants with their own API keys reuse the same connections. A context that changes an API base, request_timeout, a retry setting, http_proxy, or faraday_adapter gets connections of its own. Each forked process, such as a Puma worker, opens its own connections instead of sharing its parent’s.

Logging & Debugging

Basic Logging

RubyLLM.configure do |config|
  config.log_file = '/var/log/ruby_llm.log'  # Or set RUBYLLM_LOG_FILE
  config.log_level = :info  # :debug, :info, :warn

  # Or use Rails logger
  config.logger = Rails.logger  # Overrides log_file and log_level
end

Log levels:

  • :debug - Detailed request/response information
  • :info - General operational information
  • :warn - Non-critical issues

log_file notes:

  • Defaults to $stdout
  • Can also be set with RUBYLLM_LOG_FILE=/path/to/file.log

Debug logging is also enabled automatically when the RUBYLLM_DEBUG=true environment variable is set. No configuration change is needed.

Setting config.logger overrides log_file and log_level settings.

Advanced Logging Options

Use these options when you need deeper troubleshooting or safer handling of large debug payloads.

RubyLLM.configure do |config|
  config.log_stream_debug = true
  config.log_regexp_timeout = 1.5
end

log_stream_debug notes:

  • Shows chunk-by-chunk streaming internals (accumulator state, parsing, tool chunks)
  • Helps diagnose streaming and response parsing issues
  • Can also be enabled with RUBYLLM_STREAM_DEBUG=true

log_regexp_timeout notes:

  • Applies to regex filters used in request/response debug logging
  • Uses Regexp.timeout
  • Helps bound regex execution time when debug logs contain very large payloads

Built-in debug log redaction:

  • Large base64-like blobs are redacted as [BASE64 DATA]
  • Large embedding arrays are redacted as [EMBEDDINGS ARRAY]

Contexts: Isolated Configurations

Use a context for a tenant’s credentials or a task’s defaults without changing global configuration.

Basic Context Usage

# Global config uses production OpenAI
RubyLLM.configure do |config|
  config.openai_api_key = ENV['OPENAI_PROD_KEY']
end

ctx = RubyLLM.context do |config|
  config.openai_api_key = ENV['ANOTHER_PROVIDER_KEY']
  config.openai_api_base = "https://another-provider.com"
  config.request_timeout = 180
end

ctx_chat = ctx.chat(model: 'gpt-5.6')
response = ctx_chat.ask("Process this with another provider...")

regular_chat = RubyLLM.chat  # Still uses production OpenAI

Individual Operations

A context offers the same entry points as the top-level RubyLLM module, so the rest of the API runs under its configuration too:

image = ctx.paint("a paper boat sailing down a rainy gutter")
image.save("boat.png")

embedding = ctx.embed("Ruby is a joy to write")
transcript = ctx.transcribe("interview.mp3")

paint, animate, animate_later, embed, embed_later, moderate, speak, transcribe, ocr, rerank, upload, download, cache, and mcp all use the context’s keys, endpoints, and connection settings. So do the downloads RubyLLM performs on your behalf: fetching a URL attachment for a chat, or reading image.to_blob and video.to_blob from a provider’s hosted file, goes through the context’s http_proxy and request_timeout rather than the global ones.

Multi-Tenant Applications

class TenantService
  def initialize(tenant)
    @context = RubyLLM.context do |config|
      config.openai_api_key = tenant.openai_key
      config.default_model = tenant.preferred_model
      config.request_timeout = tenant.timeout_seconds
    end
  end

  def chat
    @context.chat
  end
end

# Each tenant gets isolated configuration
tenant_a_service = TenantService.new(tenant_a)
tenant_b_service = TenantService.new(tenant_b)

Key Context Behaviors

  • Inheritance: Contexts start with a copy of global configuration
  • Isolation: Changes don’t affect global RubyLLM.config
  • Coverage: Every entry point the context exposes uses it, chat and non-chat alike, including the file downloads those calls make
  • Thread Safety: Each context is independent and thread-safe
  • Connections: Contexts with the same connection settings share connections, as described in Connection Reuse

Next Steps