class RubyLLM::Moderation

A Moderation holds the result of screening text for potentially harmful content with a provider moderation model. Most code obtains one through RubyLLM.moderate.

result = RubyLLM.moderate("This is a safe message about Ruby programming")
result.flagged?  # => false
result.model     # => "omni-moderation-latest"

Attributes

id [R]

The provider-assigned identifier of the moderation request.

model [R]

The id of the model that performed the moderation.

results [R]

The per-input verdicts, as an array of Result objects, one per moderated input.

Public Class Methods

moderate (input = nil, model: nil, with: nil, provider: nil, assume_model_exists: false, context: nil, provider_options: {}, metadata: nil)

Source

# File lib/ruby_llm/moderation.rb, line 78
def self.moderate(input = nil, # rubocop:disable Metrics/ParameterLists
                  model: nil,
                  with: nil,
                  provider: nil,
                  assume_model_exists: false,
                  context: nil,
                  provider_options: {},
                  metadata: nil)
  attachments = Attachment.wrap(with)
  raise ArgumentError, 'must provide input text, image attachment, or both' if input.nil? && attachments.empty?

  config = context&.config || RubyLLM.config
  model ||= config.default_moderation_model
  model, provider_instance = Models.resolve(model, provider: provider, assume_model_exists: assume_model_exists,
                                                   config: config)
  payload = {
    provider: provider_instance.slug,
    provider_class: provider_instance.class.display_name,
    model: model.id,
    model_info: model,
    input: input,
    attachment_count: attachments.size,
    provider_options: provider_options,
    metadata: metadata
  }

  RubyLLM.instrument('moderation.ruby_llm', payload, config: config) do |event|
    result = provider_instance.moderate(input, model:, with: attachments, provider_options:)
    event[:result] = result
    event[:flagged] = result.flagged?
    result
  end
end

Sends input and optional image attachments to a moderation model and returns a Moderation with the provider’s verdict. Uses the configured default moderation model when model is not given. Pass provider: and assume_model_exists: true to use a model that is not in the registry.

RubyLLM::Moderation.moderate("Your content here")
RubyLLM::Moderation.moderate("User message", model: "omni-moderation-latest")
RubyLLM::Moderation.moderate("Caption", with: "screenshot.png", provider: "openai")

Public Instance Methods

category_scores ()

Source

# File lib/ruby_llm/moderation.rb, line 132
def category_scores
  results.map(&:category_scores).reduce({}) do |merged, scores|
    merged.merge(scores) { |_category, left, right| [left, right].max }
  end
end

Returns the confidence scores across all results, as a hash of category name to a score between 0.0 and 1.0. Keeps the highest score per category when there are multiple results.

result.category_scores["violence"]  # => 0.0001

flagged? ()

Source

# File lib/ruby_llm/moderation.rb, line 114
def flagged?
  results.any?(&:flagged?)
end

Returns true if any input was flagged as potentially harmful, false otherwise.

flagged_categories ()

Source

# File lib/ruby_llm/moderation.rb, line 122
def flagged_categories
  results.flat_map(&:categories).uniq
end

Returns the unique names of the categories flagged across all results.

result.flagged_categories  # => ["harassment", "violence"]