class RubyLLM::Moderation

A Moderation holds the result of screening text or images for potentially harmful content. Most code obtains one through RubyLLM.moderate.

result = RubyLLM.moderate("This is a safe message about Ruby programming")
result.flagged?  # => false

RubyLLM.moderate(with: "profile.png").flagged?

Attributes

id [R]

The provider-assigned identifier of the moderation request.

model [R]

The id of the model that performed the moderation, or nil for an operation that does not select a model.

raw [R]

The original provider response, or an array of responses when each input requires a separate request.

results [R]

The per-input verdicts, as an array of Result objects, one per moderated input.

Public Class Methods

moderate (input = nil, model: nil, with: nil, provider: nil, assume_model_exists: false, context: nil, provider_options: {}, metadata: nil)

Source

# File lib/ruby_llm/moderation.rb, line 89
def self.moderate(input = nil,
                  model: nil,
                  with: nil,
                  provider: nil,
                  assume_model_exists: false,
                  context: nil,
                  provider_options: {},
                  metadata: nil)
  attachments = Attachment.wrap(with)
  raise ArgumentError, 'must provide input text, image attachment, or both' if input.nil? && attachments.empty?

  config = context&.config || RubyLLM.config
  model, provider_instance = Models.resolve(model, provider: provider, assume_model_exists: assume_model_exists,
                                                   config: config, operation: :moderate,
                                                   default_model: config.default_moderation_model)
  empty_tokens = Tokens.new
  payload = {
    provider: provider_instance.slug,
    provider_class: provider_instance.class.display_name,
    model: model&.id,
    model_info: model,
    input: input,
    attachment_count: attachments.size,
    provider_options: provider_options,
    metadata: metadata,
    tokens: empty_tokens,
    cost: Cost.new(tokens: empty_tokens, model:)
  }

  RubyLLM.instrument('moderation.ruby_llm', payload, config: config) do |event|
    result = provider_instance.moderate(input, model:, with: attachments, provider_options:)
    event[:result] = result
    event[:flagged] = result.flagged?
    event[:tokens] = result.tokens
    event[:cost] = result.cost
    result
  end
end

Screens input and optional image attachments and returns a Moderation with the provider’s verdict. Uses the configured default moderation model when model is not given. Pass provider: and assume_model_exists: true to use a model that is not in the registry. An explicitly selected provider may instead use a configured resource without a model.

RubyLLM.moderate("User message")
RubyLLM.moderate(["First comment", "Second comment"]).results
RubyLLM.moderate("Caption", with: "screenshot.png")

Public Instance Methods

category_scores ()

Source

# File lib/ruby_llm/moderation.rb, line 159
def category_scores
  results.map(&:category_scores).reduce({}) do |merged, scores|
    merged.merge(scores) { |_category, left, right| [left, right].max }
  end
end

Returns the confidence scores across all results, as a hash of category name to a score between 0.0 and 1.0. Keeps the highest score per category when there are multiple results.

result.category_scores["violence"]  # => 0.0001

cost ()

Source

# File lib/ruby_llm/moderation.rb, line 141
def cost
  ruby_llm_usage_cost
end

Returns the moderation cost across every provider attempt.

flagged? ()

Source

# File lib/ruby_llm/moderation.rb, line 130
def flagged?
  results.any?(&:flagged?)
end

Returns true if any input was flagged as potentially harmful, false otherwise.

flagged_categories ()

Source

# File lib/ruby_llm/moderation.rb, line 149
def flagged_categories
  results.flat_map(&:categories).uniq
end

Returns the unique names of the categories flagged across all results.

result.flagged_categories  # => ["harassment", "violence"]

tokens ()

Source

# File lib/ruby_llm/moderation.rb, line 136
def tokens
  ruby_llm_usage_tokens
end

Returns provider-reported usage across every attempt. Its fields are nil when the provider did not report any.