Retrieval-Augmented Generation (RAG)

Retrieve relevant context from your own documents, then answer with that context

After reading this guide, you will know:

• How RAG fits as one step in a larger workflow.
• How to store document embeddings with neighbor and pgvector.
• How to generate embeddings automatically when a document changes.
• How to expose semantic search to an agent as a retrieval tool.
• How to build an answering agent that cites its sources.

RAG is often just one step in a larger workflow: retrieve relevant context, then answer with that context. You embed your documents once, search them by similarity at query time, and feed the closest matches to the model as grounding. For the mechanics of turning text into vectors, see Embeddings.

Setup

# Gemfile
gem 'neighbor'
gem 'ruby_llm', '2.0.0.rc1'

bin/rails generate neighbor:vector
bin/rails db:migrate

class CreateDocuments < ActiveRecord::Migration[7.1]
  def change
    create_table :documents do |t|
      t.text :content
      t.string :title
      t.vector :embedding, limit: 1536 # OpenAI embedding size
      t.timestamps
    end

    add_index :documents, :embedding, using: :hnsw, opclass: :vector_l2_ops
  end
end

Document Model with Embeddings

class Document < ApplicationRecord
  has_neighbors :embedding

  before_save :generate_embedding, if: :content_changed?

  private

  def generate_embedding
    self.embedding = RubyLLM.embed(content).vectors
  end
end

For a short scanned document, combine the model with OCR:

text = RubyLLM.ocr("refund-policy.pdf").markdown
Document.create!(title: "Refund policy", content: text)

The callback generates its embedding. Configure the OCR provider as described in Document OCR. Split long documents into passages before embedding them so each result contains focused context.

Retrieval Tool

class DocumentSearch < RubyLLM::Tool
  description "Searches knowledge base for relevant information"
  parameter :query, description: "Search query"

  def execute(query:)
    embedding = RubyLLM.embed(query).vectors

    documents = Document.nearest_neighbors(
      :embedding,
      embedding,
      distance: "euclidean"
    ).limit(3)

    RubyLLM::SearchResults.new(
      *documents.map { |doc| { title: doc.title, text: doc.content.truncate(500) } }
    )
  end
end

Return RubyLLM::SearchResults rather than a joined string and the model can cite each document it used. See Citing Tool Results.

Answering Agent

class SupportWithDocsAgent < RubyLLM::Agent
  tools DocumentSearch
  instructions "Search for context before answering. Cite sources."
end

agent = SupportWithDocsAgent.new
agent.ask("What is our refund policy?").content

To improve retrieval ordering, fetch a larger candidate set and rerank it before building SearchResults. Reranking shows the complete database-to-results example.

Next Steps

• Embeddings - Turn text into vectors for similarity search.
• Memory - The same pattern over memories the agent writes itself.
• Agentic Workflows - Compose retrieval into larger orchestrations.
• Tools - Build the retrieval tool and other capabilities.
• Agents - Define the answering agent class.