Skip to Content
RAGRetrieval Pipeline

Retrieval Pipeline

Retrieval has two stages: ingest and search. Before an agent generates a response, supporting passages are pulled from your library and the literature and sent along with the question.

Ingest

1. Add a Source

Sources come from uploads, pasted text, account sync, or PubMed search, and each one is normalized into a RagSource record.

2. Extract Text When Needed

PDF uploads are parsed in the browser with pdfjs-dist. If no usable text can be extracted, the source is still stored but excluded from normal text ingest.

3. Preserve Scope

Each source is scoped local or account. Account sources can be merged into the current folder when the user is signed in.

Search runs in this order:

  1. collect eligible text
  2. embed sources
  3. embed the query
  4. score matches
  5. return top sources
  6. build the cited prompt

Retrieval Defaults

The semantic search flow:

  • chunks source documents into passages
  • embeds text content with text-embedding-004
  • uses cosine similarity for matching
  • returns a small result set
  • builds the prompt with [Source N] labels

Retrieval Controls

  • Source filters: limit retrieval to a specific library, such as local, account, or public sources.
  • Date windows: limit to a time range, for example guidelines from the last five years.
  • Relevance thresholds: drop chunks below a minimum similarity score to keep weak matches out of the context.

PubMed Path

PubMed follows a parallel path:

  1. search PubMed
  2. load summaries and abstracts
  3. convert articles into RAG sources
  4. add the selected articles to the active context