Reference Pipeline Data

Document Intelligence Pipeline

A modular document intelligence pattern that turns papers into structured, domain-tagged, scored knowledge for retrieval and self-improving AI workflows.

View implementation ↗

Problem Research agents need to know which documents are relevant, reliable, and useful for a current goal.

Outcome A pipeline of agents that loads documents, profiles sections, classifies domains, scores papers, and selects goal-matched knowledge.

Implementation evidence

The solution is backed by inspectable code

https://github.com/ernanhughes/co-ai

This solves goal-aware document selection: ingest papers, attach structured metadata, classify by semantic domain, score quality dimensions, and return only the documents worth using for a task.

Code

pipeline:
  name: papers
  tag: "related papers import"
  description: "Import, profile, score, and select research papers"
  stages:
    - name: survey
      cls: stephanie.agents.knowledge.survey.SurveyAgent
      enabled: true
    - name: search_orchestrator
      cls: stephanie.agents.knowledge.search_orchestrator.SearchOrchestratorAgent
      enabled: true
    - name: document_loader
      cls: stephanie.agents.knowledge.document_loader.DocumentLoaderAgent
      enabled: true
    - name: document_profiler
      cls: stephanie.agents.knowledge.document_profiler.DocumentProfilerAgent
      enabled: true
    - name: paper_score
      cls: stephanie.agents.knowledge.paper_score.PaperScoreAgent
      enabled: true
    - name: knowledge_loader
      cls: stephanie.agents.knowledge.knowledge_loader.KnowledgeLoaderAgent
      enabled: true

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity


class KnowledgeLoaderAgent:
    def __init__(self, cfg, memory, logger):
        self.domain_seeds = cfg.get("domain_seeds", {})
        self.top_k = cfg.get("top_k", 3)
        self.threshold = cfg.get("domain_threshold", 0.0)
        self.include_full_text = cfg.get("include_full_text", False)
        self.use_dimensional_scores = cfg.get("use_dimensional_scores", False)
        self.dimension_weights = cfg.get("dimension_weights", {
            "relevance": 1.0,
            "usefulness": 0.8,
            "clarity": 0.6,
            "implementability": 0.7,
            "novelty": 0.5,
        })
        self.min_weighted_score = cfg.get("min_weighted_score", 0.5)
        self.memory = memory
        self.logger = logger

    async def run(self, context):
        goal = context.get("goal", {})
        goal_text = goal.get("goal_text", "")
        documents = context.get("documents", [])
        if not goal_text or not documents:
            return context

        goal_vector = self.memory.embedding.get_or_create(goal_text)
        domain_vectors = {
            domain: np.mean(
                [self.memory.embedding.get_or_create(seed) for seed in seeds],
                axis=0,
            )
            for domain, seeds in self.domain_seeds.items()
        }

        goal_domain, goal_domain_score = None, -1
        for domain, vector in domain_vectors.items():
            score = float(cosine_similarity([goal_vector], [vector])[0][0])
            if score > goal_domain_score:
                goal_domain, goal_domain_score = domain, score

        filtered = []
        for doc in documents:
            doc_domains = self.memory.document_domains.get_domains(doc["id"])
            for dom in doc_domains[: self.top_k]:
                if dom.domain != goal_domain or dom.score < self.threshold:
                    continue
                if self.use_dimensional_scores and self.compute_weighted_score(doc["id"]) < self.min_weighted_score:
                    continue
                filtered.append({
                    "id": doc["id"],
                    "title": doc["title"],
                    "domain": dom.domain,
                    "domain_score": dom.score,
                    "content": doc["text"] if self.include_full_text else doc["summary"],
                })
                break

        context["goal_domain"] = goal_domain
        context["goal_domain_score"] = goal_domain_score
        context["documents"] = filtered
        return context

    def compute_weighted_score(self, doc_id):
        scores = self.memory.document_scores.get_scores(doc_id)
        total, weight_sum = 0.0, 0.0
        for dimension, weight in self.dimension_weights.items():
            score = next((s.score for s in scores if s.dimension == dimension), None)
            if score is not None:
                total += weight * score
                weight_sum += weight
        return total / weight_sum if weight_sum else 0.0

Usage

knowledge_loader:
  domain_seeds: config/domain/seeds.yaml
  top_k: 3
  domain_threshold: 0.4
  include_full_text: false
  use_dimensional_scores: true
  min_weighted_score: 0.65

Requirements

The implementation assumes a memory layer with embeddings, document domain assignments, and document scores. The article implementation uses the co-ai/Stephanie agent framework.

Source

The implementation lives in the broader co-ai system.

Full explanation

For the full pipeline walkthrough, profiling strategy, and scoring rationale, read: Document Intelligence: Turning Documents into Structured Knowledge.