Teaching Tiny Models to Think Big: Distilling Intelligence Across Devices

🧪 Summary

As AI developers, we often face the tradeoff between intelligence and accessibility. Powerful language models like Qwen3 run beautifully on servers but what about on the edge? On devices like Raspberry Pi or old Android phones, we’re limited to small models. The question we asked was simple:

Can we teach a small model to behave like a large one without retraining it from scratch using only its outputs and embeddings?

This blog post outlines our approach: a lightweight embedding-based distillation process, where a “master” model (Qwen3) teaches a “pupil” (Phi-4-mini or similar) to align its responses through vector tuning. The setup is designed to run across distributed AI nodes from your server to your smallest Pi using shared embeddings and a clever fine-tuning trick.

🧭 Why Teach the Small Models?

Why go through all this trouble to make a small model behave like a large one?

Because we’re not just building a model we’re building a system.

And that system needs to work across different machines, in different environments, under different constraints. One deployment might run on a GPU server in a research lab. Another might be installed in a smart camera on a street corner. Another might be offline entirely, embedded in a wearable or mobile device.

Each environment is different. But the behavior of the system the decisions it makes, the judgments it applies should be aligned.

So our goal isn’t just to compress a model. It’s to build a coordinated, adaptable, modular architecture where:

• The central model sets the tone, logic, and expectations.
• The smaller models follow that guidance cheaply and reliably.
• The entire swarm behaves as a coherent whole, not a collection of disconnected parts.

When you ask the edge device to summarize a document, or monitor a camera feed, it should do it the same way the main model would just lighter and faster.

This is how we enable scale without sacrificing consistency. It’s how we offload tasks to a Raspberry Pi without handing over our AI’s values, priorities, or reasoning style.

And that brings us to the second piece of the puzzle: what kinds of tasks we can give to these subagents, and how we make the most of our compute swarm.

🧠 Divide and Conquer: Let the Pupil Handle the Heavy Lifting

In most AI setups, one big model does everything. But that’s not how humans operate and it’s not how scalable systems should work either.

Think of your AI system as a head surrounded by hands.

You might have a powerful desktop running Qwen3 or GPT-4, but you don’t want to waste that firepower on tasks like:

• Scoring a bunch of responses,
• Translating a video subtitle,
• Checking for keyword matches on YouTube,
• Running a web search,
• Summarizing a blog post.

These jobs don’t have to be perfect they just have to be done reliably and constantly. They’re what we might call non-mission-critical but high-frequency tasks.

So instead of doing it all yourself, why not build a swarm?

🐝 Enter the Raspberry Pi Cluster

Imagine a grid of Raspberry Pi devices each running a small, local LLM like Phi-4-mini or TinyLlama. Each is like a robotic assistant, ready to help out with one job:

• One Pi handles scoring hypotheses.
• Another monitors RSS feeds for new research.
• Another translates captions for real-time subtitles.
• Another does fuzzy matching on customer queries.
• Another summarizes input so your main model doesn’t have to read 20 pages.

The Master Agent sends off the work and trains each of these smaller nodes to act more like itself. Not by sending model weights but by teaching via output alignment. The Pupil doesn’t copy the Master it learns to approximate its intent in a lighter way.

This is edge AI, self-improvement, and agent swarms all rolled into one.

🧩 What Kind of Tasks Can Be Offloaded?

Here’s a sample list of jobs that can be handled by small models in this architecture:

The idea is: use small models to handle the long tail of tasks, freeing up your main model for the hard stuff the few things that actually require power and precision.

🛠️ System Setup: Master and Pupil Agents

    
graph TD
    Goal[🎯 User Question or Goal]
    Goal -->|Same prompt| Master
    Goal -->|Same prompt| Pupil

    subgraph Inference
      Master[🤖 Master Model Qwen3] -->|Generates Answer A| MasterAns[📝 Answer A]
      Pupil[🤖 Pupil Model Phi-4-mini] -->|Generates Answer B| PupilAns[📝 Answer B]
    end

    subgraph Embedding
      MasterAns -->|Embedding| EmbedA[🔷 Embedding A]
      PupilAns -->|Embedding| EmbedB[🔶 Embedding B]
    end

    EmbedA -->|Compare| Similarity[📏 Cosine Similarity]
    EmbedB -->|Compare| Similarity

    Similarity -->|Loss| LossCalc[❌ Loss Function MSE]
    LossCalc -->|Backprop| Tuner[🛠️ Pupil FineTuner]
    Tuner -->|Update| Pupil
  

We define two agents:

• MasterAgent runs on a powerful server with access to Qwen3.
• PupilAgent runs on a low-resource node (e.g. Raspberry Pi), using a small model like Phi-4-mini via Ollama.

The two agents are almost identical. This is the master.

class MasterAgent(BaseAgent):
    def __init__(self, cfg, memory=None, logger=None):
        super().__init__(cfg, memory, logger)

    async def run(self, context: dict) -> dict:
        question = context.get(self.input_key, context.get("goal", {}).get("goal_text", ""))
        answer = self.answer(question, context)
        context.setdefault(self.output_key, []).append(answer)
        self.logger.log("MasterAnswerGenerated", f"Answered: {answer[:50]}...")
        return context        

    @time_function(logger=None)
    def answer(self, question, context):
        return self.call_llm(question, context, self.cfg.get("pupil_model"))

The pupil. This will run on an edge device.

class PupilAgent(BaseAgent):
    def __init__(self, cfg, memory=None, logger=None):
        super().__init__(cfg, memory, logger)

    async def run(self, context: dict) -> dict:
        question = context.get(self.input_key, context.get("goal", {}).get("goal_text", ""))
        answer = self.answer(question, context)
        context.setdefault(self.output_key, []).append(answer)
        self.logger.log("PupilAnswerGenerated", f"Answered: {answer[:50]}...")
        return context        

    @time_function(logger=None)
    def answer(self, question, context):
        return self.call_llm(question, context, self.cfg.get("pupil_model"))

Both agents:

• Receive the same list of prompts.
• Generate responses using their respective models.
• Encode responses into embeddings (e.g. using mxbai-embed-large or Ollama’s built-in embedding).
• Compare those embeddings to calculate alignment loss.
• Use a PupilFineTuner to move the pupil’s response embedding closer to the master’s effectively “teaching” it what the master knows.

🛠️ The Agent Configs

In configs/agents/master.yaml:

Notice the master_model setting.

master:
  name: master
  
  master_model:
    name: ollama/qwen3
    api_base: http://localhost:11434
    api_key: null

  input_key: question
  output_key: master_answer
  prompt_mode: file
  prompt_file: master.txt

And for pupil.yaml:

Notice the pupil_model setting.

pupil:
  name: pupil

  pupil_model:
    name: ollama/phi4-mini
    api_base: http://localhost:11434
    api_key: null

  input_key: question
  output_key: pupil_answer
  prompt_mode: file
  prompt_file: pupil.txt

📝 Default Prompts

Inside prompts/master/master.txt:

{# prompts/master_agent/master_agent.txt #}

You are a helpful assistant

Answer this question:
{{ goal.goal_text }}

{% if preferences %}
Use these preferences:
{% for p in preferences %}
- {{ p }}
{% endfor %}
{% endif %}

{% if instructions %}
Additional instructions: 
{% for i in instructions %}
- {{ i }}
{% endfor %}
{% endif %}

And for pupil_agent/pupil_agent.txt:

{# prompts/pupil/pupil.txt #}

You are a helpful assistant

Answer this question:
{{ goal.goal_text }}

{% if preferences %}
Use these preferences:
{% for p in preferences %}
- {{ p }}
{% endfor %}
{% endif %}

{% if instructions %}
Additional instructions: 
{% for i in instructions %}
- {{ i }}
{% endfor %}
{% endif %}

Notice that we can futher tune the rpompts using preferences or extra instructions.

🎯 Training the Pupil: Embedding-Based Distillation

Once we’ve wired up the Master and Pupil agents, the next question is: how does the pupil get better over time?

We don’t want to retrain it from scratch. That’s expensive, slow, and defeats the purpose of using a small model in the first place. Instead, we use a technique called embedding-based distillation.

The idea is simple:

Don’t force the pupil to copy the master’s words just match its meaning.

🧪 The Core Idea: Learn by Meaning, Not by Imitation

Every time the Master and Pupil respond to the same prompt, we take their answers and turn them into embeddings numerical vectors that capture the meaning of their responses. If those vectors are close, it means the two models “think” similarly. If they’re far apart, we calculate a loss and use it to gently nudge the Pupil’s model weights.

This approach has three big advantages:

1. Model-agnostic: We don’t need to modify the Master model at all.
2. Lightweight: The Pupil only updates via a small adapter or LoRA head.
3. Flexible: It works across tasks summarization, scoring, filtering, etc.

🔁 The Training Loop

Each training step looks like this:

1. Take a shared prompt (e.g. “Summarize this paragraph”).
2. Ask both agents to generate an answer.
3. Convert each answer into an embedding vector.
4. Measure the distance between them (e.g. using cosine similarity or mean squared error).
5. Use that loss to update only the Pupil, so it shifts closer to the Master.

Great points here’s a refined explanation section you can add above your PupilModel and PupilFineTuner classes to address these considerations clearly:

🧠 Pupil Embedding Tuner: Simple, Fast, and Effective

The PupilModel is a lightweight neural adapter trained to align the outputs of a small model (the “Pupil”) with the behaviors of a large model (the “Master”) using only embedding similarity between their generated responses.

class PupilModel(nn.Module):
    def __init__(self, input_dim, output_dim):
        super().__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim

        self.model = nn.Sequential(
            nn.Linear(input_dim, 1024),
            nn.ReLU(),
            nn.Linear(1024, output_dim)
        )

    def forward(self, x):
        assert x.shape[-1] == self.input_dim, \
            f"Expected input dim {self.input_dim}, got {x.shape[-1]}"
        return self.model(x)

class PupilFineTuner:
    def __init__(self, input_dim=1024, output_dim=1024, lr=1e-4):
        logger.info(f"Initializing PupilModel with input_dim={input_dim}, output_dim={output_dim}")
        self.model = PupilModel(input_dim, output_dim)
        self.optimizer = optim.Adam(self.model.parameters(), lr=lr)
        self.loss_fn = nn.MSELoss()

    def train_step(self, student_input, teacher_output):
        # Assertions
        assert student_input.shape[-1] == self.model.input_dim, \
            f"Student input has wrong shape: {student_input.shape[-1]} vs expected {self.model.input_dim}"
        assert teacher_output.shape[-1] == self.model.output_dim, \
            f"Teacher output has wrong shape: {teacher_output.shape[-1]} vs expected {self.model.output_dim}"

        self.model.train()
        pred = self.model(student_input)
        loss = self.loss_fn(pred, teacher_output)

        logger.info(f"Training step loss: {loss.item():.6f}")

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss.item()

We use a simple nn.Sequential architecture:

Why this design?

• Simplicity and speed: This is meant to run on edge devices like Raspberry Pi. A minimal architecture reduces memory and compute overhead.
• Embedding alignment: Since we’re working in high-dimensional embedding space (typically 1024 dims), even small linear adjustments can have a meaningful impact.
• Adaptability: This layer can act as a proxy for an adapter or LoRA head that could be injected into the small model though here, we train it independently for clarity.

The input_dim and output_dim match the embedding sizes of the Pupil and Master respectively. If both use the same embedding model (e.g., mxbai-embed-large), these will be equal. If not, this layer can learn to bridge the dimensional gap.

We train this module using mean squared error (MSE) loss on the output embeddings: This works well because:

• MSE encourages vector similarity, and when embeddings are normalized (as they often are), minimizing MSE approximates cosine similarity.
• It’s differentiable and stable for this kind of alignment task.

Note: We update only the Pupil-side module. The Master remains frozen, acting as a behavioral anchor. In a future version, this linear module could be integrated as a trainable LoRA adapter inside the small model, allowing on-device adaptation without modifying full weights.

This training approach allows us to gradually teach the Pupil to mimic the Master’s output distribution without full finetuning a key advantage when operating under compute constraints.

The TrainerAgent is responsible for teaching the small model (the pupil) how to mimic the output behavior of the large model (the master) not by matching text, but by aligning the semantic embeddings of their answers.

class TrainerAgent(BaseAgent):
    def __init__(self, cfg, memory=None, logger=None, master=None, pupil=None):
        super().__init__(cfg, memory, logger)
        self.master = master
        self.pupil = pupil
        self.embedding_store = memory.embedding
        self.finetuner = PupilFineTuner(
            input_dim=1024,  # embedding dim of pupil
            output_dim=1024  # embedding dim of master
        )

    async def run(self, context: dict) -> dict:
        """Pipeline entrypoint (unused in this agent directly)."""
        return context


    def align_response(self, question, context=None, epochs=3, plot=True):
        master_answer = self.master.answer(question, context)
        self.logger.log("MasterAnswer", {"master_answer": master_answer})
        pupil_answer = self.pupil.answer(question, context)
        self.logger.log("PupilAnswer", {"pupil_answer": pupil_answer})

        master_emb = torch.tensor(self.embedding_store.get_or_create(master_answer), dtype=torch.float32)
        pupil_emb = torch.tensor(self.embedding_store.get_or_create(pupil_answer), dtype=torch.float32)

        losses = []
        print(f"Initial pupil answer:\n{pupil_answer}\n")

        for i in range(epochs):
            loss = self.finetuner.train_step(pupil_emb, master_emb)
            losses.append(loss)
            print(f"Epoch {i+1} Loss: {loss:.4f}")

        if plot:
            self.plot_training_curve(losses)

        return pupil_answer

    def predict_embedding(self, text):
        emb = np.array(self.embedding_store.get_or_create(text))
        input_tensor = torch.tensor(emb, dtype=torch.float32)
        with torch.no_grad():
            aligned = self.finetuner.model(input_tensor).numpy()
        return aligned

    def _approximate_generation_from_embedding(self, emb: torch.Tensor) -> str:
        """
        Dummy method: in future, this could map embeddings back to text.
        """
        return " ".join(["generated"] * 10)

    def plot_training_curve(self, losses):
        plt.figure(figsize=(8, 4))
        plt.plot(range(1, len(losses)+1), losses, marker='o', linestyle='-')
        plt.title("Training Loss Over Epochs")
        plt.xlabel("Epoch")
        plt.ylabel("Loss")
        plt.grid(True)
        plt.tight_layout()
        plt.show()

And visually:

    graph TD
    Prompt[🧾 Prompt] --> Master[🤖 Master Model]
    Prompt --> Pupil[🤖 Pupil Model]

    Master --> MA[📝 Master Answer]
    Pupil --> PA[📝 Pupil Answer]

    MA --> EA[🔷 Embed Master]
    PA --> EB[🔶 Embed Pupil]

    EA --> SIM[📏 Compare <br/> Cosine or MSE]
    EB --> SIM

    SIM --> Loss[❌ Loss]
    Loss --> Update[🛠️ Pupil Update]
    Update --> Pupil
  

🧠 Why It Works

Think of this like language-style tuning. We’re not teaching the Pupil to memorize; we’re teaching it to speak the same language of judgment as the Master.

Instead of redoing full instruction tuning or downloading giant models, we just align behavior meaning the Pupil becomes a faster, smaller proxy that captures the spirit of the Master.

This gives us a powerful architecture:

• Centralized intelligence (on the server),
• Distributed consistency (across devices),
• Adaptable, trainable agents (on the edge).

The Pupil isn’t a copy of the Master it’s a lightweight shadow that behaves in sync. A puppet on a string.

🧾 Evaluation: How Do We Know It Worked?

Training the Pupil is only half the story.

We also need to prove that the Pupil is learning that it’s getting closer to the Master, not just in output format but in meaningful behavior.

And since we’re aligning via embeddings, we can also evaluate via embeddings.

📏 Step 1: Measure Embedding Similarity

For each shared prompt, we:

1. Generate answers from both Master and Pupil.
2. Encode both into embedding vectors.
3. Compute cosine similarity between them.
4. Track the similarity score before and after fine-tuning.

If training works, we should see similarity increase over time meaning the Pupil’s answers are drifting closer to the Master’s in semantic space.

class EvaluatorAgent(BaseAgent):
    def __init__(self, cfg, memory=None, logger=None):
        super().__init__(cfg, memory, logger)
        self.master = MasterAgent(cfg, memory, logger)
        self.pupil = PupilAgent(cfg, memory, logger)
        self.trainer = TrainerAgent(
            cfg, memory, logger, master=self.master, pupil=self.pupil
        )

    async def run(self, context: dict) -> dict:
        question = context.get("goal", {}).get("goal_text", "")
        master_answer = context.get("master_answer")[0]
        pupil_answer = context.get("pupil_answer")[0]

        score_before = self.score_alignment(master_answer, pupil_answer)
        self.logger.log(
            "EvaluatorRun",
            {
                "score_before": score_before,
                "question": question,
                "master_answer": master_answer,
                "pupil_answer": pupil_answer,
            },
        )
        aligned_answer = self.trainer.align_response(question, context=context)
        score_after = self.score_alignment(master_answer, aligned_answer)
        self.logger.log(
            "EvaluationResult",
            {
                "Before": score_before,
                "After": score_after,
                "Improvement": score_after - score_before,
            },
        )
        context.setdefault("evaluation_results", []).append(
            {
                "score_before": score_before,
                "score_after": score_after,
                "improvement": score_after - score_before,
            }
        )

        return context

    def score_alignment(self, text1, text2):
        emb1 = self.memory.embedding.get_or_create(text1)
        emb2 = self.memory.embedding.get_or_create(text2)
        sim = cosine_similarity([emb1], [emb2])[0][0]
        return sim

    def evaluate_alignment(
        self, master_output: torch.Tensor, pupil_output: torch.Tensor
    ):
        similarity = (
            cosine_similarity(master_output, pupil_output, dim=-1).mean().item()
        )
        distance = torch.norm(master_output - pupil_output, dim=-1).mean().item()
        return {
            "cosine_similarity": round(similarity, 4),
            "vector_distance": round(distance, 4),
        }

You might see output like:

{
    "timestamp": "2025-06-27T23:55:10.197210+00:00", 
    "event_type": "EvaluationResult", 
    "data": {"Before": 0.7165400641572484, 
             "After": 0.8717646955387616, 
             "Improvement": 0.1552246313815132}}

📈 Step 2: Plot the Training Curve

To visualize training over time, we log the similarity after each epoch:

epochs = [1, 2, 3, 4, 5]
similarities = [0.69, 0.74, 0.81, 0.88, 0.91]

plt.plot(epochs, similarities, marker='o')
plt.title("Embedding Similarity Over Training")
plt.xlabel("Epoch")
plt.ylabel("Cosine Similarity to Master")
plt.grid(True)
plt.show()

This gives you a clear curve showing how the Pupil gets closer and closer to the Master with each round of distillation.

🧮 Test pipeline

This is an exmaple pipeline for testing this process.

# config/config.yaml
defaults:
  - _self_
  - db: postgres
  - agents/master
  - agents/pupil
  - agents/trainer
  - agents/evaluator
  - agents/trainer
  - agents/unified_mrq
  - logging/json_logger

# The goal of the pipeline, e.g., "Generate a hypothesis about the impact of climate change on biodiversity."
# This is a placeholder and should be replaced with the actual goal.
goal:
  goal_text: "If I was to develop a self improving process what would be the steps needed?"
  goal_type: "research"
  focus_area: "ai_research"
  strategy: "reasoning"
  difficulty: "medium"
  expected_formats:
    - "short_cot"
    - "code"

post_judgment:
  name: pipeline_judge
  enabled: false
  cls: co_ai.agents.pipeline_judge.PipelineJudgeAgent

paths:
  prompts: ${hydra:runtime.cwd}/prompts

report:
  generate_report: true
  path: ${hydra:runtime.cwd}/reports


embeddings:
  model: "mxbai-embed-large"
  dimension: 1024
  endpoint: "http://localhost:11434/api/embeddings"

pipeline:
  name: master_pupil
  description: "This pipeline demonstrates a master-pupil architecture where the master agent guides hypotheses the pupil generates."
  stages:
    - name: master
      description: "Master agent that guides the pupil agent."
      cls: co_ai.agents.master_pupil.master.MasterAgent
      enabled: true
      iterations: 1
    - name: pupil
      description: "Pupil agent smaller model we need to inform."
      cls: co_ai.agents.master_pupil.pupil.PupilAgent
      enabled: true
      iterations: 1
    - name: evaluator
      description: "Pupil agent smaller model we need to inform."
      cls: co_ai.agents.master_pupil.evaluator.EvaluatorAgent
      enabled: true
      iterations: 1

✅ Optional: Manual or LLM Judging

You can also spot-check by asking a third-party LLM (or human) to judge the answers:

“Given two answers to the same prompt one from a large model, one from a small how similar are they in quality, style, and accuracy?”

This gives you qualitative validation on top of the embedding metrics.

🧠 Takeaway

We don’t need to fully retrain or supervise the Pupil to prove that it works. We just track:

• ✳️ Similarity improvement (via cosine or MSE),
• 📉 Loss decrease,
• 📈 Embedding alignment curve,
• 💬 Optional LLM judgments.

When the curves go up and the loss goes down, we know our distributed swarm is learning to think like the master and we can trust it to handle more edge-side jobs.

✨ Toward Self-Improving AI at the Edge

This isn’t just a clever trick to make Raspberry Pi’s run smarter. It’s a prototype for something much bigger.

What we’ve built here is a pattern a blueprint for distributing intelligence across a swarm of AI agents without sacrificing alignment, logic, or values.

• A powerful master model sets the tone.
• Smaller pupil agents learn from it by aligning their outputs in embedding space.
• The system evolves by observing itself, adapting each layer from what it sees.

That’s self-improving AI.

No full retraining. No human-in-the-loop feedback. No perfect ground truth.

Just a reliable teacher, a willing student, and a way to measure similarity.

From that seed, a swarm can grow.

Imagine:

• An offline assistant that updates itself from your laptop’s master model.
• A wearable device that learns your tone by watching your home AI.
• A thousand micro-nodes monitoring the world, each behaving with traceable logic.

All coordinated. All evolving. All aligned.

This self-improvement isn’t a one-time alignment it’s continuous and adaptive.

The TrainerAgent can run in the background, collecting new prompts and master responses during daily use. These become new training examples. With each interaction, the pupil adapts a little more its embeddings nudged toward those of the master. No need to retrain the whole model. Just a small adapter. Just enough to stay in sync.

Over time, the pupil doesn’t just mimic it becomes resonant with the master.

This is the foundation of cooperative cognition: AI not as a single mind in a box, but a network of reasoning agents that learn from each other, teach each other, and scale together.

We’ve shown one way to start. The rest is just deployment.

🧾 Summary: What We Built and How

Let’s bring it all together. Here’s what we did step by step to create a scalable, self-improving AI system using a Master–Pupil setup:

✅ The Problem

We wanted small, low-resource models (like Phi-4-mini) to behave more like big, powerful models (like Qwen3), without retraining from scratch.

🧠 The Insight

We realized we didn’t need to copy the master’s internals we just needed to align on outputs using embeddings. If two answers “feel” similar in vector space, they can be treated as logically aligned.

🔧 The Steps We Took

🌍 What This Enables

• A powerful central brain with many lightweight edge agents.
• Real-time offloading of non-critical tasks to small models.
• Systems that stay consistent, even across wildly different hardware.
• The foundation for distributed, self-improving AI.

This isn’t just optimization it’s coordination. A roadmap for building intelligent swarms that learn together.

📊 Summary of Results

The alignment training successfully improved the Pupil model’s understanding and output quality compared to the Master model.

📈 Key Takeaways

• The Pupil model (e.g., Phi-3) learned to produce outputs much closer in meaning to the Master model (e.g., Qwen3).
• Embedding-space alignment helped reduce semantic gaps without requiring full token-by-token supervision.
• This demonstrates the effectiveness of LoRD-style distillation for practical, real-world deployment on edge devices.

💡 Example Output Comparison

Master Answer:
“Developing a self-improving process involves creating a structured, iterative system that allows you to grow, adapt, and refine your goals, habits, and strategies over time.”

Pupil Answer (Aligned):
“Developing a self-improving process involves creating a structured, iterative system that allows you to grow, adapt, and refine your goals, habits, and strategies over time.”

✅ Almost identical in meaning and structure, showing successful alignment.

⚠️ Limitations and Future Work

While this approach shows real promise, it’s important to recognize its current limitations and future potential.

• Dependence on the Master: The effectiveness of this method relies heavily on the master model consistently producing high-quality responses. If the master output is flawed, the pupil may inherit those weaknesses.

• Risk of Overfitting: As the pupil aligns more closely with the master on specific prompts, there’s a risk it could overfit to those patterns reducing its ability to generalize to unfamiliar tasks or domains.

• Training Overhead: Although the pupil model is lightweight, the TrainerAgent still requires computational resources for fine-tuning and monitoring. On very constrained edge devices, even this minimal training might need to be offloaded to slightly more capable nodes.

• Limited Textual Control: Currently, the pupil aligns only at the embedding level. Generating text from aligned embeddings (or using them to steer generation) remains an open area for exploration.

• Next Steps: Future work could explore:

  • Filtering or weighting master outputs based on quality scores or uncertainty.
  • Incorporating lightweight human-in-the-loop feedback (e.g. thumbs up/down).
  • Dynamically adjusting learning rates based on task complexity or similarity.
  • Integrating LoRA-style adapters directly into real small models like Phi-4-mini for more fine-grained tuning.

This is just the first step but it opens the door to a wide new space of self-improving, cooperative AI systems that learn in the wild.

📚 References

This whole system is grounded in the insights from the paper LoRD: Aligning Language Models via Embedding Space Distillation. Their core idea:

You don’t need a small model to match every token of a big model’s output just get the embedding close enough, and it will behave similarly.

We took that principle and turned it into a distributed architecture:

• A powerful Master model teaches small Pupil models.
• The Pupils run anywhere Raspberry Pis, mobile phones, or browser tabs.
• They don’t need massive compute just embedding alignment and a trickle of fine-tuning.

This isn’t just model compression. It’s a strategy for scalable, distributed reasoning.

We’re building systems that learn from themselves, adapt across devices, and coordinate across scale. That’s the future of edge AI. And it starts right here.

📘 Glossary