Shadow Scaling: Building Software with Copilot Agents That Can Safely Improve Themselves

Working subtitle

How component-local memory, subagent review, and forward-only implementation variants turn Copilot from a coding assistant into a scalable development workforce.


Draft thesis

The next leap in AI-assisted software development isn’t asking a coding agent to edit more files at once. It’s designing the codebase so agents can work safely, independently, and measurably.

We introduce Shadow Scaling: an agent-native development pattern where new functionality is added as parallel, forward-only variants that shadow the stable system before being promoted. Instead of asking Copilot or Codex to surgically rewrite old code, we ask it to create bounded implementations behind stable contracts, test them, benchmark them, review them, and promote only the versions that prove themselves.

The result is a development process that matches how LLM coding agents actually work best: fresh context, clear contracts, isolated tasks, deterministic feedback, and reversible changes.


1. The problem: Copilot is powerful, but not naturally reliable

Copilot Agent Mode and modern coding agents are already astonishingly useful. Given a well-scoped task, they can create files, write tests, run commands, repair failures, and move a component forward in minutes.

But they become unreliable when asked to hold an entire system in context. The failure pattern is familiar:

  • The agent edits too many files at once.
  • It changes old code without understanding knock-on effects.
  • It fixes one test and silently breaks another.
  • It loses track of design intent across sessions.
  • It produces a plausible patch that is hard to trust.

This isn’t a model limitation. It’s an environment limitation. We’re still asking agentic coding systems to work inside codebases designed for human memory, not agent context boundaries.

What would a codebase look like if it were designed for Copilot Agent Mode from the beginning?


2. From coding assistant to development workforce

The common pattern is conversational:

Human asks → Copilot edits → Human reviews → Copilot fixes

That works for small tasks. It doesn’t scale to semi-unattended development.

The stronger pattern treats Copilot as an implementation worker inside a larger harness:

Orchestrator issues bounded work order
Copilot implements inside a component boundary
Tests & benchmarks verify the result
Spec reviewer checks intent
Code reviewer checks quality
Variant archive records outcome
Router learns which expert strategy worked

Copilot is not the manager. The management layer is the system we build around it.


3. The Agentic Source-of-Truth Trio

Every component gets three local context files that form an agent-readable memory boundary:

component/
  SPEC.instructions.md   ← Permanent contract
  STATE.md               ← Live handoff marker
  PROMPTS.log            ← Append-only provenance

SPEC.instructions.md

Not documentation. An operational contract. It tells the agent:

  • What the component does and must not do
  • Owned interfaces and integration boundaries
  • Success criteria, test requirements, and performance targets
  • Whether Shadow Scaling is required

Before any Copilot session touches a component, it reads this file first.

STATE.md

The “you are here” map. Answers:

  • What was last completed
  • What currently works / what is broken
  • Which implementation is active
  • What the next action is
  • What must not be changed

Every Copilot task updates STATE.md before stopping.

PROMPTS.log

The forensic trail. Records:

  • Prompt used, task attempted, files touched
  • Tests run, outcome, follow-up needed

When something breaks, we can ask not just “what changed?” but “what instruction caused this change?”


4. Shadow Scaling: forward-only development for coding agents

Traditional development prizes clean edits to existing code. LLM coding agents are often much better at creating a clean new module from a clear specification than safely performing broad surgery on shared state.

Shadow Scaling accepts this reality and turns it into a design pattern.

Definition: New behavior is implemented as a parallel variant that shadows the stable system before being promoted.

implementations/
  baseline.py
  algorithmic_rewrite_v1.py
  vectorized_v1.py
  memory_layout_v1.py

The old system remains stable. The new system runs under tests and benchmarks. Only after the variant proves itself is it promoted.

Why this helps Copilot:

  • Old code stays available as reference
  • New ideas live in isolated files
  • Failed attempts are cheap to discard
  • Multiple strategies test in parallel
  • Promotion is evidence-based, not confidence-based
  • Rollback is trivial

This isn’t laziness. It’s agent-aware architecture.

Shadow Scaling is not an excuse to leave the codebase messy forever. It separates exploration from consolidation. During exploration, agents create parallel variants. During promotion, the best variant becomes the active implementation. During cleanup, losing variants can be archived, compressed, or removed after their benchmark and prompt lineage have been preserved.


5. Subagent-driven development: two-stage review

A reliable Copilot workflow needs implementation and review. The core loop:

Implementer subagent
  ↓
Spec compliance reviewer (must pass first)
  ↓
Code quality reviewer
  ↓
Benchmark gate
  ↓
Archive / promote / reject

Each subagent gets fresh context. Reviewers do not inherit the implementer’s conversation history. That prevents semantic contamination and forces independent verification.

Stage 1: Spec Compliance Did the implementation satisfy SPEC.instructions.md? Did it underbuild, overbuild, or violate boundaries? Did it update STATE.md and append PROMPTS.log? Did it follow Shadow Scaling?

Stage 2: Code Quality Only after spec passes: Are tests meaningful? Are edge cases covered? Is coupling minimized? Are there obvious performance regressions?

There is no point polishing code that built the wrong thing.


6. The orchestrator: the manager you leave running

The orchestrator is the unattended-or-semi-unattended process manager. It does not write code. It manages the loop:

  1. Validate component preconditions (SPEC, STATE, PROMPTS.log)
  2. Generate a sealed work order
  3. Persist run state in runs/run_<timestamp>/
  4. Wait for implementation_result.json (human paste in v1, automated in v2)
  5. Generate & ingest review results
  6. Run deterministic benchmarks (median-of-N, timeout, correctness hard gate)
  7. Archive variant, update router scores, update STATE.md
  8. Promote winner or reject, emit report.md

Run directory structure:

runs/run_20260504_001/
  run_state.json
  work_order.json / work_order.txt
  incoming/implementation_result.json
  reviews/{spec,quality}_{prompt,result}.json
  benchmark/{baseline,verifier}.json
  archive_update.json
  report.md

Resumable, auditable, and compatible with Copilot Agent Mode.


7. Strict schemas: no free-text handoffs

The orchestrator consumes structured JSON, not vague summaries. Example implementation handoff:

{
  "work_order_id": "uuid",
  "proposal_id": "uuid",
  "component": "expert_algorithmic_rewrite",
  "strategy": "algorithmic_rewrite",
  "status": "DONE",
  "files": [{"path": "proposals/abc123/new_impl.py", "content_base64": "..."}],
  "tests_run": ["pytest -q"],
  "notes": "Forward-only proposal. All tests pass.",
  "concerns": [],
  "author": "copilot-agent-mode"
}

Pydantic validation enforces contract compliance. Invalid outputs trigger structured retry loops.


8. Benchmarks are the truth signal

The system never asks an LLM whether a patch is faster. It measures it.

{
  "tests_passed": true,
  "benchmark_runs_ms": [120.1, 118.9, 119.5],
  "median_latency_ms": 119.5,
  "p95_latency_ms": 120.1,
  "correctness": 1.0,
  "speedup_vs_baseline": 3.75,
  "success": true
}

Correctness is a hard gate. Performance is measured with repeated runs. Promotion happens only when evidence crosses the threshold.


9. Variant archive & promotion policy

Every attempt is valuable. The archive stores component, strategy, prompt ID, review results, benchmark scores, and promotion status. Failed variants become negative training data for routing.

Promotion requires:

  • correctness == 1.0
  • All tests pass
  • Spec & quality review approved
  • Benchmark improvement ≥ target delta
  • Public interface unchanged

The winning variant becomes active. Others remain as evidence. This is how the system learns without updating model weights.


10. Router learning: simple, measurable, useful

The router decides which expert strategy to try next. No complex RL. A simple exponential moving average suffices:

new_score = α * reward + (1 - α) * old_score
reward = log(speedup) if tests pass
reward = -1.0 if tests fail or blocked

Reward weights map directly to the utility function from the SICA paper: U = w_score * performance + w_cost * (1 - cost) + w_time * (1 - time). Over time, the system learns which strategies work for which problem buckets. That is the practical version of a Mixture-of-Experts coding system.


11. Why this is self-improving

This system improves in three compounding layers:

  1. Target code: Copilot proposes variants → benchmarks select winners → stable system gets better.
  2. Routing policy: Router learns which expert strategies produce better variants for specific problem features.
  3. Harness itself: Because the harness is componentized, Copilot can improve the scaffolder, work-order generator, benchmark runner, review prompts, archive, and orchestrator using the exact same loop.

That’s the recursive pattern:

Copilot builds the harness
harness manages Copilot
Copilot improves the harness
harness manages Copilot better

This is non-gradient self-improvement through code, prompts, tests, and benchmark feedback. It matches the SICA architecture: archive-driven meta-improvement, fresh subagent context, asynchronous overseer checks, and utility-based promotion.


12. The competition angle

Most teams will use Codex or Copilot to write code. That’s the baseline.

The stronger submission shows a system that manages Codex/Copilot as a development workforce.

The claim:

We built a componentized, shadow-scaling development harness that turns Copilot Agent Mode into a bounded, reviewable, benchmark-driven swarm of coding subagents.

The demo compares:

Approach Target Metric
Single prompt baseline to measure
Shadow Scaling, 3 experts target: >2x success rate
Router-guided selection target: improved speedup / fewer failed attempts

What judges will see:

  • Live terminal output showing work order → Copilot paste → review gates → benchmark → promotion
  • runs/ directory proving reproducibility
  • PROMPTS.log and variant_archive showing forensic lineage
  • Router scores shifting as the system learns which strategies work
  • report.md with before/after metrics, rejected variants, and promotion rationale

That’s far more compelling than a single optimized file.


13. Minimal implementation plan (72-hour hackathon map)

MVP:

  • one component
  • one slow function
  • three Copilot-generated shadow variants
  • tests + benchmark
  • one promoted winner
  • one generated report
Phase Deliverable Success Metric
0. Seed (0–4h) AGENTS.md, pyproject.toml, components/README.md Repo ready for Copilot Agent Mode
1. Harness Foundation (4–16h) component_scaffolder, run_store, work_orders Scaffolder creates valid SPEC/STATE/PROMPTS; work orders generate sealed prompts
2. Verification & Review (16–30h) proposal_ingest, reviews, benchmark_runner Two-stage gate passes on clean impl; benchmark returns median + correctness
3. Learning & Promotion (30–46h) variant_archive, router_scores, report_builder Archive stores variants; router updates EMA; report generates
4. Orchestration (46–60h) orchestrator, overseer (timeout/cost guardrails) End-to-end mock run completes; state machine transitions cleanly
5. Competition Experts (60–72h) 3 expert components + baseline functions 3/5 functions improve ≥1 metric with 0 correctness loss; demo video ready

v1 Reality Check: The orchestrator prints work orders. You paste into Copilot Agent Mode. Drop implementation_result.json into runs/<id>/incoming/. The system picks up, reviews, benchmarks, archives, and loops. v2 automates the paste. v1 wins because it’s auditable, safe, and ships in 72 hours.


14. What makes this different

This isn’t prompt engineering. It’s a development runtime for agentic coding.

The important design moves:

  1. Component-local memory keeps agents grounded
  2. Shadow Scaling keeps changes reversible
  3. Subagent review catches spec drift and quality failures
  4. Strict schemas prevent vague handoffs
  5. Benchmarks provide truth
  6. Variant archives preserve learning
  7. Router scores improve expert selection
  8. The orchestrator makes the loop unattended

The result is a codebase that can absorb large amounts of Copilot work without collapsing under context drift.


15. Closing argument

The future of AI coding isn’t a single genius agent editing a codebase end-to-end. It’s a disciplined workflow where bounded agents operate inside clear context boundaries, create reversible shadow implementations, and are judged by tests, benchmarks, reviews, and promotion policies.

Copilot is already powerful enough to do real work. The missing layer is the system that makes that work reliable.

Shadow Scaling is that layer.

It lets us stop asking, “Can the agent make the right edit?” and start asking, “Can the system generate enough safe variants that the best one can be proven and promoted?”

That’s how Copilot becomes more than an assistant. It becomes a scalable development substrate.