# Advanced Agents From First Principles 32: Which Capabilities Are Actually Worth Building? Design a Capability Portfolio
# Book: Advanced Agents From First Principles
# Chapter URL: https://programmer.ie/books/advanced-agents-from-first-principles/32-chapter/
# Prose fingerprint (sha256, normalized): 973bfaca769421ca14adbae1eceb00436255f30cd2ac3b9d001d46a411b61ff9
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

Once an agent can learn new capabilities safely, a new problem appears.
Which capabilities should it learn?
That question is much harder than it sounds.
The naive answer is:
all of them.
If a coding agent cannot refactor a distributed transaction layer, teach it.
If a research agent cannot evaluate a new source class, teach it.
If a browser agent cannot operate a new workflow, teach it.
If a DevOps agent cannot recover a new failure mode, teach it.
But every new capability has a cost.
It needs:
experiments, benchmark cases, verifiers, tools, policies, observability, release gates, incident handling, maintenance, and eventually on-call responsibility. Some capabilities are cheap to acquire but difficult to verify.
Some are easy to verify but almost never used.
Some look valuable until you realize ordinary deterministic software can solve the problem better.
Some should remain human decisions.
Some should be explicitly prohibited.
Step 31 gave us sandboxed capability acquisition.
The agent can now explore tasks outside its production competence envelope without expanding its own authority.
That creates a planning problem:
Which unsupported capabilities are worth spending engineering, benchmark, verifier and operational capacity to acquire at all?
That is a capability portfolio problem.
The core rule for this post is:
Do not maximize what the agent can do. Maximize the useful, verifiable capability the platform should own.
The Search Problem: “What AI Agent Capabilities Should I Build First?” Agent roadmaps often grow feature by feature.
A customer asks for something.
A benchmark exposes a gap.
A model release makes a new task possible.
Someone sees a demo.
A competitor adds a feature.
The roadmap expands.
That produces a familiar architecture:
more tools ↓ more prompts ↓ more routes ↓ more memory ↓ more policies ↓ more failure modes The question is not whether a capability can be added.
The question is whether it should be.
For every proposed capability, compare at least five strategies:
1. acquire agent capability 2. implement deterministic software 3. route to a human 4. use a stronger external specialist/model 5. do not support the capability If option 2 is cheaper and more reliable, build software.
If option 3 is rare and high-risk, keep the human boundary.
If option 4 handles the workload economically, routing may beat ownership.
If option 5 is acceptable, exclusion can be the correct architecture.
Capability acquisition should have to win that competition.
Capability Is an Asset With Carrying Cost A production capability is not just something the model can do.
It is something the platform must continue to support.
That means a capability has both acquisition cost and carrying cost.
A useful model is:
capability lifecycle cost = discovery + verifier construction + benchmark maintenance + integration + rollout + observability + incident burden + dependency maintenance + regression testing + deprecation cost This matters because the cost profile is often asymmetric.
The first demo may take one afternoon.
Keeping the capability reliable for three years may take far more work.
So portfolio planning should compare lifetime value against lifetime burden, not demo cost against immediate usefulness.
Start With Capability Candidates Represent unsupported capabilities explicitly.
from dataclasses import dataclass from typing import Literal CapabilityDisposition = Literal[ "acquire", "deterministic", "human", "external_specialist", "defer", "exclude", ] @dataclass(frozen=True) class CapabilityCandidate: capability_id: str task_family: str user_value: float expected_frequency: float verifier_strength: float acquisition_cost: float maintenance_cost: float operational_risk: float blast_radius: float reversibility: float human_alternative_cost: float deterministic_alternative_quality: float external_specialist_quality: float evidence_confidence: float The numbers do not need to pretend to be exact probabilities.
Ranges or ordinal bands are often better:
user value: high usage frequency: medium verifier strength: strong acquisition cost: medium maintenance cost: low operational risk: low The important thing is that assumptions become explicit.
Do Not Collapse Everything Into One Magic Score A capability with enormous user value but no credible verifier should not automatically beat a lower-value capability with excellent verification.
Likewise, a capability that creates catastrophic external risk should not be made acceptable by adding enough value points elsewhere.
Use hard gates first.
For example:
def hard_capability_gate(candidate: CapabilityCandidate) -> str | None: if candidate.operational_risk > 0.95 and candidate.verifier_strength < 0.8: return "exclude" if candidate.blast_radius > 0.9 and candidate.reversibility < 0.2: return "human" return None Only after hard constraints pass should you compare softer trade-offs.
This follows a rule we have used repeatedly throughout the series:
Hard safety, authority and verification constraints should dominate weighted optimization.
Expected Capability Value For candidates that pass hard constraints, a rough expected-value model can help.
Something like:
expected capability value ≈ expected task volume × value per successful task × expected verified success - acquisition cost - maintenance cost - expected incident loss - verifier cost You can add opportunity cost:
net portfolio value = expected capability value - value of best alternative use of engineering capacity That last term is crucial.
A capability does not compete against zero.
It competes against:
fixing false-success incidents, improving a verifier, reducing latency, simplifying routing, adding a deterministic implementation, improving existing high-volume capabilities. The correct comparison is always against the best available alternative use of scarce capacity.
Verifier Availability Changes the Portfolio Suppose your coding agent has two unsupported capabilities:
A. update dependency versions safely B. perform broad architecture modernization Capability A may have excellent external verification:
dependency resolver, compile, unit tests, integration tests, vulnerability scanner. Capability B may have weak verification:
vague design quality, long-horizon maintainability, uncertain architecture consequences. The model may be more impressive on B.
But A can still be the much better production investment.
This gives us another useful principle:
The capability portfolio is partly a verifier portfolio.
If you cannot verify a valuable capability today, you may decide that the right investment is not capability acquisition.
It is verifier acquisition.
valuable unsupported capability ↓ verification too weak ↓ build verifier first ↓ then reconsider capability acquisition That is very different from simply buying a stronger model.
Some Capabilities Should Become Ordinary Software Imagine a research agent frequently needs to normalize publication dates from known metadata formats.
You could teach the model to do this more reliably.
Or you could write:
from datetime import datetime KNOWN_FORMATS = [ "%Y-%m-%d", "%Y/%m/%d", "%d %b %Y", "%B %d, %Y", ] def parse_publication_date(value: str) -> datetime | None: for fmt in KNOWN_FORMATS: try: return datetime.strptime(value, fmt) except ValueError: pass return None If the invariant is known, deterministic software has major advantages:
repeatability, low cost, easy testing, transparent failure, simple maintenance. A capability portfolio should actively search for these opportunities.
Do not ask:
Can the agent learn this?
Ask:
Should this remain an agent problem at all?
Human Work Can Be the Correct End State Some tasks are:
rare, consequential, context-heavy, difficult to verify, politically or organizationally sensitive. Automating them may have poor economics even if technically possible.
Suppose a DevOps agent sees a one-off recovery procedure that could delete production data if applied incorrectly.
You may be able to create a sandboxed acquisition program around it.
But if the event occurs once every three years, requires deep organizational context, and has enormous blast radius, the portfolio decision may be:
agent prepares diagnosis agent proposes recovery plan human incident commander authorizes and executes That is not a failed capability roadmap.
It is a correct authority allocation.
External Specialists Can Beat Internal Capability Not every capability needs to be owned internally.
Suppose your local agent stack handles 95% of coding tasks economically.
The remaining 5% require unusually strong reasoning.
You could spend months acquiring that tail capability.
Or route those cases to a stronger specialist model.
The portfolio comparison becomes:
internal acquisition vs specialist routing cost If the workload is rare, external routing may dominate.
But remember Step 10’s mixture-of-agents lesson:
Specialist routing earns its place only if heterogeneity creates measurable value.
A second provider with the same failure modes is not meaningful diversification.
Exclusion Is a Capability Decision Some tasks should be explicitly unsupported.
This is different from merely not having implemented them yet.
Represent exclusion as a first-class portfolio state.
@dataclass(frozen=True) class CapabilityPolicy: capability_id: str disposition: CapabilityDisposition reason: str policy_version: str Example:
capability: autonomous irreversible financial transfer policy: exclude reason: verifier and authority requirements not satisfiable That makes the boundary inspectable and testable.
It also prevents the agent from treating every unsupported task as an invitation to improvise.
Capability Dependencies Form a Graph Capabilities are rarely independent.
A higher-level capability may depend on several lower-level ones.
For example:
autonomous dependency upgrade ├── detect package ecosystem ├── resolve compatible versions ├── modify manifest ├── run build ├── run tests ├── inspect vulnerability changes └── verify lockfile consistency So maintain a capability dependency graph.
CAPABILITY_GRAPH = { "autonomous_dependency_upgrade": { "detect_package_ecosystem", "resolve_versions", "modify_manifest", "run_build", "run_tests", "scan_vulnerabilities", "verify_lockfile", } } This prevents portfolio planning from selecting an attractive top-level capability while ignoring missing prerequisites.
It also exposes reusable lower-level investments.
A strong verifier or tool may unlock several capabilities at once.
That creates option value.
Infrastructure Capabilities Can Have High Option Value Suppose three future capabilities all depend on reliable browser state capture.
You could independently attack each capability.
Or improve the shared browser observation layer.
The second investment may unlock multiple roadmap items.
So include dependency leverage:
portfolio value += downstream capabilities enabled Examples of high-option-value investments:
stronger verifier infrastructure, reproducible sandboxes, repository snapshots, browser state capture, provenance systems, tool schema validation, common retrieval pipelines. The most valuable capability investment may not itself be user-facing.
Prioritize Bottlenecks, Not Demos Suppose the model can generate an excellent SQL migration plan.
But your platform cannot reliably:
snapshot schema state, simulate the migration, estimate lock impact, verify postconditions, roll back safely. Then the missing capability is not “better SQL reasoning.”
The bottleneck is the execution/verification substrate.
This is another recurring pattern across the series:
model capability > platform verification capability When that happens, investing in model intelligence produces diminishing operational value.
Build the bottleneck.
Use Observed Demand, Not Imagined Demand Capability roadmaps are vulnerable to speculative feature inflation.
Track unsupported requests.
@dataclass(frozen=True) class UnsupportedTaskEvent: task_family: str requested_authority: str user_or_workload: str frequency_weight: float current_fallback: str fallback_cost: float outcome: str Then ask:
how often does this happen? who is affected? what is the current fallback? how expensive is that fallback? how much time is actually lost? how often does the unsupported capability block a high-value workflow? A capability requested twice a year should not automatically beat a mundane capability needed 30,000 times a day.
But Frequency Is Not Everything Rare high-consequence capabilities may still matter.
An incident-response capability might be used infrequently but produce enormous value when needed.
So distinguish:
frequency from criticality A good portfolio contains different capability classes:
high-frequency efficiency capabilities high-value workflow capabilities rare resilience capabilities measurement/verifier capabilities platform-enabling capabilities Do not let one class consume the entire roadmap.
Build a Portfolio, Not a Ranked List A simple rank order tends to overconcentrate investment.
Instead, allocate capability budget across categories.
For example:
40% existing high-volume capability improvement 20% verifier / measurement capability 15% new high-value acquisition 10% resilience / incident capabilities 10% platform-enabling capability 5% exploratory sandbox research Those percentages are examples, not universal targets.
The principle is diversification.
If all investment goes into new autonomy, reliability debt compounds.
If all investment goes into hardening, the platform stops growing.
Portfolio thinking makes the trade-off explicit.
Acquisition Confidence Should Affect Position Size Suppose two candidate capabilities have similar expected value.
One has strong evidence:
repeated unsupported demand, clear verifier, low-risk sandbox, known deterministic baselines. The other is speculative:
uncertain demand, unclear success criteria, weak verifier, expensive experiments. Do not allocate equal budgets.
Start the uncertain capability with a smaller experimental position.
high confidence → larger acquisition budget low confidence → cheap discovery experiment first This is exactly where Expected Value of Information from Step 18 reappears.
The first investment may simply be to learn whether the capability is worth pursuing.
Portfolio Experiments Should Answer Decision Questions A capability-discovery experiment should change a portfolio decision.
Bad experiment:
See if the model can do Kubernetes recovery.
Better experiment:
On 30 representative sandboxed recovery scenarios, can the candidate system identify a safe recovery plan with zero false-success events under the required verifier, at less than 2× current human triage cost?
That experiment can support a decision.
The first one merely produces a demo.
Measure Acquisition Cost per Promoted Capability Track the full acquisition funnel.
candidate capabilities ↓ sandbox experiments ↓ competence candidates ↓ held-out validation ↓ shadow ↓ limited authority ↓ promoted production capability Useful metrics include:
acquisition cost per promoted capability, experiments per promotion, verifier-building cost, promotion failure rate, time to competence evidence, time to production authority, false-competence rate, post-promotion incident rate, maintenance cost per capability, utilization after promotion. A capability that looked valuable but is barely used after promotion is portfolio evidence.
Use it.
Measure Realized Value After Promotion Do not stop measurement at successful rollout.
Compare predicted and realized value.
@dataclass(frozen=True) class CapabilityRealization: capability_id: str predicted_monthly_value: float realized_monthly_value: float predicted_maintenance_cost: float realized_maintenance_cost: float predicted_incident_loss: float realized_incident_loss: float If you repeatedly overestimate demand or underestimate maintenance burden, the portfolio process itself needs calibration.
This mirrors Step 28’s reliability-prioritization calibration.
Planning quality should also be measurable.
Capabilities Can Be Retired Competence is not permanent.
Neither is portfolio membership.
A capability should be considered for retirement when:
usage collapses, maintenance becomes expensive, verifier quality degrades, dependencies disappear, a deterministic replacement becomes available, an external specialist becomes cheaper, reliability falls outside its SLO, authority can no longer be justified. Add explicit lifecycle states:
CANDIDATE EXPERIMENTAL VALIDATING LIMITED PRODUCTION DEPRECATED RETIRED EXCLUDED A system that can only add capabilities will accumulate permanent complexity.
Deletion must be part of the portfolio lifecycle.
Retirement Is Different From Competence Contraction These are related but distinct.
Competence contraction says:
we no longer have enough evidence for this authority level Capability retirement says:
we no longer want to own this capability A capability may still work technically and yet be retired because its economics are poor.
That is an architectural business decision, not a model failure.
Coding Agent Example Suppose your coding platform sees unsupported requests for:
A. dependency upgrades B. database schema migrations C. frontend accessibility remediation D. broad architecture rewrites E. release-note generation Portfolio analysis might produce:
A → acquire high frequency strong verifier reversible in branch B → limited acquisition high value stronger risk controls required C → acquire repeatable checks available D → human + agent proposal weak verifier large blast radius E → deterministic/template + model assist low need for autonomous agent capability That is more useful than a roadmap that simply says:
make the coding agent better.
Research Agent Example Unsupported capabilities:
A. summarize unfamiliar source types B. perform source provenance analysis C. make causal scientific claims D. translate documents E. detect conflicting evidence Possible dispositions:
A → acquire carefully B → high priority platform capability C → human/research-method gate D → external model or deterministic pipeline E → acquire; strong user value and verification potential Notice that the glamorous capability—causal scientific reasoning—may not be the first one to own.
Better provenance and contradiction detection may create more dependable value.
Browser Agent Example Unsupported workflows:
A. read account balances B. update profile settings C. submit support forms D. purchase items E. transfer money Portfolio outcome:
A → observe capability B → reversible bounded action C → acquire with submit verification D → human-before-commit E → exclude or dual-control only Again:
capability portfolio planning is partly authority planning.
DevOps Agent Example Unsupported tasks:
A. summarize deployment health B. restart stateless service C. scale replicas D. modify database failover topology E. rotate production secrets Potential portfolio:
A → acquire immediately B → acquire with health verifier C → acquire with policy bounds D → proposal + human incident authority E → deterministic secure workflow with dual control The best agent roadmap does not maximize autonomous operations.
It allocates the correct implementation mode to each operation.
Mixture-of-Agents Changes the Portfolio Question In a multi-agent runtime, the question is not just:
Should we acquire capability X?
It becomes:
Which component should own capability X?
Possible answers:
local model frontier model retrieval specialist code-analysis specialist critic verifier deterministic tool human reviewer Ownership should minimize correlated failure and operational burden.
Do not duplicate capability across specialists unless redundancy has measured value.
Avoid Capability Inflation Through Composition Advanced systems can appear to support a capability because several partial components happen to cooperate successfully once.
That does not automatically establish a stable production capability.
For composed capabilities, evaluate joint competence.
router × specialist × tool × memory × verifier × scheduler If one dependency is weak, the composed capability is weak.
Step 30’s competence-envelope logic applies to the full behavioral system.
Put Capability Portfolio State in the Release System Capability policy should be versioned with behavioral releases.
@dataclass(frozen=True) class CapabilityPortfolioEntry: capability_id: str lifecycle_state: str disposition: CapabilityDisposition competence_claim_ref: str | None max_authority: str required_verifiers: tuple[str, ...] owner: str policy_version: str That makes several questions answerable:
Which capabilities does release 42 own? Which are experimental only? Which require human approval? Which are excluded? Which verifier versions are required? Which capability changed between releases? This also improves Step 25 replay and Step 26 forensics.
Portfolio Changes Should Have Promotion Gates Too A capability roadmap decision is not the same as production activation.
Use stages:
PORTFOLIO_CANDIDATE ↓ DISCOVERY_APPROVED ↓ SANDBOX_ACQUISITION ↓ COMPETENCE_CANDIDATE ↓ PROMOTION_REVIEW ↓ LIMITED_CAPABILITY ↓ PRODUCTION_CAPABILITY And explicitly support:
DEFERRED REJECTED EXCLUDED RETIRED The roadmap itself becomes evidence-driven.
Capability Budget Should Respond to Reliability State Step 27 gave us error budgets.
Use them here.
If the platform is healthy:
more exploratory acquisition budget If false-success burn rises:
less new capability work more verifier / containment / remediation work If the system enters FREEZE:
no production authority expansion Sandbox experiments may continue if they cannot affect production reliability, but promotion should stop.
That connects capability growth directly to operational health.
Security and Safety Capabilities Deserve Separate Treatment Do not let portfolio value override hard exclusions.
Some capabilities require organizational or legal controls regardless of expected value.
Examples may include:
privileged credential operations, irreversible financial actions, destructive production mutations, sensitive data movement. The portfolio system can classify them as:
human-only controlled workflow prohibited rather than trying to optimize them into autonomy.
Build a Capability Review Packet A capability proposal should include enough evidence for independent review.
Capability: autonomous dependency upgrade Observed demand: - 14,200 eligible tasks / month Current fallback: - engineer performs manually Expected value: - high Verifier availability: - compile - unit tests - integration tests - dependency resolution - vulnerability scan Acquisition estimate: - medium Maintenance estimate: - low-medium Risk: - bounded repository changes Alternatives: - deterministic updater: partial coverage - external specialist: higher marginal cost - human-only: expensive at current volume Proposed disposition: - sandbox acquisition That is a much stronger planning artifact than:
We should teach the agent dependency upgrades.
Independent Portfolio Review Matters The team that wants to build a capability may systematically overestimate its value.
So separate:
capability proposer from portfolio reviewer For larger systems, the reviewer should challenge:
expected demand, verifier quality, maintenance burden, risk assumptions, deterministic alternatives, human fallback cost, downstream dependencies. This is the capability equivalent of independent release promotion.
Do Not Let the Agent Choose Its Own Portfolio The agent may help discover demand.
It may propose capability candidates.
It may run sandbox experiments.
It may estimate where it struggles.
But it should not control:
portfolio objectives, prohibited capability classes, risk weights, verifier requirements, acquisition budgets, authority ceilings, promotion thresholds. Those remain external control-plane policy.
Otherwise self-improvement turns into self-expansion.
Capability Portfolio Failure Modes Failure 1: Demo-driven roadmap A capability is funded because it looks impressive.
Fix:
Measure demand, verifier strength, alternatives and carrying cost.
Failure 2: Capability maximalism Everything technically possible becomes a roadmap item.
Fix:
Require acquisition to beat deterministic, human, external-specialist and exclusion alternatives.
Failure 3: Verifier blindness High-value capability is acquired without strong acceptance evidence.
Fix:
Treat verifier availability as a portfolio dependency.
Failure 4: Maintenance blindness Acquisition cost is estimated but long-term support cost is ignored.
Fix:
Use lifecycle cost.
Failure 5: Frequency bias Only common tasks receive investment.
Fix:
Include criticality and resilience value.
Failure 6: Prestige bias Sophisticated reasoning capabilities outrank mundane infrastructure capabilities.
Fix:
Include dependency leverage and option value.
Failure 7: Irreversible roadmap growth Capabilities are never retired.
Fix:
Support deprecation and retirement as normal portfolio actions.
Failure 8: Agent-controlled expansion The system proposes and approves its own new authority.
Fix:
Keep portfolio, competence and authority promotion external.
Test the Portfolio Process The portfolio mechanism itself can be wrong.
Track historical decisions.
For each promoted capability, compare:
predicted demand vs realized demand predicted acquisition cost vs actual acquisition cost predicted maintenance burden vs actual maintenance burden predicted reliability vs actual reliability predicted user value vs actual user value If the organization systematically mispredicts one dimension, calibrate the decision process.
Minimal Capability Portfolio Runtime You do not need sophisticated optimization initially.
A small explicit registry is enough.
class CapabilityPortfolio: def __init__(self, entries: list[CapabilityPortfolioEntry]): self._entries = {e.capability_id: e for e in entries} def get(self, capability_id: str) -> CapabilityPortfolioEntry | None: return self._entries.get(capability_id) def acquisition_allowed(self, capability_id: str) -> bool: entry = self.get(capability_id) if entry is None: return False return entry.lifecycle_state in { "DISCOVERY_APPROVED", "SANDBOX_ACQUISITION", } def production_allowed(self, capability_id: str) -> bool: entry = self.get(capability_id) if entry is None: return False return entry.lifecycle_state == "PRODUCTION_CAPABILITY" That alone creates a strong separation between:
we are interested in learning this and:
we allow this in production The Architecture So Far The Advanced Agents series has now moved far beyond prompting tricks.
The platform looks something like:
task ↓ competence envelope ↓ authority policy ↓ platform admission ↓ run scheduler ↓ agent/search/tools ↓ verification ↓ fenced commit ↓ postcondition verification ↓ trajectory + provenance ↓ SLO / error-budget accounting ↓ incident / drift / release systems Outside the production path sits another loop:
unsupported demand ↓ capability portfolio ↓ sandbox acquisition ↓ competence evidence ↓ independent promotion ↓ limited authority ↓ production capability This is controlled system growth.
Not unconstrained self-improvement.
The Key Principle The goal of an advanced agent platform is not to become capable of everything.
That target creates endless complexity and increasingly weak evidence.
The better objective is:
Own the smallest portfolio of capabilities that produces the most useful verified value at acceptable operational risk and lifetime cost.
Sometimes the correct investment is a new capability.
Sometimes it is a better verifier.
Sometimes it is deterministic software.
Sometimes it is human expertise.
Sometimes it is a stronger routed specialist.
Sometimes it is deleting a capability you no longer need.
And sometimes the correct answer is:
We should not do this at all.
That is not a limitation of good architecture.
It is part of good architecture.
Next: Capability Dependencies and Platform Architecture Once capabilities become an explicit portfolio, another question emerges.
Capabilities share infrastructure.
They depend on:
common tools, common verifiers, common memory, common retrieval, common execution environments, common policies. A change to one shared substrate can unlock—or break—many capabilities at once.
So the next stage is to model the capability dependency graph as a first-class platform architecture:
Which shared primitives unlock the most verified capability, and which shared dependencies create dangerous correlated failure?
That takes us from managing a list of agent capabilities to managing the architecture underneath the capability portfolio.