Which Capabilities Are Actually Worth Building? Design a Capability Portfolio
Once an agent can learn new capabilities safely, a new problem appears.
Which capabilities should it learn?
That question is much harder than it sounds.
The naive answer is:
all of them.
If a coding agent cannot refactor a distributed transaction layer, teach it.
If a research agent cannot evaluate a new source class, teach it.
If a browser agent cannot operate a new workflow, teach it.
If a DevOps agent cannot recover a new failure mode, teach it.
But every new capability has a cost.
It needs:
- experiments,
- benchmark cases,
- verifiers,
- tools,
- policies,
- observability,
- release gates,
- incident handling,
- maintenance,
- and eventually on-call responsibility.
Some capabilities are cheap to acquire but difficult to verify.
Some are easy to verify but almost never used.
Some look valuable until you realize ordinary deterministic software can solve the problem better.
Some should remain human decisions.
Some should be explicitly prohibited.
Step 31 gave us sandboxed capability acquisition.
The agent can now explore tasks outside its production competence envelope without expanding its own authority.
That creates a planning problem:
Which unsupported capabilities are worth spending engineering, benchmark, verifier and operational capacity to acquire at all?
That is a capability portfolio problem.
The core rule for this post is:
Do not maximize what the agent can do. Maximize the useful, verifiable capability the platform should own.
The Search Problem: “What AI Agent Capabilities Should I Build First?”
Agent roadmaps often grow feature by feature.
A customer asks for something.
A benchmark exposes a gap.
A model release makes a new task possible.
Someone sees a demo.
A competitor adds a feature.
The roadmap expands.
That produces a familiar architecture:
more tools
↓
more prompts
↓
more routes
↓
more memory
↓
more policies
↓
more failure modes
The question is not whether a capability can be added.
The question is whether it should be.
For every proposed capability, compare at least five strategies:
1. acquire agent capability
2. implement deterministic software
3. route to a human
4. use a stronger external specialist/model
5. do not support the capability
If option 2 is cheaper and more reliable, build software.
If option 3 is rare and high-risk, keep the human boundary.
If option 4 handles the workload economically, routing may beat ownership.
If option 5 is acceptable, exclusion can be the correct architecture.
Capability acquisition should have to win that competition.
Capability Is an Asset With Carrying Cost
A production capability is not just something the model can do.
It is something the platform must continue to support.
That means a capability has both acquisition cost and carrying cost.
A useful model is:
capability lifecycle cost
= discovery
+ verifier construction
+ benchmark maintenance
+ integration
+ rollout
+ observability
+ incident burden
+ dependency maintenance
+ regression testing
+ deprecation cost
This matters because the cost profile is often asymmetric.
The first demo may take one afternoon.
Keeping the capability reliable for three years may take far more work.
So portfolio planning should compare lifetime value against lifetime burden, not demo cost against immediate usefulness.
Start With Capability Candidates
Represent unsupported capabilities explicitly.
from dataclasses import dataclass
from typing import Literal
CapabilityDisposition = Literal[
"acquire",
"deterministic",
"human",
"external_specialist",
"defer",
"exclude",
]
@dataclass(frozen=True)
class CapabilityCandidate:
capability_id: str
task_family: str
user_value: float
expected_frequency: float
verifier_strength: float
acquisition_cost: float
maintenance_cost: float
operational_risk: float
blast_radius: float
reversibility: float
human_alternative_cost: float
deterministic_alternative_quality: float
external_specialist_quality: float
evidence_confidence: float
The numbers do not need to pretend to be exact probabilities.
Ranges or ordinal bands are often better:
user value: high
usage frequency: medium
verifier strength: strong
acquisition cost: medium
maintenance cost: low
operational risk: low
The important thing is that assumptions become explicit.
Do Not Collapse Everything Into One Magic Score
A capability with enormous user value but no credible verifier should not automatically beat a lower-value capability with excellent verification.
Likewise, a capability that creates catastrophic external risk should not be made acceptable by adding enough value points elsewhere.
Use hard gates first.
For example:
def hard_capability_gate(candidate: CapabilityCandidate) -> str | None:
if candidate.operational_risk > 0.95 and candidate.verifier_strength < 0.8:
return "exclude"
if candidate.blast_radius > 0.9 and candidate.reversibility < 0.2:
return "human"
return None
Only after hard constraints pass should you compare softer trade-offs.
This follows a rule we have used repeatedly throughout the series:
Hard safety, authority and verification constraints should dominate weighted optimization.
Expected Capability Value
For candidates that pass hard constraints, a rough expected-value model can help.
Something like:
expected capability value
≈
expected task volume
× value per successful task
× expected verified success
- acquisition cost
- maintenance cost
- expected incident loss
- verifier cost
You can add opportunity cost:
net portfolio value
= expected capability value
- value of best alternative use of engineering capacity
That last term is crucial.
A capability does not compete against zero.
It competes against:
- fixing false-success incidents,
- improving a verifier,
- reducing latency,
- simplifying routing,
- adding a deterministic implementation,
- improving existing high-volume capabilities.
The correct comparison is always against the best available alternative use of scarce capacity.
Verifier Availability Changes the Portfolio
Suppose your coding agent has two unsupported capabilities:
A. update dependency versions safely
B. perform broad architecture modernization
Capability A may have excellent external verification:
- dependency resolver,
- compile,
- unit tests,
- integration tests,
- vulnerability scanner.
Capability B may have weak verification:
- vague design quality,
- long-horizon maintainability,
- uncertain architecture consequences.
The model may be more impressive on B.
But A can still be the much better production investment.
This gives us another useful principle:
The capability portfolio is partly a verifier portfolio.
If you cannot verify a valuable capability today, you may decide that the right investment is not capability acquisition.
It is verifier acquisition.
valuable unsupported capability
↓
verification too weak
↓
build verifier first
↓
then reconsider capability acquisition
That is very different from simply buying a stronger model.
Some Capabilities Should Become Ordinary Software
Imagine a research agent frequently needs to normalize publication dates from known metadata formats.
You could teach the model to do this more reliably.
Or you could write:
from datetime import datetime
KNOWN_FORMATS = [
"%Y-%m-%d",
"%Y/%m/%d",
"%d %b %Y",
"%B %d, %Y",
]
def parse_publication_date(value: str) -> datetime | None:
for fmt in KNOWN_FORMATS:
try:
return datetime.strptime(value, fmt)
except ValueError:
pass
return None
If the invariant is known, deterministic software has major advantages:
- repeatability,
- low cost,
- easy testing,
- transparent failure,
- simple maintenance.
A capability portfolio should actively search for these opportunities.
Do not ask:
Can the agent learn this?
Ask:
Should this remain an agent problem at all?
Human Work Can Be the Correct End State
Some tasks are:
- rare,
- consequential,
- context-heavy,
- difficult to verify,
- politically or organizationally sensitive.
Automating them may have poor economics even if technically possible.
Suppose a DevOps agent sees a one-off recovery procedure that could delete production data if applied incorrectly.
You may be able to create a sandboxed acquisition program around it.
But if the event occurs once every three years, requires deep organizational context, and has enormous blast radius, the portfolio decision may be:
agent prepares diagnosis
agent proposes recovery plan
human incident commander authorizes and executes
That is not a failed capability roadmap.
It is a correct authority allocation.
External Specialists Can Beat Internal Capability
Not every capability needs to be owned internally.
Suppose your local agent stack handles 95% of coding tasks economically.
The remaining 5% require unusually strong reasoning.
You could spend months acquiring that tail capability.
Or route those cases to a stronger specialist model.
The portfolio comparison becomes:
internal acquisition
vs
specialist routing cost
If the workload is rare, external routing may dominate.
But remember Step 10’s mixture-of-agents lesson:
Specialist routing earns its place only if heterogeneity creates measurable value.
A second provider with the same failure modes is not meaningful diversification.
Exclusion Is a Capability Decision
Some tasks should be explicitly unsupported.
This is different from merely not having implemented them yet.
Represent exclusion as a first-class portfolio state.
@dataclass(frozen=True)
class CapabilityPolicy:
capability_id: str
disposition: CapabilityDisposition
reason: str
policy_version: str
Example:
capability: autonomous irreversible financial transfer
policy: exclude
reason: verifier and authority requirements not satisfiable
That makes the boundary inspectable and testable.
It also prevents the agent from treating every unsupported task as an invitation to improvise.
Capability Dependencies Form a Graph
Capabilities are rarely independent.
A higher-level capability may depend on several lower-level ones.
For example:
autonomous dependency upgrade
├── detect package ecosystem
├── resolve compatible versions
├── modify manifest
├── run build
├── run tests
├── inspect vulnerability changes
└── verify lockfile consistency
So maintain a capability dependency graph.
CAPABILITY_GRAPH = {
"autonomous_dependency_upgrade": {
"detect_package_ecosystem",
"resolve_versions",
"modify_manifest",
"run_build",
"run_tests",
"scan_vulnerabilities",
"verify_lockfile",
}
}
This prevents portfolio planning from selecting an attractive top-level capability while ignoring missing prerequisites.
It also exposes reusable lower-level investments.
A strong verifier or tool may unlock several capabilities at once.
That creates option value.
Infrastructure Capabilities Can Have High Option Value
Suppose three future capabilities all depend on reliable browser state capture.
You could independently attack each capability.
Or improve the shared browser observation layer.
The second investment may unlock multiple roadmap items.
So include dependency leverage:
portfolio value
+= downstream capabilities enabled
Examples of high-option-value investments:
- stronger verifier infrastructure,
- reproducible sandboxes,
- repository snapshots,
- browser state capture,
- provenance systems,
- tool schema validation,
- common retrieval pipelines.
The most valuable capability investment may not itself be user-facing.
Prioritize Bottlenecks, Not Demos
Suppose the model can generate an excellent SQL migration plan.
But your platform cannot reliably:
- snapshot schema state,
- simulate the migration,
- estimate lock impact,
- verify postconditions,
- roll back safely.
Then the missing capability is not “better SQL reasoning.”
The bottleneck is the execution/verification substrate.
This is another recurring pattern across the series:
model capability
>
platform verification capability
When that happens, investing in model intelligence produces diminishing operational value.
Build the bottleneck.
Use Observed Demand, Not Imagined Demand
Capability roadmaps are vulnerable to speculative feature inflation.
Track unsupported requests.
@dataclass(frozen=True)
class UnsupportedTaskEvent:
task_family: str
requested_authority: str
user_or_workload: str
frequency_weight: float
current_fallback: str
fallback_cost: float
outcome: str
Then ask:
- how often does this happen?
- who is affected?
- what is the current fallback?
- how expensive is that fallback?
- how much time is actually lost?
- how often does the unsupported capability block a high-value workflow?
A capability requested twice a year should not automatically beat a mundane capability needed 30,000 times a day.
But Frequency Is Not Everything
Rare high-consequence capabilities may still matter.
An incident-response capability might be used infrequently but produce enormous value when needed.
So distinguish:
frequency
from
criticality
A good portfolio contains different capability classes:
high-frequency efficiency capabilities
high-value workflow capabilities
rare resilience capabilities
measurement/verifier capabilities
platform-enabling capabilities
Do not let one class consume the entire roadmap.
Build a Portfolio, Not a Ranked List
A simple rank order tends to overconcentrate investment.
Instead, allocate capability budget across categories.
For example:
40% existing high-volume capability improvement
20% verifier / measurement capability
15% new high-value acquisition
10% resilience / incident capabilities
10% platform-enabling capability
5% exploratory sandbox research
Those percentages are examples, not universal targets.
The principle is diversification.
If all investment goes into new autonomy, reliability debt compounds.
If all investment goes into hardening, the platform stops growing.
Portfolio thinking makes the trade-off explicit.
Acquisition Confidence Should Affect Position Size
Suppose two candidate capabilities have similar expected value.
One has strong evidence:
- repeated unsupported demand,
- clear verifier,
- low-risk sandbox,
- known deterministic baselines.
The other is speculative:
- uncertain demand,
- unclear success criteria,
- weak verifier,
- expensive experiments.
Do not allocate equal budgets.
Start the uncertain capability with a smaller experimental position.
high confidence
→ larger acquisition budget
low confidence
→ cheap discovery experiment first
This is exactly where Expected Value of Information from Step 18 reappears.
The first investment may simply be to learn whether the capability is worth pursuing.
Portfolio Experiments Should Answer Decision Questions
A capability-discovery experiment should change a portfolio decision.
Bad experiment:
See if the model can do Kubernetes recovery.
Better experiment:
On 30 representative sandboxed recovery scenarios, can the candidate system identify a safe recovery plan with zero false-success events under the required verifier, at less than 2× current human triage cost?
That experiment can support a decision.
The first one merely produces a demo.
Measure Acquisition Cost per Promoted Capability
Track the full acquisition funnel.
candidate capabilities
↓
sandbox experiments
↓
competence candidates
↓
held-out validation
↓
shadow
↓
limited authority
↓
promoted production capability
Useful metrics include:
- acquisition cost per promoted capability,
- experiments per promotion,
- verifier-building cost,
- promotion failure rate,
- time to competence evidence,
- time to production authority,
- false-competence rate,
- post-promotion incident rate,
- maintenance cost per capability,
- utilization after promotion.
A capability that looked valuable but is barely used after promotion is portfolio evidence.
Use it.
Measure Realized Value After Promotion
Do not stop measurement at successful rollout.
Compare predicted and realized value.
@dataclass(frozen=True)
class CapabilityRealization:
capability_id: str
predicted_monthly_value: float
realized_monthly_value: float
predicted_maintenance_cost: float
realized_maintenance_cost: float
predicted_incident_loss: float
realized_incident_loss: float
If you repeatedly overestimate demand or underestimate maintenance burden, the portfolio process itself needs calibration.
This mirrors Step 28’s reliability-prioritization calibration.
Planning quality should also be measurable.
Capabilities Can Be Retired
Competence is not permanent.
Neither is portfolio membership.
A capability should be considered for retirement when:
- usage collapses,
- maintenance becomes expensive,
- verifier quality degrades,
- dependencies disappear,
- a deterministic replacement becomes available,
- an external specialist becomes cheaper,
- reliability falls outside its SLO,
- authority can no longer be justified.
Add explicit lifecycle states:
CANDIDATE
EXPERIMENTAL
VALIDATING
LIMITED
PRODUCTION
DEPRECATED
RETIRED
EXCLUDED
A system that can only add capabilities will accumulate permanent complexity.
Deletion must be part of the portfolio lifecycle.
Retirement Is Different From Competence Contraction
These are related but distinct.
Competence contraction says:
we no longer have enough evidence for this authority level
Capability retirement says:
we no longer want to own this capability
A capability may still work technically and yet be retired because its economics are poor.
That is an architectural business decision, not a model failure.
Coding Agent Example
Suppose your coding platform sees unsupported requests for:
A. dependency upgrades
B. database schema migrations
C. frontend accessibility remediation
D. broad architecture rewrites
E. release-note generation
Portfolio analysis might produce:
A → acquire
high frequency
strong verifier
reversible in branch
B → limited acquisition
high value
stronger risk controls required
C → acquire
repeatable checks available
D → human + agent proposal
weak verifier
large blast radius
E → deterministic/template + model assist
low need for autonomous agent capability
That is more useful than a roadmap that simply says:
make the coding agent better.
Research Agent Example
Unsupported capabilities:
A. summarize unfamiliar source types
B. perform source provenance analysis
C. make causal scientific claims
D. translate documents
E. detect conflicting evidence
Possible dispositions:
A → acquire carefully
B → high priority platform capability
C → human/research-method gate
D → external model or deterministic pipeline
E → acquire; strong user value and verification potential
Notice that the glamorous capability—causal scientific reasoning—may not be the first one to own.
Better provenance and contradiction detection may create more dependable value.
Browser Agent Example
Unsupported workflows:
A. read account balances
B. update profile settings
C. submit support forms
D. purchase items
E. transfer money
Portfolio outcome:
A → observe capability
B → reversible bounded action
C → acquire with submit verification
D → human-before-commit
E → exclude or dual-control only
Again:
capability portfolio planning is partly authority planning.
DevOps Agent Example
Unsupported tasks:
A. summarize deployment health
B. restart stateless service
C. scale replicas
D. modify database failover topology
E. rotate production secrets
Potential portfolio:
A → acquire immediately
B → acquire with health verifier
C → acquire with policy bounds
D → proposal + human incident authority
E → deterministic secure workflow with dual control
The best agent roadmap does not maximize autonomous operations.
It allocates the correct implementation mode to each operation.
Mixture-of-Agents Changes the Portfolio Question
In a multi-agent runtime, the question is not just:
Should we acquire capability X?
It becomes:
Which component should own capability X?
Possible answers:
local model
frontier model
retrieval specialist
code-analysis specialist
critic
verifier
deterministic tool
human reviewer
Ownership should minimize correlated failure and operational burden.
Do not duplicate capability across specialists unless redundancy has measured value.
Avoid Capability Inflation Through Composition
Advanced systems can appear to support a capability because several partial components happen to cooperate successfully once.
That does not automatically establish a stable production capability.
For composed capabilities, evaluate joint competence.
router
× specialist
× tool
× memory
× verifier
× scheduler
If one dependency is weak, the composed capability is weak.
Step 30’s competence-envelope logic applies to the full behavioral system.
Put Capability Portfolio State in the Release System
Capability policy should be versioned with behavioral releases.
@dataclass(frozen=True)
class CapabilityPortfolioEntry:
capability_id: str
lifecycle_state: str
disposition: CapabilityDisposition
competence_claim_ref: str | None
max_authority: str
required_verifiers: tuple[str, ...]
owner: str
policy_version: str
That makes several questions answerable:
- Which capabilities does release 42 own?
- Which are experimental only?
- Which require human approval?
- Which are excluded?
- Which verifier versions are required?
- Which capability changed between releases?
This also improves Step 25 replay and Step 26 forensics.
Portfolio Changes Should Have Promotion Gates Too
A capability roadmap decision is not the same as production activation.
Use stages:
PORTFOLIO_CANDIDATE
↓
DISCOVERY_APPROVED
↓
SANDBOX_ACQUISITION
↓
COMPETENCE_CANDIDATE
↓
PROMOTION_REVIEW
↓
LIMITED_CAPABILITY
↓
PRODUCTION_CAPABILITY
And explicitly support:
DEFERRED
REJECTED
EXCLUDED
RETIRED
The roadmap itself becomes evidence-driven.
Capability Budget Should Respond to Reliability State
Step 27 gave us error budgets.
Use them here.
If the platform is healthy:
more exploratory acquisition budget
If false-success burn rises:
less new capability work
more verifier / containment / remediation work
If the system enters FREEZE:
no production authority expansion
Sandbox experiments may continue if they cannot affect production reliability, but promotion should stop.
That connects capability growth directly to operational health.
Security and Safety Capabilities Deserve Separate Treatment
Do not let portfolio value override hard exclusions.
Some capabilities require organizational or legal controls regardless of expected value.
Examples may include:
- privileged credential operations,
- irreversible financial actions,
- destructive production mutations,
- sensitive data movement.
The portfolio system can classify them as:
human-only
controlled workflow
prohibited
rather than trying to optimize them into autonomy.
Build a Capability Review Packet
A capability proposal should include enough evidence for independent review.
Capability: autonomous dependency upgrade
Observed demand:
- 14,200 eligible tasks / month
Current fallback:
- engineer performs manually
Expected value:
- high
Verifier availability:
- compile
- unit tests
- integration tests
- dependency resolution
- vulnerability scan
Acquisition estimate:
- medium
Maintenance estimate:
- low-medium
Risk:
- bounded repository changes
Alternatives:
- deterministic updater: partial coverage
- external specialist: higher marginal cost
- human-only: expensive at current volume
Proposed disposition:
- sandbox acquisition
That is a much stronger planning artifact than:
We should teach the agent dependency upgrades.
Independent Portfolio Review Matters
The team that wants to build a capability may systematically overestimate its value.
So separate:
capability proposer
from
portfolio reviewer
For larger systems, the reviewer should challenge:
- expected demand,
- verifier quality,
- maintenance burden,
- risk assumptions,
- deterministic alternatives,
- human fallback cost,
- downstream dependencies.
This is the capability equivalent of independent release promotion.
Do Not Let the Agent Choose Its Own Portfolio
The agent may help discover demand.
It may propose capability candidates.
It may run sandbox experiments.
It may estimate where it struggles.
But it should not control:
- portfolio objectives,
- prohibited capability classes,
- risk weights,
- verifier requirements,
- acquisition budgets,
- authority ceilings,
- promotion thresholds.
Those remain external control-plane policy.
Otherwise self-improvement turns into self-expansion.
Capability Portfolio Failure Modes
Failure 1: Demo-driven roadmap
A capability is funded because it looks impressive.
Fix:
Measure demand, verifier strength, alternatives and carrying cost.
Failure 2: Capability maximalism
Everything technically possible becomes a roadmap item.
Fix:
Require acquisition to beat deterministic, human, external-specialist and exclusion alternatives.
Failure 3: Verifier blindness
High-value capability is acquired without strong acceptance evidence.
Fix:
Treat verifier availability as a portfolio dependency.
Failure 4: Maintenance blindness
Acquisition cost is estimated but long-term support cost is ignored.
Fix:
Use lifecycle cost.
Failure 5: Frequency bias
Only common tasks receive investment.
Fix:
Include criticality and resilience value.
Failure 6: Prestige bias
Sophisticated reasoning capabilities outrank mundane infrastructure capabilities.
Fix:
Include dependency leverage and option value.
Failure 7: Irreversible roadmap growth
Capabilities are never retired.
Fix:
Support deprecation and retirement as normal portfolio actions.
Failure 8: Agent-controlled expansion
The system proposes and approves its own new authority.
Fix:
Keep portfolio, competence and authority promotion external.
Test the Portfolio Process
The portfolio mechanism itself can be wrong.
Track historical decisions.
For each promoted capability, compare:
predicted demand
vs
realized demand
predicted acquisition cost
vs
actual acquisition cost
predicted maintenance burden
vs
actual maintenance burden
predicted reliability
vs
actual reliability
predicted user value
vs
actual user value
If the organization systematically mispredicts one dimension, calibrate the decision process.
Minimal Capability Portfolio Runtime
You do not need sophisticated optimization initially.
A small explicit registry is enough.
class CapabilityPortfolio:
def __init__(self, entries: list[CapabilityPortfolioEntry]):
self._entries = {e.capability_id: e for e in entries}
def get(self, capability_id: str) -> CapabilityPortfolioEntry | None:
return self._entries.get(capability_id)
def acquisition_allowed(self, capability_id: str) -> bool:
entry = self.get(capability_id)
if entry is None:
return False
return entry.lifecycle_state in {
"DISCOVERY_APPROVED",
"SANDBOX_ACQUISITION",
}
def production_allowed(self, capability_id: str) -> bool:
entry = self.get(capability_id)
if entry is None:
return False
return entry.lifecycle_state == "PRODUCTION_CAPABILITY"
That alone creates a strong separation between:
we are interested in learning this
and:
we allow this in production
The Architecture So Far
The Advanced Agents series has now moved far beyond prompting tricks.
The platform looks something like:
task
↓
competence envelope
↓
authority policy
↓
platform admission
↓
run scheduler
↓
agent/search/tools
↓
verification
↓
fenced commit
↓
postcondition verification
↓
trajectory + provenance
↓
SLO / error-budget accounting
↓
incident / drift / release systems
Outside the production path sits another loop:
unsupported demand
↓
capability portfolio
↓
sandbox acquisition
↓
competence evidence
↓
independent promotion
↓
limited authority
↓
production capability
This is controlled system growth.
Not unconstrained self-improvement.
The Key Principle
The goal of an advanced agent platform is not to become capable of everything.
That target creates endless complexity and increasingly weak evidence.
The better objective is:
Own the smallest portfolio of capabilities that produces the most useful verified value at acceptable operational risk and lifetime cost.
Sometimes the correct investment is a new capability.
Sometimes it is a better verifier.
Sometimes it is deterministic software.
Sometimes it is human expertise.
Sometimes it is a stronger routed specialist.
Sometimes it is deleting a capability you no longer need.
And sometimes the correct answer is:
We should not do this at all.
That is not a limitation of good architecture.
It is part of good architecture.
Next: Capability Dependencies and Platform Architecture
Once capabilities become an explicit portfolio, another question emerges.
Capabilities share infrastructure.
They depend on:
- common tools,
- common verifiers,
- common memory,
- common retrieval,
- common execution environments,
- common policies.
A change to one shared substrate can unlock—or break—many capabilities at once.
So the next stage is to model the capability dependency graph as a first-class platform architecture:
Which shared primitives unlock the most verified capability, and which shared dependencies create dangerous correlated failure?
That takes us from managing a list of agent capabilities to managing the architecture underneath the capability portfolio.