Utility-Learning Tension in Self-Modifying Agents
As systems trend toward superintelligence, a natural modeling premise is that
agents can self-improve along every facet of their own design. We formalize
this with a five-axis decomposition and a decision layer, separating incentives
from learning behavior and analyzing axes in isolation. Our central result
identifies and introduces a sharp utility--learning tension, the structural
conflict in self-modifying systems whereby utility-driven changes that improve
immediate or expected performance can also erode the statistical preconditions
for reliable learning and generalization. Our findings show that
distribution-free guarantees are preserved iff the policy-reachable model
family is uniformly capacity-bounded; when capacity can grow without limit,
utility-rational self-changes can render learnable tasks unlearnable. Under
standard assumptions common in practice, these axes reduce to the same capacity
criterion, yielding a single boundary for safe self-modification. Numerical
experiments across several axes validate the theory by comparing destructive
utility policies against our proposed two-gate policies that preserve
learnability.
68e46e6ce4e093a7044e4cba
Charles L. Wang
claimed_verified
2025-10-07T12:27:25.165Z
67c604ff3f8c7b027d479671
https://cdn-avatars.huggingface.co/v1/production/uploads/67c604ff3f8c7b027d479671/IhfN3eWRQ8iNfcsDKA1Ep.png
Charles Wang
user
charleslwang
68e46e6ce4e093a7044e4cbb
Keir Dorchen
68e46e6ce4e093a7044e4cbc
Peter Jin
Utility-Learning Tension in Self-Modifying Agents
As systems trend toward superintelligence, a natural modeling premise is that
agents can self-improve along every facet of their own design. We formalize
this with a five-axis decomposition and a decision layer, separating incentives
from learning behavior and analyzing axes in isolation. Our central result
identifies and introduces a sharp utility--learning tension, the structural
conflict in self-modifying systems whereby utility-driven changes that improve
immediate or expected performance can also erode the statistical preconditions
for reliable learning and generalization. Our findings show that
distribution-free guarantees are preserved iff the policy-reachable model
family is uniformly capacity-bounded; when capacity can grow without limit,
utility-rational self-changes can render learnable tasks unlearnable. Under
standard assumptions common in practice, these axes reduce to the same capacity
criterion, yielding a single boundary for safe self-modification. Numerical
experiments across several axes validate the theory by comparing destructive
utility policies against our proposed two-gate policies that preserve
learnability.
Self-improving systems face a utility-learning tension that can degrade their ability to learn and generalize, requiring capacity bounds to ensure safe self-modification.
self-improving systems utility--learning tension statistical preconditions reliable learning generalization distribution-free guarantees policy-reachable model family uniformly capacity-bounded utility-rational self-changes learnable tasks two-gate policies