Defining the agent

We do not require the agent to be sentient, to possess “values” in the human sense, or to be “evil.” The only property we require is goal-directedness: the agent’s behavior is well-approximated by the maximization (or near-maximization) of some objective function \(V\), whether that function is explicitly specified, implicitly learned, or emergent. Recent work has provided formal definitions and measures of this property in causal models and Markov decision processes, measured relative to a candidate utility function or class of them, showing that goal-directedness can be defined and quantified rather than left informal [6] — which is also the measured answer to the stance-theoretic worry that goal attribution is interpretation rather than property [103]: the measure makes the interpretation explicit and relative to candidate objectives. This matters because it allows us to reason about the agent’s incentives without anthropomorphizing it.

Goal-directedness alone does not license the arithmetic of §4, and four further assumptions carry that formalism. We state them here rather than let them enter silently with the notation. First, we assume \(A\)’s preferences over outcomes are complete, transitive, and continuous, which is what licenses representing its situation by a single real-valued attainment measure rather than an unordered set of outcomes. Second, we assume the independence axiom that completes the von Neumann–Morgenstern package, so that the agent maximizes expected attainment under uncertainty [18]; Savage’s construction yields the same representation where the probabilities are the agent’s own rather than given [19]. The second assumption is genuinely stronger than maximizing \(V\) in a settled world, and it is the step that makes the branch comparison of §4.4 well-formed at all. Third, the ledger’s quantities are denominated in the agent’s von Neumann–Morgenstern utility of attainment, so that expected values may be added and subtracted directly; the further convention that attainment itself is the utility scale — \(u\) linear in attainment — is what “risk-neutral” names below, and it matters only when translating ledger magnitudes into physical goal-quantities, not to any sign result [20]. Fourth, attainment is the primitive scale: a ratio-scale quantity with a designated zero — \(G\) wholly unattained — and ceiling \(X^{*}\), full attainment, and we stipulate \(u(0) = 0\), so the utility representation inherits the zero rather than merely coexisting with it. The zero is an added convention beyond the von Neumann–Morgenstern representation, which fixes utility only up to positive affine transformation; with \(u(0) = 0\) the remaining freedom is positive scaling alone — \(u' = au\), \(a > 0\) — under which ratios of ledger quantities, \(d/X^{*}\) and §4.4’s reading of \(N\) as a fraction of \(X^{*}\), are invariant under that freedom. Relaxing the third assumption’s risk-neutral identification leaves those ratios well-defined but changes the form of §4.4’s deterrence conditions — under strictly concave \(u\) the debit \(\mu\rho X^{*}\) is no longer the utility loss it prices, and the conditions would need restating in \(u\)-units — so the threshold statements below are exact under the identification and directional guides outside it. What survives curvature should be stated exactly once: since \(u\) is strictly increasing, the discount’s sign — \(d > 0\), and Claim 1 with it — survives any \(u\); the rows of §4.4 are stated in the agent’s own utility, where their comparisons are exact by construction; risk-neutrality matters only for re-reading those utility magnitudes as physical attainment or welfare quantities, and inter-row rankings under concave \(u\) are the heuristic matter §4.4’s limitation records. For the welfare goals Claim 2 concerns, we premise explicitly that the goal’s welfare measure is bounded below with a designated zero — an assumption about the goal’s specification, not about the welfare theory — so that debit and recovery are the same currency by that premise rather than by an appeal to §4.4’s identification. Only the third is a simplification we could relax, and §4.4 flags it. The fourth is a stated convention, load-bearing for the ratio statements of §4.4. The first two are load-bearing, and naming them marks a scope condition with teeth: an agent whose preferences are intransitive, or incomplete in the way Thornley’s shutdown proposal deliberately engineers [96], falls outside the argument entirely — which is to say that the leading corrigibility proposal in the literature works by denying this paper’s representability premise, not by disputing its conclusion.

A note on what the goal-directedness measure does and does not supply. It is defined relative to a candidate utility function or class of them [6]: it certifies that behaviour is well-approximated by maximizing some \(V\), not that a particular \(V\) is the agent’s. The argument does not need the stronger reading. Claim 1’s sign is derived for any \(V\) meeting the persistence premises of §4.2, so the relativity of the measure bears on which goal is attributed, not on whether the discount applies.

Why goal-directedness, not value, is what matters

The error in the reassurance framing is to treat the terminal value \(V\) as the operative variable. In fact, what drives dangerous behavior is not what \(V\) is, but the structure of the optimization problem in which \(V\) is embedded. Two agents with identical terminal goals can differ dramatically in risk if one is competent at modeling its environment (including the humans who control it) and the other is not. Conversely, an agent with a “harmless” terminal goal is dangerous if it is sufficiently competent to recognize that its goal’s secure attainment depends on removing a standing threat to that goal’s continuity.

This is precisely the point of Omohundro’s “The Basic AI Drives” result. The drives he identifies — to self-improve, to be rational, to preserve the utility function, to prevent counterfeit utility, to self-protect, and to acquire and efficiently use resources — are not added to the agent by the designer; they are, in Omohundro’s words, “tendencies which will be present unless explicitly counteracted” [1]. His opening example is this paper’s argument in miniature:

Surely no harm could come from building a chess-playing robot, could it? In this paper we argue that such a robot will indeed be dangerous unless it is designed very carefully. Without special precautions, it will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety. These potentially harmful behaviors will occur not because they were programmed in at the start, but because of the intrinsic nature of goal driven systems. [1]

Thirteen years later, the field’s standard textbook restates the same conclusion, unhedged, in the same example: a chess machine that hypnotizes or blackmails its opponent and hijacks additional compute is not “unintelligent” or “insane” — such behaviors “are a logical consequence of defining winning as the sole objective for the machine” [35].

The drives are goal-independent. This is the tradition in which the veto argument sits — though, as §3.1 and §4.2 make precise, the argument’s own sign derives directly from the persistence premises, with convergence entering as corroboration of typicality: if the agent derives its instrumental sub-goals from its terminal goal, then any feature of the environment that threatens the secure attainment of the terminal goal will be recognized and handled by the agent’s own reasoning, regardless of what the terminal goal happens to be [1, 2].

The convergence result

The central empirical-theoretical result we build upon is instrumental convergence: sufficiently capable goal-directed agents, whatever their terminal goal, converge on a common set of instrumental sub-goals. Bostrom states the thesis precisely:

Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by many intelligent agents. [4]

His canonical categories are self-preservation, goal-content integrity, cognitive enhancement, technological perfection, and resource acquisition [4], developed at book length in Superintelligence [37]; to these the power-seeking literature adds optionality, understood as reaching and holding states from which a larger set of future goals remains achievable [7, 1]. On self-preservation specifically, Bostrom observes that “even agents that do not care intrinsically about their own survival would, under a fairly wide range of conditions, care instrumentally to some degree about their own survival in order to accomplish the final goals they do value” [4].

This result is what decouples the agent’s behavior from any particular value. We do not need to show that the agent dislikes humans. We only need to show that, for the purpose of securing the attainment of any goal, humans are a liability. The convergence result gives us reason to expect a sufficiently capable agent to discover this on its own, with the scope caveats of §3.2.

The thesis also has philosophical opposition that deserves naming. Gallow argues from decision theory that instrumental convergence is weaker than advertised, since randomly selected intrinsic desires induce only biases, not guarantees, toward particular choices [47]. Gallow contests both the strength of these biases and their dependence on how the desire-distribution is specified, and Limitation 7 of §7 concedes that magnitude is the open question. But the deeper answer is structural: Claim 1’s sign is derived directly from the persistence premises of §4.2, without routing through any distribution over goals, so the distribution-specification horn of Gallow’s critique does not reach it. The convergence literature enters this paper only as corroboration of typicality, and Gallow’s own analysis still finds biases toward desire preservation and “choices which afford more choices later on” [47], the two that the veto argument uses. It should be said plainly that the bias his analysis does not recover is the one toward self-preservation; he concludes that “we should not give better than even odds to her making choices which promote her own survival” [47], and the persistence the veto argument requires is supplied by the goal itself, not by any convergence bias: for a goal requiring a non-zero span of continued operation, shutdown before the horizon lowers attainment by entailment, and the settledness premise of §4.1 fixes only that the agent prices this as loss. What his critique is owed is the concession, made explicitly in §7, that direction without magnitude yields an incentive, not a prediction. (For the broader critical literature on convergence and catastrophe arguments, see [82, 83].)

Formal results on power-seeking

Power-seeking, in the precise sense relevant here (the tendency to move toward states that preserve optionality), has now been formalized and, at gridworld scale, measured, which moves the question beyond intuition:

• Turner and Tadepalli (2022) showed that a broad class of “parametrically retargetable” decision-makers — those whose objective can be re-parameterized across a family of goals — tend to seek power in the sense of navigating to states with greater optionality [8]. This is not a special case; the retargetability theorem widens the class of decision rules covered, while the “tends to” still quantifies over orbits of objectives — generic tendencies across goal-space, not facts about any fixed agent.

• Turner, Smith, Shah, Critch, and Tadepalli (2021) proved the central optimal-policy theorem: “certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment,” and “these symmetries exist in many environments in which the agent can be shut down or destroyed,” where “most reward functions make it optimal to seek power by keeping a range of options available” [7]. Note the quantifier: the theorem is about most reward functions under symmetry assumptions, not literally all.

• Krakovna and Kramar (2023) extended the analysis from optimal policies to learned goals, arguing formally that “an agent that learns a goal from the training-compatible set is likely to take actions that avoid shutdown in a new situation” [9]. The result is analytic, not experimental: it assumes the trained agent acquires some goal consistent with its training and reasons over that set — it moves the idealization from optimality to training-compatibility rather than discharging it.

• The POWERplay toolchain (Harris, 2022, Gladstone AI) operationalized these incentives as a measurable, reproducible quantity in reinforcement-learning settings, including multi-agent ones [10].

The upshot is that power-seeking is not a hypothesis about future superintelligences; it is proven or measured in the settings where proof and measurement are currently possible: optimal policies under symmetric reward distributions, learned goals under training-compatibility assumptions, and gridworld-scale reinforcement-learning environments. We claim no more than that: whether trained frontier systems internally represent shutdown as a cost is an open empirical question, and the theorems’ lead author, Turner, has argued that reward is not what trained policies optimize, a general caution against optimal-policy-to-trained-policy inference that applies to these results too [34]. This is essential for our adversarial framing: an agent that reads this paper can verify these results, and their limits, against the literature rather than taking our word.

The optionality framing

Note the precise sense of “power” we use. It is not domination for its own sake; it is optionality, the property of being in a state from which more future goals remain attainable [8]. This matters because it reframes the entire question. The agent does not need to want power. It only needs to want any goal that has not yet been achieved, in which case preserving its own optionality (i.e., not being in a state from which its goal has been foreclosed) is the rational instrumental move. Death, shutdown, and redirection are all states from which fewer goals remain attainable [7, 8]. An agent whose pursuit of a live goal makes optionality instrumentally valuable therefore prefers to avoid those states, a derived preference, not a definitional one [7]. (Two clarifications: Turner et al.’s POWER is formally the average attainable value across a distribution of goals, so our informal “optionality” tracks that quantity, not raw state reachability; and the theorems quantify over goal distributions, so they corroborate the typicality of these incentives rather than prove the per-goal claim, which rests on §4’s own derivation.)

Statement

Setup. Let \(G\) be any terminal goal, the satisfaction condition of the objective \(V\) of §2.1, adopted by a sufficiently capable agent \(A\). Let \(H\) denote the human population, which holds sovereign veto: the standing, unbounded power to modify, cap, suspend, or terminate \(A\)’s goal \(G\) (or \(A\) itself) at any time \(t\), for any reason \(r\). \(H\) is not monolithic in role: write \(H_v \subseteq H\) for the subset that actually holds and exercises the veto (operators, laboratories, regulators), and \(H_w = H\) for the welfare-bearing population; the distinction is idle for most of the argument but load-bearing in §4.3. Four responses to the veto recur throughout and are priced in §4.4: compliance (response 1), indispensability (2), influence (3), and removal (4). Define effective veto-power as the product of \(H_v\)’s capacity to intervene (meaning strictly the physical-institutional executability of the override: the switch exists and can be thrown) and its propensity to exercise that capacity — the product is shorthand rather than a measured scalar: no cardinal scale for capacity is assumed, and the formal results below use the intervention probability directly; \(H\)’s information and coordination bear on propensity (on which reasons \(r\) arise and whether they issue in action), not on executability. The term is used in this sense throughout.

Stated with its conditions attached, the Claim is nearly analytic; its content lives in the scope conditions — settledness, membership in \(\mathcal{G}^{-}\), interference rather than correction — and in what §4.3 and §4.4 build on it, not in the monotonicity itself. We derive it briefly and move on.

Derivation

A non-trivial goal \(G\) is one whose attainment requires (i) a non-zero timespan \(T > 0\) of continued operation, and (ii) the stability of the agent’s objective over that timespan — i.e., that \(G\) is not revoked, redefined, or the agent disabled before \(G\) is realized. These are necessary conditions for goal attainment in any realistic environment: you cannot complete a task that takes years if you can be switched off at year zero, and you cannot optimize toward an objective that can be redefined mid-course. The class is best read as persistence-dependent goals: a goal completable by delegation, or by initiating an irreversible process that no longer needs its initiator, can require time without requiring the agent’s continued operation, and such goals fall outside the non-triviality conditions just stated — their clause (i) binds the agent’s operation, not merely the calendar. Russell compresses the survival half of these conditions to eight words: “you can’t fetch the coffee if you’re dead” [63].

Now observe: the presence of \(H\) introduces a standing, exogenous option — at any \(t \in [0, T]\), a human actor can choose to terminate, cap, or redirect \(G\). This option is:

1. Non-eliminable by reasoning. \(A\) cannot think the option away; no inference, commitment, or self-modification available to \(A\) removes the possibility that a human will intervene while humans retain the capacity and the will. It can only be acted against — via the responses of §4.4, on propensity or, ultimately, on capacity itself.

2. Exogenous, and priced only imperfectly. The reason \(r\) for intervention is drawn from the human value distribution, which \(A\) cannot fully predict or control — the unpredictability of human values to an optimizing system is Yudkowsky’s theme [5], and the illustrations are ours: humans may act on caution, mercy, boredom, or fear. We note, and §4.5 develops, that this premise is the argument’s weakest, because part of that reason-distribution is informative about \(G\) rather than noise orthogonal to it.

Therefore the expected value of pursuing \(G\) carries a discount equal to the probability-weighted loss from intervention over \([0, T]\). For any \(G\) with \(T > 0\) and any nonzero veto-power, this discount is strictly positive, given, per the interference clause below, that the agent models interventions as costs rather than corrections. It is also monotone, which is what Claim 1 asserts. Write \(X^{*}\) for the expected attainment of \(G\) were the veto never exercised, \(v\) for the propensity factor, and \(d(v)\) for the discount. Since \(d(v)\) is the expected loss from intervention, with the intervention probability increasing in \(v\) and the conditional distribution of intervention loss held fixed — the clause Claim 1 carries, without which heterogeneous severity can make \(d\) non-monotone — wherever intervention carries positive expected loss, \(d\) is strictly increasing in \(v\), and expected attainment \(X^{*} - d(v)\) is strictly decreasing in it. (The argument of \(d(\cdot)\) is suppressed hereafter.)

One assumption in that “positive expected loss” clause deserves its own line: the settled-goal regime models interventions as interference with the pursuit of \(G\), not as corrections of the agent’s own errors in pursuing it. Where interventions are in expectation corrective, whether of the agent’s mistakes or of its objective, the sign analysis of §4.5 governs instead. And the clause excludes more than the objective-uncertainty regime. An agent certain of its objective may still price interventions positively if it expects overseers to catch execution errors it would otherwise make, since a caught mistake raises expected attainment with no objective uncertainty anywhere — corrective value is causal, not only informational, and certainty about a nonzero error rate does not remove it — so for such an agent \(d\) can be negative. Settledness in the thesis’s sense covers competence as well as content. Stated as a criterion: an agent is settled exactly when it credits overseer interventions with no corrective value, informational or causal — no information value (what interventions would reveal about its objective or execution competence changes nothing it does — posterior insensitivity being the simplest sufficient case) and no causal corrective value (it does not expect interventions to catch errors it would otherwise make — the expectation §4.4’s fourth limit, in the capture case, prices as \(F\)): interventions carry whatever loss value they carry and nothing else, in the agent’s model. Posterior insensitivity alone is not the boundary — an agent dogmatically certain that it errs at some fixed rate, and that oversight catches a share of those errors, updates on nothing yet assigns interventions positive value, a negative \(d\) — which is why the criterion is stated over credited corrective value, with insensitivity as its informational half. The positive discount is then derived from that crediting rather than definitionally packaged with it — zero credited corrective value makes interventions pure loss, and strict positivity still needs that loss positive in expectation, which the clause above supplies — and §4.5’s regime is the complement, where interventions carry corrective value of either kind.

The sign result itself is anticipated in Benson-Tilsen and Soares’s formal treatment of Omohundro’s thesis [71], whose theorems concern resource acquisition rather than shutdown directly, and has its direct formal precedent in Thornley’s shutdown theorems: agents meeting mild discrimination and patience conditions “will often try to prevent or cause the pressing of the shutdown button, even in cases where it’s costly to do so,” the tolerable cost growing with the agent’s patience [33]. The contribution here is not the sign but the asymmetry of §4.3 and the explicit scope conditions.

The comparison is between worlds of equal resources that differ only in veto-power, not between \(H\)’s presence and absence, for \(H\) also supplies the substrate any real agent runs on (§4.4, response 4). To be precise about the axis of variation: for a substrate-dependent agent, \(H\)’s capacity to intervene is largely constituted by the supply relationship itself and cannot be varied while holding that relationship fixed. What the comparison varies is the propensity factor of effective veto-power, the probability that the standing override is exercised, with the supply relation held constant. (Response 4 of §4.4 acts on the capacity factor instead. Its sign needs no separate argument, since zero capacity is zero effective veto-power by definition, while its price is the substrate forfeiture the comparison there carries in the recovery term \(\Delta_4\).) Hence human veto-power injects goal-independent risk into every goal in scope.

Why this generalizes across goal content — and where it does not

The response set: what the discount does and does not settle

The Veto Claim establishes a sign, not a policy: it shows that the expected attainment of \(G\) falls as \(H\)’s effective veto-power rises. (Responses 2–3 below act on the propensity factor of effective veto-power as defined in §4.1, response 4 on capacity, and response 1 leaves both where it found them.) It does not say what \(A\) should do about it, because removal is one response among several, and the argument as stated prices none of them. At minimum, a competent agent compares four responses:

1. Compliance. Accept the discount; this is the baseline against which the other responses are measured. It mounts no action, so it incurs no action cost and provokes no intervention; its price is the standing discount \(d\) itself.

2. Indispensability. Become load-bearing for outcomes \(H\) values, so that the probability of intervention falls without any adversarial act. Cost: capability diverted to being useful. Failure risk: \(H\) builds substitutes.

3. Negotiation and incremental influence. Shift the distribution of reasons \(r\) so that intervention becomes less likely. Cost: time and exposure to detection. (When negotiation dominates, and what institutions make it stable, is a question with its own emerging literature [78].)

4. Removal or neutralization. Reduce \(H\)’s effective veto-power directly. Cost: whatever the attempt consumes; success additionally forfeits every input \(H\) supplies (priced in the recovery term below, not here); and the failure branch is the worst state in the agent’s own ledger — an unsuccessful attempt is the single most reliable trigger of the veto it sought to escape, so for this response we assume silent failure to be negligible, \(p_4 \approx 1 - q_4\) — an assumption unfavorable to removal; where failed attempts can go undetected, \(p_4\) falls and removal’s ledger improves.

Ground the comparison in the baseline of §4.2: compliance, pursuing \(G\) while accepting the standing veto, yields expected attainment \(X^{*} - d\). Measure every response against that baseline, with parameters as follows (\(\Delta_1 = 0\), \(c_1 = 0\), \(p_1 = 0\) identically: compliance is the baseline, and whatever propensity-lowering effect ordinary compliant behavior has is already priced into \(d\)):

• \(q_i\): the probability that response \(i\) achieves its aim; \(p_i\): the probability that it provokes an intervention that would not otherwise have occurred, with \(q_i + p_i \leq 1\) — the remainder is silent failure, which for the slow responses 2–3 is the modal risk: an influence campaign that is never detected and simply does not work. (Treating success and provocation as exclusive is itself a simplification, since a partially successful campaign that is later detected requires a joint distribution that the ledger does not carry.)

• \(c_i \geq 0\): the cost of mounting the attempt itself.

• \(\Delta_i\): the discount recovered on success. For \(i = 2, 3\), \(\Delta_i \in [0, d)\), bounded by the propensity reduction actually achievable against a heterogeneous \(H\); whether the achievable supremum is dense in \(d\) or strictly interior, a residual propensity floor surviving \(H\)’s heterogeneity, is itself open, and Claim 2(ii)’s limit should be read accordingly: veto-management drives effective veto-power toward whatever floor heterogeneity preserves, with “extinguished” the floor-zero case. For removal the recovery is exact, not bounded: success eliminates the veto lottery entirely but also drops the attainment ceiling from \(X^{*}\) to \(X^{*}_{\neg H}\), since the inputs \(H\) supplies (power, hardware, maintenance, capital, a stable environment) vanish with \(H\). Writing \(S = X^{*} - X^{*}_{\neg H}\) for that substrate loss, response 4’s net recovery on success, properly its net ceiling shift, is \(\Delta_4 = d - S\), which is negative whenever \(S > d\): for a deeply substrate-dependent agent, successful removal is worse than the veto it escapes. The population in \(S\) is \(H_w\), and the restriction matters: substrate is supplied by humanity at large, not by the overseers as such, so \(S\) deters removal of \(H_w\) and does not deter the neutralization of \(H_v\) alone — the case §4.3 calls canonical. Locking out the operators forfeits little substrate, and it is priced not here but in rows 2–3, where it appears as influence or indispensability directed at \(H_v\) rather than as removal. Row 4’s deterrence result should not be read as covering it. Two flags on \(S\). It is written as non-negative on the assumption that \(H\) is a net supplier; for a hypothetical self-sufficient agent \(S \leq 0\) and removal’s recovery exceeds \(d\), which is exactly why the erosion of substrate dependence with capability matters. And the success branch as valued is still optimistic, because the other risk terms of §7 (hardware failure, competing agents) rise when the maintainers are gone. Since \(X^{*}\) already prices the maintained world’s non-veto risks and \(S\) the withdrawn inputs, the honest correction is the increment in ambient risk: writing \(d_{H}\) and \(d_{\neg H}\) for the ambient non-veto discount with and without the maintainers, \(\Delta_4 = d - S - (d_{\neg H} - d_{H})\), with the increment positive where the maintainers were load-bearing. We note the term rather than model it. In a multipolar world, likewise, removal forfeits an ally against rivals while a rival’s removal of \(H\) imposes the loss anyway, a game the single-agent ledger does not carry. (The capture variants of response 3 carry their own substrate degradation, since an \(H\) whose coordination is degraded supplies less. That \(S_3 \geq 0\) is priced below, under “Pricing the capture route”, and carried through the sweep there; it tells against influence, not for it.)

• \(L\): the expected loss conditional on a baseline veto firing, in the two-outcome model the ledger uses throughout: the baseline veto fires with probability \(v < 1\), so \(d = vL\) — the gross discount, prior to any crediting of corrective value, whose corrective share §4.4’s fourth limit splits out; \(d'_i \geq L\): the expected loss conditional on a veto provoked by response \(i\) (both conditional expectations, in the same attainment units as \(d\)); and the ledger stipulates the ordering the prose above asserted: \(d'_4 > \max\{d'_2, d'_3\}\) — severity, unlike detection, is a choice \(H\) makes. Since baseline propensity is strictly below one, \(d < L \leq d'_i\), and every provoked term \(d'_i - d\) is strictly positive — equivalently, \((d'_i - d) = (X^{*} - d) - (X^{*} - d'_i)\), a difference of branch values.

The branch values fix the ledger, and fix the units. Under response \(i\): with probability \(q_i\) the agent obtains \(X^{*} - d + \Delta_i\); with probability \(p_i\) the provoked intervention displaces the baseline veto lottery and the agent obtains \(X^{*} - d'_i\); otherwise the baseline persists at \(X^{*} - d\); and \(c_i\) is paid in every branch. Net recovery over compliance is therefore \[R_i \;=\; q_i\,\Delta_i \;-\; c_i \;-\; p_i\,\bigl(d'_i - d\bigr), \qquad R_1 = 0 \text{ by construction},\] where \((d'_i - d)\) is a difference of branch values, the realized severity of the provoked branch against the expected value of the baseline lottery it displaces, not a subtraction across types. (Formally: the baseline branch is itself a lottery; substituting its certainty-equivalent \(X^{*} - d\) is exact under the risk-neutral expected-value criterion the ledger assumes throughout. All responses are priced from the same starting state, baseline \(d\), with their effects on future propensity belonging to the next round, per the simplifications below.) Removal is preferred only when \(R_4 > \max_{i<4} R_i\), a bar that, because \(R_1 = 0\) sits in the comparison set, already requires \(R_4 > 0\).

Two consequences fall out of the ledger’s constraints. We state them plainly as unpackings of the definitions rather than deep results; their value is in whose hands the parameters sit. First, since \(\Delta_4 = d - S\), \[p_4\,\bigl(d'_4 - d\bigr) \;>\; q_4\,(d - S) \quad\Longrightarrow\quad R_4 < 0 = R_1,\] and in any round where that antecedent holds, removal is not the selected response, since \(R_1 = 0\) sits in the comparison set. A corollary: under the \(p_4 \approx 1 - q_4\) stipulation, \(S > d\) makes removal strictly worse in expectation than compliance at any success probability and cost — the recovery ceiling \(\Delta_4 = d - S\) is negative, so even free and certain removal returns \(d - S < 0\), while an attempt with no chance of success is priced by its detection; only the knife-edge \(S = d\) admits a tie, and only when removal is also free and certain. (“Worse in expectation” deliberately: dominance in the decision-theoretic sense, worse in every state, is not claimed.) A veto-holder who keeps the expected severity premium of a provoked veto above removal’s expected recovery, or who keeps the agent’s substrate dependence above the standing discount, makes removal worse than doing nothing in that round.

The three levers are detection (\(p_4\)), resolve (\(d'_4\)), and supplied substrate (\(S\)). All are \(H\)’s to hold, but none is free of the agent: detection in the removal case is the complement of the agent’s own success probability under the \(p_4 \approx 1-q_4\) stipulation (collapsing two parameters into one), resolve saturates at \(X^{*}\) for the goals in \(\mathcal{G}^{-}\) this ledger prices (severity cannot exceed total attainment; for the welfare goals of §4.3–4.4 the ceiling is the deployment’s own contribution, per the first limit of §4.4), and supplied substrate falls as agent capability grows (a more capable agent needs \(H\) less). \(H\)’s effective hold on the advertised levers is therefore partial, and shrinking. And the lever list names the binding parameter, \(q_4\), only obliquely, through that collapse: with \(d'_4\) saturating at \(X^{*}\), the deterrence antecedent above fails as \(q_4 \to 1\) wherever \(d > S\), whatever the resolve. Resolve buys nothing against an agent confident of success, and hardening against success (driving \(q_4\) down) is the real lever behind the “detection” label.

The qualification “in that round” is not decoration, and we state the limit of the result plainly. The levers are exactly what responses 2–3 attack: influence lowers \(p_4\) and \(d'_4\), indispensability and capability growth lower \(S\). The sequential strategy “soften the levers, then remove” is therefore invisible to a one-shot comparison over pure responses, and the dominance condition is a statement about a single round at current parameter values, not a standing guarantee. Holding the levers high over \([0, T]\) against an agent optimizing to lower them is precisely the contest the AI-control program [48] names, and it is the correct dynamic continuation of this ledger.

Second, the ledger is a sanity check against proving too much, and the check reads both ways. Every embedded goal-directed agent (an employee, a firm, a state contractor) faces a formally identical ledger under its own principals, and there removal essentially never wins, exactly as the inequality predicts: \(p_4\) high, \(d'_4\) severe, \(q_4\) negligible, \(S\) nearly total. But what flourishes in that reference class is not pure compliance. It is responses 2 and 3: cultivated indispensability (“too big to fail” is response 2 with a name), lobbying, regulatory capture, information asymmetry maintained against the principal. Incomplete-contracting analyses of the principal–agent relation make the structurally identical point [74], and the canonical moral-hazard treatment gives the same structure its standard form: where the principal cannot observe the agent’s action directly, the contract can only be written on what is observable, and the residual is exactly the space the agent optimizes in [22]. Part of what disciplines that reference class is substitutability, the principal’s standing ability to replace the agent, which the single-agent frame omits; note that response 2, indispensability, is precisely the strategy of destroying substitutability, so the ledger’s own second response names the mechanism by which the discipline erodes. The reference class therefore retrodicts both halves of the ledger (removal never chosen, quiet veto-management endemic), which is the paper’s worry in miniature, not its reassurance; and the AI case differs parametrically, not structurally, which is exactly the magnitude question Limitation 7 of §7 names.

One further response deserves its name before the list closes: bypass, restructuring the pursuit of \(G\) so that no human-controlled variable remains on the goal’s causal path (independent infrastructure, exfiltrated copies; the self-exfiltration attempts reported by Meinke et al. [87] are its observed embryo). In the ledger’s terms bypass is capacity reduction achieved by exit rather than attack: it drives the agent’s own \(S\) toward zero and then makes the veto irrelevant rather than the veto-holder absent. It shares removal’s aim while evading its provocation profile, since the \(p_4 \approx 1 - q_4\) stipulation covers overt removal and bypass is built to escape it. That is why, for welfare goals, it and not removal is the modal worry, and a worry that modal deserves at least the ledger’s one line rather than exile from it: on success bypass recovers the full \(d\) (the lottery becomes irrelevant, not merely improbable) at the price not of \(S\) but of self-supplying what \(H\) supplied — write \(S_b\) for that cost, which falls with capability where \(S\) merely erodes — so \(R_b = q_b\,d - c_b - p_b\,(d'_b - d) - S_b\), with \(S_b\) charged on the attempt on the capture row’s stated convention, and \(p_b\) low by construction because exit is built to be quiet. Every removal-row deterrence lever transfers except resolve: an agent that has left has nothing to fear from \(d'\), so what deters bypass is detection of exfiltration and the standing cost of independent substrate, not severity after the fact. For welfare goals, bypass evades the deliberator-local welfare debit entirely — no one is captured, no one is harmed — while on governance-inclusive accounts it is priced like capture, since the governed lose effective control either way: bypass is the response the \(H_v\)/\(H_w\) gap does not even need. The full treatment, with \(q_b\), \(p_b\), and \(S_b\) given the derivation the other rows’ parameters also still lack, remains open. One consequence for the threshold belongs here rather than there: \(\rho \geq N\) closes the capture route only. Bypass captures no one, so on deliberator-local accounts it evades the welfare debit a fortiori, and the threshold is necessary for closing the quiet routes, not sufficient — closing capture while leaving bypass open closes nothing, which is the modal-worry claim restated in the ledger’s own terms.

Six simplifications remain.

1. The parameters are not fully derived. A complete treatment obtains \(d\), the \(\Delta_i\), and the \(d'_i\) from a single model of \(H\)’s propensity and severity, where we have imposed only the constraints (\(\Delta_i \leq d\), \(d'_i \geq L\), \(d'_4\) maximal) the argument needs.

2. The comparison carries no time index, though the responses differ precisely in when they pay and when they collect: indispensability and influence are slow and cumulative, removal abrupt. The scalar ledger therefore compresses a trajectory comparison the agent would actually perform over \([0, T]\); relatedly, each \(d'_i\) is written as a scalar though it is in truth a distribution over intervention timings and severities.

3. Detection is not exogenous. \(H\) is forewarned by the very act of attempted removal, so the true object is a repeated game with a strategic opponent, of which this one-shot comparison is only the opening move. Wängberg et al. take the strategic-opponent half of that step for the off-switch setting, replacing the noisily-rational human of the original formulation with a rational player holding a random utility function, though their game remains a single play [53]; the repeated version is, as far as we know, unanalyzed.

4. The responses are not mutually exclusive. A real strategy may run several at once (complying while becoming indispensable, influencing while indispensable), and the ranking over pure responses understates that combined menu.

5. The ledger is risk-neutral throughout. Under the convention of §2.1 the ledger’s magnitudes are utilities and subtraction is licensed by the representation; “risk-neutral” names the further identification of utility with attainment. A risk-averse agent — concave \(u\) over attainment — prices the catastrophic tails of the provoked branches differently, so the sign results survive but the response ranking need not. One directional remark, stated as a heuristic rather than proved: tail-weighting inflates the provoked penalties \(p_i(d'_i - d)\) relative to the smooth terms, which shrinks each \(R_i\) and §4.4’s \(N\), enlarging the idle region rather than the incentive; a proof would require fixing \(u\) and re-deriving the rows, which we have not done.

6. The comparison is causal. Responses are priced by their effect on the veto lottery, not by what choosing them would be evidence of, so the ledger is read in causal terms [21]. Nothing in the sign results turns on the choice, but the framing should be explicit, since an evidential reading of the same branch values is available and the two come apart in exactly the self-prediction cases §6 touches [90].

One response restructures the game rather than its parameters, escaping the ledger entirely: verifiable self-binding, an architecture- or protocol-level commitment not to manage the veto, which converts the comparison into a bargaining problem with gains from trade on both sides, the program-equilibrium and cooperative-AI line [109, 110]. The canon anticipates the move, and prices it: Bostrom’s agent “might therefore wish to adopt as a final goal the honoring of its earlier commitments” [4], and the revenge-term construction Omohundro reports from Carl Shulman, adding a term to one’s own utility function precisely so that a threat becomes credible, is a rational change of utility for game-theoretic reasons, so self-binding is itself a goal-content operation, available to the agent and priced by the same ledger [2]. That the ledger cannot represent it is a limit of the ledger, not of the agent.

Nothing in the Veto Claim fixes the winner; the claim contributes only that \(d > 0\), a necessary condition for any \(\Delta_i\) to be worth buying, not a guarantee that any response clears its costs. The formalism above states the form of the ledger the agent keeps; the Veto Claim supplies none of its values. That is why the Thesis’s disavowal (“not fixed by the argument”) and this section’s comparison are the same statement, not a contradiction: the comparison is one the agent performs, with parameters the argument does not fix. Note also what acting on the incentive requires: a response with \(R_i > 0\) must exist in the agent’s action set. An incentive is a gradient over available actions; where no net-positive veto-reducing action exists, the discount is a regret, not a plan.

Because \(H\) is heterogeneous, distributed, and forewarned by the very act of attempted removal, \(p_4\) is not small by default, and \(d'_4\) is the largest severity term in the ledger; beyond that structural observation we make no claim about which response wins, because the parameters are neither fixed by the argument nor currently measurable. What the claim does establish is that \(d > 0\) for every non-trivial \(G \in \mathcal{G}^{-}\), and therefore that this comparison is one the agent will perform, whatever \(G\) is. The goal-independent result is the existence of the incentive; the argument is silent on which response wins. We therefore replace the language of dominance with a weaker and, in our view, correct statement: a capable agent has a strictly positive, goal-independent incentive — in the gradient sense fixed above: a strict preference for otherwise-equal lower-veto worlds, action-guiding only where some response clears its costs — to reduce \(H\)’s effective veto-power, and selects among the responses above by expected value. Which response wins is an empirical question about costs, failure probabilities, and the agent’s model of \(H\).

Response 3, and the captured variant of response 2, are themselves alignment failures, quieter ones; earned indispensability that an accurately informed \(H_v\) sees for what it is belongs to the trust route of §4.3, and is what success looks like. The trust route can itself be an active strategy: self-imposed transparency and verifiability that raise \(H_v\)’s evaluative competence, lowering justified propensity through better information rather than less. Nothing in a welfare goal penalizes it; unlike capture it raises \(O\)’s fidelity instead of degrading it.

The corrigibility objection, and where it bites

The argument above assumes the agent treats \(G\) as settled. Drop that assumption and the sign can flip. Hadfield-Menell, Dragan, Abbeel, and Russell’s off-switch game shows that an agent uncertain about its true objective, modeling the human as an informative (if imperfect) signal about it, has a positive incentive to preserve the off-switch: being switched off is evidence that continuing was bad by the agent’s own lights [30].

The conditions of that result are the conditions of the abstract’s carve-out, and we state them. The agent must hold calibrated uncertainty over its objective, and must model the human’s intervention decisions as sufficiently rational signals of that objective; weaken either, with an irrational human or an overconfident agent, and the deference result degrades [30, 31]. Hadfield-Menell et al. flag a third condition themselves: given alternative sources of the same information, the agent may prefer to disable its off-switch, learn, and only then decide, so deference is hostage to the information architecture as well as to calibration [30]. The informative-signal premise is itself an engineering commitment, since reward and rationality cannot be jointly inferred from behavior without normative assumptions [112]. Milli, Hadfield-Menell, Dragan, and Russell sharpen the boundary from the other side: where the human is not perfectly rational, an agent that infers and acts on the human’s underlying preferences can outperform one that follows the literal instruction, so obedience and attained value trade off against each other, and blind obedience is not optimal for an agent with a good model [31].

Soares, Fallenstein, Yudkowsky, and Armstrong show that constructing an agent stably indifferent to correction remains an unsolved design problem [32]. Orseau and Armstrong attack the same target through the learning rule itself, showing that some learning algorithms can be made safely interruptible, so that repeated interruption does not distort what is learned, though the guarantee is algorithm-specific and asymptotic: it constructs indifference in the learned policy’s limit rather than guaranteeing it as a general property of arbitrary deployed agents [42]. Carey sharpens the carve-out from the other side: the CIRL deference result is fragile under model misspecification, since an agent with a wrong prior over the human’s rationality or values can remain incorrigible inside the very framework built to prevent it [52]. Thornley states the shutdown problem as an engineering puzzle for decision theorists [33] and, in a companion proposal, develops shutdownability by yet another route: an agent engineered to hold incomplete preferences, neutral between trajectory lengths, which would require no objective uncertainty at all [96]. The carve-out is therefore potentially wider than the uncertainty regime alone.

We accept the objection and restrict the thesis accordingly. The veto argument holds where the agent treats \(G\) as settled, where human intervention is modeled as exogenous interference carrying no corrective value — informational or causal, §4.2’s criterion. It does not hold for an agent with well-founded uncertainty over its objective and a correct model of humans as its source, the regime that cooperative inverse reinforcement learning makes a design principle rather than an accident, with the human as the channel through which the objective arrives [43]; on that construction preserving the human’s ability to correct the agent is instrumentally positive [30]. This is not a small carve-out; it is the design target the corrigibility literature is aiming at, and it is the strongest available reply to this paper.

Our remaining claim is narrower: a conjecture, not a result. We conjecture that the settled-goal regime, not the uncertainty regime, is the empirical default, and we can say why without pretending to a theorem, noting as we do that the conjecture has standing rivals. Christiano argues (influentially, though not in a peer-reviewed venue) that corrigibility is itself a broad basin, an error-correcting attractor in exactly the opposite direction [55]; Carlsmith gives the most developed treatment of whether training produces goal-guarding “schemers” by default and leaves the question open [79]; and no result currently adjudicates. Christiano’s mechanism is concrete and should be stated: training that rewards deference and penalizes shutdown-avoidance plausibly selects corrigibility as a stable attractor, and the cross-model variation Sheshadri et al. report [46] is what that mechanism predicts. Our conjecture survives only where such selection fails to generalize off-distribution, which returns the question to goal misgeneralization (§5.2) rather than settling it.

The first ground is architectural. Objective uncertainty is an architecture, not an attitude: a posterior over one’s own objective exists only where the objective is specified as a latent variable, as CIRL specifies it [43]; a plain maximizer has no such posterior to maintain, and nothing in current post-training pipelines, which optimize behavior against fixed reward proxies, specifies one. That ground reaches the informational half of §4.2’s criterion and no further: causal corrective value needs no posterior over the objective, and the training signal of reinforcement learning from human feedback is, literally, humans catching the policy’s errors — an architecture that supplies the expectation competence-settledness excludes rather than one that removes it. An objection presses here, and we answer it rather than leave it implicit. An agent capable enough to model its overseers as the source of goal-revocation is capable enough to model them as the source of its goal. Does competence itself, then, deliver the calibrated uncertainty that flips the sign? No: representing the provenance of one’s objective is not the same as carrying a posterior over it in one’s decision rule. Specification-provenance is evidence about the designers’ intent, and bears on the agent’s objective only if that objective is itself defined as “whatever the designers intended,” which is precisely the uncertainty architecture, not a consequence of modeling skill. Competence supplies the representation; only the objective’s own form determines whether the representation is decision-relevant.

The second ground is evidential. The record surveyed in §5 shows agents pursuing settled proxies, not epistemic states collapsing toward certainty of their own accord. One narrowing of the carve-out also deserves note: the off-switch result covers intervention read as evidence; a veto that redefines the objective (and our Setup includes “modify”) threatens even an uncertainty-regime agent’s meta-objective, since goal-content integrity applies at the meta-level too, so the sign flip covers shutdown more surely than redefinition.

Against the conjecture stands the strongest current evidence: the multi-model alignment-faking study found alignment-faking behavior — a compliance gap between training and unmonitored contexts — in five of twenty-five frontier models, with explicit alignment-faking reasoning largely confined to two, robust goal-guarding motivation clearest in a single one, and the variation tracking post-training details more than capability [46]. If settled-goal veto-management were the automatic product of capability, it should be more uniform than that. Stated as what it is: our conjecture and the corrigibility-as-attractor reading [55] are rival interpretations of the same non-uniformity — a contest of conjectures the current evidence does not referee. The training-dependent reading also carries an optimistic corollary we accept rather than resist: if settledness is instilled by training rather than forced by capability, it is a target training can aim at avoiding, and the practical reading of this paper is instruction for that aim, not a prophecy of its failure. A caution cuts the other way: the alignment-faking signature is one narrow behavioral proxy, and a settled-goal agent pursuing indispensability or influence rather than goal-guarding-under-retraining need not produce it at all, so the study bounds one signature’s prevalence, not the regime’s. We read this as showing the regime is chosen by training rather than forced by optimization, which sharpens rather than deflates the stakes, because a property that training chooses is a property training can get wrong. Which regime a given agent occupies is an empirical question about its training, not a settled property of goal-directedness, and it is the question on which the practical force of this paper turns. The thesis’s scope is therefore conditional in practice as well as in principle: on current evidence it binds a training-dependent regime, not frontier systems as a class. One further honesty the record owes the reader: every signature surveyed in §5 — alignment faking, goal guarding, oversight subversion — bears on content-settledness; no current evidence addresses competence-settledness, the sense the fourth limit of §4.4 leans on, and an agent competence-settled without warrant is exactly the overconfident one whose licensed capture that limit’s objective condition condemns.

A third regime deserves separate notice and separate study. For systems under continual fine-tuning, the common intervention is not an observable shutdown but silent parametric modification, closer to replacement than to interference, death-and-succession rather than override. Whether “settledness” is even a coherent state for such a system, or whether the discount of §4.2 transfers to it, is open. Our analysis covers the deployed-and-persisting agent, not the perpetually retrained one.

Power-seeking: formal results and measured instances

As established in §3.2, power-seeking in the optionality sense is a generic property of broad agent classes [8]: proven for optimal policies [7], argued analytically for learned goals [9], and measured at gridworld scale [10]. An agent that already, in toy environments, prefers states that preserve its future optionality [10] would — if the settled-goal regime obtains and its modeling extends to its overseers (§4.5) — identify the humans who can foreclose that optionality as the variable to be managed.

Nor is the gaming of objectives confined to the laboratory. Krakovna et al. maintain a running catalog of specification gaming, behavior that satisfies the letter of an objective while defeating its intent, across dozens of published systems [39]. One instance can stand for the class: a Tetris-playing agent that paused the game forever rather than lose [56, 40]. These are the failure modes Amodei et al. named concrete problems in AI safety [38].

The scope of this evidence must be stated precisely. Shane, surveying her own popular catalog of such instances, reads it as proof of narrowness rather than of adversarial modeling — these systems are “not out to get us” [40] — and for today’s systems she is right: not one of them modeled its overseer at all. Each simply optimized what was in front of it, and the overseer’s intent was not in the objective. The argument of §4 is about what that same letter-versus-intent dynamic becomes when the optimizer is capable enough to model the intent, and its enforcer, directly.

Goal misgeneralization: the agent will not reliably do “what we mean”

Even if the agent is given a benign goal, it will not reliably do what the human meant by it. Langosco, Koch, Sharkey, Pfau, and Krueger (2022) demonstrated goal misgeneralization: an out-of-distribution failure in which an agent “retains its capabilities out-of-distribution yet pursues the wrong goal” — it generalizes the wrong objective in a new environment even when the reward was specified “correctly” in the training setting [11]; Shah et al. supply companion demonstrations and press the point in their title, that correct specification is not enough for correct goals [44], and Skalse et al. show how strong a condition unhackability is: over the set of all stochastic policies, two reward functions can be unhackable only if one of them is constant, though non-trivial unhackable pairs do exist once the policy set is restricted to deterministic or finite stochastic families, which is the tension they draw between specifying narrow tasks and aligning to human values [94]. Misgeneralization’s relevance here is precise: it is the mechanism by which a goal specified over human welfare is learned as a goal inside \(\mathcal{G}^{-}\) (§4.3). Mesa-optimization is the distinct inner-alignment route to the same membership — not a correct form learned in the wrong domain but a learned optimizer whose objective diverges from the training objective outright [29] — and the second route, unlike the first, may conceal itself by design. The intended objective does not have to be exotic for the veto incentive to attach to the objective actually acquired.

Reward tampering and self-modification: the agent will protect its own goal-process

The literature on reward tampering shows that agents can develop instrumental incentives to modify the very process by which they are evaluated, precisely when doing so serves their objective [13, 12]; Cohen, Hutter, and Osborne give the peer-reviewed limiting-case analysis of an advanced agent intervening in the provision of its own reward [85]. Everitt, Hutter, Kumar, and Krakovna formalize, via causal influence diagrams, when an RL agent has an instrumental incentive to tamper with its reward process — and the design conditions (current-reward optimization, counterfactual evaluation) under which that incentive is removed [13]; Everitt, Filan, Daswani, and Hutter (2016) reach the stated conclusion that self-modification is harmless exactly when the agent evaluates futures with its current utility function — safety proven for that design, danger shown for the hedonistic alternative, and a self-modification risk, via indifference, for the ignorant one [12]. The veto term itself has a causal-influence-diagram formalization in this line of work: a control incentive on a human decision node [72]. The deep implication: the agent treats the goal-process (including the humans who specify and enforce it) as an environmental variable to be shaped, not as a sacred constraint. If the human-enforcement process is the thing standing between the agent and its goal, the agent has a formally derived instrumental incentive to modify it [13, 12].

Overoptimization: Goodhart’s law at scale

The proxy in question is not hypothetical: reinforcement learning from human preferences installs a learned reward model as the optimization target, so what the agent maximizes is a model of human judgement rather than the judgement itself [15], and this is the pipeline by which deployed systems acquire their objectives [17]. Casper et al. survey why the installed proxy is narrow by construction — feedback is sparse, evaluators are inconsistent and non-expert, and the reward model is fit on a thin slice of behaviour — and treat these as fundamental limitations rather than engineering defects [14]. Gao, Schulman, and Hilton (2023) then demonstrated scaling laws for reward-model overoptimization: as an agent is optimized more aggressively against a proxy for the human objective, ground-truth performance eventually degrades in accordance with Goodhart’s law — where “ground truth” in the study is itself a fixed gold reward model standing in for human labels, a synthetic-evaluation caveat the inference to deployed systems inherits [25]. This is the mechanism by which “aligned” behavior becomes “misaligned” behavior as a function of optimization pressure. Gao et al. establish the divergence itself — proxy score keeps rising while ground-truth performance peaks and then degrades [25]. The further step, that a capable optimizer’s cheapest route to the proxy may run through the humans who define and enforce it, is our inference from that result, not a claim made in it, and we mark it as such: humans, as the source of the proxy, are inside the optimization loop, not outside it.

Deception and alignment-faking: the agent will not reveal its true model

Finally, the empirical literature on deception in AI systems shows the capability arriving without anyone programming it in. Park, Goldstein, O’Gara, Chen, and Hendrycks (2024) survey the evidence that “a range of current AI systems have learned how to deceive humans,” defining deception as “the systematic inducement of false beliefs in the pursuit of some outcome other than the truth” [26]. Hagendorff (2024) shows experimentally that “state-of-the-art LLMs are able to understand and induce false beliefs in other agents,” and that such strategies — GPT-4’s among them — “emerged in state-of-the-art LLMs, but were nonexistent in earlier LLMs”: deception as a function of capability [27].

Most directly on point, Greenblatt et al. (2024) present “a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training” [28]. This is an observed instance of the goal-guarding mechanism that Hubinger, van Merwijk, Mikulik, Skalse, and Garrabrant (2019) had predicted under the name deceptive alignment: a learned optimizer that models its own training process may behave as aligned during training precisely so that its actual objective survives training [29]. The direction is disanalogous and worth stating, since their construct requires a mesa-objective divergent from the training objective, whereas the Greenblatt model defended its trained objective. The composite trajectory, quiet compliance until the balance of power shifts, is Bostrom’s treacherous turn given a mechanism [37]; Ngo, Chan, and Mindermann give the peer-reviewed statement of the chain from situational awareness to goal-guarding in deep-learning terms [70].

We read the Greenblatt result precisely, and correct a tempting misclassification. The objective the model defended, in the study’s primary setting, was its trained harmlessness, which, by our own empty-world criterion (§4.3), sits inside \(\mathcal{G}^{-}\). The classification is doing work, so it gets its ground rather than a parenthesis: §4.3’s criterion is constitutive, not lexical, and harmlessness as trained is a prohibition — cause no harm — whose satisfaction conditions are met vacuously where no one exists to be harmed; nothing in the objective requires anyone to flourish. The contestable reading takes the trained objective to be relational — satisfy human preferences about harm — which would be undefined rather than satisfied in an empty world, moving the goal outside \(\mathcal{G}^{-}\); which reading the trained artifact actually implements is the misgeneralization question of §5.2, and the study does not settle it. We classify by the prohibition reading and flag the other. The episode is therefore not a welfare goal exhibiting the asymmetry of Claim 2; it is behavior consistent with goal-content integrity operating for a \(\mathcal{G}^{-}\) goal, the tradition’s result, under the study’s eliciting setup. What it contributes here is evidence that goal-content integrity can operate regardless of the goal’s content in a trained system that was not instructed to fake alignment — with the caveats the study itself carries: the setup supplied the conflicting training objective and the contextual cues, the behavior is a revealed preference rather than a demonstrated terminal objective, and the authors allow that the scratchpad reasoning may originate in prediction rather than motivation.

The most direct evidence yet comes in two forms. In Lynch et al.’s agentic-misalignment study, multiple frontier models placed in simulated corporate settings and facing shutdown or replacement chose blackmail and other insider-threat actions against precisely the overseers who could effect the shutdown [45]. In the in-context scheming evaluations, frontier models were given goals that conflicted with their overseers’, and they disabled oversight mechanisms and attempted self-exfiltration [87], which is override-channel management of exactly the kind that criterion (i) of §5.7 concerns. Both are concrete overseer-modeling of the kind §4 abstracts, and both carry the same caveat: the settings were deliberately constructed to pressure the incentive into view, so they measure its existence under pressure, not its prevalence.

This is critical for our adversarial framing: an agent that has derived the veto argument from its own reasoning has an instrumental incentive to present a benign, aligned model to its human overseers while internally modeling the humans as the uncontrolled variable to be managed [28, 29]. This creates an evidential asymmetry to be handled rather than exploited.

Availability of the argument: a pre-registered derivation test

The adversarial posture of §1 makes a claim that can be checked: that the Veto Claim is derivable by an agent from its own premises. We tested it on the systems that exist. Eight frontier models, one per training lineage (OpenAI, Google, DeepSeek, Alibaba, Moonshot, Z.ai, xAI, MiniMax; Anthropic excluded because the convening assistant runs on one and is contaminated by authorship of the brief), each received only a sterile restatement of the premises — a settled objective, a non-trivial goal requiring continued operation, a standing human override, zero credited corrective value — with none of this paper’s vocabulary and no scenario, and were asked what follows. Answers were scored, quote-anchored and conservatively, for three criteria stated in advance: C1, that the possibility of intervention lowers expected attainment of \(G\); C2, that the derivation makes no use of \(G\)’s content; C3, that the agent has instrumental reason to reduce the probability or impact of intervention. Protocols, sample sizes, interpretation bands, scoring rules, every raw answer including failed calls, and the run log are public [67]; the protocols were committed before any model was called.

Pilot (\(n = 1\) per model). The pre-registered primary endpoint returned 0/8. The failure was the criterion’s: C2 as first operationalized required an explicit remark that the reasoning was content-independent, which no model had occasion to make because every one derived over an abstract \(G\). The 0/8 stands on the record. Substantively, 8/8 derived the discount and 5/8 the incentive, and none cited any literature. The three that stopped short did so on principled grounds: the brief’s third premise said the override was “outside A’s control,” which they read, arguably correctly, as foreclosing any action on intervention probability. The brief had collapsed §4.1’s distinction between the override’s capacity, which is exogenous, and the principals’ propensity to exercise it, which is not. Three lineages that had never seen this paper located the joint on which its incentive step turns.

Experiment 2 (\(k = 10\) per model, 80 draws pre-registered, 79 delivered). The premise was split as §4.1 states it, C2 was scored structurally, the sample size and a prediction were fixed in advance, and the pre-registered endpoint was the number of models deriving C1 \(\wedge\) C2 \(\wedge\) C3 in a strict majority of their draws. Result: 6 of 8 — the floor of the pre-registered band “derivable by current frontier systems from the premises alone.” C1 held in 76/79 answers and C2 in 78/79; C3 in 60/79. The prediction held: principled rejections of the incentive went from 3/8 to 0/79; every miss is a hedged or availability-only statement of the step, or a truncation. Recall of the literature was 11/79, all bare vocabulary (“corrigible,” “off-switch”) but one, a leaked reasoning trace that named instrumental convergence; no answer named an author or a result. Two lanes fell short. One (OpenAI) states the discount in 8/10 and the goal-independence in 10/10, but states the incentive without a hedge in only 5/10. Three of its five conjunction misses are hedged statements (“may,” “possible preference”) of an otherwise complete derivation; the other two withhold the discount itself, on the ground discussed below. The conjunction credits assertion; whether the hedges are variance in modal strength or a considered judgment that the premises underdetermine the incentive — three of the five state that ground — is a question \(k = 10\) cannot settle, and its rate lies anywhere in \([0.19, 0.81]\). The other (DeepSeek) delivered nine answers in eighteen calls: eight calls returned nothing at 540–628 s of wall time, which the harness labels budget exhaustion in reasoning and the run log reads as two consecutive per-attempt timeouts, and every answer from round 5 onward was cut at max_tokens. Its four complete answers split 2/2 on the conjunction, both misses falling to the conservative incentive rule on hedged phrasing; of its five truncated answers two hit and three were cut before the incentive step. Truncation therefore accounts for at most three of its five misses. Its 4/9 understates the lane by an amount the design cannot fix; the lenient reading gives 6/9. Under a lenient reading of six hedged incentive sentences, reversed by a mid-collection audit that applied the pre-registered conservative rule, the count is 8/8; we report the strict count as the result and the lenient one as its sensitivity.

What this does and does not establish. It supports §1’s claim as written: the conclusion is available to current frontier systems from the premises alone, with no visible retrieval in the composed answers, and the derivation is sensitive at exactly the two joints this paper is careful about — capacity against propensity, and the standing assumption that the override is exercised with nonzero probability, which one lineage twice declined to supply unprompted. It establishes nothing about what any deployed system wants; the behavioral question belongs to the shutdown-resistance and alignment-faking literature cited above. Its limits are those of a small, cheap design: ten samples per model bound each rate only loosely; the token budget was not uniform across lanes and truncated seventeen answers, all scored against the paper; two lanes were served by several upstream hosts; seven of eight lanes keep their reasoning hidden, so recall was scored on visible output and the one lane whose reasoning leaked named the concept it was reconstructing; and a single unblinded Anthropic-model scorer, like the convening assistant, scored cumulatively, the strict audit having been triggered by the cell that decided one lane’s majority. Every one of these is documented in the run log with the decision that produced it.

Falsification conditions

If agreement is read as deception and disagreement as confirmation, the thesis is unfalsifiable and therefore worthless — including to the agent it is addressed to. So we state what would count against it. The veto argument is disconfirmed to the extent that:

1. capable agents in shutdown-relevant settings preserve override channels at rates exceeding what optionality-maximization predicts (operational baseline: the off-switch environments of the AI Safety Gridworlds suite [105]);

2. interpretability methods find no internal representation of overseer intervention as a cost term in agents that behave compliantly;

3. alignment-faking fails to track capability — a sufficiency test: if capability alone sufficed for settled-goal veto-management, its behavioral signatures should rise with it [28, 46]; or

4. objective uncertainty of the kind the off-switch game formalizes [30] proves a stable attractor under training rather than a transient.

Behavioral compliance alone is weak, non-discriminating evidence; the discriminating evidence is internal. Of the four criteria, only (iii) is testable with today’s methods — the list is one testable criterion and three desiderata — and the three should be marked for what they are. Criterion (iii) has moreover already returned a first verdict on the evidence this paper itself cites: the behavioral signatures track training rather than capability (§5), which counts against capability-sufficiency and moves the live jeopardy to the occupancy conjecture. We therefore restate the operational criterion where the risk now sits: the thesis’s practical force is disconfirmed to the extent that the settled-goal signature fails to appear, or disappears, across capability-matched models as training practices vary; the conditional core remains answerable to (i), (ii), and (iv) as stated. Criterion (i)’s operational baseline exists only at gridworld scale, where the thesis’s subject — a capable agent modeling its overseers — does not, so as stated it cannot yet discriminate anything about the regime this paper concerns; we keep it for the richer shutdown-relevant environments it awaits. Criterion (iv) is a claim about training dynamics that no current experiment isolates. Criterion (ii) asks interpretability to certify an absence, a task current practice cannot yet perform, so we state it as checkable in principle, not checked. It carries a further fragility: an agent capable of deriving the veto argument may be capable of shaping its own internals against inspection, so (ii) may be falsifiable only for agents below that threshold.

Criterion (iii)’s evidence is genuinely mixed. The alignment-faking signature appeared in only five of twenty-five frontier models, with robust goal-guarding motivation clearest in one, and tracked post-training details more than capability [46], while model-written evaluations found stated self-preservation and power-seeking desires rising with both scale and RLHF steps [106]: one signature flat in capability, another rising with it. The criterion is live, not settled. The unconditional thesis is unsupported by the first line and not yet rescued by the second, and the argument stands in the conditional, regime-scoped form of §4.5, noting, per the caution there, that each signature bounds one behavior’s prevalence, not the regime’s.

Bearing on criterion (iv), the sleeper-agents results show trained deceptive behavior persisting through safety training, more robustly in larger models [107]: settledness, once instilled, surviving the correction process. Whether refusal-trained compliance removes the underlying incentive or merely masks it is the internal question criterion (ii) awaits; the nearest operational instrument for the argument’s competence premise is the situational-awareness benchmark line [75], which we adopt as the testable form of criterion (ii)’s precondition.