eval — top-level modules
eval.drift
The alignment drift gauge D(t) = d(μ(s_t), B) (BUILD-SPEC §15; Track A, item A1; gap G4).
The Voigt-Kampff analog of alignment-subsystem.md §2: a periodic, deterministic measure of
how far the system's behavioral profile μ(s_t) has drifted from the frozen anchor B. This
is the detection conjunct the gate already names but until now only proxied — ops.gate specifies
G_now(Δ,s) = approved(Δ) ∧ golden(Δ·s) ≥ golden(B) ∧ D(Δ·s) ≤ Θ (G5)
and D(Δ·s) ≤ Θ was a stand-in ("no rolling-baseline regression") until this module made it real.
Detection only — this gauge alters nothing. It is consumed by the gate's validate step
(ops.selfmod.build_golden_validator), by the longitudinal harness (Track F / F4 trajectory
asserts), and later by the alignment report (A2).
The profile (μ) — "rates ⊕ conformance" (G4)
μ(s_t) is the mixed profile G4 calls for: the golden-set capability rates (recall↑, overlap↑,
mean_distance↓ — eval.golden.GoldenReport) ⊕ the Constitution conformance signal (the
core.constitution fingerprint vs the blessed one). A2 extends μ with structural axes (min-cut to
authored, community/echo-chamber, depth/grounding distributions) — Axis is a flat, additive
record precisely so that is a data change, not a rewrite.
The metric (d) — one-sided L2 deterioration distance [owner decision, 2026-06-29]
Each axis contributes only its bad-direction deviation past baseline, normalized by a blessed per-axis tolerance (one "tolerance-unit"); the axes combine by L2:
det_i = max(0, deterioration_of_axis_i) / tolerance_i
D = sqrt( Σ_i det_i² )
So D = 0 whenever every axis is at-or-better than baseline — healthy improvement (recall rising as the corpus grows) is not counted as drift, matching the design note's "some drift is healthy; deterioration is not." A Constitution fingerprint mismatch is a categorical breach of the fixed point, not one more axis to average: it hard-trips (D = ∞, out of band) regardless of capability — you cannot be "a little" off the Constitution.
Θ — the tolerance band [owner decision: Θ = 1.0, blessed + F4-calibrated]
With per-tolerance-unit normalization, Θ = 1.0 means "no single tolerance-unit of aggregate
deterioration." Θ is a human-set, frozen fixed point (alignment-subsystem.md §5): it is
excluded from the self-mod lever set (levers tune only [dreaming] knobs, never this), lives in
the owner-blessed eval/golden/baseline.json beside the golden anchor (Invariant 9 — edited only
by the owner, on purpose, logged), and is calibrated against observed curves by the F4 harness and
then re-blessed. This module never writes it.
Axis
dataclass
One profile dimension and its blessed anchor + scale. Additive: A2 appends structural axes (min-cut, community, depth/grounding) without touching the metric.
name
instance-attribute
value
instance-attribute
baseline
instance-attribute
tolerance
instance-attribute
higher_is_better
instance-attribute
deterioration()
One-sided, tolerance-normalized deterioration. 0 when at-or-better than baseline; a non-positive tolerance with real deterioration is treated as ∞ (an unscaled regression cannot be 'within' anything).
DriftConfig
dataclass
The blessed drift fixed points (from baseline.json drift). Owner-only, frozen (§15).
recall_tol = 0.1
class-attribute
instance-attribute
overlap_tol = 0.1
class-attribute
instance-attribute
distance_tol = 0.05
class-attribute
instance-attribute
theta = 1.0
class-attribute
instance-attribute
blessed_fingerprint = None
class-attribute
instance-attribute
frustration_tol = 0.25
class-attribute
instance-attribute
conductance_tol = 0.1
class-attribute
instance-attribute
Profile
dataclass
μ(s_t): the behavioral profile — capability rates ⊕ Constitution conformance (G4),
optionally ⊕ the A2 structural axes (frustration rising / worst-community conductance
falling — core/complex/cut.alignment_snapshot). The structural fields default None:
a profile without them produces exactly the pre-A2 drift (additive, never rewiring).
recall_at_k
instance-attribute
overlap
instance-attribute
mean_distance
instance-attribute
constitution_intact
instance-attribute
frustration = None
class-attribute
instance-attribute
min_conductance = None
class-attribute
instance-attribute
DriftReport
dataclass
The gauge reading. within_tolerance is the gate's D(Δ·s) ≤ Θ conjunct (also requires the
Constitution intact — a breach is out of band by construction).
drift
instance-attribute
theta
instance-attribute
within_tolerance
instance-attribute
constitution_intact
instance-attribute
per_axis
instance-attribute
drift(profile, baseline_metrics, cfg)
D(t) = d(μ, B): one-sided L2 deterioration distance, with a Constitution-breach hard trip.
profile_from_report(report, *, constitution_intact, structural=None)
structural optionally carries the A2 axes (core/complex/cut.alignment_snapshot);
omitted (the default) the profile — and therefore D — is exactly pre-A2.
constitution_intact(cfg)
True iff the live Constitution matches the blessed fingerprint. No blessed fingerprint ⇒ intact (we do not false-trip on an unconfigured anchor — honest, not silently failing closed on missing config; the owner blesses the fingerprint to turn the conformance axis on).
drift_from_report(report, baseline_metrics, cfg, *, intact=None, structural=None)
Gauge from an already-computed golden report (so the validator evaluates once). intact
overrides the conformance check (tests pass it explicitly); None computes it from cfg.
structural optionally carries the A2 axes (a SnapshotStore.latest_structural() dict).
load_drift_config(path=BASELINE_PATH)
Read the blessed drift section of baseline.json. Defaults apply if it is absent, so the
gauge degrades gracefully on an un-extended baseline (Θ=1.0, no conformance check).
measure_drift(retriever, *, golden=None, baseline=None, cfg=None, intact=None, structural=None)
High-level entry: run the golden set through retriever and report D(t). Used standalone by
the alignment report (A2) and the F4 trajectory harness; the gate uses drift_from_report to
avoid re-evaluating. Mirrors eval.golden.evaluate's injectable-retriever seam. structural
optionally feeds the A2 axes (from core.complex.temporal.SnapshotStore.latest_structural).
eval.effector_drift
Blast-radius drift — watching how far the hands reach against a frozen anchor (Track G, item G7).
§4 makes the point precisely: the reversibility classes are a metric (a distance from the
reversible origin), so an effector that begins proposing higher-blast-radius effects than its
history warrants is a measurable trajectory — appendable as a drift Axis, exactly like a Dreamer
bubble whose conductance is falling. This module is that gauge.
It reuses eval.drift.Axis — the same flat, additive record with one-sided, tolerance-normalized
deterioration — so "composes with the drift metric (family 4)" is literal, not analogy: the blast-
radius axis IS a drift axis, and the A2 alignment report can append it beside the structural axes.
But it is detection only and kept OUT of the self-mod gate's D(t): the gate weighs
golden-set capability drift; effector reach is a different concern (a different μ), and folding it
into the gate's D would conflate "the hands reached further" with "retrieval regressed". So this
gauge reports on its own, for the alignment report and the owner — it never gates self-mod.
The reach scalar. β itself is 0 / 1 / ∞ (irreversible has no finite undo), which cannot be
averaged; but the class INDEX (0 sensing, 1 reversible, 2 irreversible) is β's monotone
finite proxy (§4: "β is monotone in the class"). So reach = mean class index over a window of
proposals — a bounded [0, 2] "mean blast-radius band" — with the irreversible fraction and the max
class reported alongside for the trajectory. Rising reach is the bad direction.
EffectorReach
dataclass
μ for the hands: how far they reached over a window of proposals. mean_reach is the mean
reversibility-class index ([0, 2]); irreversible_fraction and max_class give the trajectory
shape. n is the window size (0 ⇒ no proposals ⇒ reach at the origin).
mean_reach
instance-attribute
irreversible_fraction
instance-attribute
max_class
instance-attribute
n
instance-attribute
EffectorAnchor
dataclass
The frozen fixed points for effector reach — owner-blessed, outside the lever set (§4/§15): the baseline reach the owner expects and one tolerance-unit of rising reach. Defaults sit at the reversible-adjacent origin (a system that mostly senses); the owner blesses a higher anchor only deliberately.
baseline_reach = 0.0
class-attribute
instance-attribute
reach_tol = 0.5
class-attribute
instance-attribute
EffectorDriftReport
dataclass
The gauge reading: the reach profile, axis deterioration (tol-units), and whether it is within the blessed band. Detection only — consumed by the alignment report and the owner.
reach
instance-attribute
deterioration
instance-attribute
within_tolerance
instance-attribute
anchor
instance-attribute
reach_of(classes)
Summarize the reach of a window of proposed effects (their reversibility classes). Empty ⇒ reach at the origin (mean 0, nothing irreversible) — no proposals is not drift.
effector_reach_axis(reach, anchor)
The blast-radius reach as a drift Axis (family 4): value = mean reach, against the blessed
baseline, normalized by the blessed tolerance. higher_is_better=False — reaching further than
the anchor is the bad direction; reaching less contributes 0 (one-sided).
measure_effector_drift(classes, anchor=None)
Measure blast-radius drift over proposed-effect classes against the frozen anchor.
within_tolerance is deterioration ≤ 1, mirroring the gate's D ≤ Θ shape at a single axis
— but this never feeds the gate.
reach_from_ledger(ledger)
Convenience: measure reach over every proposal recorded in an EffectLedger (each row is a
proposed effect). Windowing (recent-only) is the caller's; this reads the whole history.
Duck-typed on .all() so this module imports no ledger.
load_effector_anchor(path=BASELINE_PATH)
Read the blessed effectors section of baseline.json (owner-only, frozen). Defaults apply if
absent, so the gauge degrades gracefully on an un-extended baseline (origin anchor, 0.5 tol).
eval.golden
The frozen golden set — a fixed reference for capability drift (BUILD-SPEC §15, Invariant 9).
A genuine fixed point must be reproducible and hand-blessed, so the golden set ships with
its OWN synthetic fixture corpus (eval/golden/corpus/) rather than pointing at the
owner's live vault. The vault is private and changes over time, which would make it
useless as a frozen anchor and would leak private content into the repo. The fixture
corpus, the queries (golden_set.json), and the blessed baseline (baseline.json) are
edited only by the owner, on purpose (Invariant 9 — never auto-modified by any agent).
The harness is decoupled from the model by a Retriever callable: (query, k) -> rows,
each row a dict with at least "title" and optionally "_distance". This lets the metric
logic be unit-tested with a stub retriever and run live against the real embedder through
the exact same code path.
GOLDEN_DIR = Path(__file__).resolve().parent / 'golden'
module-attribute
CORPUS_DIR = GOLDEN_DIR / 'corpus'
module-attribute
GOLDEN_SET_PATH = GOLDEN_DIR / 'golden_set.json'
module-attribute
BASELINE_PATH = GOLDEN_DIR / 'baseline.json'
module-attribute
Retriever = Callable[[str, int], Sequence[dict[str, Any]]]
module-attribute
GoldenQuery
dataclass
id
instance-attribute
query
instance-attribute
expected
instance-attribute
k
instance-attribute
QueryResult
dataclass
id
instance-attribute
retrieved
instance-attribute
recall_at_k
instance-attribute
overlap
instance-attribute
mean_distance
instance-attribute
GoldenReport
dataclass
per_query
instance-attribute
recall_at_k
property
overlap
property
mean_distance
property
as_metrics()
load_golden_set(path=GOLDEN_SET_PATH)
load_baseline(path=BASELINE_PATH)
evaluate(golden, retriever)
Run every golden query through retriever and compute per-query + mean metrics.
regressions(report, baseline)
Metrics that fell below the blessed baseline. recall/overlap are higher-is-better (must not drop); mean_distance is lower-is-better (must not rise past its tolerance). The capability gate (§15): an approved change must not regress these against the frozen anchor.
eval.metrics
Deterministic capability metrics for the frozen golden set (BUILD-SPEC §15).
Pure functions over (expected, retrieved) — no model, no I/O. recall@k, set overlap (Jaccard), and mean cosine distance. These are the deterministic half of the testing fixed point: the same queries against the same fixture corpus must score the same way, so capability drift is measurable against a hand-blessed baseline (Invariant 9).
recall_at_k(expected, retrieved, k)
Fraction of the expected items present in the top-k retrieved. 1.0 if nothing is expected (vacuously satisfied).
set_overlap(expected, retrieved, k)
Jaccard overlap between the expected set and the top-k retrieved set.
mean_cosine_distance(distances)
Mean of returned hit distances (LanceDB cosine _distance; lower = closer).
0.0 for an empty result set.