ops.lifecycle
ops.lifecycle
Operational lifecycle — the one-command start/stop for the whole mind-palace.
ops/ is the outermost orchestration layer (it may import scheduler + core; nothing imports
it back). This package turns the already-built pieces — the supervisor, the durable queue, the
vault watcher, the stores — into a single supervised process with:
- a run ledger (
runs.py) pinning which git commit each run executed under, and whether it shut down cleanly (the basis for recovery mode); - preflight (
preflight.py) — ensure our own components, verify the external daemons (Vault / Ollama / podman) read-only and fail closed with a clear checklist; - the launcher (
launcher.py) —start(preflight → record run → rebuild-if-empty → supervise) with a graceful shutdown hook (SIGTERM/SIGINT → finish at a job boundary → mark the run clean → optional final snapshot),stop,status, andreset(the surgical fresh-start wipe that guards the production Vault Raft store).
children
Supervised child processes — the thin-master/child runtime model.
palace is the always-on master; components that MUST be separate processes (the network-facing edge
monitor — Invariant 2) are spawned, liveness-checked, and gracefully SIGTERM'd by palace, which
waits for them to drain before it exits (ASG-style). The core itself stays one process (it is the
model-slot arbiter — the ceiling has a single owner); only the genuinely-separate components are
children. spawn is injectable so the whole lifecycle is testable without real OS processes.
Spawn = Callable[[list[str]], Proc]
module-attribute
Proc
Bases: Protocol
A Popen-like: .pid, .poll() (None = alive), .terminate(), .wait(timeout), .kill().
Structural on purpose — tests inject a bare fake (no subprocess.Popen inheritance) that
satisfies this shape, so Spawn stays swappable without real OS processes.
pid
instance-attribute
poll()
terminate()
wait(timeout=None)
kill()
Child
dataclass
One supervised child process. Idempotent start (won't double-spawn a live child); stop
is graceful (SIGTERM → wait → SIGKILL on timeout) and never raises (shutdown must not crash).
name
instance-attribute
argv
instance-attribute
spawn = _default_spawn
class-attribute
instance-attribute
stop_timeout_s = 10.0
class-attribute
instance-attribute
pid
property
start()
alive()
stop()
launcher
The launcher — one supervised process for the whole mind-palace (operational lifecycle).
start → preflight (ensure own, verify externals, fail-closed) → record the run pinned to the
git commit → reconcile the corpus (a catch-up vault sync; rebuilds an empty cache) → run the
supervisor + watcher with a graceful shutdown hook (SIGTERM/SIGINT → stop claiming new work,
let the in-flight job finish at its boundary — the scheduler is already cooperative — then mark
the run CLEAN). stop signals the live run's pid. status shows preflight + the last runs.
reset is the surgical fresh-start wipe.
Recovery (nervous-system-and-ambassador.md §1): if the previous run never marked itself stopped
(crash / kill -9 / power loss), start comes up in recovery mode — scheduler halted, watcher
off, read-only — and asks the owner to inspect, then --force to resume. State itself lives in
the stores/files, so a clean restart just resumes; recovery is the cautious response to an
unclean exit, not the normal path.
SupervisorLike
Bases: Protocol
scheduler.supervisor.Supervisor's real surface here — structural so tests inject a bare
_FakeSupervisor without subclassing the real Supervisor. Narrowed to a no-arg run()
(the only call shape launcher.py uses: c.supervisor.run()), not the real Supervisor's
fuller run(*, max_ticks=...) — a Protocol is only as wide as its actual call sites here.
run()
WatcherLike
Bases: Protocol
core.ingest.watch.VaultWatcher's real surface here (structural, same reasoning).
start() narrowed to no-arg (the only call shape here: c.watcher.start()).
start()
stop()
QueueLike
Bases: Protocol
scheduler.queue.JobQueue's real surface here (only .close() is called through
Components.queue — build_components calls .depth() on its own JobQueue directly).
close()
ChildLike
Bases: Protocol
ops.lifecycle.children.Child's real surface here — structural so
tests/integration/test_lifecycle.py's bare _FakeChild satisfies it without subclassing.
name
instance-attribute
pid
property
start()
alive()
stop()
Components
dataclass
What serve drives. Injectable so tests exercise the lifecycle without models.
supervisor
instance-attribute
watcher
instance-attribute
queue
instance-attribute
enqueue_catchup = lambda: None
class-attribute
instance-attribute
enqueue_housekeeping = lambda: None
class-attribute
instance-attribute
health_check = lambda: []
class-attribute
instance-attribute
children = field(default_factory=list)
class-attribute
instance-attribute
snapshot = lambda _run, _flags: None
class-attribute
instance-attribute
Launcher
dataclass
cfg
instance-attribute
runs
instance-attribute
repo_root
instance-attribute
components_factory = build_components
class-attribute
instance-attribute
preflight_fn = run_preflight
class-attribute
instance-attribute
tick_seconds = 1.0
class-attribute
instance-attribute
housekeeping_interval_s = _HOUSEKEEPING_INTERVAL_S
class-attribute
instance-attribute
health_interval_s = 60.0
class-attribute
instance-attribute
snapshot_interval_s = 5.0
class-attribute
instance-attribute
on_shutdown = None
class-attribute
instance-attribute
gate_cmd = ('uv', 'run', 'pytest', '-q', '-m', 'not live and not podman and not needs_vault and not needs_restic')
class-attribute
instance-attribute
ci_check_cmd = ('uv', 'run', 'scripts/ci_witness.py', 'check')
class-attribute
instance-attribute
ci_release_cmd = ('uv', 'run', 'scripts/ci_witness.py', 'release')
class-attribute
instance-attribute
deploy_wait_s = 60.0
class-attribute
instance-attribute
deploy_poll_s = 0.5
class-attribute
instance-attribute
launchd_label = 'com.mind-palace.palace'
class-attribute
instance-attribute
start(*, force=False, max_ticks=None)
deploy(*, skip_tests=False)
Apply committed code/infra to the always-on system by a GRACEFUL cycle (owner rule 2026-07-11) — never a kill. Gate, then drain, then verify.
The gate: an active run exists; the working tree is clean; the branch is main; HEAD
differs from the live run's commit; the fast ratchet is green (--skip-tests is the
emergency hatch). Under launchd (KeepAlive) the graceful stop IS the restart — drain →
exit → relaunch on the new code — so deploy just waits for the successor run and
verifies its pinned SHA. Infra half: if the repo plist drifted from the installed
copy, the cycle is bootout → cp → bootstrap instead, so plist changes deploy the same
way code does. (Corollary the owner should know: under KeepAlive, palace stop means
RESTART; a true stop is launchctl bootout gui/$UID/com.mind-palace.palace.)
stop()
status()
reset_targets()
The corpus + its derived/chain layer + the stale queue. Computed from cfg.paths; each is asserted to be under data/ and outside the guard set (never the Vault Raft).
reset(*, confirm)
build_components(cfg)
Wire the full daemon: vault_sync (+watcher), the delegating Ambassador inbox, the delegated-task worker, and the trough dream/curate handlers — all on one supervisor.
build_launcher(config=None, **kw)
preflight
Preflight — ensure our own components, VERIFY the external daemons (fail-closed).
Owner's chosen scope: the launcher manages the mind-palace's own pieces (data dirs, the queue, the supervisor/watcher loop) and only checks the external daemons it depends on — Vault, Ollama, podman — which have their own lifecycles (LaunchAgents / the Ollama app / a podman VM). A required external being down is a fail-closed refusal with a clear checklist, not an attempt to start it.
ops/ is outside the core import firewall, so a loopback health probe here is fine (Vault and
Ollama both listen on 127.0.0.1; this never reaches off-box). The check functions are injectable
so tests assert the aggregation/fail-closed logic without any live daemon.
CheckFn = Callable[[object], Check]
module-attribute
Check
dataclass
name
instance-attribute
required
instance-attribute
ok
instance-attribute
detail
instance-attribute
render()
Preflight
dataclass
checks
instance-attribute
ok
property
All REQUIRED checks pass (a failed optional check is a warning, not a blocker).
failures()
render()
check_ollama(cfg)
check_vault(cfg)
check_podman(cfg)
check_own(cfg)
check_constitution(cfg)
Fail-closed integrity check on the fixed point: the live CONSTITUTION.md must match the
owner-blessed fingerprint in eval/golden/baseline.json (BUILD-SPEC §15, Invariant 9).
This runs the drift gauge's Constitution-breach comparison AT STARTUP, so a tampered or un-blessed Constitution is caught before any agent is framed with it — the runtime half that was missing: the fingerprint was recorded post-hoc in attestations but never compared in the live loop (a tampered file would otherwise be served to every agent after the next restart).
run_preflight(cfg, *, ollama=check_ollama, vault=check_vault, podman=check_podman, constitution=check_constitution)
Assemble the full preflight. External checks are injectable for tests.
runs
The run ledger — every supervised run pinned to a git commit (operational lifecycle).
The system's state (corpus, vectors, queue, ledgers, secrets) already persists in stores + files, so a normal restart just resumes. What was MISSING is an operational record of the runs themselves: which commit executed, when it started/stopped, and — crucially — whether it shut down cleanly. That last bit is the basis for recovery mode (nervous-system-and-ambassador.md §1): a previous run that never marked itself stopped (killed -9, panic, power loss) means the next start should come up cautious rather than assume consistency.
A small dedicated SQLite (data/runs.sqlite), the same pattern as the other state stores. It is
append-mostly: open_run inserts; mark_stopped closes the row. A row with stopped_at IS NULL
is either the live run or a crashed one — last_was_clean() distinguishes by recency.
RunRecord
dataclass
id
instance-attribute
commit_sha
instance-attribute
dirty
instance-attribute
pid
instance-attribute
started_at
instance-attribute
stopped_at
instance-attribute
clean_shutdown
instance-attribute
recovery
instance-attribute
note
instance-attribute
active
property
Still running (or crashed without closing) — stopped_at was never set.
RunLedger
dataclass
path
instance-attribute
open_run(*, commit_sha, dirty, pid, recovery=False, note='')
mark_stopped(run_id, *, clean, note='')
get(run_id)
last()
The most recently started run (the live one, if any).
last_was_clean()
Did the PRIOR run end cleanly? True if there is no prior run. A prior run still marked
active (stopped_at IS NULL) means it crashed — unclean → the caller should recover.
Call this BEFORE open_run (then last() is the prior run) — the launcher does.
recent(n=10)
close()
git_state(repo_root)
(commit_sha, dirty) for repo_root. Best-effort: a non-git checkout reports
('unknown', False) rather than raising — the launcher must still run.
open_run_ledger(config=None)
snapshot
Status snapshot — the core→edge monitoring handoff (Invariant 2).
The launcher writes a small JSON snapshot of operational METADATA to a file each health tick; the edge monitor process READS only that file to render its dashboard. Same asymmetry as the airlock: the core emits, the networked side never reads a store. The snapshot carries only what the OpsView already narrates — action counts, health, the shape of recent activity, queue depth, memory headroom — plus dream/finding counts. NO note text, NO authored-note titles, NO secrets/tokens.
build_status(*, ops_view, dreams_view, queue_depth, run=None, mem_available_gb=None, flags=())
Assemble the snapshot dict from the read-only views. Pure — no I/O — so it is unit-testable
against in-memory stores. run is the active RunRecord (commit-pinned); flags are the OS
watchdog's crossed-threshold flags.
write_status(path, data)
Write the snapshot atomically — the edge reader never sees a partial file (rename swap).