core.ingest
core.ingest
Zone A — ingest: the storage engine's write path (BUILD-SPEC §8, §9).
Parses the owner's Logseq vault into the immutable raw store (content-addressed, dedup), extracts the explicit graph layer the owner authored (tags, [[links]]), and chunks notes for embedding. Everything ingested here is provenance AUTHORED — the mirror's ground truth. Embedding + LanceDB indexing build on these records (next increment).
amend
Versioned amendment as a chunk-level diff (ingest-identity-and-amendment.md §4).
Today an amendment is destroy-and-replace: a changed note gets a new whole-note digest, so EVERY chunk re-embeds and the old rows are dropped (core/ingest/sync.py:88-101; proven by tests/integration/test_index_keying.py). §4 wants the opposite — unchanged chunks keep their points, only changed/new chunks embed, removed chunks are marked superseded — so the stable parts of a frequently-edited note never churn and provenance is enhanced, not destroyed.
This module is the PURE core of that: given the old and new chunk sets of ONE document, compute what to retain / embed / supersede, keyed by chunk-content-hash SCOPED to the document id. It does not read or write any store and does not embed — those are the wiring (gated behind a re-index and an owner-ratified §4 go, build plan PD5). Family 2: derived + regenerable, no network.
The scoping is load-bearing (build plan R1, resolving the §3-vs-§7 tension): identity is (doc_id, chunk_hash), NOT a bare global chunk-hash. Two DISTINCT documents that happen to share a verbatim paragraph keep BOTH points — independent provenance agreeing is corroboration (§7), never coalesced. Only re-occurrences of a chunk WITHIN one document's version chain coalesce.
AmendmentPlan
dataclass
The re-projection an amendment implies, as sets of chunk-content-hashes for ONE document.
retained — in both versions (same hash): keep the existing point, do NOT re-embed.
added — new this version: embed a new point.
removed — gone this version: mark the point superseded (kept as provenance, never deleted).
retained
instance-attribute
added
instance-attribute
removed
instance-attribute
is_noop
property
No content changed — nothing to embed or supersede (an idempotent re-ingest, e.g. a touch or a reorder-without-edit).
text_hash(text)
SHA-256 of chunk text — the chunk content-hash for a raw text string, identical to
Chunk.content_hash for the same text. Lets the amendment path recognize an already-stored
row (a plain dict carrying text) by content, without re-wrapping it in a Chunk.
chunk_point_id(doc_id, chunk)
The document-scoped, content-addressed identity of a chunk point: (doc_id, chunk_hash).
Stable across versions of the SAME document (an unchanged chunk keeps its id) and distinct
across documents (shared verbatim text stays two points — §7). This REPLACES the current
f"{note_digest}:{chunk_index}" key (core/ingest/index.py:33), whose note-digest prefix is
exactly why an amendment re-keys every chunk. doc_id is a stable document identity (the
catalog source_path, or a minted doc-id — never the whole-note content-hash, §4).
plan_amendment(old, new)
Diff one document's old vs new chunk sets by content hash. Pure and order-insensitive:
identity is the chunk's content, so a moved-but-unchanged chunk is retained, not churned —
the property the current index-position key cannot provide.
chunk
Chunk note text for embedding (BUILD-SPEC §8 derived layer).
Block-aware and deterministic: split on blank lines (Logseq blocks), hard-split any block larger than the budget, then greedily pack blocks up to a character budget with overlap. Chunks are a regenerable derived representation — re-chunk from the raw store if the strategy or embedding model changes. Token-aware sizing (per the embedding model's tokenizer) is a later refinement; characters are a stable proxy now.
Chunk
dataclass
index
instance-attribute
text
instance-attribute
content_hash
property
The chunk's content-address: SHA-256 of its text. The identity a content-addressed
derived index keys on (ingest-identity-and-amendment.md §3–§4; build plan Item 1b), so an
unchanged chunk is recognizable across versions of a note. A property, not a stored field:
additive and construction-preserving — chunk_text and every existing caller are
unchanged, and the derived layer stays regenerable from raw (§8).
chunk_text(text, *, max_chars=1200, overlap_chars=150)
curated
Ingest the system's own white papers + design notes as a curated self-knowledge graph
(Track B / B4; nervous-system-and-ambassador.md §4).
The Ambassador can explain its own architecture by reading this graph — "fittingly, the white
papers + design notes ARE the corpus" for self-narration. It is CURATED, not authored: it is
the system's design prose, kept in its OWN graph and never merged into the authored mirror
(curated ∉ MIRROR_READABLE — the same firewall as book dreaming). The Ambassador reaches it
only via a deliberate, non-default provenances={CURATED} query.
Reuses the parametrized ingest pipeline (provenance=CURATED) into the existing multi-provenance VectorStore — no new store. Only PROSE docs are ingested (Constitution, Conventions, the docs/ tree). Config and anything that could hold a secret are never sourced here (there are none in the .md tree; the note's own caveat: explain the design, never expose live keys).
CuratedReport
dataclass
ingested = 0
class-attribute
instance-attribute
chunks = 0
class-attribute
instance-attribute
curated_sources(repo_root)
The self-knowledge corpus: the Constitution, the Conventions, and the whole docs/ tree (white papers + design notes). Prose only — never config or secrets.
ingest_curated(paths, raw, store, embedder, catalog, *, repo_root, attestor=None)
Ingest each doc as CURATED. Titles are repo-relative (e.g. docs/design-notes/...).
Idempotent: delete-then-index per digest, so re-running after a doc edit re-embeds cleanly.
build_and_ingest_curated(config=None)
Wire + run the curated ingest against the configured stores + embedder (needs the live embedder — owner-run, see scripts/ingest_self_knowledge.py).
dialogue
Capture owner↔Ambassador dialogue into the corpus as authored-dialogue (Track B).
Chatting with the Ambassador is itself a form of feeding the system — "your words to it are
more yours than its words to you" (nervous-system-and-ambassador.md §4). So the owner's
messages are captured as a distinct provenance, AUTHORED_DIALOGUE, which IS mirror-readable
(it is the owner's own writing) — closing the capture loop.
It rides the SAME deterministic path as vault ingest — ingest_note (now provenance-parametric)
→ index_records → VaultCatalog.record — never a bespoke writer (the §1 split is exactly what
makes this one-line provenance swap possible). Raw is sacred: the message bytes are stored
content-addressed; identical text dedups. The attestor (optional) stamps a capture action so
the dialogue leaf is part of the same attestation chain as authored notes.
DialogueCapture
dataclass
raw
instance-attribute
store
instance-attribute
embedder
instance-attribute
catalog
instance-attribute
attestor = None
class-attribute
instance-attribute
capture(text, *, conversation='default')
Store one owner message as authored-dialogue; return its content digest.
Idempotent on content: identical text is one raw object + one set of vectors (delete- then-index, the store's own re-index idiom); each call still records a distinct catalog entry keyed by timestamp so the conversation's turns are individually tracked.
build_dialogue_capture(config=None)
Wire a DialogueCapture against the configured stores + embedder.
embed
Embedding adapter (BUILD-SPEC §8 derived layer).
Wraps the local embedding model. Documents are embedded plain; queries are wrapped in the model's instruction format (Qwen3-Embedding is instruction-aware on the query side, which materially improves retrieval). Embeddings are a regenerable derived representation — re-embed from the raw store if the model changes (§8).
Embedder
dataclass
client
instance-attribute
config
instance-attribute
dim
property
embed_documents(texts)
embed_query(text)
build_embedder(config=None)
founding
Founding-corpus ingest — authoring the initial condition (founding-corpus.md; build plan Item 3).
The founding corpus is NOT model training and NOT steady-state ingest (founding-corpus.md §1–§2): it is a hand-selected batch of the owner's musings, authored across a long past, injected at once as the graph's initial condition. It must be a dated, supersession-linked sequence (§2.1 — reconstruct the partial order of thought, not a bag stamped "today"), and it MUST share the steady-state ingest path or the provenance model fractures at the origin (§4 / Q3).
So this driver is a thin batch over the ONE pipeline — ingest_note (provenance AUTHORED_SOLO) →
index_records → VaultCatalog.record, exactly the curated.py shape, never a bespoke writer. Two
founding-specific disciplines it enforces:
- Dated, not a bag (§2.1). Every item carries a reconstructed date, recorded as a
date::property IN the note, so the raw content-addressed blob carries it (permanent provenance) andparse_textlifts it intoproperties['date']. An undated item is refused — the timestamp lie (collapsing years into simultaneous peers) is exactly what founding must avoid. (The temporal layer reading these dates is dormant today; the dates are recorded now regardless.) - Supersession-linked (§2.1). When a later musing revises an earlier one, it is recorded as an
OWNER-DECLARED authored-historical supersession (
core/stores/authored_supersession.py; 8f / PD11) — a K₀↔K₀ RELATION between two authored documents, so the active projection shows the current musing and the earlier lives on in history. It is NOT a claim-supersede(no warrant, no derived alternative) and NOT a note-versionsupersedes(two documents, not one doc's versions). Founding is an owner entry point, so it mints anOwnerDeclaration; the store is owner-declared only and refuses any machine caller at its boundary (the-edge-model.md §4a).
Provenance is AUTHORED_SOLO — the owner's writing, the mirror's ground truth. The founding corpus is deliberately biased-coherent, so it CANNOT be the Track-L control corpus (§2.3): the control is a separate, non-curated CURATED-graph act. This driver writes only AUTHORED_SOLO and never the control — the two acts stay mechanically distinct. Ingest, not fine-tuning; the weights never move.
UndatedFoundingItem
Bases: ValueError
A founding item with no reconstructed date — refused (§2.1: a dated sequence, not a bag stamped 'today'; the timestamp lie is exactly what founding must avoid).
ForwardSupersession
Bases: ValueError
A founding item supersedes one not yet ingested — refused: the sequence is ordered (a musing can only revise an EARLIER one), so a forward reference is a manifest error.
FoundingItem
dataclass
One dated musing in the founding sequence. body is the text; date its reconstructed
original date (a date:: property in the note); supersedes the source_path of an
EARLIER item this one revises, or None.
source_path
instance-attribute
title
instance-attribute
body
instance-attribute
date
instance-attribute
supersedes = None
class-attribute
instance-attribute
FoundingReport
dataclass
ingested = 0
class-attribute
instance-attribute
chunks = 0
class-attribute
instance-attribute
superseded = 0
class-attribute
instance-attribute
ingest_founding(items, raw, store, embedder, catalog, *, supersession_store=None, attestor=None)
Ingest the founding sequence through the STEADY-STATE path (no bespoke writer): each item
rides ingest_note (AUTHORED_SOLO) → index_records → VaultCatalog.record, exactly like
curated/dialogue ingest. Dated (undated refused) and supersession-linked (recorded as an
OWNER-DECLARED authored-historical supersession when a later musing revises an earlier — 8f).
Ingest, not fine-tuning — weights never move (§1); AUTHORED_SOLO, never the control (§2.3).
build_and_ingest_founding(items, config=None)
Wire + run the founding ingest against the configured stores + embedder (owner-run; needs the live embedder — see scripts/ingest_founding.py).
index
Index ingest records into the vector store + provenance-aware semantic search (BUILD-SPEC §8, §9).
Vectors are derived and regenerable: to re-index after a model/strategy change, rebuild the vector store from the raw corpus (§8) rather than mutating in place.
Chunk points are keyed by a DOC-SCOPED content address (source_path, chunk_hash) (§3/§4,
build plan R1): stable across versions of a note (an unchanged chunk keeps its point) and
distinct across documents (two notes sharing a verbatim chunk keep both points — corroboration,
§7). index_amendment uses that to re-embed only the chunks that actually changed.
index_records(records, embedder, store)
Embed each record's chunks and add them to the vector store. Returns rows added. Identical-content notes (same digest) are embedded once; within a note, chunks are deduplicated by content hash — one point per canonical chunk (§3).
index_amendment(record, existing_rows, embedder, store)
Re-index one note as a chunk-level amendment (ingest-identity-and-amendment.md §4).
Reuse the vector of any chunk whose content is unchanged from the note's current projection
(existing_rows) — NO re-embed — embed only genuinely new chunks, dedup this version's chunks
by content, and replace the note's projection under its stable source_path. Returns
(embedded, reused). The stable parts of a frequently-edited note therefore never re-embed and
keep a stable point id; only changed/new chunks cost an embedding call.
rekey_preview(store)
(total rows, count whose id would change) under the doc-scoped re-key — a read that mutates nothing. The dry-run half of the Item-1c migration.
rekey_store(store)
Re-key every stored row to its doc-scoped content id (source_path, chunk_hash) IN PLACE,
preserving vectors (NO re-embed) — the migration off the old {digest}:{chunk_index} scheme
(build plan Item 1c). Identical chunks within a source coalesce to one point (§3). Idempotent:
a row already under the new key re-keys to itself, so re-running is a no-op. Returns
(rows_read, points_written); the raw store and catalog are untouched (this is a derived-layer
re-key, not a re-ingest — so it needs no embedder and cannot be defeated by catalog change-
detection the way a reset + rescan() would be). Regenerable from raw regardless (§8).
semantic_search(query, embedder, store, *, k=5, provenances=MIRROR_READABLE)
Search the thought-graph. Defaults to MIRROR_READABLE (AUTHORED only) — the introspective default that keeps observed exhaust out of the mirror. Pass provenances=None for the assistant tier to search across all classes.
grouped_semantic_search(query, embedder, store, *, k=5, provenances=MIRROR_READABLE)
Semantic search returning results grouped by SOURCE OBJECT instead of flat chunks.
The explicit opt-in to source-grained retrieval: flat semantic_search stays the default
and is untouched (byte-identical), and this is a separate entry point rather than a flag on
it — the recursive-strata I3 floor-zero posture (the grouped mode adds nothing to the flat
path). k is the flat chunk budget; the returned sources are those chunks grouped by
digest, so a query hitting two chunks of one note yields one source with two members. Source
order follows search rank (each source at its best hit; see group_sources). Defaults to
MIRROR_READABLE like semantic_search. To expand a hit to its full membership rather than
only the matched chunks, call source_set(store, hit.digest).
logseq
Parse a Logseq markdown vault into the explicit (authored) layer (BUILD-SPEC §8).
Logseq is the owner's hand-made skeleton: page text, the tags they applied, the [[links]] they drew. We extract those as ground truth and seed the thought-graph from them rather than imposing an ontology up front. Interpreted edges (similarity, inferred themes) are a separate, derived layer added later.
DEFAULT_PATTERN = '**/*.md'
module-attribute
DEFAULT_EXCLUDE_DIRS = frozenset({'.git', 'bak', '.recycle', 'assets', '.obsidian'})
module-attribute
ParsedNote
dataclass
source_path
instance-attribute
title
instance-attribute
text
instance-attribute
tags
instance-attribute
links
instance-attribute
properties
instance-attribute
raw_bytes = b''
class-attribute
instance-attribute
parse_text(text, *, source_path, title, raw_bytes)
Parse an in-memory note (tags, links, properties) — the path-free core of parse_note.
Used by programmatic ingest that builds note text directly rather than reading a vault file (the founding-corpus driver, and anywhere else authored text is composed in memory), so those paths get the SAME tag/link/property extraction as vault ingest — no bespoke parser.
parse_note(path, vault)
iter_vault(vault, *, pattern=DEFAULT_PATTERN, exclude_dirs=DEFAULT_EXCLUDE_DIRS)
Yield in-scope markdown files, deterministically ordered. pattern and
exclude_dirs are the §20.8 sub-scoping knob (which graphs are ingested).
pipeline
Ingest pipeline: vault -> raw store (dedup) -> chunks (BUILD-SPEC §8, §9).
The deterministic write path. Embedding + LanceDB indexing consume IngestRecord.
ingest_note is provenance-PARAMETRIC (default AUTHORED_SOLO, the mirror's ground truth):
vault notes are solo-authored, but the same chunk/embed path is reused — never a bespoke
writer — for the Ambassador's AUTHORED_DIALOGUE capture and the CURATED self-knowledge
ingest, which pass their own provenance through this one pipeline (the §1 spectrum split).
IngestRecord
dataclass
digest
instance-attribute
source_path
instance-attribute
title
instance-attribute
provenance
instance-attribute
tags
instance-attribute
links
instance-attribute
chunks
instance-attribute
is_new
instance-attribute
derive_chunks(raw_bytes, *, max_chars=1200, overlap_chars=150)
The chunks a raw blob yields — the ONE authoritative raw→chunks derivation, deterministic
and reproducible from the immutable raw store alone: decode (the tolerant text view, §8) then
chunk. ingest_note performs exactly this (it chunks note.text, which parse sets to
_decode(note.raw_bytes)); factoring it here lets the retrieval-integrity check
(core.ingest.verify) re-derive a source's chunks and confirm a stored row still matches —
"derived is regenerable from raw" made checkable.
ingest_note(note, raw, *, provenance=Provenance.AUTHORED_SOLO, max_chars=1200, overlap_chars=150)
ingest_vault(vault, raw, *, pattern='**/*.md', max_chars=1200, overlap_chars=150)
purge
Purge-raw — deliberate, owner-gated TRUE deletion (design-notes/vault-sync-and-capture.md).
The watcher never deletes raw: a vault delete only TOMBSTONES (derived dropped, raw kept) so nothing is lost and a re-add dedups. But for genuine privacy deletion the owner must be able to destroy the source bytes too. That is this action — and it is deliberately NOT the watcher's default, mirroring the propose/approve posture of the self-modification gate (Invariant 4): destroying ground truth is consequential and irreversible, so it requires an explicit owner act and refuses to fire on content still in use.
Two gates, both fail-closed:
1. confirm=True must be passed explicitly (no accidental purge).
2. the digest must have zero active references — an active note still holds this content,
so tombstone/delete it from the vault first. (Purge operates on already-tombstoned data.)
On success it drops derived rows, removes the raw blob, and deletes the tombstoned catalog
rows for that digest. scripts/purge_raw.py is the owner-facing entry.
PurgeRefusedError
Bases: RuntimeError
The purge was refused by a safety gate (no confirm, or content still referenced).
PurgeResult
dataclass
digest
instance-attribute
raw_removed
instance-attribute
paths_removed
instance-attribute
purge_raw(digest, *, raw, store, catalog, confirm=False)
Permanently remove a note's raw bytes + derived rows. Owner-gated; see module docstring.
run
Run a full ingest of the configured vault into the real stores (BUILD-SPEC §8, §9).
Rebuild semantics: the raw store is append-only and content-addressed (immutable, dedup); the vector store is rebuilt from scratch each run, because vectors are a derived layer regenerable from raw. This is the entry the scheduler (Phase 3) will drive as a job.
IngestSummary
dataclass
notes
instance-attribute
new_raw
instance-attribute
chunks_indexed
instance-attribute
vector_rows
instance-attribute
run_ingest(config=None, *, rebuild=True)
sync
Incremental vault sync — re-ingest changed notes (design-notes/vault-sync-and-capture.md).
Core-side, LOCAL filesystem only: it reads vault files and writes the local stores. No
network, no edge, no sockets — the seal holds and the import-lint proves it. This is the
deterministic engine; the watcher (core/ingest/watch.py) only triggers it, and the
scheduler runs it as a background job so all store mutation stays on the single writer.
Idempotency rides on the existing content-addressing plus the vault catalog:
- unchanged (same digest, still active) → no-op: no re-embed, no new rows.
- changed / new → (re)embed the note's chunks; the previous digest's derived rows are dropped iff no other active file still references that content.
- deleted → tombstone: derived rows dropped, the catalog row marked inactive, and the
raw blob kept (raw is sacred) so a later re-add dedups. True deletion is the separate,
owner-gated purge (
core/ingest/purge.py), never done here.
Everything ingested is authored-solo — the existing AUTHORED provenance tag (the spectrum
split is deferred, see PROGRESS.md). The mirror firewall is unaffected: these are the owner's
own notes, the mirror's ground truth.
SyncOutcome
Bases: Enum
UNCHANGED = 'unchanged'
class-attribute
instance-attribute
INDEXED = 'indexed'
class-attribute
instance-attribute
TOMBSTONED = 'tombstoned'
class-attribute
instance-attribute
SyncReport
dataclass
indexed = 0
class-attribute
instance-attribute
unchanged = 0
class-attribute
instance-attribute
tombstoned = 0
class-attribute
instance-attribute
tally(outcome)
VaultSync
dataclass
vault
instance-attribute
raw
instance-attribute
store
instance-attribute
catalog
instance-attribute
embedder
instance-attribute
pattern = '**/*.md'
class-attribute
instance-attribute
exclude_dirs = DEFAULT_EXCLUDE_DIRS
class-attribute
instance-attribute
max_chars = 1200
class-attribute
instance-attribute
overlap_chars = 150
class-attribute
instance-attribute
attestor = None
class-attribute
instance-attribute
version_store = None
class-attribute
instance-attribute
sync_path(path)
Re-ingest one note as a chunk-level amendment; unchanged content is a no-op.
handle_deleted(source_path)
A vault file disappeared: tombstone it and drop its projection (by source_path).
Source-scoped, so an identical-content file elsewhere keeps its own rows. Raw is kept
(sacred); true deletion is the separate, owner-gated purge (core/ingest/purge.py).
rescan()
Full catalog-vs-vault reconciliation. The watcher triggers this; it is the idempotent backbone (an unchanged re-scan does no work) and also the catch-up path for changes that happened while no watcher was running.
build_vault_sync(config=None)
Wire a VaultSync against the configured vault + real stores + embedder.
verify
Retrieval-time content integrity (BUILD-SPEC §8, closes prompt-integrity audit G9.5).
The raw store is the immutable, content-addressed source of truth (H(raw) = digest); the vector
store's chunk rows are a derived, mutable layer. Retrieval reads the chunk text field at face
value, so a mutated LanceDB row would reach the prompt while the read attestation logged the clean
digest — a false-fidelity record, and an injection vector once lower-trust provenances become
retrievable. This verifies each retrieved row against its raw source before it can be cited: the
text must be one of the chunks re-derived from the raw blob it claims. The check is exact — the
same raw bytes → _decode → chunk_text derivation the ingest pipeline used (derive_chunks), so a
legitimate row always passes and only text that does not reproduce from the sacred layer is dropped.
IntegrityDrop
dataclass
A row that could not be verified against the raw store and was withheld from the prompt.
digest
instance-attribute
chunk_index
instance-attribute
reason
instance-attribute
verify_rows_against_raw(rows, raw, *, max_chars=1200, overlap_chars=150)
Split rows into (verified, dropped).
A row is verified iff its text is one of the chunks re-derived from the raw blob identified by
its digest — the exact ingest derivation (derive_chunks). The raw blob is fetched and
re-chunked once per digest (a query hitting several chunks of one note pays one raw read).
A missing raw blob ("raw-missing") or a text that does not reproduce ("text-not-in-raw") is
dropped fail-closed — unverified content never reaches a model. Verified rows keep input order
(so a distance-ranked result stays ranked); dropped carries each withheld row's reason.
watch
Local vault watcher — core-side, LOCAL filesystem only, NO network (vault-sync task).
Watches the configured vault path and signals when notes change, so the system keeps the
owner's embeddings current as they write. It does not mutate the stores itself and does
not import the scheduler: on a change it just calls an injected on_change callback. The
scheduler wiring (scheduler/vault_sync.py) supplies a callback that enqueues a background
vault_sync job, so all store writes stay on the single supervisor writer.
Seal integrity: this module imports no edge, no sockets, no http — only the local
filesystem (the import-lint proves it). watchdog (FSEvents/inotify) is an OPTIONAL real-time
backend, imported lazily; without it the watcher falls back to polling (a timer that
triggers a periodic rescan). Either way on_change ultimately runs the idempotent
VaultSync.rescan, so missed/duplicate events are harmless.
OnChange = Callable[[], None]
module-attribute
ObserverLike
Bases: Protocol
The slice of a watchdog Observer this watcher drives (watchdog is an OPTIONAL dependency, so its own types never appear in signatures here).
stop()
join(timeout=...)
VaultWatcher
dataclass
vault
instance-attribute
on_change
instance-attribute
debounce_s = 1.0
class-attribute
instance-attribute
poll_interval_s = 5.0
class-attribute
instance-attribute
backend = field(default='', init=False)
class-attribute
instance-attribute
notify()
A change was observed. Arm/re-arm the debounce timer so a save burst fires once.
start(*, prefer='auto')
Begin watching. prefer: 'auto' (watchdog if importable, else poll), 'watchdog',
or 'poll'. Returns the backend actually started.