Projection and the materialiser, one log entry at a time.

What it is for

The screens in NovusLens show a twinTwinNovusLens’s model of one organisation, built from records about its teams, services and how they rely on one another. It is only as current as those records. as it stood at the last projectionProjectionThe regular pass that brings the read model up to date with the canonical log. It runs every 30 seconds by default. pass. The meshMeshA set of connected Mycelium services that exchange updates and keep the NovusLens log. commit log is the record of what was asserted, measured and corrected. The projection loop updates the read model at regular intervals. Corrections therefore reach the board without a manual refresh. Every surface shows the age of its picture from the latest pass, so a reader can see how current the figures are.

How it works

The work splits into three parts: a loop that runs passes, a materialiserMaterialiserThe component that reads recorded changes in order and updates the database. It records its place so it can continue after a restart. that applies log entries to PostgresPostgresThe database software that stores the tables read by the NovusLens screens, along with document bodies, access records and protected personnel records., and a set of post-apply stages that recompute derived views.

The projection loop

./lens exec -- python -m conway_reasoning.project runs the loop. The Compose service projector starts it and passes --interval from NOVUSLENS_PROJECTION_INTERVAL (default 30). The module itself does not read that variable.

Each pass, for each org:

1. A row is inserted into projection_runs before any work starts, so a crashed pass still leaves a record.
2. materialiser.retry_quarantine re-attempts the org’s quarantinedQuarantineA holding area for a log entry that could not be applied. The entry keeps its error and waits for a retry or an operator. entries that are due.
3. materialiser.materialise pulls the commits newer than the cursorCursorA saved position in a sequence of records. A program uses it to continue from where it stopped. and applies them.
4. Only if step 3 succeeded, the agent-evidence outboxOutboxA database list of saved messages waiting to be published. A message stays on the list until its progress is recorded. is delivered (agent_delivery.deliver).
5. If entries were applied or retried, operations._refresh_engines runs cpl, fidelity, history, meaning-sweep and derived, in that order. cpl computes critical path lengthCPLCritical path length. The total coordination weight of the longest chain of dependencies in the model. NovusLens reports it as index points, not elapsed time., the dependencyDependencyA link where one unit needs something from another. A dependency is not a reporting line: it shows where work relies on something else.-chain index. A pass that applied nothing recomputes nothing. A failing stage is recorded and the next stage still runs.
6. The projection_runs row is finished. It records applied and retried counts, the cursor, an error string and a stages document. The error names failed stages, capped at 500 characters. The stages document holds each result and any warnings.

latest(conn, org, successful=True) returns the newest pass with no error at all. A surface can therefore tell “last attempted” from “last succeeded”.

The materialiser

materialiser.materialise(conn, org, host, port) reads the org’s commit stream from the mesh. The stream key comes from keys.commit_stream(org). The reads use the mesh agent’s scan_log(stream, from_hlc).

• An organisation marked as forgotten is skipped, and the call returns 0.
• The starting point is the row in materialiser_cursor for the stream, read as last_hlc. The scan begins one HLCHybrid logical clockA number attached to a log entry to order it against other entries. It combines clock time with a counter, so entries can be ordered even when clocks do not agree exactly. above it. HLC is the stamp that orders commits in the log.
• When a pass finds nothing new, a coherence probe checks that the entry at the cursor still exists in the log. If it is missing, the probe retries up to 6 times, waiting 2 seconds between attempts. A restarted node may still be replaying its log. If the entry never reappears, LogDivergedError is raised. The message says the canon is gone and the org must be rebuilt.
• Each entry is applied in its own transactionTransactionA group of database changes that is saved together. If the group fails, its changes are not saved.. The transaction takes a per-org advisory lockAdvisory lockA database lock that cooperating programs take to prevent conflicting work at the same time. They must explicitly use the lock., checks the applied-event ledger, applies the entry, records the entry in materialiser_applied, and advances the cursor. The cursor update is GREATEST(...), so a stale pass never moves it backwards. The entry, the ledger row and the cursor move together or not at all.

The materialiser is idempotentIdempotentRepeating the same operation has the same effect as doing it once: it creates no extra copies or repeated changes.. UpsertsUpsertAdd a record if its identifier is new, or update the record with that identifier if it already exists. are keyed on stable identifiers, and the ledger stops a second application.

Late arrivals and --full

The cursor records the highest log position applied so far. A delayed entry can arrive below the cursor’s HLC after a network interruption or a slow node. An ordinary pass misses it. The defined recovery is a full rescan:

./lens graph materialise --host mesh-0 --org <org> --full

--full scans from HLC 0 and applies only the entries that materialiser_applied does not hold. Each run checks the whole log against the list of entries already applied.

Quarantine and retry

Entries are sorted into three kinds on failure. The classification is made by _classify.

A quarantined entry is recorded in materialiser_quarantine with its kind, payloadPayloadThe data carried inside a message, apart from the surrounding fields that identify and date it., error and retry flag. A finding of kind materialiser-quarantine is opened with the id materialiser-quarantine-<org>-<hlc>. The finding’s text says whether the entry is held for retry or waits for an operator. Entries already in the quarantine are skipped by the normal pass, so only retry_quarantine touches them.

Retry follows a fixed schedule. Each row starts with attempts set to 1, which counts the original failure.

The 1800-second step in RETRY_BACKOFF_SECS is written on the fifth failed retry, but the attempt cap stops any retry from reaching it. A retry that fails with a non-retryable error clears the retry flag, so the entry waits for an operator from then on. A successful retry applies the entry, deletes the quarantine row, and resolves the finding with the note “recovered: the entry applied on retry”.

Operators use two commands:

• ./lens graph retry --org <org> re-attempts the retryable entries that are due now.
• ./lens graph requeue --host mesh-0 --org <org> --hlc <n> applies one quarantined entry by HLC, whatever its kind. The entry is read back from the log, not from the quarantine copy. The finding is resolved with the note “recovered: requeued by an operator”.

Correction-epoch fencing

A correction pass holds a lock and stamps its writes with a fencing tokenFencing tokenA number that identifies a turn to update shared data. NovusLens rejects an earlier correction’s writes after a newer correction is applied.. The token is the HLC of the lock commit. The materialiser keeps the highest token it has applied per organisation in correction_epoch.

Fenced observed-layer writes are checked against that correction epochCorrection epochThe highest correction token already applied for one organisation. An observation with a lower token belongs to an older correction pass and is refused.. These are observed edges, observed C/A/RC/A/RThree separate scores for a team or service: Complexity, Agility and Recoverability. Each uses current evidence and the customer’s stated scoring rules. (Complexity, Agility and RecoverabilityRecoverabilityThe R in C/A/R: how well a team or service restores acceptable service after a failure. Recovery times and tested alternatives can supply evidence.) rows, axis observations and network flow observations. A write whose token is below the highest correction token already applied is refused and quarantined as poison. A write with a token at or above that value is applied and records the greater token. A write with no token is admitted without that protection. An observed edge missing a token or observation windowObservation windowThe period in which a measurement counts as current. The default is 30 days. After it passes, the measurement remains as history and is counted as expired. is stored at the told rungRungOne of three levels of support for a recorded value: told, shown or measured. The level appears beside the value.. Its source adds “unfenced/unwindowed: not a measured observation”.

Provenance rungs

The rung is the evidence level behind a value. The stored names are asserted, documented and measured. engine.py shows them as toldToldEvidence that somebody stated the value. It is the weakest level, and it is always attributed to its speaker., shownShownEvidence that a document supports a value. A recovery plan that states a target shows the target, but it does not show that the target was achieved. and measuredMeasuredThe top evidence level: a connected source recorded the observation within a stated period. A measurement shows what was observed on that occasion, not that it always holds..

Evidence cannot be replaced by a claim at a lower rung. A write at a lower rung than the one held is refused, and the stored row is left untouched. When the refused source differs from the held source, a finding of kind provenance-conflict is opened, so the contradiction is kept rather than dropped. The ratchet applies to attribute provenanceProvenanceThe record of where a value came from, who stated it and when. Every value keeps this record., to edges and to C/A/R layers. Cell value columns for structure, authorityAuthorityWhat a team or agent is allowed to decide or do. For an agent, this comes from a named person or system that grants permission. and operational ownerOperational ownerThe person who runs a team or service day to day. This is not automatically the accountable owner, and NovusLens records the two separately. are written only when their rung is admitted.

The one deliberate fall is an artifact purgePurgeRemoving a stored document’s contents while keeping a record that it was removed and why. Conclusions that used it must be checked again.. When an evidence body is purged, every row that cited it drops back to asserted, and the source is marked “(source purged)”. A measurement whose source is gone is no longer treated as a measurement.

Interfaces

Commands:

./lens forget-org <org> --reason '<why>' --yes requires --reason and --yes. It runs forget-org in a running reasoning or projector service.

Tables this area reads or writes:

Calls out: the materialiser reads the mesh through the agent’s scan_log and TwinWriter. The post-apply stages call engine.compute_cpl, compute_fidelity and record_history, plus the meaning sweep and the derived findings.

Agent cursors are not the materialiser cursor

conway_graph.agent_cursor tracks producer progress for agent-evidence batches. Each row in agent_evidence_cursors links a source’s previous cursor to its next one, per stream. advance takes an advisory lock for the org, refuses a cursor that does not continue from the stream head, and refuses to reuse a position. It is called from conway_reasoning.agent_evidence. It has nothing to do with materialiser_cursor, which is the read model’s position in the mesh log.

Invariants and failure behaviour

• The log is the source of truth. The read model can be rebuilt from the log. A reset does not touch the log.
• Entry, ledger and cursor move together. One transaction covers each entry. A crash between entries leaves the cursor at the last committed entry.
• Transient failures stop the pass. The cursor does not advance and nothing is quarantined. Note that run_once catches a transient failure with its general handler, so it is recorded as an error on the pass row. The loop does not add a backoff for it. The next pass tries again after the normal interval.
• Nothing is dropped silently. A failed entry is quarantined and surfaced as a finding, and the stream moves on past it.
• A missing log is refused. If the entry at the cursor has vanished, LogDivergedError is raised rather than serving a detached twin.
• A database outage does not end the loop. After connection failure, waits grow through 5, 10, 20 and 40 seconds, then stay at 60 seconds. Success resets the wait. If the database is lost in the middle of a pass, the loop prints a message and reconnects on its next turn. The interrupted pass’s projection_runs row is never finished, so latest does not count it. With --once, a failed connection returns exit code 1.
• A failed stage is recorded. A post-apply stage that raises is written to the pass row. The pass is not reported as healthy.
• Forgotten organisations stay forgotten. The projection loop skips them, and materialise refuses to rebuild them.
• Organisation scopeScopeThe work, data, activity or access that a rule, statement or calculation covers. For access, it names the records an account can see. For team measures, “Own operations” covers the selected unit. “Including descendants” adds the units beneath it.. Every projected row carries its organisation, and every pass is scoped to one.

Reset

./lens graph reset truncates the rebuildable tables. These include cells, edges, cell_car, attribute_provenance, findings, materialiser_cursor, materialiser_applied, materialiser_quarantine and correction_epoch, together with the agent-evidence and twin tables. A rebuild then starts from the log on the next materialise pass.

The following are not truncated, because nothing rebuilds them from the log:

• evidence_artifacts, which holds the evidence bodies. The log carries only content-addressedContent addressAn identifier calculated from a record's contents. It lets the program check that the retrieved bytes are the same bytes originally stored. citations.
• sealed_identity_join, which holds the sealed link between people and cells. Persons are not put on the mesh log.
• query_audit, the audit trail.

The reset also leaves projection_runs, forgotten_orgs and actor_map in place.

Forgetting an organisation

forget_org is the operator’s erasure act. In one transaction it:

1. writes an audit row under the operator’s identity, with the reason;
2. deletes the audit rows written by that organisation’s accounts;
3. deletes the organisation’s rows from every table with an org column, in a fixed order that puts children before parents;
4. deletes the organisation’s actor_map rows, and the materialiser_cursor rows whose stream name contains the organisation name;
5. records the organisation in forgotten_orgs, unless --allow-reuse is given. That flag is for development and rehearsal only.

A blank reason, the operator wildcard and an empty organisation name are refused.

What is not erased: the canonical logCanonical logThe permanent history of changes to an organisation’s model. New entries are added at the end. Earlier entries stay when the database is rebuilt. in the mesh data directories. It is append-only and still carries the organisation’s commits, including labels, owners’ names and finding subjects. Erasing it means stopping the services and destroying the mesh data directories. The function’s own output says so.

Open the diagram

For the full list of commands and environment variablesEnvironment variableA named setting made available to a running program. The customer launcher reads these settings from the installation environment file., see the conway_graph command line. For the layout of the read model as a whole, see the read model.