The schema, built one ordered file at a time.
What it is for
The read model is the PostgresPostgresThe database software that stores the tables read by the NovusLens screens, along with document bodies, access records and protected personnel records. database that the web application, the projectionProjectionThe regular pass that brings the read model up to date with the canonical log. It runs every 30 seconds by default. worker and the reasoning service read. The 51 files in apps/graph/migrations define the application tables. The migrationMigrationAn ordered change to a database's tables or rules. Applied migration files are recorded so the same file is not run again. runner also creates schema_migrations to record which files it has applied. The schemaSchemaThe declared structure of data: which fields or tables exist and which values they accept. defines which tables reset empties and the database privacy floorPrivacy floorThe rule that a unit with fewer than five mapped people is shown as a small team, never as a number. The key-person finding says fewer than five and never says who.. It also determines whether a service refuses an outdated database.
How it works
The migration files
Each file is named with a four-digit number and a short name, from 0001_init.sql to 0051_network_flow_observations.sql. The number is the order, and the header comment of each file says what it adds and why. Later files change earlier ones: 0011 drops a default organisation that 0002 had added, and 0016 removes development viewers that 0003 had seeded. The set is therefore a history, not a snapshot of the final schema.
The files are grouped by theme below. Every migration is in exactly one row.
| Migration | Theme | What it adds |
|---|---|---|
0001–0002 | Foundation and organisation scopeScopeThe work, data, activity or access that a rule, statement or calculation covers. For access, it names the records an account can see. For team measures, “Own operations” covers the selected unit. “Including descendants” adds the units beneath it. | cells, cell_car, attribute_provenance, edges, cell_lineage, findings, materialiser_cursor, twin_cpl, twin_fidelity, query_audit and their search indexes. 0002 adds an org key to the core tables, evidence_artifacts, sealed_identity_join and the cell_headcount() function. |
0003–0004 | Scenarios, viewers and artifact requests | scenarios, actor_map (the viewer-to-role table), artifact_requests, and a disputed marker on attribute_provenance. |
0005–0006, 0013 | CouplingCouplingA link where one team or service depends on another. Its Loose, Buffered or Rigid class describes the time needed for a handoff., names and service types | org_meta, the Unknown handoffHandoffWork or information passed from one team or service to another. Its measured or stated completion time determines the dependency class. value on edges, cell_aliases, and service_type on cells. |
0007–0008 | FederationFederationSharing selected signed facts and summaries between separate regional twins, while their detailed records stay in their own regions. and agents | groups, group_members and federation_facts, which has no foreign keysForeign keyA database rule requiring a reference in one record to point to an existing record in another table. to cells. The agent cellCellA box on the map that stands for a team, service, supplier or AI agent. It is a unit you can inspect, not necessarily a department or a person. kind. |
0009, 0016, 0017 | Operators, viewers and login | pending_proposals; the operator and board roles on actor_map; a nullable passphrase_hash on actor_map. 0016 deletes the four development viewers from databases that had them. |
0010, 0015 | Business domainsDomainA business area defined by the work it does, such as purchasing or customs. It may involve several teams, and one team may serve several areas. | domains, cell_domains, and accountable_owner on domains. |
0011, 0012, 0014 | Integrity and provenanceProvenanceThe record of where a value came from, who stated it and when. Every value keeps this record. | Adds materialiser_quarantine. Makes lineage, artifacts and personnel links unique by their identifying fields. Adds artifact_id and locator to edges and attribute_provenance. |
0018–0019 | StewardshipStewardshipCare of recorded information, such as correcting or withdrawing entries. The action allowed depends on the record type and the reader’s role. and the meaning canonMeaning canonThe organisation’s supplied written descriptions of its operations, systems and terms. NovusLens stores these separately from records of teams, dependencies and measurements. | The retracted findingFindingA difference in the records that needs a decision. It includes its sources and can describe a slow handoff, disputed fact or missing owner. status; purgePurgeRemoving a stored document’s contents while keeping a record that it was removed and why. Conclusions that used it must be checked again. columns on evidence_artifacts; meaning_entries. |
0020 | Timeline, finance and regulation | twin_history; financial_class on cells; regulations and cell_regulations. |
0021 | Key-person riskKey-person riskA finding about a team on the measured critical path with one to four people represented in the records. It never names those people. | The critical_path_owner_count() function, the second named read of the sealed join. |
0022–0023 | Measurement integrity | materialiser_applied, correction_epoch, the observation windowObservation windowThe period in which a measurement counts as current. The default is 30 days. After it passes, the measurement remains as history and is counted as expired. and fencing_token on edges, extra columns on twin_cpl, responsibility on cells, and provenance and ingested_at on cell_car. |
0024, 0050 | Questions | question_requests, with the question scope held as data. |
0025 | Change records in history | record_state and changes on twin_history. |
0026–0027 | Assessment evidence | car_assessment_evidence, and car_axis_observations: one value per Complexity, Agility and RecoverabilityRecoverabilityThe R in C/A/R: how well a team or service restores acceptable service after a failure. Recovery times and tested alternatives can supply evidence. (C/A/RC/A/RThree separate scores for a team or service: Complexity, Agility and Recoverability. Each uses current evidence and the customer’s stated scoring rules.) axis and per source. |
0028–0032 | Agent evidence intake | agent_evidence_outbox, agent_evidence_batches, agent_evidence_records, agent_evidence_delivery, agent_evidence_policies, agent_evidence_cursors and boundary_evidence_heads. |
0033 | Notifications | notification_subscriptions and notification_deliveries. |
0034–0036, 0038 | Externality, recovery and ownership | is_external on cells becomes three-valued, with NULL meaning not stated. projection_runs, retry columns on materialiser_quarantine and notification_deliveries, operational_owner on cells, and excluded_infra_slots on twin_fidelity. |
0037 | Privacy floor and forgetting | cell_headcount() default floor raised from three to five; forgotten_orgs. |
0039–0040 | GuideGuideThe help pages inside NovusLens. They explain the pages, input formats and tasks in the version you use. They are product instructions, not evidence about the organisation. and projection stages | guide_pages; stages on projection_runs. |
0041–0044 | RemitRemitA written list of an agent’s allowed actions, resources and destinations for information. The list states whether it is complete. authoring | product_sources, remit_templates, remit_declarations, activity_catalogues and remit_exports. One approved declaration per subject at a time. |
0045–0047 | People export snapshots | people_exports; export_id on sealed_identity_join; a team index. A complete export withdraws the people in its teams that it does not name. |
0048–0049 | Finding identity | finding_aliases, and a backfill that gives older dependencyDependencyA link where one unit needs something from another. A dependency is not a reporting line: it shows where work relies on something else. findings their consumer and provider. |
0051 | Network egressEgressTraffic leaving a system or network for another destination. | network_flow_observations, with the observation point fixed to network. |
The main tables, by purpose
The list below names the tables that carry the product’s main reads. It is not exhaustive.
| Purpose | Tables |
|---|---|
| The twinTwinNovusLens’s model of one organisation, built from records about its teams, services and how they rely on one another. It is only as current as those records.’s structure | cells, edges, cell_car, attribute_provenance, cell_lineage, cell_aliases, org_meta |
| Measurement outputs | twin_cpl (critical path lengthCPLCritical path length. The total coordination weight of the longest chain of dependencies in the model. NovusLens reports it as index points, not elapsed time., CPL, per layer), twin_fidelity, twin_history |
| Findings and their identity | findings, finding_aliases, notification_subscriptions, notification_deliveries |
| Evidence | evidence_artifacts, artifact_requests, car_axis_observations, car_assessment_evidence, network_flow_observations, boundary_evidence_heads, and the agent_evidence_* tables |
| Sealed personnel | sealed_identity_join, people_exports |
| Organisations, access and federation | actor_map, pending_proposals, query_audit, groups, group_members, federation_facts |
| Domains, meaning and regulation | domains, cell_domains, meaning_entries, regulations, cell_regulations |
| Remits | product_sources, remit_templates, remit_declarations, activity_catalogues, remit_exports |
| Projection state | materialiser_cursor, materialiser_applied, materialiser_quarantine, correction_epoch, projection_runs, forgotten_orgs |
| Other | scenarios, question_requests, guide_pages |
Records held only in Postgres
The rebuild from the log is described on the projection page. The reset() docstring in db.py names three stores held only in Postgres. These are evidence_artifacts (document bodies), sealed_identity_join (protected personnel links) and query_audit (access history). Migration comments describe further tables as primary intake or operational state: the intake state in agent_evidence_outbox, agent_evidence_delivery, agent_evidence_policies and agent_evidence_cursors, plus boundary_evidence_heads and product_sources. reset() leaves all of these in place. It does truncate agent_evidence_records and agent_evidence_batches. It truncates a specific list of 27 derived tables, and the tables outside that list are not touched.
The sealed identity join and the privacy floor
Person-level records are stored only in sealed_identity_join. Each row holds saltedSaltExtra data mixed into a value before its fingerprint is calculated. It makes matching or guessing the original value harder. hashesHashA fixed-length fingerprint calculated from content. Comparing fingerprints helps detect changes and match records. of a person’s identity and of their handle, a team, a Cell and a retention limit (retain_until). Rows never go on the meshMeshA set of connected Mycelium services that exchange updates and keep the NovusLens log. log or into the key-value storeKey-value storeStorage where a named key points to a value. Reading that key retrieves the value currently stored under it.. The people ingestIngestRead supplied records or documents and add their information to NovusLens, after the required checks. in conway_connectors.personnel writes the join, and resolve_actor maps a handle to a Cell without returning a person.
The headcount read is cell_headcount(). Below the floor it returns NULL, not a small number. Migration 0037 raised the default floor to five, and PRIVACY_FLOOR in keyperson.py is also five. The key-person check reads critical_path_owner_count() and reports that one to four people are mapped, never the count. It skips cells with no mapped people. Missing ownership stated in an interview can raise unowned-capability separately.
Running migrations
./lens graph migrate calls db.migrate(). The function creates schema_migrations if it is missing, then runs each file in apps/graph/migrations whose name is not recorded, in sorted filename order. For each file it runs the SQLSQLStructured Query Language: the language used to read and change records in a database., records the file name, and commits. The file and its record commit together. If a file fails, the error is raised before its record is written, so the file is not recorded as applied.
The command-line entry points never create tables. Every conway_graph subcommand other than migrate, and the connectorsConnectorSoftware that reads records from a source system, such as an incident tool. A customer-written file connects those records to teams or services in NovusLens.’ commands, call db.assert_schema() before they run. That check runs in two steps. It first confirms that five core tables exist (cells, edges, materialiser_cursor, twin_cpl and schema_migrations). It then compares every .sql file name in the directory with the names in schema_migrations, and it refuses to start if any are missing.
The connection comes from DATABASE_URL. If that is not set, connect() uses a development default on 127.0.0.1, port 5433, which matches the port mapping in deploy/dev/postgres.compose.yml. Under CONWAY_PROFILE=customer, a missing DATABASE_URL is an error instead. Run the command with ./lens run -- <command> to use the installation’s saved configuration. The connection has a five-second timeout.
Interfaces
| Name | Kind | What it does |
|---|---|---|
DATABASE_URL | Environment variableEnvironment variableA named setting made available to a running program. The customer launcher reads these settings from the installation environment file. | The Postgres connection string. Read by connect(), never written to disk. |
CONWAY_PROFILE | Environment variable | When set to customer, a missing DATABASE_URL is refused rather than defaulted. |
./lens graph migrate | Command | Applies the pending migrations and prints their names, or that nothing was pending. |
db.assert_schema(conn) | Function | Startup check. Refuses if the core tables or any migration file is missing from the record. |
db.reset(conn) | Function | Truncates the 27 derived tables, leaving the mesh log and the Postgres-primary stores as they are. Run by the reset subcommand. |
For the commands that use these functions, see the conway_graph command line. For how the derived tables are filled, see projection and the materialiser.
Invariants and failure behaviour
- Files run in filename order, and each runs once. The record is the file name only. The runner stores no checksum, so an edited file that has already been applied is not detected.
- A file and its record commit together. A failed file leaves no record and does not stop the earlier files from being recorded.
- The command-line entry points refuse to run while a migration file is unapplied. They do not create what is missing.
assert_schema()checks only the names of the files. A record for a file that has since been deleted does not cause a refusal.reset()empties derived tables and keeps the log. The Postgres-primary stores survive it, because nothing could rebuild them from the log.