Skip to Content
ArchitectureConnectors and meaningIngest of organisational inputs
Architecture · Acquisition

Organisational inputs, at the rung they earn.

Each input, its command, and the rung its claims are recorded at. Simulated fixtures are labelled as such in the diagram.

What it is for

Much of what NovusLens knows about an organisation arrives as documents rather than telemetryTelemetryMeasurements and activity records sent by a running system, such as ticket timings, network traffic or workflow results.. Inputs include org charts, personnel exports, business-domain lists, finance classifications, regulation mappings and operations reports. Infrastructure files show where services run. The ./lens connectors sub-commands in this page read those files and write them into the twinTwinNovusLens’s model of one organisation, built from records about its teams, services and how they rely on one another. It is only as current as those records.. Each claim keeps a citation to the document that supplied it, so a reader of a cellCellA box on the map that stands for a team, service, supplier or AI agent. It is a unit you can inspect, not necessarily a department or a person. or a dependencyDependencyA link where one unit needs something from another. A dependency is not a reporting line: it shows where work relies on something else. edge can open the source.

The inputs are files on disk. Every ingestIngestRead supplied records or documents and add their information to NovusLens, after the required checks. command takes a --fixture path. None of them calls a live system; polling operational systems belongs to the acquirersAcquirerA program that reads a customer's operational records, calculates quantities from them and records the results in NovusLens..

How it works

The evidence rungs

Each claim in the twin sits on one rungRungOne of three levels of support for a recorded value: told, shown or measured. The level appears beside the value. of the evidence ladderEvidence ladderThe three levels of support a record can have, in order: told, shown and measured.. The code names the rungs asserted, documented and measured (Rung in apps/mesh/src/conway_mesh/model.py). This page uses the product words for them:

  • toldToldEvidence that somebody stated the value. It is the weakest level, and it is always attributed to its speaker.: a statement made in conversation or interview, stored as asserted.
  • shownShownEvidence that a document supports a value. A recovery plan that states a target shows the target, but it does not show that the target was achieved.: a declared artefact, such as a chart, a report or infrastructure codeInfrastructure codeFiles that describe the computing resources an organisation intends to create and how they should be configured., stored as documented.
  • measuredMeasuredThe top evidence level: a connected source recorded the observation within a stated period. A measurement shows what was observed on that occasion, not that it always holds.: instrumented data from a running system.

Some sub-commands record a rung on the commit and some do not. The Interfaces table says which is which. The C/A/RC/A/RThree separate scores for a team or service: Complexity, Agility and Recoverability. Each uses current evidence and the customer’s stated scoring rules. (Complexity, Agility and RecoverabilityRecoverabilityThe R in C/A/R: how well a team or service restores acceptable service after a failure. Recovery times and tested alternatives can supply evidence.) scores that an ops report supplies stay at the shown rung until telemetry measures them. The scorecard module states that the materialiserMaterialiserThe component that reads recorded changes in order and updates the database. It records its place so it can continue after a restart. keeps that rung one-way.

Canonical units

Every infrastructure source reduces to one provider-neutral shape, InfraUnit in ingest/canonical.py. A unit is either a group or a deployment. A group is a containerContainerA running program with its own packaged software and isolated environment, managed by Docker on the host., such as a cloud account or project. A deployment is a workload, such as a database, queue or clusterClusterA group of computers running together as one installation. Each regional NovusLens installation keeps its own records.. Adapters in ingest/sources.py translate provider-specific identifiers and structures into that shared shape. Anything provider-specific stays in namespacedNamespaceA naming area that keeps one set of records apart from another. NovusLens uses the conway/ prefix so its keys do not occupy Mycelium's reserved names. attributes. Each unit carries source_ref, pointing to its place in the source document. depends_on lists the identifiers of units it needs.

The --source choices of ingest map to five adapters:

--sourceAdapterWhat the fixtureFixtureA prepared set of records used for a test or demonstration. A fixture is sample data, not a live customer's records. holdsArtifact kind
awsAwsOrganizationsSourceAn Organizations description: organisational unit tree, accounts and resourcesaws-organizations
gcpGcpResourceSourceFolders, projects and resources with labelsgcp-resources
azureAzureResourceSourceManagement groups, subscriptions, resource groups and resourcesazure-resources
terraformTerraformStateSourceTerraform state. Each count or for_each instance is a unitterraform-state
cmdbCmdbExportSourceconfiguration management databaseCMDBConfiguration management database: an inventory of systems and the relationships recorded between them. (CMDB) or data centre infrastructure managementDCIMData centre infrastructure management: records and tools describing physical data centre equipment and its use. (DCIM) recordscmdb-export

What each input becomes

  • Infrastructure (ingest --source …). The body is stored as a content-addressedContent addressAn identifier calculated from a record's contents. It lets the program check that the retrieved bytes are the same bytes originally stored. evidence artifactArtifactA stored source file or evidence body that a record can cite. Keeping the exact content lets a reader check what supported the record.. Each deployment whose domain tag matches a domain cell becomes an infra cell, nested under the groups that contain it. Groups are created per domain they serve. Each backed domain cell’s structure climbs to the shown rung and cites the artifact. Dependencies become edges, typed by the kind of resource depended on. Queues and brokers are BufferedBufferedA handoff that takes more than one day and up to four days. It adds 1 point to the dependency-chain index., stores and databases are RigidRigidA handoff that takes more than four days. It adds 3 points to the dependency-chain index., and object or analytical stores are LooseLooseA handoff that takes about one day or less. It adds 0.5 points to the dependency-chain index.. An unrecognised type stays Unknown. These types are estimates from the code, and telemetry can correct them.
  • Org chart (orgchart). Each unit becomes a team-level domain cell at the shown rung. AuthorityAuthorityWhat a team or agent is allowed to decide or do. For an agent, this comes from a named person or system that grants permission. is recorded as well when the chart names a head, responsibilityResponsibilityThe work or result that a team or agent answers for. NovusLens records the person answerable for it. or granted authority. The chart adds no edges. The chart declares units without dependencies. A twin built only from it therefore has no critical path lengthCPLCritical path length. The total coordination weight of the longest chain of dependencies in the model. NovusLens reports it as index points, not elapsed time. (CPL), the dependency-chain index. Individuals never become cells. A name enters only as an accountable head.
  • People (people). Person-level rows are sealed. See the next section.
  • Domain taxonomyDomain taxonomyThe list of business areas, their owners and the teams or services that do the work in each area. (domain-taxonomy). Business domainsDomainA business area defined by the work it does, such as purchasing or customs. It may involve several teams, and one team may serve several areas. are added or updated at the shown rung, with their many-to-many mapping to cells. The taxonomy is the business’s own declared claim. NovusLens does not treat it as a system of record.
  • Finance (finance). Each cell is classed cost_centre or profit_centre. The class is written as a cell.finance claim that cites the stored page. The commit records no rung.
  • Regulation (regulation). The catalogue of regulations is reference data, kept in PostgresPostgresThe database software that stores the tables read by the NovusLens screens, along with document bodies, access records and protected personnel records.. The scopeScopeThe work, data, activity or access that a rule, statement or calculation covers. For access, it names the records an account can see. For team measures, “Own operations” covers the selected unit. “Including descendants” adds the units beneath it. mapping is the compliance function’s claim, with who mapped it and why. The posture check is computed. For an obligation that requires recoverability evidence, a cell in scope with no measured C/A/R rung raises a findingFindingA difference in the records that needs a decision. It includes its sources and can describe a slow handoff, disputed fact or missing owner..
  • Operations report (ops-report). Each scored cell gets its complexity, agility and recoverability on the asserted layer at the shown rung. The report’s as_of_ms is the observation date. Upload time is not used.

The sealed identity join

people reads a personnel export into a sealed identity joinSealed identity joinStored links between employee fingerprints and teams or services. The fingerprints replace readable employee identifiers. Product pages do not show the links for individual people.. This is what lets telemetry land on the right cell without any person-level data leaving Postgres.

  • The fixture holds teams (each team mapped to a cell), and people (each with employee_id, email and team). An optional complete flag and export_id control how withdrawals are handled.
  • Person-level rows go only into the Postgres table sealed_identity_join. They never reach meshMeshA set of connected Mycelium services that exchange updates and keep the NovusLens log. KVKey-value storeStorage where a named key points to a value. Reading that key retrieves the value currently stored under it. or the commit log. The module takes a database connection and no mesh agent, so the seal holds by construction.
  • Raw identities are never stored. Each person is kept as a saltedSaltExtra data mixed into a value before its fingerprint is calculated. It makes matching or guessing the original value harder. SHA-256SHA-256A method that turns content into a fixed-length fingerprint. Comparing fingerprints helps check whether content changed. hashHashA fixed-length fingerprint calculated from content. Comparing fingerprints helps detect changes and match records. of employee_id, and each handle as a hash of the e-mail address. The salt is CONWAY_IDENTITY_SALT. If it is unset, the module uses a public development salt and prints a warning. The customer profileDeployment profileA setting that selects customer or development behaviour. Customer mode refuses development credentials and uses the signed-in session as the identity. refuses to run without a salt.
  • By default an export is an increment. It adds and moves people and removes nobody. An export with "complete": true is a snapshot of the teams it lists. Any person in those teams that the file does not name has their retention ended. Teams the file does not list are untouched.
  • Each row is kept for 180 days (RETENTION_DAYS). After that, the permitted read functions cannot see it.
  • Reads go through two functions only. resolve_actor maps a handle to a cell and never returns a person. The Python helper cell_headcount defaults to a floor of 3 and returns nothing below it. The database function and key-person findings use a floor of 5. The product withholds counts below five; these defaults differ.

The command returns aggregate statistics only.

link --estate SOURCE=FIXTURE runs once, after every source has been ingested. It is repeatable, and each pair names one of the five source choices and its fixture.

  1. A per-source ingest resolves a depends_on only against cells of its own source. A dependency that crosses a domain or a provider can resolve only when all infra cells exist.
  2. link re-reads every named fixture, builds the units, and loads the infra cells already in the twin. It resolves every dependency against that whole set. An edge is written only when both ends already exist as infra cells, so a source that has not been ingested yet contributes no cross-links.
  3. Each edge cites the consumer’s document, using the newest stored artifact of that kind.
  4. A dependency that crosses a provider or a region raises a resilience-cross-boundary finding. The finding is recoverability evidence drawn from the topology, not a verdict.
  5. The declared-versus-actual drift comparison (estate_drift.analyse) then runs on the same computing resources.

Running the link pass again creates no duplicate edges: it adds or updates each edge under its existing identifier. Because link reads the fixture files you name, pass the same files that were ingested.

Interfaces

Sub-commandMain inputsWhat it writesRung
ingest --source {aws,gcp,azure,terraform,cmdb} --fixture …the adapter’s fixture, optional --solicited-byinfra cells, edges, domain structure, findings, artifactshown (structure and edges)
orgchart --fixture …units with parent, head, responsibility, authorityteam cells, authorityshown
people --fixture …teams, people, optional complete and export_idsealed_identity_join and people_exports in Postgres onlynone (sealed)
domain-taxonomy --fixture …domains with cellsdomains and cell-to-domain mappingshown
finance --fixture …cells with classcell.finance claims, starved findingsnone recorded
regulation --fixture …catalogue and scoperegulations, cell_regulations, posture findingsnone recorded
ops-report --fixture … [--solicited-by]cells with 0 to 100 scoresasserted C/A/R at the shown rung, findingsshown
link --estate SOURCE=FIXTUREall the computing resources, repeatableservice edges, resilience and drift findingsshown (edges)

Global options apply to every sub-command. --org is required. --host and --port default to NOVUSLENS_MESH_HTTP_ADDR and NOVUSLENS_MESH_HTTP_BASE, then to 127.0.0.1 and 8400.

Invariants and failure behaviour

  • Nothing is guessed onto the mapMapThe view of teams, services and their dependencies for the selected part of the organisation. Details appear beside the map.. An unknown domain, cell or team is reported as a finding (undeclared-domain, undeclared-domain-cell, unknown-scored-cell), and the row is not applied.
  • Name collisions are held out. When the same unit id appears in two accounts or projects, both are removed from the map and a duplicate-unit-id finding is raised. They are not merged.
  • Structural defects refuse the whole chart. A missing or duplicate id rejects the chart before anything is written. An undeclared parent or a loop in reporting lines also rejects it. A finance row with a class other than cost_centre or profit_centre is rejected in the same way.
  • Ages come from the document. Observation dates come from as_of_ms, and a missing date stays unknown.
  • A blank score is not a zero. An ops report that omits a dimension documents nothing about it.
  • Findings are evidence, not verdicts. A structurally-starved finding asks for a review of funding and decision rights and says that funding adequacy has not been assessed. A regulatory-posture-gap finding says that no measured evidence exists. It never states a verdict on a cell’s standing under the regulation.
  • People data stays sealed. A malformed person row or an unknown team is skipped and reported by field name or team name. A reused export_id with different content is refused. A complete export with no people is refused, because it would withdraw everyone.
Last updated