Telemetry, measured from the systems that run the organisation.

TelemetryTelemetryMeasurements and activity records sent by a running system, such as ticket timings, network traffic or workflow results. consists of measurements reported by operational systems. A value at this rungRungOne of three levels of support for a recorded value: told, shown or measured. The level appears beside the value. comes from a system the organisation already runs: a ticket queue, a metrics server, a continuous integrationContinuous integrationAutomated checks run when code changes, usually including building the code and running tests. (CI) service, an incident tool, or a network flow log. NovusLens reads that system’s own data, and the customer’s mapping fileMappingA customer-written file that connects names in a source system to teams or services in NovusLens. For scores, it also specifies the scoring rules. says which parts of it belong to which cellCellA box on the map that stands for a team, service, supplier or AI agent. It is a unit you can inspect, not necessarily a department or a person.. A cell is one unit of the organisation’s structure. Complexity, Agility and RecoverabilityRecoverabilityThe R in C/A/R: how well a team or service restores acceptable service after a failure. Recovery times and tested alternatives can supply evidence. (C/A/RC/A/RThree separate scores for a team or service: Complexity, Agility and Recoverability. Each uses current evidence and the customer’s stated scoring rules.) are the three scores each cell carries.

Two things stay with the customer. The mapping, which joins twinTwinNovusLens’s model of one organisation, built from records about its teams, services and how they rely on one another. It is only as current as those records. cells to the sourceSourceA document, interview statement, measurement or export that supports a value. Each source has a date and a stated scope.’s names, is written by the customer. The calibrationCalibrationThe customer's own rule that turns raw figures, such as a gauge, a workflow history or an incident log, into a C/A/R score. A score moves when the calibration moves, and the change is recorded., which says which series or quantity stands for each score and how it scales, is also the customer’s. The acquirersAcquirerA program that reads a customer's operational records, calculates quantities from them and records the results in NovusLens. do not invent either. Anything unmapped is reported, never guessed.

measuredMeasuredThe top evidence level: a connected source recorded the observation within a stated period. A measurement shows what was observed on that occasion, not that it always holds. is the rung this page is about. Everything from a telemetry run lands there, labelled with the source it came from.

What you will achieve

By the end of this page you will have:

• run one acquisition against a source, or a composed set of sources,
• turned the feed it wrote into mapMapThe view of teams, services and their dependencies for the selected part of the organisation. Details appear beside the map. commits with the correctionCorrectionA change to a recorded value or record. The earlier version stays in the history, and any conclusion that depended on it is checked again. command,
• seen where the result shows on the map, and
• read what a refusal said and what to change.

Docker commands

For the Compose installation, run commands from the checkout that holds .novuslens.env and compose.yml. Prepare the input described below, then import it with ./lens, which uploads the mapping into the running reasoning containerContainerA running program with its own packaged software and isolated environment, managed by Docker on the host.:

./lens connectors acquire --org harbour --kind prometheus --url https://prom.example --mapping metrics.json
./lens connectors correct --org harbour

Use your organisation id and the chosen account wherever a command asks for them. Use a source address reachable from the container. For a source on the Docker host, use host.docker.internal. Upload a flow log first with ./lens exec -- sh -c 'cat > /tmp/flow.log' < flow.log, then pass --flow-log /tmp/flow.log.

GraphGraphA model made of units and links between them. In NovusLens it records teams, services and their dependencies. materialise and requeue commands also need --host mesh-0. Follow the complete Operations manual for accounts, configuration and command access.

Before you start

You need the installation running, and a mapping file for each source you want to read. The mapping is yours to write. The examples on this page are illustrative and are not the values of any real organisation.

Run the commands from the installation directory:

./lens connectors acquire --org harbour --kind prometheus --url https://prom.example --mapping metrics.json

The --org flag is required and comes before the subcommand. --host and --port default to the values of NOVUSLENS_MESH_HTTP_ADDR (or 127.0.0.1) and NOVUSLENS_MESH_HTTP_BASE (or 8400). --refresh-ms defaults to 30000.

The flags that matter for acquire:

Run an acquisition

A run reads the source, derives scores from what it read, and writes feed entries. It does not write commits to the map. The feed entries wait for the correction command, described under The sequence below.

On success the command prints a summary of what it wrote and exits 0. A refusal names what was wrong, as described under When something is refused.

For a composed run, several sources feed one set of cells:

./lens connectors acquire --org harbour --kind composite --source prometheus=https://prom.example --source pagerduty=https://api.pagerduty.com --mapping composed.json

pagerduty is accepted by --kind on its own as well, but a standalone run refuses any cell whose calibration needs another source’s quantities. Use --kind composite when a cell needs more than one source.

The mapping file

The customer writes two sets of rules in the mapping:

• cells links names. It says which twin cell corresponds to which job, repository, service or capabilityCapabilityWhat a team can provide or an agent can do. NovusLens records the claim and the evidence that supports it. in the source.
• axes is the calibration. It names inputs for Complexity, Agility and Recoverability. It also defines how each raw value becomes a score from 0 to 100. Each axis is a base plus linear terms. Each term has a per_unit multiplier. The result is clamped to 0 to 100 and rounded.

A cell missing any calibrated input is refused whole. The tool never fills missing scores with values from another rung. If the mapping sets "partial_axes": true, the axes that can be scored are written and the rest stay unknownUnknownA score or value that cannot be calculated from current evidence. It is shown as unavailable, never as zero., shown as unavailable rather than zero. Without that flag, a mapping that does not calibrate all three axes is refused before anything is acquired.

Tickets: ServiceNow and Jira

Ticket queues become per-capability lead timesLead timeThe elapsed time between starting a request or handoff and completing it.. The mapping joins the source’s assignment groups to the twin’s capabilities.

{"groups": {"Customs Service Desk": {"capability": "declaration-filing"},
            "Warehouse Ops": {"capability": "stock-movements"}}}

Open tickets are skipped. A group that the mapping does not name is counted and reported, not attributed.

Metrics: Prometheus

cells joins each cell to the job label its exporters stamp. axes calibrates over exported series.

{"cells": {"customs": {"job": "customs-stack"}, "warehousing": {"job": "wms"}},
 "axes": {"complexity": {"base": 0, "terms": [{"series": "estate_systems_total", "per_unit": 1.0}]},
          "agility": {"base": 100, "terms": [{"series": "change_lead_time_days", "per_unit": -8.0}, {"series": "change_failure_rate_pct", "per_unit": -0.5}]},
          "recoverability": {"base": 100, "terms": [{"series": "incident_recovery_hours", "per_unit": -5.0}]}}}

Continuous integration: GitHub

cells joins each cell to the repository whose pipelines deliver it. axes calibrates over quantities the acquirer computes from the raw workflow runs: ci_pipelines_total, ci_change_lead_days, ci_change_failure_rate_pct and ci_recovery_hours.

Recovery is measured from a failed run to the next successful run of the same workflow on the same branch. A success elsewhere closes nothing. A failure with no such success is counted as unrecovered. Listings are read page by page. A repository whose listing could not be completed is refused with the counts, rather than scored over a partial page.

{"cells": {"customs": {"repo": "harbour/customs-filing"}},
 "axes": {"complexity": {"base": 0, "terms": [{"quantity": "ci_pipelines_total", "per_unit": 4.5}]},
          "agility": {"base": 100, "terms": [{"quantity": "ci_change_lead_days", "per_unit": -8.0}, {"quantity": "ci_change_failure_rate_pct", "per_unit": -0.5}]},
          "recoverability": {"base": 100, "terms": [{"quantity": "ci_recovery_hours", "per_unit": -5.0}]}}}

The cells entry uses repo where the metrics kind uses job.

Composed: several sources, one cell

Combine sources when no single source measures every score. Each cell names its identity in each source, and each axis draws from the source that actually measures it. Incidents come from PagerDuty by service, as the pd_mttr_hours quantity.

{"cells": {"warehousing": {"job": "wms", "service": "wms-oncall"}},
 "axes": {"complexity": {"base": 0, "terms": [{"series": "estate_systems_total", "per_unit": 1.0}]},
          "agility": {"base": 100, "terms": [{"series": "change_lead_time_days", "per_unit": -8.0}]},
          "recoverability": {"base": 100, "terms": [{"quantity": "pd_mttr_hours", "per_unit": -5.0}]}}}

A series term is answered by Prometheus. A quantity term belongs to the source that derives it. Each score records its source.

Network flows

network-flow reads a flow-log export, not an APIAPIApplication programming interface: a defined way for one program to request data or actions from another.. It accepts Cilium Hubble JSONJSONJavaScript Object Notation: a text format for data, using named fields, lists and values. Configuration files and service messages use it. (hubble observe -o json, --format hubble) or AWSAWSAmazon Web Services: a cloud platform whose account and resource exports NovusLens can read. VPCVPCVirtual private cloud: a private network defined inside a cloud provider's infrastructure. Flow Logs in the version 2 default format (--format vpc-flow-v2), passed with --flow-log and repeatable.

The mapping joins each cell to the workloads that send for it, and declares the cell’s egressEgressTraffic leaving a system or network for another destination.. A Hubble selector names a namespaceNamespaceA naming area that keeps one set of records apart from another. NovusLens uses the conway/ prefix so its keys do not occupy Mycelium's reserved names. and labels. A VPC selector names source address ranges. A workload that matches no cell, or more than one, is reported and not attributed. The declared egress lists destinations as host:port or an address, with a trailing * for a prefix, and says whether the list is complete. destinations gives your own names to addresses.

{"cells": {"eval-agents": {
   "hubble": {"namespace": "eval-fleet", "labels": ["k8s:app=eval-agent"]},
   "vpc": {"addresses": ["10.42.1.0/24"]},
   "declared_egress": {"allowed": ["gateway.node-gateway.svc.cluster.local:8443"],
                       "excluded": [], "complete": true,
                       "source": "confined-fleet profile"}}},
 "destinations": {"10.42.0.10": "gateway.node-gateway.svc.cluster.local"}}

The source of declared_egress names where the declaration came from, and the acquirer requires it. The acquirer checks each classified flow against the declaration. The result is the network’s own account of each cell’s egress and the findingsFindingA difference in the records that needs a decision. It includes its sources and can describe a slow handoff, disputed fact or missing owner. it raised, described under Network egress.

The sequence

Four steps take a reading from the source to the map. Each is a separate step, and each can be inspected.

1. Acquire. acquire reads the source and writes feed entries. For tickets it writes one entry per capability. For the score kinds it writes the derived cell scores, one entry per cell, and the per-axis observations keyed by cell, axis and source. Each entry carries synthetic, source, acquired_from, refresh_interval_ms and written_at_ms. Nothing reaches the map yet.
2. Correct. Run ./lens connectors correct --org <org>. The correction pass reads the feed, takes the distributed lockDistributed lockA lock that coordinates programs running in separate processes or on separate machines, so only one holder is recognised at a time. conway-correct-<org>, and receives a fencing tokenFencing tokenA number that identifies a turn to update shared data. NovusLens rejects an earlier correction’s writes after a newer correction is applied.. It then commits observed-layer edges and cell scores. Each commit carries an observation windowObservation windowThe period in which a measurement counts as current. The default is 30 days. After it passes, the measurement remains as history and is counted as expired.: valid_until is the observation time plus CONWAY_OBSERVATION_TTL_DAYS days, which defaults to 30. Findings open where a measured value differs from what was asserted, or where a value is measured for the first time.
3. Materialise. The projectionProjectionThe regular pass that brings the read model up to date with the canonical log. It runs every 30 seconds by default. service reads committed history into the map. The conway_reasoning.project loop runs every NOVUSLENS_PROJECTION_INTERVAL seconds, which defaults to 30. Set it in .novuslens.env. Each pass retries quarantinedQuarantineA holding area for a log entry that could not be applied. The entry keeps its error and waits for a retry or an operator. commits, then materialises new ones.
4. Show. The map shows the result after the next projection pass. A commit whose fencing token is older than the one already applied for its axis is refused as stale and quarantined. A stale holder’s write never reaches the map.

Run the correction command after each source reading. See keeping the twin current for when changes appear. An acquisition alone does not change the map.

What a re-run does

When something is refused

A refusal stops the run and names the cause. Nothing is written for the refused part. The table below lists the refusals the code raises.

The ticket run has a particular behaviour. A partial listing is refused, and the run writes zero entries. The message reads partial collection: ... — not written.

Network egress

A flow observation is compared with its cell’s declaration. The classification is:

• an explicitly excluded destination is outside the declaration,
• a listed destination is inside it,
• an unlisted destination is outside the declaration only when the list is marked complete,
• anything else is not established.

Two findings can open. undeclared-egress opens when a forwarded flow leaves the declared set. egress-blocked opens when a flow was dropped. A flow that is not established goes to review and never opens a finding. Findings are not closed automatically.

Where this lives