Connectors, one shape for every source.

A connectorConnectorSoftware that reads records from a source system, such as an incident tool. A customer-written file connects those records to teams or services in NovusLens. reads systems the customer already runs: ticket queues, metrics servers, continuous integrationContinuous integrationAutomated checks run when code changes, usually including building the code and running tests. (CI) services or incident tools. It maps their measurements to named capabilitiesCapabilityWhat a team can provide or an agent can do. NovusLens records the claim and the evidence that supports it. or cellsCellA box on the map that stands for a team, service, supplier or AI agent. It is a unit you can inspect, not necessarily a department or a person.. This page guides integrators through supported sourceSourceA document, interview statement, measurement or export that supports a value. Each source has a date and a stated scope. readers, their mappingsMappingA customer-written file that connects names in a source system to teams or services in NovusLens. For scores, it also specifies the scoring rules. and their limits. It also explains how to rehearse a connection with simulated data.

Open the pipeline diagramSource, customer-authored mapping, shipped acquirer, correction and materialisation, with the labels each stage writes.

Docker commands

For the Compose installation, run commands from the checkout that holds .novuslens.env and compose.yml. Prepare the input described below, then import it with ./lens, which uploads the mapping into the running reasoning containerContainerA running program with its own packaged software and isolated environment, managed by Docker on the host.:

./lens connectors acquire --org <org> --kind prometheus --url https://prom.example --mapping mapping.json
./lens connectors correct --org <org>
./lens graph materialise --host mesh-0 --org <org> --host mesh-0

Use your organisation id and the chosen account wherever a command asks for them. Use a source address reachable from the container. For a source on the Docker host, use host.docker.internal. Upload a flow log first with ./lens exec -- sh -c 'cat > /tmp/flow.log' < flow.log, then pass --flow-log /tmp/flow.log.

GraphGraphA model made of units and links between them. In NovusLens it records teams, services and their dependencies. materialise and requeue commands also need --host mesh-0. Follow the complete Operations manual for accounts, configuration and command access.

What it is for

A twinTwinNovusLens’s model of one organisation, built from records about its teams, services and how they rely on one another. It is only as current as those records. is only as current as the evidence behind it. People and documents describe owners and dependenciesDependencyA link where one unit needs something from another. A dependency is not a reporting line: it shows where work relies on something else.. Operational systems measure work times, failed changes and recovery times. Connectors bring in those measurements with their sources and dates. A reader can check what was measured and how recent it is.

NovusLens maps and advises. A connector reads a system and writes an observation. It does not change the source system, and nothing a connector writes is a verdict on the customer’s systems.

The one shape

Every connection has the same four parts. Only the source and the mapping differ.

The acquire command is the same for every kind:

./lens connectors acquire --org <org> --kind <kind> --url <base-url> --mapping <mapping.json> [--token <token>]

Then run correctionCorrectionA change to a recorded value or record. The earlier version stays in the history, and any conclusion that depended on it is checked again. and materialisation:

./lens connectors correct --org <org>
./lens graph materialise --org <org>

The --org flag is required on the connectors CLICLICommand-line interface: commands you type into a terminal to operate software. and applies to every subcommand. --host defaults to the value of NOVUSLENS_MESH_HTTP_ADDR (falling back to 127.0.0.1) and --port to NOVUSLENS_MESH_HTTP_BASE (falling back to 8400). --refresh-ms defaults to 30000.

--kind accepts servicenow, jira, prometheus, github, pagerduty, composite and network-flow. The composite kind takes repeated --source KIND=URL arguments instead of a single --url. network-flow reads a flow log through --format (hubble or vpc-flow-v2) and --flow-log. The boundary subcommand is a separate read-only client of an estate gatewayGatewayA service that accepts requests and passes them into another system. The Mycelium gateway gives NovusLens access to the mesh., described at the end of this page.

The shipped acquirers

Each acquirer has its own mapping. The mapping is the customer’s statement of which source object means which twin object.

Tickets: ServiceNow and Jira

The ticket acquirer reads resolved incidents and derives a lead timeLead timeThe elapsed time between starting a request or handoff and completing it. per mapped group: the mean of resolved minus opened. The ServiceNow path is /api/now/table/incident?sysparm_display_value=true. The Jira path is /rest/api/2/search?jql=resolution%20is%20not%20EMPTY. Both page in batches of 100.

The mapping names each group the tickets carry and the capability it stands for:

{
  "groups": {
    "Forecasting Service Desk": {"capability": "demand-curves"},
    "SCADA Data Ops": {"capability": "scada-feeds"}
  }
}

Two rules matter when reading the output. Open tickets are skipped, because they have no resolution time to measure. Groups that appear in the tickets but not in the mapping are reported as unmapped, not guessed at. The mapping may also set window_days to limit how far back the acquirer reads. A partial collection is refused, and nothing is written for that pass.

The lead time is written as an edge observation under conway/telemetry/<org>/edge/<capability>.

Metrics: Prometheus

The metrics acquirer queries the Prometheus HTTP API. It lists the jobs present with /api/v1/label/job/values, then reads each series with /api/v1/query?query=<series>{job="<job>"}.

The mapping has two parts. cells maps a Prometheus job to a twin cell. axes maps each of the three C/A/RC/A/RThree separate scores for a team or service: Complexity, Agility and Recoverability. Each uses current evidence and the customer’s stated scoring rules. (Complexity, Agility and RecoverabilityRecoverabilityThe R in C/A/R: how well a team or service restores acceptable service after a failure. Recovery times and tested alternatives can supply evidence.) axes to a base value and per-unit terms, with optional clamp and round settings. The three axes are written together, as a triple. If one is missing for a cell, the cell is not scored, and the output says which axis is absent. A series that returns more than one sample is ambiguous and is refused rather than summed.

Cell observations are written as a score-observation artefactArtifactA stored source file or evidence body that a record can cite. Keeping the exact content lets a reader check what supported the record. plus an entry under conway/telemetry/<org>/cell/<cell_id>.

Delivery: GitHub Actions

The GitHub acquirer reads workflow runs through /repos/{repo}/actions/runs and lists repositories through /orgs/{owner}/repos. Each mapped repository gives four quantities:

Runs still in progress are skipped. A quantity that cannot be measured is left out of the output, not defaulted. For example, a success with no head commit timestamp adds no lead time. The mapping key is repo.

Incidents: PagerDuty

The PagerDuty acquirer reads /services and then /incidents?service_ids[]=<id>. It produces one quantity, pd_mttr_hours, the mean time to resolve across resolved incidents. Unresolved incidents are skipped. A service with no resolved incident has no pd_mttr_hours, which is different from a service that had quiet incidents. On its own, the PagerDuty acquirer usually cannot score all three measures. Its one recovery quantity does not measure Complexity or Agility. It is most useful inside a composite.

Composite

The composite acquirer runs the Prometheus, GitHub and PagerDuty acquirers together and joins their outputs for each cell. Its source label is the kinds and hosts it read, for example prometheus:<host>+github:<host>, joined with +. The composite is labelled simulated if any of its contributors is simulated.

The shipped composed mapping for the demonstration organisation is apps/connectors/fixtures/meridian_gridops_composed_mapping.json. Meridian GridMeridian GridThe fictional demonstration organisation used by the demonstration walk and its sample data. Nothing about Meridian Grid describes a real company. is a fictional demonstration organisation. Its mapping is an example of the shape a customer authors, not a claim about a real estate.

Network flow

The network-flow acquirer reads a flow log in Hubble or VPCVPCVirtual private cloud: a private network defined inside a cloud provider's infrastructure. flow format. It produces two findingsFindingA difference in the records that needs a decision. It includes its sources and can describe a slow handoff, disputed fact or missing owner., undeclared-egress (traffic to a destination the declaration does not cover) and egress-blocked (traffic the control refused). The coverageCoverageWhat the available records cover, and what is missing. A count such as “3 of 5 dependencies measured” names the measured portion and the gap. of the flow log travels with each finding, so a finding is never stronger than the logs that produced it. Under an incomplete declaration, destinations are reported as not established, not as absent.

Boundary feed

The boundary subcommand is a read-only client of a customer estate’s gateway (--gateway HOST:PORT). It reads each agentAgentA program that takes actions for an organisation. NovusLens records what it is allowed to do and what its activity reports show.’s signed envelopeEnvelopeThe surrounding fields that identify a message, its sender, its type and its time, alongside the message's data. of facts and observed behaviour and writes it under conway/estate/boundary/<agent_id>. A legacy envelope without a signatureDigital signatureA check calculated with a signing key that can detect whether a record has changed. Identifying the signer also requires a trusted link between that key and its owner. is labelled as coming from an unverified publisher. It is evidence the gateway served, not evidence NovusLens has verified the agent.

Honesty rules

These rules are enforced by the code, not left to the author of a mapping.

• Simulated flags propagate. A simulated source marks its responses with x_novuslens_simulated: true in the body and sends an X-NovusLens-Simulated: true header. The acquirer copies the flag into every entry’s synthetic field and adds the simulated-source suffix. Simulated data therefore keeps its simulation label.
• The source is written on every entry. Entries record the kind and host they came from. Simulated sources add the suffix (simulated source).
• The refresh time is recorded. Each entry carries written_at_ms and refresh_interval_ms. A reader treats an entry older than three times its refresh interval as stale, and an entry with no refresh-time information as stale too. Stale and absent entries are omitted from the twin’s current reading.
• Partial collection is refused. If an acquirer cannot read everything it was asked for, it writes nothing for that pass.
• Unmeasured is not zero. A quantity that cannot be measured is absent. The calibrationCalibrationThe customer's own rule that turns raw figures, such as a gauge, a workflow history or an incident log, into a C/A/R score. A score moves when the calibration moves, and the change is recorded. then refuses to score the cell.
• Signals are not verdicts. An observation shows what a system reported over a window. The correction step records where observed and asserted structure differ, and a human decides what changes.

Correction and freshness

Observations are written with an observation windowObservation windowThe period in which a measurement counts as current. The default is 30 days. After it passes, the measurement remains as history and is counted as expired.: observed_at_ms is the source’s own write time when it declared one, and valid_until_ms is that plus the observation TTLTTLTime to live: the period after which a temporary record or lock is no longer treated as current.. The time-to-live (TTL), meaning the period an observation counts as current, is set by CONWAY_OBSERVATION_TTL_DAYS and defaults to 30 days.

correct runs one correction pass. It takes a short distributed lockDistributed lockA lock that coordinates programs running in separate processes or on separate machines, so only one holder is recognised at a time. (conway-correct-<org>, time-to-live 30 seconds) only to obtain a fencing tokenFencing tokenA number that identifies a turn to update shared data. NovusLens rejects an earlier correction’s writes after a newer correction is applied.. Every edge event it writes carries that token. The materialiserMaterialiserThe component that reads recorded changes in order and updates the database. It records its place so it can continue after a restart. checks the token before applying a write. It refuses and quarantines writes with an outdated token. The correction records two kinds of finding. asserted-observed-delta marks a place where measured and asserted structure disagree. first-measurement marks the first time a capability has an observation at all.

The lock does not establish whether the measurement is correct. The protection is the fencing token, and the disagreement is surfaced for people to resolve.

Calibration

The calibration decides which quantities make a cell complete and how they combine into the three C/A/R axes. A mapping names the cells and the terms. The metrics acquirer also accepts a partial_axes setting, which allows a cell to be written with fewer than three axes. The telemetry format explains how partial scores are stored. Check the mapping carefully before using this option.

Connect a real source

Moving from a simulated source to a real one changes two things: the URL and the token. Write the same kind of mapping file using the customer’s real identifiers. Check it carefully.

1. Confirm the source is reachable from the host that runs the acquirer, and that the token has read access to the objects the mapping names.
2. Write the mapping from the customer’s own group, repository, job or service names. Do not copy a fixtureFixtureA prepared set of records used for a test or demonstration. A fixture is sample data, not a live customer's records.’s names.
3. Run acquire once against the real URL and read the output. It reports the number of resolved tickets per capability, the unmapped groups and any refused pass.
4. Run correct and materialise, then check the result in the application against the source.
5. Schedule the acquire command on the refresh interval the source can support. Set --refresh-ms to match, because readers treat an entry as stale after three times that interval.

Nothing in the acquirer changes for a real source except the URL, the token and the mapping. An entry from a real source carries no simulated flag, so it is labelled with the real source’s kind and host.

To write an acquirer for a system that is not listed here, see adding a connector.

Simulated sources

The repository includes a simulated source for each system an acquirer reads. They exist so the acquirers can be built, tested and rehearsed before a pilot hands over credentials. The simulated sources are served by:

./lens exec -- python -m conway_connectors.sim_sources --fixture <fixture.json> [--port 8600]

Each simulated instance serves one fixture and speaks the API shape of one system: ServiceNow, Jira, Prometheus, GitHub or PagerDuty. The default port is 8600. Every simulated response carries the marker described under honesty rules above, so the output is labelled simulated throughout the twin. See Simulated data for how to tell simulated from real and what the labels mean.

When something goes wrong