From a log file to an admitted batch, one decision at a time.

The reference exporter is a starting implementation. It shows the decisions an exporter has to make between a sourceSourceA document, interview statement, measurement or export that supports a value. Each source has a date and a stated scope.’s log and an agentAgentA program that takes actions for an organisation. NovusLens records what it is allowed to do and what its activity reports show.-evidence batch, against a deliberately messy fixtureFixtureA prepared set of records used for a test or demonstration. A fixture is sample data, not a live customer's records.. It is not a certified vendor connectorConnectorSoftware that reads records from a source system, such as an incident tool. A customer-written file connects those records to teams or services in NovusLens., and it does not read a real gatewayGatewayA service that accepts requests and passes them into another system. The Mycelium gateway gives NovusLens access to the mesh.. A customer or integrator who builds an exporter for their own log keeps the decisions and writes their own parser.

What you will achieve

Run the exporter on the fixture and inspect refusals before output. The ten decisions below explain why each line was accepted or refused. You will have an unsigned batch on disk. You will know what the source signs and which exporter rules need adapting for your system.

Integration environment

The source team runs export and signing commands in a source-owned Docker development runner. Its private keyPrivate keyA secret value used to create digital signatures. The source keeps it so others cannot sign evidence in its name. and producer checkpoint must stay in durable storage controlled by that team. Use ./lens ops --version dev exec -- for the source checkout. Customer-side checks use the configured Compose reasoning service.

For a Docker demonstration, follow the Operations manual to enrol and activate the source. The Docker demo keeps the reasoning port private; it does not publish a source intake endpointEndpointA particular address on a service that accepts a defined kind of request.. Before sending from another machine, agree an authenticated HTTPSHTTPSHTTP carried over an encrypted connection, so the contents are protected while travelling between the caller and the server. intake endpoint with the deployment operator. The producer accepts plain HTTPHTTPHypertext Transfer Protocol: the rules a browser or program uses to send a request to a web service and receive its response. only for loopbackLoopbackA network address that refers to the same host, such as 127.0.0.1. Other hosts cannot reach a service through that address. addresses.

Before you start

• A source-owned checkout, Docker and its development runner, and Python 3.9 or newer to start the runner through ./lens ops. No other host Python packages are required. Keep the source key and producer state on that source-owned system.
• The fixture apps/connectors/fixtures/reference_gateway_access.jsonl. Its twelve lines include malformed text, duplicates, missing line numbers, varied HTTP responses and a negative amount.
• An organisation identifier for your twinTwinNovusLens’s model of one organisation, built from records about its teams, services and how they rely on one another. It is only as current as those records., and an enrolment for the source if you intend to run the checker. Enrolment is covered in Agent evidence contract v1.

Steps

1. Run the exporter

./lens ops --version dev exec -- uv run python -m conway_connectors.reference_exporter \
  --log apps/connectors/fixtures/reference_gateway_access.jsonl \
  --org ORG --source gateway --subject invoice-agent --stream gateway/access \
  --out batch.json

The exporter prints counts and warns that its output is unsigned.

2. Read the refusals first

The command prints a summary like this one:

6 record(s) from 12 line(s)   cursor None → 14
  refused, and why: malformed=1, duplicate=1, unmapped_path=2, unmapped_status=1,
                    other_principal=1, value_omitted=2, line_gaps=['4..4', '7..8']
  UNSIGNED — the source signs this; run `agent-source check` first

Twelve lines in and six records out, with every other line accounted for. An exporter that drops lines silently presents partial capture as complete. This one reports each refusal, and the batch carries a coverage block so the receiver can see what it did not get.

3. Follow the decisions

1. An operation name is declared, never inferred. The exporter maps POST /invoices/approve to the operation invoice.approve on the resource account, and nothing else. The fixture’s GET /invoices/1002 and POST /payments/release are not in that table. They are counted as unmapped_path and dropped, because nothing is guessed from a URL.

2. A status becomes a state only where the meaning is certain. A 2xx response becomes completed, a 403 becomes denied and a 5xx becomes failed. The fixture’s 302 has none of those meanings, so it is counted as unmapped_status rather than rounded to a state.

3. A value carries its unit, or it does not travel. The exporter declares GBP for invoice.approve. A USD amount still produces a completed approval record. Its value is omitted because the declared unit is GBP. A negative amount is treated the same way. The value_omitted count records both.

4. Export only the declared caller’s records. An access log records who called. It does not record who authorised. The exporter accepts only the principalPrincipalThe account or service identity that made a request. It identifies the caller, which may differ from the person who authorised the action. in its SUBJECT_PRINCIPALS set, here svc-invoice-bot. The fixture’s svc-payroll-bot line is counted as other_principal and dropped. No line becomes an intervention_authorisation, because that would manufacture an approval nobody gave.

5. Record identifiers stay the same and include the stream. A record id is the stream and the request id, for example gateway/access:r-1002. Running the exporter again gives the same ids, and a second stream cannot collide with the first. The fixture repeats r-1002 on line 5, so it is emitted once and counted as duplicate.

6. A malformed line is skipped and counted, never repaired. Line 4 is not JSONJSONJavaScript Object Notation: a text format for data, using named fields, lists and values. Configuration files and service messages use it.. It is counted as malformed. A real log will contain lines like it.

7. The cursorCursorA saved position in a sequence of records. A program uses it to continue from where it stopped. says where you were and how far you got. The cursor has the form {stream, previous, next}. previous is the line the receiver last acknowledged and is null on a first run. next is the highest line read. A resumed run passes --previous, and lines up to that point are not exported again. The receiver refuses a batch that does not extend the stream’s head, so a resumed run must resume rather than restate.

8. The batch id is the cursor range. The batch id has the form stream:<previous|start>..<next>, for example gateway/access:start..14. Sending the same batch twice gives the receiver the same batch id, and the receiver reports a duplicate rather than applying it twice.

9. The exporter does not sign. It produces the unsigned document. The source signs it with its own key. If NovusLens held that key, it could forge the source’s evidence, and the source’s records would be worth nothing as evidence against it.

10. An event in the future is refused. The intake allows a five-second difference between clocks and refuses a record whose time is further ahead, with the message Event timestamp exceeds clock allowance. The fixture’s timestamps are instants in UTCUTCCoordinated Universal Time: a shared time reference that does not change with daylight-saving time.. A machine running an hour behind, using local time, would produce refusals that look like bad data but are clock errors. Export timestamps as instants, and check the clock of the machine that runs the exporter.

4. Check the batch against the enrolment

Before signing, upload the unsigned batch into the configured customer containerContainerA running program with its own packaged software and isolated environment, managed by Docker on the host.. ./lens uploads the enrolment itself in the next step. Keep the private key on the source-owned system.

./lens exec -- sh -c 'cat > /tmp/batch.json' < batch.json

Check the batch format and enrolment. The checker reports the problems it finds together:

./lens connectors agent-source check --org ORG \
  --batch /tmp/batch.json --enrolment gateway.json

The enrolment declares which record types, subjects and parties the source may name. The checker compares those fields with the enrolment. It does not authorise individual operation names.

The checker validates the shape of the batch. The intake validates the moment it arrives, so a batch can pass the checker and still be refused on the clock.

5. Sign and send

Signing is the source’s job. The reference code includes a helper, sign_batch, for a pilot where the same team runs both sides:

./lens ops --version dev exec -- uv run python -c "import json; from conway_connectors.agent_source import sign_batch; \
  json.dump(sign_batch(json.load(open('batch.json')), open('gateway.key').read().strip()), \
  open('signed.json','w'))"

On the source-owned system, replace OPERATOR_LOGIN and REASONING_ENDPOINT with the approved setup account and HTTPS reasoning endpoint. Keep producer.state in the bound checkout. The producer saves progress and checks the receiptReceiptA recorded acknowledgement of a request or action, with its identity, time and result. An evidence intake receipt confirms storage. It does not establish acceptance or prove that an agent acted. before retrying:

./lens ops --version dev exec -- uv run python -m conway_connectors.agent_producer --batch signed.json \
  --state producer.state --viewer OPERATOR_LOGIN --url https://REASONING_ENDPOINT

The producer reads the service tokenService tokenA secret value sent with requests between NovusLens services. The receiving service refuses requests with a missing or wrong value. from CONWAY_SERVICE_TOKEN. The customer profileDeployment profileA setting that selects customer or development behaviour. Customer mode refuses development credentials and uses the signed-in session as the identity. refuses every request without it, and --viewer must name a real signed-in account, not a convenience override. The custodian supplies the service token through the approved secret process. The installer does not create an external HTTPS intake route automatically. If none is configured, remote delivery is blocked until the operator provides one. HTTP to reasoning:8500 is refused, and the reasoning container has no persistent producer-state mount.

The producer keeps its pending envelopeEnvelopeThe surrounding fields that identify a message, its sender, its type and its time, alongside the message's data. in producer.state until the receipt confirms it. Receipts and projectionProjectionThe regular pass that brings the read model up to date with the canonical log. It runs every 30 seconds by default. states are described in Agent evidence contract v1.

What you will see

After step 2, the summary and the coverage block tell you what the exporter did not get. After step 4, the checker lists each reason a batch would be refused, or reports that it would be admitted. After step 5, the producer reports the receipt state for the batch, which moves from stored to projected when the twin has read it.

What this example deliberately does not do

• It does not read a real gateway. Your system’s log format needs your own parser, and the decisions above still apply to it.
• It emits only operation records. Grants, interventions and outcomes come from systems that hold those facts, not from a log that records traffic.
• It does not infer. Every field it cannot map is counted and dropped, and the count is the most useful thing it produces.

When something goes wrong