Skip to Content
OperatingTroubleshooting
Operating · Troubleshooting

Troubleshooting, from the words the platform actually prints.

Run every command on this page from the installation directory.

What you will achieve

Find the message you saw in the tables below. Each row gives its cause and the next action. The tables group messages by the action you were attempting.

What it is for

A refusal in the customer profileDeployment profileA setting that selects customer or development behaviour. Customer mode refuses development credentials and uses the signed-in session as the identity. is deliberate. The platform refuses unsafe settings, undeclared network access and identities it cannot verify. Read the full message and find its row below. It names the setting to change.

Installation diagnostics

Run ./lens doctor from the installation directory. It checks the host first, then the running installation. Each failed check names what it found and the command that repairs it. Deployment runs the same installation checks before showing sign-in details.

CheckWhat must be trueRepair
profileThe services use the customer profile and strong secrets.Correct the named secret in .novuslens.env, then run ./lens start.
releaseAll three sidecars, the SDKSDKSoftware development kit: a package of code that helps another program use a service. NovusLens uses mycelium-py to talk to Mycelium. wheelWheelA packaged Python library ready to install. A vendored wheel is stored in this repository so its exact bytes can be checked. and its installed version match the release manifestRelease manifestThe file recording the exact software versions and fingerprints used for a NovusLens release candidate. Installation checks compare files against it..Pull the tested release again with ./lens start, or rebuild from the pinnedPinA recorded choice of one exact dependency version. It prevents an update from changing that dependency without a deliberate decision. source.
guideThe release includes a GuideGuideThe help pages inside NovusLens. They explain the pages, input formats and tasks in the version you use. They are product instructions, not evidence about the organisation. with pages.Run ./lens start to restore the release image.
web-bindA network web port has a declared HTTPSHTTPSHTTP carried over an encrypted connection, so the contents are protected while travelling between the caller and the server. proxy.Set NOVUSLENS_TLS_PROXY=1 behind an HTTPS proxy, or restore NOVUSLENS_UI_BIND=127.0.0.1. Then run ./lens start.
gatewayInternal services have no published host ports.Remove their port entries from the Compose override, then run ./lens start.
modelThe configured model endpointEndpointA particular address on a service that accepts a defined kind of request. answers; an Ollama server holds the configured models.Start the server or pull the models named in the findingFindingA difference in the records that needs a decision. It includes its sources and can describe a slow handoff, disputed fact or missing owner..
sdk-sourceThe SDK comes from the pinned wheel.Remove CONWAY_SDK_FROM_CHECKOUT from the Compose override, then run ./lens start.
schemaThe database has every migrationMigrationAn ordered change to a database's tables or rules. Applied migration files are recorded so the same file is not run again. shipped with the release.Run ./lens start to apply the migrations.

If a required service is stopped, run ./lens start before repeating ./lens doctor. An installation without a model server can still import data and browse the twinTwinNovusLens’s model of one organisation, built from records about its teams, services and how they rely on one another. It is only as current as those records..

Read the analysis status

./lens status reports setup progress and containersContainerA running program with its own packaged software and isolated environment, managed by Docker on the host.. To inspect analysis stages, query the reasoning service inside its container:

./lens exec -- python - <<'PYTHON' import http.client connection = http.client.HTTPConnection("127.0.0.1", 8500, timeout=10) connection.request("GET", "/ready") response = connection.getresponse() print(response.status) print(response.read().decode()) PYTHON

The response lists organisation errors and failed stages. ready means the current checks passed. degraded means a recent analysis stage failed. failed means records cannot advance. stale means the recorded projectionProjectionThe regular pass that brings the read model up to date with the canonical log. It runs every 30 seconds by default. pass is too old.

If a stage failed, collect its name and error below. Send them to your demo provider before presenting the affected analysis.

If the reasoning container is unavailable, collect the container list and logs below instead.

Reset the initial operator passphrase

If sign-in rejects the recorded credentials, check the login name first. For this demo, it is demo-owner unless you chose another name.

If you need to reset it, use the command below. This example preserves the initial operator’s global role and organisation scope. Use it only for that initial operator.

Replace demo-owner with your initial operator login. Run the command from the installation directory. Enter a new passphrase at the hidden prompt:

./lens exec -- python -c ' import getpass, subprocess, sys phrase = getpass.getpass("New operator passphrase (12+ characters): ") if len(phrase) < 12: raise SystemExit("Use at least 12 characters.") subprocess.run([ sys.executable, "-m", "conway_graph", "viewer", "--viewer", "demo-owner", "--role", "operator", "--org", "*", "--passphrase", phrase, ], check=True) '

Save the new passphrase in your password manager. Sign in again with the same login name. If sign-in reports too many attempts, wait ten minutes before retrying.

For another account, follow Accounts and stewardship with its existing role and organisation scope.

Collect a diagnostic packet

Create a private directory for the command output:

umask 077 mkdir -p demo-diagnostics ./lens version > demo-diagnostics/revision.txt ./lens doctor > demo-diagnostics/compose-version.txt ./lens status > demo-diagnostics/setup-status.txt 2>&1 ./lens compose -- ps -a > demo-diagnostics/containers.txt ./lens compose -- logs --no-color --tail 100 > demo-diagnostics/services.log 2>&1

If a command fails, retain its output and continue with the next diagnostic command.

Record the failed setup step, exact error message, image release and whether the failure followed a restart or configuration change. For a missing journalInstallation journalA saved record of setup steps, their attempts and whether they started, completed or failed. It records progress, not a backup of application data., record which file or volume is missing. Keep the original journals and configuration untouched.

Review the packet for credentials and organisation data. Remove those values before sharing it. Keep .novuslens.env, passphrases and registry tokens private.

Send the redacted packet to the person who provided the demo. Alternatively, report the fault through the repository issues  if you have access. Ask that maintainer to assess the preserved state.

Choose the next action

FailureNext action
Docker is unavailableStart the engine. Run ./lens doctor before retrying.
A private image pull is refusedCheck that your GitHub token can read the release packages. Run resume and enter the registry credentials in the wizard.
A download was interruptedRestore network access. Run ./lens resume.
The web port is occupiedRun configure. Follow the port change steps. The wizard retains credentials and preloads saved choices.
Setup journal is malformedPreserve the original file and stores. Collect the packet. Ask the demo provider to assess recovery.
Initialisation journal is missing while records remainPreserve the records and configuration. Ask the demo provider to assess the missing metadata.
Canonical meshMeshA set of connected Mycelium services that exchange updates and keep the NovusLens log. history is missing while PostgresPostgresThe database software that stores the tables read by the NovusLens screens, along with document bodies, access records and protected personnel records. remainsStop recovery attempts. Ask the installation operator to restore the matching stores and metadata from a verified backup. Use the original secrets.

The installer has no automatic journal repair command. To recover from a backup, the installation operator runs ./lens restore. Otherwise ask your installation operator or demo provider for approved recovery instructions for the preserved installation. Without those instructions, retain the original state and wait for maintainer assessment.

Configuration refusals

Each ./lens command checks .novuslens.env before it runs Compose, and names the line it refuses. Configuration lists each refusal with its fix. The secret rules are in Secrets. The profile rules behind them are in The customer profile.

Deployment failures

./lens start names the step that failed and the command’s own output. Correct the cause, then run ./lens resume.

MessageCauseAction
Application readiness could not be confirmed. Run status or logs, then resume.A service did not pass its health check within 300 seconds.Run ./lens status to find the service, then ./lens compose -- logs --tail 200 <service> for its error.
CI registry pulls need GHCR_TOKEN or GITHUB_TOKEN with read:packages permission.An unattended run has no registry token.Set GHCR_TOKEN to a token that can read the NovusLens packages.
volume <name> belongs to the installation in <directory>Another installation on this host uses the same COMPOSE_PROJECT_NAME, so Compose would reuse its database and mesh volumes. Deployment, backup and restore stop before they start a service or touch data.Set a new COMPOSE_PROJECT_NAME in .novuslens.env, then run ./lens resume. If the other installation is no longer needed, run the ./lens delete --directory command the message names. If this installation moved from that directory, move it back.
GitHub login and token are required for private images.The login or token answer was empty.Run ./lens resume and enter both.
The model server shows unavailable in ./lens monitorA configured model server does not answer.Start the model server, or correct CONWAY_LLM_ENDPOINT with ./lens setup --interview.
The model server shows missing in ./lens monitorA configured model is not loaded on the server.Load the model on the server. For a local model server, run ./lens resume; it downloads both models.

Sign-in

Sign-in is the only identity the customer profile accepts. These are the messages on the sign-in page and at the proxy.

MessageCauseAction
sign-in refused — the viewer/passphrase pair does not check outthe account name or passphrase is wrong, or one of them is empty. The platform does not say whichcheck the account name and passphrase with the customer. For the initial operator, use Reset the initial operator passphrase. For other accounts, use Accounts and stewardship
too many sign-in attempts — wait ten minutes, then try againfailed attempts for the account, or from the same address, have reached the limit within the last ten minuteswait ten minutes, then try again
CONWAY_SESSION_SECRET is unset, the public dev value, or too short — refused in the customer profilethe web application will not sign sessions with the value it hasrestore the line in .novuslens.env from your protected copy, then run ./lens start. Secrets explains the rules
no signed-in viewer — sign in first; a viewer named in the request is not trusted (401)a request reached the reasoning proxy without a sessionsign in. A request that names a viewer in its body is not trusted, and the proxy replaces the viewer with the session’s
service boundary: missing or wrong service token — requests reach this service only through the NovusLens UI (401)a request reached the reasoning service without the service tokenService tokenA secret value sent with requests between NovusLens services. The receiving service refuses requests with a missing or wrong value.the request should come through the web application. Both read CONWAY_SERVICE_TOKEN from the same line in .novuslens.env. If you edited it, run ./lens start to restart both

Readiness and health

./lens monitor shows each service’s health check and the age of the read modelRead modelThe database tables used by the map, Ask and Find. NovusLens rebuilds these tables from its history of recorded changes. for each organisation. Starting, stopping and health explains /health, /ready and their states.

If readiness is degraded or failed, read the errors with Read the analysis status. Then check that the database, mesh and model server answer. Retry quarantinedQuarantineA holding area for a log entry that could not be applied. The entry keeps its error and waits for a retry or an operator. items for one organisation:

./lens graph retry --org <org>

To retry one item, name its log position:

./lens graph requeue --org <org> --hlc <position> --host mesh-0

Both actions are audited.

Backup and restore

Backup and restore lists the backup and restore messages, with the action for each. Restore replaces the database and all four state volumes. Add --yes to confirm. stops a restore before it starts. It changes nothing.

Refused record changes

These come from the reasoning service and carry the role rule in their text.

MessageCauseAction
retraction is the steward's or the speaker's act (403)the account is not the organisation’s steward and did not make the findinga steward or the speaker must perform the retractionRetractionThe withdrawal of a finding or meaning entry, with the reason recorded. The record keeps who withdrew it, when and why; it is not deleted.
retracting a session purges its transcript — the steward's act (cxo/operator) (403)the account is not cxo or operatorthe steward performs the act
purging a document is the steward's act (cxo/operator) (403)the same rule, for an artefactArtifactA stored source file or evidence body that a record can cite. Keeping the exact content lets a reader check what supported the record.as above
the deep sweep is the steward's act (cxo/operator) (403)the same rule, for a sweepas above
viewer '<name>' holds the board role — read-onlya board account attempted a writeboard accounts read. Use a writer account for the change

Notifications not arriving

Each delivery is retried up to eight times in total, with the waits given in Accounts and stewardship. List the most recent deliveries to see what was issued and its outcome. Use the operator login chosen during setup:

./lens connectors notify deliveries --org <org> --viewer <operator-login> --limit 20

The platform refuses a destination that breaks any of these rules, with one of these messages:

  • Endpoint may not be a cloud metadata service, and Endpoint may not be a link-local address. The platform never sends to these.
  • A loopback receiver needs NOVUSLENS_ALLOW_LOOPBACK_WEBHOOK=1 in the deployment. Inside the installation, a loopbackLoopbackA network address that refers to the same host, such as 127.0.0.1. Other hosts cannot reach a service through that address. address is the service’s own container. Use the receiver’s network address instead.

Forgetting an organisation

Erasure cannot be undone. Add --yes to confirm. means ./lens forget-org was run without --yes. It deletes data, so the flag is required. A forgotten organisation is marked, and the materialiserMaterialiserThe component that reads recorded changes in order and updates the database. It records its place so it can continue after a restart. refuses to rebuild it: organisation <org> is marked forgotten (F7): nothing rebuilds it from the log.

When the cause is not in this list

Do not work around a refusal by editing the code, turning off a check, or copying a value from another installation. Collect the message in full and the diagnostic packet, and send them to Novus. Do not send a secret value; describe the variable name instead.

Last updated