Guides

Troubleshooting

Match a symptom to its cause and the action that fixes it, for build problems, preparation and proving errors, rejected proofs, encoding failures and performance problems.

Each table on this page pairs a symptom with its usual cause and the action that resolves it. Installation problems are also covered in Installation, and mistakes in building requests in Proving state operations.

Before you start#

  • Print errors with {:?}. Most error types implement only Debug, and the variant with its fields usually names the cause.
  • Note whether the problem is a failed proving call or a false from verification. They have different causes and are covered in different sections below.
  • Record the source revision you build (for example the commit your Cargo.lock pins), the configuration, the operating system, the toolchain and the exact command. Every report needs them.

Build and toolchain#

SymptomCauseAction
Cargo reports that a package requires a newer Rust version, or does not recognize edition 2024The compiler is older than 1.85, the minimum that the manifests declareUse the pinned toolchain, 1.96.1, as described in Installation
rustup show reports a toolchain other than 1.96.1A +<toolchain> argument, RUSTUP_TOOLCHAIN or a directory override takes precedence over rust-toolchain.tomlRemove the override. In your own project, add a rust-toolchain.toml of its own
Cargo reports that Cargo.lock needs to change while you build with --lockedThe manifest or the toolchain differs from the one the lock file was made withKeep --locked, find the difference and do not regenerate the lock file to silence the error
prove_batch_parallel, wrap::settlement, wrap::risc0_destination_policy or wrap::risc0_route_b_manifest cannot be foundThe dependency turns off default features, which removes the host featureKeep the default features in host builds
Building, testing or proving takes far longer than expectedA debug buildBuild and run with --release

Preparation and proving errors#

SymptomCauseAction
SyncError::Compile(UnsupportedConfig { reason: "tree depth must be positive" })smt.depth is 0Set a depth of at least 1
SyncError::Compile(UnsupportedConfig { reason: "leaf_max_fields must be >= 10 (tag + 9 keccak limbs)" })The leaf bound is below 10Use a bound of at least 10; the default is 31
SyncError::Compile(UnsupportedConfig { reason: "v0.1 implements strategy A (one data-parallel instance per tree level) only" })layer_strategy is LayerStrategy::B or LayerStrategy::CUse LayerStrategy::A
SyncError::Witness(PathLengthMismatch { expected, found })The path has found siblings and the tree has expected levelsFetch the path for the configured depth, with the leaf-level sibling first
SyncError::Witness(LeafEncoding(EncodingTooLong { len, max }))The leaf encoding is longer than the leaf bound. An occupied leaf encodes one tag, its sync-state fields and nine identity limbsKeep the sync-state fields at or below the bound minus 10. A larger bound means a new configuration and a new tree
A debug build stops on an assertion inside prove_sync_op_prepared or make_job_preparedThe request's operation kind differs from the preparation'sSelect the preparation by request.operation.kind()
prove_batch returns fewer proofs than jobsprove_batch returns an empty vector when the configuration does not compileCompare counts, then call prepare to see the SyncError
Proving succeeded for a request that should failProving does not judge whether the operation holdsTreat only a true verification result as acceptance

A configuration error returns the same error on every attempt, so fix the configuration instead of retrying. Errors describes every variant and the calls that return it.

Verification returns false#

The facade's verification calls report every failure as false. Work through these causes in order:

SymptomCauseAction
Every proof is rejected after a deployment or configuration changeProver and verifier use different configurations, or a preparation was built before prover.config changedBuild both sides' preparations from one configuration, and rebuild them after every change
The prover's own check accepts, and the receiving side rejects the encoded bytesThe bytes were damaged or framed wrongly, were made under another configuration, or were checked against another requestDiagnose the bytes as described in Encoding and transport
One request is always rejectedThe operation does not hold, or the public inputs do not match the operation and witnessNarrow it down with the function in Narrow down a rejected result, then check the common mistakes in Proving state operations
Requests with large keys are rejectedThe key is at or above 2^depth. Witness generation does not check the range, so proving succeeds and verification rejectsAllocate keys below 2^depth
verify_sync_op and verify_sync_op_reference disagree on the same request and resultBoth share one verifier body and differ only in the wiring oracle, so they are expected to agreeOpen a public issue with a minimal reproduction; if the disagreement accepts an invalid proof, report it privately instead
A tampered or mismatched proof is acceptedA potential vulnerabilityStop and report it privately through the Security policy, never in a public issue

Narrow down a rejected result#

The function below repeats the cheap checks, in a useful order, for a typed result that verify_sync_op_prepared rejected. It is a diagnostic: the reference check at the end recompiles the circuit, so keep the function off the request path, and keep the verifier's false as the verdict.

src/diagnose.rsRust
use statesync_gkr::compiler::smt_valid_native;
use statesync_gkr::primitives::hash::Poseidon2Gadget;
use statesync_gkr::{PreparedSync, StateSyncProver, SyncRequest, SyncResult};

/// Narrow down why `verify_sync_op_prepared` returned `false`. Diagnostics only.
pub fn explain_rejection(
    prover: &StateSyncProver,
    prepared: &PreparedSync,
    request: &SyncRequest,
    result: &SyncResult,
) -> String {
    if request.operation.kind() != prepared.kind() {
        return "the request's operation kind differs from the preparation's".to_owned();
    }
    if result.public_inputs != request.public_inputs {
        return "the result carries different public inputs than the request".to_owned();
    }
    let params = &prover.config.smt;
    let hasher = Poseidon2Gadget::new(params.leaf_max_fields as usize);
    match smt_valid_native(
        &hasher,
        params,
        &request.operation,
        &request.public_inputs.old_root,
        &request.public_inputs.new_root,
        &request.witness,
    ) {
        Err(error) => return format!("the request is malformed: {error:?}"),
        Ok(false) => return "the operation does not hold for these roots and this witness".to_owned(),
        Ok(true) => {}
    }
    if prover.verify_sync_op_reference(request, result) {
        return "the reference verifier accepts: the preparation does not match this \
                prover's configuration, or the two verifiers disagree (report it)"
            .to_owned();
    }
    "the operation holds natively: check op_kind_tag, asset_id and value_digest, \
     then the proof itself"
        .to_owned()
}

smt_valid_native checks the path, the leaf condition and the root equations, but not op_kind_tag, asset_id or value_digest. When it returns true and verification still fails, those three fields or the proof itself are the likely cause.

Encoding and received bytes#

SymptomCauseAction
EncodeError::CountOverflow from encode_sync_result or circuit_identityleaf_max_fields is above 65,535, the largest value the circuit identity storesUse a leaf bound of at most 65,535
EncodeError::BadRoundPolyArityA round polynomial does not have exactly five coefficients, so the proof did not come unchanged from this proverEncode only proofs returned by the prover
EncodeError::MessageTooLongThe encoding would exceed u32::MAX bytesMeasured configurations stay far below this limit; check the configuration that produced the proof
decode_inner_proof returns DeclaredLengthMismatch or HeaderTooShortBytes were cut, added or concatenated in transitSend one complete encoding per message and frame it, as described in Encoding and transport
decode_inner_proof returns BadMagic, UnsupportedProofEncodingVersion, UnsupportedProtocolVersion, UnsupportedCircuitVersion, UnsupportedLeafEncodingVersion, UnsupportedProofKind or UnsupportedLayerStrategyThe payload is not an inner-proof-v1 message, or it uses a version this release does not decodeCheck what the sender transmits; this release decodes only version 1
decode_inner_proof returns NonCanonicalFieldElement, BadRoundPolyArity, UnknownOpKindTag, OpKindMismatch, OversizedCount, Truncated or TrailingBytesThe content is malformed or was modified, including counts that disagree with the declared lengthReject the bytes and check the sender

Performance#

SymptomCauseAction
Each request takes far longer than the measured timesThe fresh path compiles the circuit and computes its commitment on every call, or the build is a debug buildUse the prepared path, described in Prepared execution, and --release
Verification throughput is lowverify_sync_op or verify_sync_op_reference recompiles on every callUse verify_sync_op_prepared, or verify_encoded_sync_op with a preparation
Changing the worker count changes nothingprove_batch_parallel runs outside your pool's installFollow Batching and parallelism
Latency keeps rising during a run while throughput stays flatThe offered rate is above what the setting sustains, so the queue growsLower the rate, add workers or scale out, then confirm with a longer run, as described in Performance tuning
p99 latency rises by seconds after you lower the worker count or the batch capToo few workers or too small batches let the queue growReturn to one worker per core and a batch cap equal to the worker count, then retune
Memory grows with concurrencyEvery job in flight holds its own witness and prover stateLimit the jobs in flight and measure under your memory limit
Short tests meet the latency budget and longer runs miss itThe queue builds up slowly over the runConfirm settings with longer runs
A load of one operation kind behaves differently from the mixed loadUpdates prove two paths and do more work per request than membership or non-membershipMeasure with the operation mix you expect

Proof bytes differ between builds#

SymptomCauseAction
proof_digest prints different output for a baseline and a native build of the same commitProof bytes depend on the build, which the repository's determinism check rules outOpen a public issue (opens in a new tab) with the commit, the host and both outputs, as described in Installation

Report a problem#

  • For a reproducible defect or a documentation error, open a public issue (opens in a new tab) with the revision, operating system, toolchain, command, expected and observed output and a minimal reproduction.
  • For a suspected vulnerability, including any accepted tampered proof, follow the Security policy and never use a public issue.
  • For support with a production deployment, contact Oraclizer Labs through StateSync-GKR licensing.

Next steps#