Guides
Troubleshooting
Match a symptom to its cause and the action that fixes it, for build problems, preparation and proving errors, rejected proofs, encoding failures and performance problems.
Each table on this page pairs a symptom with its usual cause and the action that resolves it. Installation problems are also covered in Installation, and mistakes in building requests in Proving state operations.
Before you start#
- Print errors with
{:?}. Most error types implement onlyDebug, and the variant with its fields usually names the cause. - Note whether the problem is a failed proving call or a
falsefrom verification. They have different causes and are covered in different sections below. - Record the source revision you build (for example the commit your
Cargo.lockpins), the configuration, the operating system, the toolchain and the exact command. Every report needs them.
Build and toolchain#
| Symptom | Cause | Action |
|---|---|---|
| Cargo reports that a package requires a newer Rust version, or does not recognize edition 2024 | The compiler is older than 1.85, the minimum that the manifests declare | Use the pinned toolchain, 1.96.1, as described in Installation |
rustup show reports a toolchain other than 1.96.1 | A +<toolchain> argument, RUSTUP_TOOLCHAIN or a directory override takes precedence over rust-toolchain.toml | Remove the override. In your own project, add a rust-toolchain.toml of its own |
Cargo reports that Cargo.lock needs to change while you build with --locked | The manifest or the toolchain differs from the one the lock file was made with | Keep --locked, find the difference and do not regenerate the lock file to silence the error |
prove_batch_parallel, wrap::settlement, wrap::risc0_destination_policy or wrap::risc0_route_b_manifest cannot be found | The dependency turns off default features, which removes the host feature | Keep the default features in host builds |
| Building, testing or proving takes far longer than expected | A debug build | Build and run with --release |
Preparation and proving errors#
| Symptom | Cause | Action |
|---|---|---|
SyncError::Compile(UnsupportedConfig { reason: "tree depth must be positive" }) | smt.depth is 0 | Set a depth of at least 1 |
SyncError::Compile(UnsupportedConfig { reason: "leaf_max_fields must be >= 10 (tag + 9 keccak limbs)" }) | The leaf bound is below 10 | Use a bound of at least 10; the default is 31 |
SyncError::Compile(UnsupportedConfig { reason: "v0.1 implements strategy A (one data-parallel instance per tree level) only" }) | layer_strategy is LayerStrategy::B or LayerStrategy::C | Use LayerStrategy::A |
SyncError::Witness(PathLengthMismatch { expected, found }) | The path has found siblings and the tree has expected levels | Fetch the path for the configured depth, with the leaf-level sibling first |
SyncError::Witness(LeafEncoding(EncodingTooLong { len, max })) | The leaf encoding is longer than the leaf bound. An occupied leaf encodes one tag, its sync-state fields and nine identity limbs | Keep the sync-state fields at or below the bound minus 10. A larger bound means a new configuration and a new tree |
A debug build stops on an assertion inside prove_sync_op_prepared or make_job_prepared | The request's operation kind differs from the preparation's | Select the preparation by request.operation.kind() |
prove_batch returns fewer proofs than jobs | prove_batch returns an empty vector when the configuration does not compile | Compare counts, then call prepare to see the SyncError |
| Proving succeeded for a request that should fail | Proving does not judge whether the operation holds | Treat only a true verification result as acceptance |
A configuration error returns the same error on every attempt, so fix the configuration instead of retrying. Errors describes every variant and the calls that return it.
Verification returns false#
The facade's verification calls report every failure as false. Work through these causes in order:
| Symptom | Cause | Action |
|---|---|---|
| Every proof is rejected after a deployment or configuration change | Prover and verifier use different configurations, or a preparation was built before prover.config changed | Build both sides' preparations from one configuration, and rebuild them after every change |
| The prover's own check accepts, and the receiving side rejects the encoded bytes | The bytes were damaged or framed wrongly, were made under another configuration, or were checked against another request | Diagnose the bytes as described in Encoding and transport |
| One request is always rejected | The operation does not hold, or the public inputs do not match the operation and witness | Narrow it down with the function in Narrow down a rejected result, then check the common mistakes in Proving state operations |
| Requests with large keys are rejected | The key is at or above 2^depth. Witness generation does not check the range, so proving succeeds and verification rejects | Allocate keys below 2^depth |
verify_sync_op and verify_sync_op_reference disagree on the same request and result | Both share one verifier body and differ only in the wiring oracle, so they are expected to agree | Open a public issue with a minimal reproduction; if the disagreement accepts an invalid proof, report it privately instead |
| A tampered or mismatched proof is accepted | A potential vulnerability | Stop and report it privately through the Security policy, never in a public issue |
Narrow down a rejected result#
The function below repeats the cheap checks, in a useful order, for a typed result that verify_sync_op_prepared rejected. It is a diagnostic: the reference check at the end recompiles the circuit, so keep the function off the request path, and keep the verifier's false as the verdict.
use statesync_gkr::compiler::smt_valid_native;
use statesync_gkr::primitives::hash::Poseidon2Gadget;
use statesync_gkr::{PreparedSync, StateSyncProver, SyncRequest, SyncResult};
/// Narrow down why `verify_sync_op_prepared` returned `false`. Diagnostics only.
pub fn explain_rejection(
prover: &StateSyncProver,
prepared: &PreparedSync,
request: &SyncRequest,
result: &SyncResult,
) -> String {
if request.operation.kind() != prepared.kind() {
return "the request's operation kind differs from the preparation's".to_owned();
}
if result.public_inputs != request.public_inputs {
return "the result carries different public inputs than the request".to_owned();
}
let params = &prover.config.smt;
let hasher = Poseidon2Gadget::new(params.leaf_max_fields as usize);
match smt_valid_native(
&hasher,
params,
&request.operation,
&request.public_inputs.old_root,
&request.public_inputs.new_root,
&request.witness,
) {
Err(error) => return format!("the request is malformed: {error:?}"),
Ok(false) => return "the operation does not hold for these roots and this witness".to_owned(),
Ok(true) => {}
}
if prover.verify_sync_op_reference(request, result) {
return "the reference verifier accepts: the preparation does not match this \
prover's configuration, or the two verifiers disagree (report it)"
.to_owned();
}
"the operation holds natively: check op_kind_tag, asset_id and value_digest, \
then the proof itself"
.to_owned()
}smt_valid_native checks the path, the leaf condition and the root equations, but not op_kind_tag, asset_id or value_digest. When it returns true and verification still fails, those three fields or the proof itself are the likely cause.
Encoding and received bytes#
| Symptom | Cause | Action |
|---|---|---|
EncodeError::CountOverflow from encode_sync_result or circuit_identity | leaf_max_fields is above 65,535, the largest value the circuit identity stores | Use a leaf bound of at most 65,535 |
EncodeError::BadRoundPolyArity | A round polynomial does not have exactly five coefficients, so the proof did not come unchanged from this prover | Encode only proofs returned by the prover |
EncodeError::MessageTooLong | The encoding would exceed u32::MAX bytes | Measured configurations stay far below this limit; check the configuration that produced the proof |
decode_inner_proof returns DeclaredLengthMismatch or HeaderTooShort | Bytes were cut, added or concatenated in transit | Send one complete encoding per message and frame it, as described in Encoding and transport |
decode_inner_proof returns BadMagic, UnsupportedProofEncodingVersion, UnsupportedProtocolVersion, UnsupportedCircuitVersion, UnsupportedLeafEncodingVersion, UnsupportedProofKind or UnsupportedLayerStrategy | The payload is not an inner-proof-v1 message, or it uses a version this release does not decode | Check what the sender transmits; this release decodes only version 1 |
decode_inner_proof returns NonCanonicalFieldElement, BadRoundPolyArity, UnknownOpKindTag, OpKindMismatch, OversizedCount, Truncated or TrailingBytes | The content is malformed or was modified, including counts that disagree with the declared length | Reject the bytes and check the sender |
Performance#
| Symptom | Cause | Action |
|---|---|---|
| Each request takes far longer than the measured times | The fresh path compiles the circuit and computes its commitment on every call, or the build is a debug build | Use the prepared path, described in Prepared execution, and --release |
| Verification throughput is low | verify_sync_op or verify_sync_op_reference recompiles on every call | Use verify_sync_op_prepared, or verify_encoded_sync_op with a preparation |
| Changing the worker count changes nothing | prove_batch_parallel runs outside your pool's install | Follow Batching and parallelism |
| Latency keeps rising during a run while throughput stays flat | The offered rate is above what the setting sustains, so the queue grows | Lower the rate, add workers or scale out, then confirm with a longer run, as described in Performance tuning |
| p99 latency rises by seconds after you lower the worker count or the batch cap | Too few workers or too small batches let the queue grow | Return to one worker per core and a batch cap equal to the worker count, then retune |
| Memory grows with concurrency | Every job in flight holds its own witness and prover state | Limit the jobs in flight and measure under your memory limit |
| Short tests meet the latency budget and longer runs miss it | The queue builds up slowly over the run | Confirm settings with longer runs |
| A load of one operation kind behaves differently from the mixed load | Updates prove two paths and do more work per request than membership or non-membership | Measure with the operation mix you expect |
Proof bytes differ between builds#
| Symptom | Cause | Action |
|---|---|---|
proof_digest prints different output for a baseline and a native build of the same commit | Proof bytes depend on the build, which the repository's determinism check rules out | Open a public issue (opens in a new tab) with the commit, the host and both outputs, as described in Installation |
Report a problem#
- For a reproducible defect or a documentation error, open a public issue (opens in a new tab) with the revision, operating system, toolchain, command, expected and observed output and a minimal reproduction.
- For a suspected vulnerability, including any accepted tampered proof, follow the Security policy and never use a public issue.
- For support with a production deployment, contact Oraclizer Labs through StateSync-GKR licensing.
Next steps#
- Errors: every error type and variant.
- Encoding and transport: diagnose received bytes.
- Performance tuning: choose and confirm operating settings.