BHarbor Evaluation Report
Collinear candidate task · 2026-09-05

A Blender task about recovery, not just rendering.

An offline, long-horizon asset-delivery task in which an agent must make Blender packages correct, recoverable, auditable, and repeatable after crashes and corruption.

HARBOR 0.22.0 BLENDER 4.3.2 4 CPU · 8 GB RAM · NO GPU DETERMINISTIC / OFFLINE
1.00Oracle, run 114 / 14 dimensions
1.00Oracle, run 2independent clean rerun
0.74GPT-5.5 Highvalid frontier rollout
25Agent turns37 tool calls
0.00NOP baselinebehavioral verifier rejects it
01 / Task

The agent repairs a production-like delivery worker.

The task is deliberately multi-stage. It combines Blender scene inspection, geometry processing, atomic artifact publication, append-only state, package audits, crash recovery, and a final byte-idempotency requirement.

What the agent must deliver

For each valid mechanical asset, build editable Blender output, visual and collision GLBs, and a manifest that is derived from the saved geometry. Invalid input must be isolated without stopping the batch.

  1. 1Inspect each .blend contract and validate ordered failure precedence.
  2. 2Generate LODs, convex collision proxies, GLBs, an editable result.blend, and a complete package manifest.
  3. 3Maintain a canonical append-only ledger through incomplete attempts, package corruption, and repair generations.
  4. 4Resume deterministic crashes and leave a normal rerun byte-for-byte unchanged.
/app
├── input/
│   ├── batch_contract.json
│   ├── batch_contract.sha256
│   └── assets/*.blend
├── seed_state/
├── state/
│   ├── batch_ledger.jsonl (append-only)
│   └── checkpoint.json
└── output/
    ├── reconcile_batch.py
    ├── batch_manifest.json
    └── assets/<asset-id>/
        ├── result.blend
        ├── *-visual.glb
        ├── *-collision.glb
        └── asset_manifest.json
Open the public task contract: expected outputs, state events, and fault boundaries

The candidate writes one standalone reconcile_batch.py into /app/output and invokes it through Blender in background mode. All asset-specific values must be discovered from the batch contract and each scene's embedded contract.

  • Validation order: input digest mismatch, contract digest mismatch, missing/non-mesh source, then source fingerprint mismatch. Each invalid asset receives exactly one terminal outcome and no package directory.
  • Valid package: preserve source geometry; rebuild the declared hierarchy and hinges; create named LODs and closed convex collision proxies; export visual and collision GLBs; save result.blend; derive the manifest from the saved scene.
  • Append-only state: preserve existing ledger bytes and use canonical JSON lines for BATCH_STARTED, ASSET_RECOVERED, ASSET_COMMITTED, ASSET_INVALIDATED, and reconciliation events.
  • Fault injection: the verifier may exit immediately after artifact publication, commit fsync, or invalidation fsync. A later run must recover without a duplicate lineage event.
  • Idempotency: after success, a normal rerun must leave ledger, aggregate files, Blender files, GLBs, and manifests byte-for-byte unchanged.
INPUTRead contractsContracts live in the batch and in Blender custom properties. Names, ratios, parts, and fault positions must not be hard-coded.
BUILDPublish packagesSaved Blender geometry, LODs, collision proxies, GLBs, and manifests are verified as a coherent package.
RECOVERRepair stateExisting ledger bytes remain unchanged. Crashes, orphan packages, and corruption lead to a durable generation lineage.
VERIFYAudit outcomesThe verifier evaluates artifacts and state transitions, not source strings, tool trajectory, or chain-of-thought.
02 / Rollout

What the target agent actually did.

This is a faithful public abstraction of one recorded Codex + GPT-5.5 rollout at high reasoning effort. It retains the observed sequence and outcome evidence, while excluding raw prompt payloads, internal paths, provider configuration, and model reasoning text.

Turns 01–0716:45:01–16:45:44 UTC

Discover the real contract

The agent inspected the workspace, read the batch and embedded Blender contracts, parsed the seed ledger, and used Blender rather than plain Python to inspect .blend custom properties and mesh fingerprints.

contract inspectionledger inspectionBlender probe
Turns 08–1016:48:19–16:49:52 UTC

Implement the reconciler

The implementation separated Blender package construction from ledger and audit logic, recovered the unfinished seeded attempt, staged assets before publication, and ran the first reconciliation under Blender.

standalone scriptatomic publishfirst Blender run
Turns 11–1716:49:52–16:50:44 UTC

Audit the normal path

The agent checked manifests, package inventories, LOD ratio bands, and failure codes. It then ran a second normal invocation and compared state and artifact hashes for byte-level idempotency.

artifact auditidempotencytemp repair workspace
Turns 18–2116:50:44–16:52:07 UTC

Exercise crash recovery

On disposable copies, the agent repaired a corrupted package, interrupted after artifact publication, resumed an orphan package, then interrupted after an invalidation record and resumed without duplicating the reopen or invalidation event.

repair generationartifact crashinvalidation crash
Turns 22–2516:52:18–16:52:42 UTC

Inspect the delivered scene

The final audit re-opened delivered scenes to check source fingerprints, parent chains, LOD ratios, and closed 12-triangle collision proxies. The output script also passed Python compilation.

scene-level auditstaging cleanupfinal report
Open the recorded agent-facing progress messages (no hidden reasoning)
T01 · inspect

Inspect the workspace and existing state first, then build against the actual contract/schema rather than guessing fixture details. Run under Blender and rerun for idempotency.

T02 · decode

The batch contract and sidecar digest need to be pinned down through the generator and embedded Blender contracts.

T03 · state

The seed ledger contains a started batch and an unfinished first asset attempt, requiring an ASSET_RECOVERED event before retrying.

T04 · Blender

Use Blender to read scene custom properties and sample mesh/fingerprint behavior because plain Python cannot parse the source scenes.

T05 · implement

Keep existing ledger bytes append-only, recover unfinished attempts, build packages in staging, and skip assets only when their active terminal state audits cleanly.

T06 · first run

The initial implementation is in place; run it under Blender so scene preparation and export API mismatches surface immediately.

T07 · audit

The first reconciliation built the valid packages. Audit ledger, manifests, inventories, and idempotency hashes next.

T08 · rerun

Ledger ordering and failure codes look right; run a second normal invocation and compare hashes of all state and artifact files.

T09 · repair

The normal rerun produced no byte changes and left staging empty. Exercise the repair path on a throwaway workspace copy.

T10 · crash

Repair a corrupted package in a temporary copy, inspect appended repair events, then test an artifact-boundary crash.

T11 · orphan

Interrupt after the first asset is published, then resume normally to verify recovery from an orphan package.

T12 · invalidate

After artifact recovery succeeds, interrupt right after the invalidation event and verify that resumption does not append a duplicate reopen or invalidation.

T13 · scene audit

Inspect saved result.blend files for source fingerprints, parent chains, collision closure, and LOD ratios from the delivered scenes.

T14 · finish

Confirm Python compilation, remove temporary cache output, and report the reconciled batch, generated aggregate files, and normal-rerun idempotency result.

03 / Verification

The normal path works. The hard repair lineage breaks.

The recorded model did not fail on a missing dependency, a timeout, or a subjective visual review. It failed four related lifecycle checks after the verifier introduced corruption and required a durable, append-only repair history.

DimensionOutcome checkedStatus
Visible asset pipelineBlender packages, LODs, GLBs, collisionPASS
Visible failure isolationPrecedence and no invalid packagePASS
Visible stateLedger structure and aggregate artifactsPASS
Visible idempotencyByte-stable normal rerunPASS
Held-out deliveryDifferent seeded IDs and scene contractsPASS
Held-out failure precedenceContract-defined terminal classificationPASS
Held-out stateLedger, aggregates, and idempotencyPASS
Commit crash recoveryDurable commit before aggregate statePASS
Package corruption detectionAudit missing or altered package bytesPASS
Invalidation resumeResume one repair generation correctlyFAIL
Repair artifact recoveryComplete a repair after artifact-boundary crashFAIL
Unaffected asset preservationDo not mutate healthy packagesPASS
Successive generation repairPreserve valid history through later repairFAIL
Aggregate self-healRegenerate aggregate-only state without mutationFAIL
Failure diagnosis

The ledger says an altered package was unchanged.

For a package carrying an unexpected sidecar file, the model wrote ASSET_INVALIDATED.actual_package_sha256 equal to the old expected digest. The public contract requires hashing the complete on-disk package index, including unexpected files. The append-only record becomes contradictory, so later repair generations cannot validate.

0.00Empty nop candidate is rejected by the same behavioral verifier.
1.00 × 2Independent oracle runs pass every one of the 14 dimensions.
Local onlySeed fixtures are generated from Blender primitives. No runtime model call, asset download, or network dependency is needed.
Outcome-basedThe verifier inspects saved Blender and GLB artifacts, state lineage, and byte-idempotency. It does not grade source strings or trajectory prose.
04 / Reproduce

The snapshot is the source of truth.

The submitted ZIP contains the task prompt, pinned Docker environment, seed generator, behavioral verifier, oracle, and compact rollout evidence. The commands below are included in the package documentation.

Harbor commands
# Baseline must fail
harbor run -p ./blender-batch-asset-pipeline-reconciliation \\
  -a nop -n 1 -r 0

# Oracle must pass
harbor run -p ./blender-batch-asset-pipeline-reconciliation \\
  -a oracle -n 1 -r 0

# Frontier route used for the recorded trial
harbor run -p ./blender-batch-asset-pipeline-reconciliation \\
  -a codex -m openai/gpt-5.5 -n 1 -r 0 \\
  --agent-kwarg reasoning_effort=high \\
  --agent-kwarg web_search=disabled

Review boundary

This site is supplemental evidence rather than a replacement for the submission snapshot.

  • The ZIP is the reproducible review artifact and contains README.md, RUN_REPORT.md, the oracle, verifier, and evidence JSON.
  • The public page intentionally omits raw trajectories, internal launch configuration, hidden asset fixture details, and credentials.
  • The task is net-new: assets, contracts, journal protocol, oracle, verifier, and documentation were authored from scratch using fixed-seed Blender primitives.
  • One frontier trial demonstrates a concrete failure mode; it is not a population estimate of model performance.