Skip to content
TraceDojo

TraceDojo limitations

TraceDojo tests agent behavior inside a defined simulation. Use its reports as evidence about the tested conditions, not a certification of production safety.

What the demo establishes

The booking demo runs a scripted reference agent twice against the same tools, starting state, checks, and lost-confirmation fault. A fresh operation key on each retry creates two bookings; a stable key preserves one. Both no-fault controls must pass. The demo does not use a language model or establish that a model independently discovered the fix. The GIF is an edited walkthrough of recorded CLI output and actual report evidence; its 45-second length is presentation pacing.

Execution boundary

  • Agents run locally or in CI. The hosted dashboard stores and inspects reports; it does not execute arbitrary customer code.
  • Adapters are trusted JavaScript with access to your environment and network. Cancellation and deadlines are cooperative. They cannot forcibly terminate synchronous blocking code. This is not a security sandbox.
  • Tool simulations may return promises and act on in-memory state drafts. Whole transactions are serialized in call order; overlapping service transactions are not simulated. Late results cannot commit to a closed trial, but ignored cancellation cannot forcibly stop handler code or external side effects. Model calls in the agent may be asynchronous and may incur provider charges.
  • A passing simulation does not establish that external services implement the same consistency, authorization, or idempotency behavior.
  • Virtual time is opt-in and applies only to context.clock.now() and context.clock.sleep(). Native timers, Date.now(), and external I/O remain real. The clock models cooperative delays, not parallel service transactions. Each trial has bounded virtual time/sleep counts and retains its real wall deadline.

Outcomes and comparisons

  • Automatic matrices enumerate faults from declared tool kinds. They preserve supplied checks rather than infer correct business outcomes. They do not infer tool reachability, all call occurrences, attack payloads, or stale-read history. Unreached tools remain incomplete. Each suite produces a normal report; the matrix coverage index is not a dashboard report or an aggregate regression gate.

  • Workflow checks support exact equality, array counts, unchanged state, and forbidden tool attempts. Tool ordering and arbitrary predicates are not implemented.

  • Prompt injection replaces successful tool-output data with configured JSON. The included scripted failure/fix verifies the harness, not model security. Coverage depends on the supplied attacks and checks; it is not a security certification.

  • Duplicate delivery executes a write twice sequentially and returns the first response. Stale reads replay the first captured successful response for matching inputs when later data differs. These model immediate redelivery and a pinned snapshot, not queue reordering, concurrent writes, or distributed consistency.

  • A failed control prevents fault trials. A fault that is never reached is inconclusive. Neither is evidence of successful recovery.

  • The dashboard compares individual reports under matching conditions. CLI/CI comparisons gate regressions against matching base-commit batches; generated workflows update a persistent PR comment. Statistical significance is not measured. See Security boundaries for the CI trust model and required-check setup.

  • Limits on tool calls, repeated calls, duration, and evidence size can stop a run.

  • Scenario trends cover loaded reports only and display the latest 12 commit groups by evidence time. Incomplete controls/trials or conflicting evidence suppress the rate. Commit labels are supplied by the caller; they do not verify checkout contents. Rates describe one scenario's observed trials, not overall suite health or statistical reliability. Legacy unlabelled reports are excluded. Partial evidence remains distinguishable from completed trials.

  • Token usage and provider error handling in the separate support-agent adapter are not measurements of model reliability. Live-model results require actual trials and inspection; mock-provider tests do not establish live performance.

Trace import and evidence

  • LangSmith CLI imports generate runnable playback stubs and observed signatures; the dashboard and --draft-only produce review drafts. Stubs do not reconstruct your agent or application. Users must supply starting state, simulated tools, and independent expected outcomes. The default failing timeout is a playback demonstration, not evidence of a defect in the original agent. Recorded outputs are observations, not automatically correct expectations. Ambiguous, missing, or error outputs block the generated control until reviewed.
  • Imported reports are externally supplied evidence. Schema checks reject contradictions but do not independently prove that reported events occurred.
  • Only observable messages and actions are recorded. TraceDojo does not expose hidden model reasoning.
  • Playback reveals saved events. A fresh execution requires the corresponding adapter, configuration, and dependencies; playback itself makes no new calls.

Privacy and storage

  • Redaction uses patterns. It is not a complete personal-data scrubber. Use synthetic fixtures and review reports before uploading or sharing them.
  • Hosted browser imports are previews. CLI uploads persist reports in a private project. Credentials stay in the CLI environment and should never be committed.
  • CI artifacts inherit the repository's access settings and can contain application data. The generated workflow requests 30-day retention; the separate opt-in live-model workflow requests seven days, subject to repository settings.
  • Local JSON storage is intended for a developer workspace. Hosted projects have upload quotas. Automatic customer-configurable retention and team sharing are not implemented. Free and Pro storage allowances apply to hosted accounts; paid checkout availability depends on the deployment.

Licensing and release

The SDK and command-line launcher are Apache-2.0. The dashboard and branding are proprietary; see LICENSE for the boundary. The README records whether public npm publishing has completed. A packed archive or a successful local registry test is not evidence that a package is available on npm.