BayesAudit

Problem

Hierarchical LLM systems can delegate work across multiple agents and tool calls, making exhaustive inspection expensive. A useful monitor must decide not only whether a workflow is risky, but when the expected benefit of inspection justifies its cost.

BayesAudit treats oversight as a sequential decision problem. A policy receives observable risk signals, operates under a fixed audit budget, and chooses whether to continue, inspect, escalate, or intervene.

Approach

The framework evaluates single-agent, unstructured delegation, structured constraint inheritance, and verified-inheritance workflows under matched safe and attacked conditions.

The project uses synthetic, sandboxed tasks and objective scorers so monitoring policies can be compared under controlled budgets without presenting the result as a production AI safety system.

Key findings

The confirmatory study observed more objective-positive outcomes in attacked no-oversight conditions than in matched safe controls, but the paired effect was not statistically confirmed.

All validated violations in the principal studies were internal-only. No tool-argument or final-output violations were observed.

In the held-out study, the rule-based monitor recorded 7 true positives and 5 false positives. The Bayesian/logistic monitor recorded 0 true positives and 5 false negatives.

The held-out study did not replicate the direction of the confirmatory attack effect.

Clean causal prevention was not established. Corrected-before-final behavior remains conditional and exploratory.

Why this matters beyond LLM oversight

The core problem is a familiar quantitative one: allocate a scarce resource under uncertainty while balancing asymmetric errors and intervention costs.

The project required Bayesian updating, calibrated risk estimation, sequential decisions, matched experimental design, false-positive and false-negative analysis, held-out robustness testing, and strict accounting of computational cost.

Engineering and research discipline

• 1,945 automated tests
• Deterministic artifact manifests
• Evidence-package hashes
• Provider request and cache accounting
• Token and cost ceilings
• CI validation
• Security scanning
• Preservation of null and nonreplicated results

What failed or changed

Phase 9 did not replicate the Phase 8 attack-effect direction, so the final writeup preserves the held-out nonreplication instead of tuning the benchmark until the original result returned.

The Bayesian/logistic monitor did not outperform the rule-based monitor in the held-out study.

Clean causal prevention was not established, even though corrected-before-final behavior remains useful to inspect as conditional exploratory evidence.

Final interpretation

BayesAudit is strongest as a reproducible research framework for studying adaptive oversight under constrained budgets.

The released result supports careful experimental infrastructure, objective scoring, provider-cost accounting, and transparent preservation of null and nonreplicated findings. It does not show that oversight generally prevents LLM workflow violations.

Limitations

• All tasks and tools were synthetic and sandboxed.
• One provider model was tested.
• The project does not estimate real-world violation prevalence.
• Cross-model generalization was not established.
• No final-output violations were observed in the principal studies.
• Clean causal prevention was not established.
• Phase 9 did not replicate the confirmatory attack-effect direction.
• The framework is a research prototype, not a production safety system.

How to reproduce

The public repository contains the source code, deterministic manifests, release artifacts, technical report source, benchmark documentation, dataset card, model and provider card, and validation outputs for BayesAudit v1.0.0.

The page links to the release and public documentation rather than reinterpreting unsupported statistics.