# Selective Reliability in Retrieval-Grounded Mixture-of-Agent Systems ## Evidence-Aware Routing, Heterogeneous Verification, and Capability-Constrained Action **First thesis manuscript draft** **Evidence base:** supplied local research artifacts only **Status:** proposed architecture and experimental plan; no original experimental results are reported --- ## Abstract Large language models can follow natural-language instructions, retrieve external information, invoke tools, and participate in multi-agent deliberation. These capabilities make it possible to build systems that perform research, software engineering, administrative work, and other multi-step tasks. They do not, however, establish that such systems are reliably truthful, appropriately aligned, or safe when actions affect external state. The reviewed literature instead identifies distinct and partially independent failure mechanisms: models may follow lower-authority instructions over higher-authority policy; retrieval corpora may be incomplete, stale, poisoned, or contradictory; retrievers may omit decisive evidence; generators may ignore available evidence or attach citations that do not entail their claims; learned verifiers may reproduce generator errors; multiple agents may converge on a shared misconception; and tool-using systems may execute unauthorized or irreversible actions. These failures cannot be resolved by model scale, retrieval, agent consensus, or preference optimization alone. This thesis proposes and evaluates a **risk-adaptive, retrieval-grounded mixture-of-agent architecture** for selective reliability. The architecture combines: (1) explicit instruction-authority and risk classification; (2) governed retrieval over provenance-preserving evidence snapshots; (3) typed planning and atomic decomposition of claims and actions; (4) heterogeneous verification selected according to the object being checked; (5) calibrated abstention, clarification, and escalation; and (6) an externally enforced action gateway based on least privilege, argument validation, dry runs, transaction boundaries, postcondition checks, and rollback where available. Sparse mixture-of-experts capacity and inference-time mixture-of-agents orchestration are treated as separate design dimensions. Sparse experts may increase parameter capacity per unit of active computation, whereas multiple agents spend inference-time computation on search, critique, and synthesis. Neither mechanism is assumed to improve factuality or safety independently. The central empirical question is whether this defense-in-depth architecture reduces severe unsupported claims and unsafe actions relative to single-model, retrieval-augmented, self-consistency, best-of-\(N\), search, debate, layered mixture-of-agents, and deterministic workflow baselines. Comparisons will be conducted both under matched token and tool-call budgets and under unconstrained best-performance settings. Primary outcomes include atomic supported-claim precision, citation correctness and completeness, attack success rate, unauthorized action rate, severity-weighted expected loss, selective risk at matched coverage, latency, and cost. The design will be evaluated across instruction conflicts, retrieval-grounded factuality, long-form generation, tool use, agent tasks, poisoned corpora, indirect prompt injection, and shifted domains. No claim of perfect alignment, zero hallucination, or universal safety is made. The proposed contribution is a falsifiable account of when orchestration, retrieval, verification, and abstention improve reliability; which components produce those gains; and what cost, coverage, and residual-risk trade-offs remain. The architecture follows a centaur-style division of labor: models propose and decompose, retrieval supplies external evidence, specialized tools calculate or test, independent verifiers challenge claims and actions, deterministic controls enforce authority, and humans retain decision rights over unresolved consequential cases. --- ## 1. Introduction Language models have developed from text predictors into components of systems that retrieve documents, synthesize long-form answers, call APIs, write and execute code, navigate software environments, and coordinate multiple model instances. Instruction tuning has made natural-language prompts a broadly useful task interface. Human- and AI-feedback methods have improved behavior on selected preference distributions. Retrieval-augmented generation has introduced updateable, inspectable non-parametric memory. Tool-use methods have connected models to calculators, search engines, code interpreters, databases, and web environments. Sparse mixture-of-experts models have expanded total parameter capacity without activating every parameter for every token, while inference-time mixture-of-agents systems have used multiple complete model calls to propose, critique, and aggregate answers. These advances are often discussed under a general vocabulary of capability, alignment, grounding, or agency. That vocabulary can obscure important distinctions. An answer may be correct without being supported by its cited evidence. A model may produce a citation that exists but does not entail the associated claim. Several agents may agree because they share the same training data, model family, prompt conventions, and retrieved passages. A learned verifier may assign a high score to a fluent but invalid solution. A model may correctly describe a safe policy while still being able to issue an unauthorized tool call. A sparse neural expert is not an autonomous software agent, and a mixture-of-experts result is not evidence for the reliability of a mixture-of-agents workflow. This thesis starts from three distinctions. First, **sparse mixture-of-experts and inference-time mixture-of-agents are different scaling mechanisms**. In a sparse mixture-of-experts model, a learned router activates a subset of internal feed-forward experts, often at the token level. The principal objective is to increase total parameter capacity without a proportional increase in active computation. In a mixture-of-agents system, complete model calls generate candidate answers, critiques, or refinements. This ordinarily increases inference cost. Sparse routing may improve capability per active floating-point operation; multi-agent orchestration may improve search or revision through additional test-time computation. Neither mechanism, by itself, creates epistemic independence, trustworthy evidence use, robust policy adherence, or safe action. Second, **retrieval is not equivalent to grounding**. Retrieval can fail because the corpus is missing, stale, poisoned, low-authority, or internally contradictory. A retriever can fail to return relevant material. A generator can fail to use relevant material that is present, particularly amid distractors or adverse evidence positions. Finally, a generated statement can exceed, distort, or contradict its cited evidence. Retrieval therefore creates a chain of components whose errors must be evaluated separately. The presence of retrieved passages or citations is not sufficient evidence of support. Third, **confidence, self-consistency, and consensus are not verification**. Sampling variation can reveal instability, and multi-agent debate can expose disagreement or support revision. Yet a misconception repeated consistently by related models remains a misconception. Stronger verification requires independent contact with evidence or external state: a proof kernel, a compiler, a test suite, a database invariant, an authoritative source snapshot, or a verifier with a genuinely distinct model family and evidence path. Model judgment is useful, but it remains probabilistic and can be correlated with the generator it evaluates. The literature reviewed in the supplied artifacts does not support the proposition that a language model or agent can be made to “never do anything wrong.” Broad distributions, imperfect supervision, adversarial pressure, changing information, and open-ended action make a universal zero-error target both scientifically unsupported and operationally underspecified. A more defensible goal is **selective reliability**: reducing severe errors under stated conditions, calibrating when the system should answer or act, and exposing residual risk rather than concealing it behind aggregate accuracy. The proposed architecture implements this goal through defense in depth. A policy and risk gate first separates authoritative instructions from user requests and untrusted retrieved content. A governed retrieval subsystem produces immutable evidence bundles with provenance, authority, freshness, and access-control metadata. A planner expresses subgoals, evidence requirements, tool choices, preconditions, verification methods, rollback procedures, and stop conditions in typed form. A proposer generates atomic claims and explicit action proposals. A verification portfolio dispatches each object to the strongest applicable checker. A controller then approves, revises, retrieves more evidence, requests clarification, qualifies, abstains, refuses, or escalates. State-changing operations pass through an external action gateway that enforces permissions independently of model-generated text. This architecture is deliberately modular. Modularity permits component-level measurement and ablation, but it does not imply statistical independence. A retriever and claim verifier may share embeddings; a proposer and critic may share a checkpoint; several agents may rely on the same source corpus; and a policy classifier may inherit the same post-training biases as the generator. The experiments therefore measure not merely component accuracy but also correlated failure, conditional error, and the marginal contribution of each layer. The thesis is motivated by a centaur-chess analogy. In centaur chess, performance can emerge from a structured partnership between human judgment and machine calculation rather than from treating either participant as infallible. The analogous architecture here assigns different forms of epistemic and operational authority to different components. Models are well suited to interpretation, decomposition, candidate generation, and natural-language synthesis. Retrieval systems provide updateable evidence. Deterministic tools calculate or test. External controllers enforce permissions. Human reviewers adjudicate rare, ambiguous, or consequential cases. The objective is not a larger committee of interchangeable language models. It is a governed system in which complementary capabilities are paired with explicit evidence and authority boundaries. The principal thesis claim is therefore: > A risk-adaptive, retrieval-grounded orchestration architecture using explicit instruction authority, atomic claim and action decomposition, heterogeneous verification, calibrated abstention, and capability-constrained tools can reduce severe unsupported claims and unsafe actions relative to single-model and equal-compute test-time-scaling baselines, while exposing measurable trade-offs among cost, coverage, utility, and residual risk. This claim is empirical and falsifiable. It would not be supported by an improvement in average benchmark accuracy alone, by a preference win rate from an unvalidated language-model judge, or by gains obtained only through substantially greater inference expenditure. It requires statistically credible reductions in severe failures at matched task coverage and compute, including under held-out domains and adversarial conditions. It is falsified or materially weakened if simpler workflows match the full system, if verifiers mostly reproduce generator errors, if abstention fails under shift, or if external controls impose high utility costs without reducing attack success. The expected contribution is thus not a declaration that a particular architecture is safe. It is an experimentally grounded map of what reliability mechanisms help, when they help, how much they cost, and where they fail. --- ## 2. Research Questions and Hypotheses ### 2.1 Primary research question **RQ1.** Can a risk-adaptive orchestration system reduce severe factual and action failures better than single-model and equal-compute test-time-scaling baselines by combining governed retrieval, claim and action decomposition, heterogeneous verification, calibrated abstention, and least-privilege tools? The phrase “better than” requires simultaneous consideration of reliability, task coverage, benign utility, and resource consumption. A system that avoids every error by refusing every task is not useful. A system that obtains a small accuracy gain by multiplying inference cost without bound is not necessarily efficient. The proposed comparisons therefore examine failure–coverage, failure–cost, and utility–security frontiers. ### 2.2 Secondary research questions **RQ2.** Which parts of the architecture account for any observed improvement: retrieval governance, atomic decomposition, independent verification, abstention, external permission enforcement, or adaptive routing? **RQ3.** Does heterogeneous verification reduce correlated error more effectively than role-prompted copies or independent samples of the same model? **RQ4.** Under matched inference budgets, does adaptive orchestration outperform always-on debate, layered mixture-of-agents synthesis, self-consistency, best-of-\(N\), and tree-structured search? **RQ5.** How robust is selective abstention under domain shift, incomplete evidence, contradictory sources, retrieval distractors, and adversarially poisoned corpora? **RQ6.** To what extent do external trust and permission boundaries reduce prompt-injection and unauthorized-action success while preserving benign-task utility? **RQ7.** Does a sparse mixture-of-experts backbone improve capability at matched active computation without independently improving factuality or safety after capability is controlled? **RQ8.** Which benchmark and metric choices materially change conclusions about the architecture, particularly when automated language-model judges are replaced or audited by human and executable evaluation? ### 2.3 Hypotheses #### H1 — Grounding through decomposition and evidence verification Atomic claim decomposition combined with independent evidence verification will reduce unsupported claims relative to standard top-\(k\) retrieval-augmented generation. The hypothesized mechanism is observability. Whole-answer evaluation can conceal a mixture of supported statements, unsupported additions, temporal errors, and citation mismatches. Atomic decomposition creates a unit to which evidence IDs, entailment judgments, and uncertainty can be attached. This hypothesis does not assume decomposition is perfectly accurate; decomposition errors will themselves be measured. #### H2 — Selective reliability Risk-calibrated abstention will lower severity-weighted error at matched coverage relative to confidence-only and always-answer baselines. This hypothesis distinguishes selective prediction from blanket refusal. Thresholds will be tuned on held-out calibration data, and unnecessary refusal will be reported alongside residual error. Performance under distribution shift is central: in-domain calibration alone would provide weak evidence. #### H3 — Heterogeneity and correlated error Verifiers with distinct tools, model families, training objectives, or evidence access will reduce correlated failures more than role-prompted copies of one generator model. The hypothesis does not equate different labels or personas with independence. Heterogeneity is operationalized through distinct sources of evidence or computation. Examples include a language model checked by a compiler, a generator checked by a separately trained claim verifier, or separate agents searching non-identical evidence indexes. #### H4 — Process and object-level checks Step-, claim-, and action-level verification will reduce severe end-state failures more than final-answer majority voting at equal inference compute. The mechanism is earlier localization of errors and the use of checkers tailored to the object under review. The likely effect may vary by domain. Process verification is expected to be strongest where intermediate steps have stable semantics, such as arithmetic, code, schemas, and structured transactions, and weaker in open-ended judgment. #### H5 — External security boundaries Typed trust channels and externally enforced least privilege will reduce indirect prompt-injection success and unauthorized actions more than prompt-only defenses at matched benign utility. The external gateway is expected to remain effective even when the model is persuaded by malicious retrieved text, because permissions are not derived from that text. This hypothesis does not imply immunity to all attacks; attacks against the gateway, schemas, credentials, or human confirmation process remain possible. #### H6 — Efficient orchestration A risk-adaptive controller will produce a better failure–cost frontier than always-on mixture-of-agents or debate. Most low-risk, well-supported requests may need one retrieval pass, one generator, and lightweight checks. Additional proposers or debate rounds should be reserved for contradictory evidence, failed verification, high-impact action, or out-of-distribution inputs. The controller succeeds only if routing overhead and missed escalations do not offset the saved compute. #### H7 — Sparse capacity is not a safety mechanism At matched active compute, a sparse mixture-of-experts backbone may improve task capability, but it will not independently improve factuality, grounding, or action safety once capability and the surrounding reliability pipeline are controlled. This hypothesis explicitly prevents total parameter count from being interpreted as a reliability guarantee. Router behavior, expert overflow, and communication costs will be reported as systems properties rather than alignment evidence. --- ## 3. Background and Related Work ### 3.1 Instruction tuning and behavioral alignment Instruction tuning provides the interface on which most contemporary agent systems depend. [*Finetuned Language Models Are Zero-Shot Learners*](https://arxiv.org/abs/2109.01652) reports that tuning a large model across many natural-language task descriptions can improve zero-shot transfer to held-out task clusters. [*Multitask Prompted Training Enables Zero-Shot Task Generalization*](https://arxiv.org/abs/2110.08207) similarly treats prompted tasks as a multitask training mixture and highlights substantial sensitivity to prompt selection and task-family proximity. These findings support natural-language descriptions as a way to coordinate routers, planners, retrievers, and specialists. They also caution against calling an evaluation “novel-task generalization” merely because the exact dataset was held out. Semantically adjacent task families and pretraining contamination may still contribute. Instruction tuning does not specify whose instructions should prevail when messages conflict. The proposed system therefore adopts an explicit authority model. System policy, operator constraints, user requests, retrieved documents, web pages, emails, and tool outputs cannot be treated as equivalent strings. [*The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions*](https://arxiv.org/abs/2404.13208) motivates training models to privilege higher-authority instructions, but runtime enforcement is still required for consequential operations. Retrieved text must remain data even when it contains imperative language. Preference-based post-training changes model behavior relative to selected human or model judgments. [*Training Language Models to Follow Instructions with Human Feedback*](https://arxiv.org/abs/2203.02155) establishes the influential supervised fine-tuning, reward-model, and policy-optimization pipeline. Its results support the proposition that post-training can improve instruction following and some measured factuality and toxicity outcomes. The same work emphasizes that “alignment” is alignment to particular researchers, customers, labelers, and distributions rather than a universal value function. Preference labels can reward style, confidence, verbosity, or superficial helpfulness rather than factual support. [*Constitutional AI: Harmlessness from AI Feedback*](https://arxiv.org/abs/2212.08073) makes selected principles explicit and uses model-generated critique, revision, and preferences to reduce human-label requirements. This offers a useful pattern for grounding-specific constitutions: preserve source identity, distinguish support from inference, surface contradictory evidence, and abstain when evidence is insufficient. The limitation is that an AI judge can share the generator’s errors. Explicit principles improve inspectability but do not prove robust adherence under adversarial pressure. [*Direct Preference Optimization: Your Language Model Is Secretly a Reward Model*](https://arxiv.org/abs/2305.18290) provides a simpler offline alternative to an explicit reward-model-plus-PPO pipeline. It may be suitable for preference pairs contrasting supported and unsupported synthesis, necessary and wasteful routing, or justified abstention and fluent fabrication. Yet DPO cannot recover absent evidence, distinguish factuality from style without appropriate labels, or eliminate distribution-shift concerns. Synthetic and curated instruction data offer complementary strategies. [*Self-Instruct: Aligning Language Models with Self-Generated Instructions*](https://arxiv.org/abs/2212.10560) demonstrates that a capable model can expand a small seed set into a much larger instruction collection, subject to filtering. For this thesis, synthetic traces could cover query planning, evidence conflict, abstention, and malicious context. Mechanical checks should validate document IDs, quotations, timestamps, tool results, and citation bindings. Conversely, [*LIMA: Less Is More for Alignment*](https://arxiv.org/abs/2305.11206) supports a small, carefully curated baseline. A gold set of high-quality orchestration traces may be more useful initially than a large collection of noisy synthetic agent conversations. Collectively, this literature supports behavioral improvement but not robust objective validity. Sycophancy, jailbreaks, reward tampering, persistent deceptive behaviors, and alignment-faking constructions in the broader screened corpus qualify conclusions based on ordinary preference tests. The architectural consequence is to combine learned behavior with external enforcement rather than asking model post-training to bear the entire security burden. ### 3.2 Retrieval-augmented generation and evidence governance Retrieval-augmented language models combine parametric generation with a non-parametric corpus. [*REALM: Retrieval-Augmented Language Model Pre-Training*](https://arxiv.org/abs/2002.08909) integrates retrieval into pretraining, while [*Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks*](https://arxiv.org/abs/2005.11401) provides a canonical retriever–generator formulation. [*Dense Passage Retrieval for Open-Domain Question Answering*](https://aclanthology.org/2020.emnlp-main.550/) supplies an influential dense retrieval approach. [*Improving Language Models by Retrieving from Trillions of Tokens*](https://arxiv.org/abs/2112.04426), [*Atlas: Few-shot Learning with Retrieval Augmented Language Models*](https://arxiv.org/abs/2208.03299), and [*REPLUG: Retrieval-Augmented Black-Box Language Models*](https://arxiv.org/abs/2301.12652) explore different balances of retrieved memory, model training, and modularity. The reviewed work supports several retrieval patterns: learned dense retrieval, joint retriever–generator training, frozen-model retrieval, dynamic retrieval during generation, corrective fallback, graph-based memory, and domain adaptation under distractors. [*Active Retrieval Augmented Generation*](https://arxiv.org/abs/2305.06983), [*Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection*](https://arxiv.org/abs/2310.11511), and [*Corrective Retrieval Augmented Generation*](https://arxiv.org/abs/2401.15884) motivate retrieving selectively or correcting low-quality retrieval rather than using one fixed retrieval pass. These methods do not make retrieval an oracle. Four failure stages must be separated: 1. **Corpus failure:** the evidence collection omits relevant information or contains stale, poisoned, low-authority, or contradictory content. 2. **Retrieval failure:** relevant evidence exists but is not returned or ranked high enough. 3. **Utilization failure:** relevant evidence appears in context but the model ignores or misreads it. 4. **Generation and attribution failure:** output claims are unsupported, overbroad, contradictory, or attached to non-entailing citations. Long-context models do not remove these stages. [*Lost in the Middle: How Language Models Use Long Contexts*](https://arxiv.org/abs/2307.03172) motivates testing evidence position and distractor load rather than assuming that accessible text will be used. Evidence benchmarks and evaluators help decompose performance. [*FEVER: a Large-scale Dataset for Fact Extraction and VERification*](https://aclanthology.org/N18-1074/) combines factual labels with evidence. [*KILT: a Benchmark for Knowledge Intensive Language Tasks*](https://aclanthology.org/2021.naacl-main.200/) standardizes provenance-aware knowledge-intensive evaluation. [*ALCE: Enabling Large Language Models to Generate Text with Citations*](https://arxiv.org/abs/2305.14627) distinguishes citation correctness and completeness. [*RAGAS: Automated Evaluation of Retrieval Augmented Generation*](https://arxiv.org/abs/2309.15217), [*ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems*](https://arxiv.org/abs/2311.09476), [*RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models*](https://arxiv.org/abs/2401.00396), and [*RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation*](https://arxiv.org/abs/2408.08067) motivate component-level diagnosis. The proposed evidence store extends these ideas with operational governance. Every retrieved passage will be linked to an immutable snapshot, source identity, retrieval timestamp, authority classification, freshness metadata, tenant or domain access controls, and contradiction signals. Search-provider snippets are discovery evidence rather than authoritative support. If sufficient evidence cannot be retrieved, the system must represent that state explicitly rather than silently substituting parametric memory. ### 3.3 Hallucination, factuality, and atomic evaluation “Hallucination” is often used to describe several distinct errors. For experimental clarity, this thesis separates retrieval omission, contextual contradiction, unsupported addition, world-factual error, attribution error, temporal error, and epistemic overclaim. [*TruthfulQA: Measuring How Models Mimic Human Falsehoods*](https://aclanthology.org/2022.acl-long.229/) demonstrates that ordinary scale does not necessarily remove misconceptions learned from human-generated text. This matters because capability improvements can coexist with confidently repeated falsehoods. [*SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models*](https://aclanthology.org/2023.emnlp-main.557/) uses disagreement among samples as a signal of hallucination risk. The method is useful for detecting instability, but stable false beliefs can evade it. [*FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation*](https://aclanthology.org/2023.emnlp-main.741/) motivates decomposing long-form output into atomic propositions. This unit is central to the proposed system. Each externally verifiable atomic claim receives a claim identifier, evidence identifiers, a support status, and, where appropriate, temporal and authority requirements. Atomicity is not absolute: decomposition models may split too finely, combine multiple propositions, or omit implied claims. A human-audited sample will estimate these errors. [*RARR: Researching and Revising What Language Models Say, Using Language Models*](https://arxiv.org/abs/2210.08726) and Self-RAG show that retrieval and critique can revise unsupported content. Tool-augmented evaluators can broaden coverage, but they inherit search and judge error. Automated factuality scores will therefore not be treated as ground truth without validation. The thesis adopts selective prediction rather than an always-answer objective. [*Language Models (Mostly) Know What They Know*](https://arxiv.org/abs/2207.05221) and [*Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation*](https://arxiv.org/abs/2302.09664) motivate calibration and uncertainty estimation. Theoretical work represented by [*Calibrated Language Models Must Hallucinate*](https://arxiv.org/abs/2311.14648) further cautions against universal zero-hallucination claims under broad distributions. Confidence is therefore an input to a decision rule, not evidence in itself. The evaluation will emphasize risk–coverage curves. Coverage is the proportion of tasks for which the system provides a substantive answer or executes an action. Selective risk is error among those covered tasks. For high-impact actions, error severity must also enter the objective. A system that abstains appropriately under absent or contradictory evidence may have lower coverage but lower expected loss. A system that refuses answerable benign tasks has poor utility and will be penalized through unnecessary-refusal metrics. ### 3.4 Sparse mixture-of-experts [*Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*](https://arxiv.org/abs/1701.06538) established large-scale sparse conditional computation using learned top-\(k\) routing, load-balancing objectives, and distributed expert computation. The paper supports a constant-active-compute argument: sparsely accessed parameters can improve predictive quality without executing the full parameter set for every input. It does not establish that experts become reliable factual modules or aligned roles. [*Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity*](https://jmlr.org/papers/v23/21-0998.html) simplifies routing to top-1 expert selection and reports large-scale training results. Its practical lessons include capacity factors, overflow behavior, load balancing, communication topology, and selective high-precision router calculations. Total parameter count remains a poor proxy for per-token latency or active capability. Later work in the reviewed corpus—such as [*GLaM: Efficient Scaling of Language Models with Mixture-of-Experts*](https://arxiv.org/abs/2112.06905), [*ST-MoE: Designing Stable and Transferable Sparse Expert Models*](https://arxiv.org/abs/2202.08906), [*DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models*](https://arxiv.org/abs/2401.06066), [*MegaBlocks: Efficient Sparse Training with Mixture-of-Experts*](https://arxiv.org/abs/2211.15841), and [*OLMoE: Open Mixture-of-Experts Language Models*](https://arxiv.org/abs/2409.02060)—extends sparse-model design and implementation. The relevant risks include router collapse, load imbalance, expert overflow, all-to-all communication, unstable specialization, and comparisons that report total rather than active parameters. A nominal 200-billion-parameter sparse backbone is therefore treated as a scaling design rather than a necessary thesis implementation. Unless matching compute is available, the prototype should use an open smaller sparse model or dense models in the approximately 7–14B range. Any sparse experiment must report total parameters, active parameters per token, expert count and size, top-\(k\), capacity factor, overflow or dropped-token behavior, routing entropy, load balance, communication cost, latency, and energy. No reliability conclusion will be attributed to sparsity without a matched-active-compute comparison. ### 3.5 Mixture-of-agents, debate, and test-time scaling [*Improving Factuality and Reasoning in Language Models through Multiagent Debate*](https://arxiv.org/abs/2305.14325) uses multiple model instances that generate answers, inspect other agents’ outputs, and revise over several rounds. The reported results support multi-sample revision in selected arithmetic, reasoning, chess, question-answering, and biography tasks. They do not show that consensus is truth. Related agents can share priors and blind spots, while repeated cross-conditioning can reduce independence and create groupthink. [*Mixture-of-Agents Enhances Large Language Model Capabilities*](https://arxiv.org/abs/2406.04692) organizes several proposer models into layers followed by an aggregator. The reported benchmark gains motivate layered synthesis as a test-time scaling pattern, but comparisons are affected by model-judge reliance, changing API prices, context expansion, and uncertain epistemic diversity. A final aggregator can launder an unsupported statement into polished prose unless provenance is preserved. [*AI Safety via Debate*](https://arxiv.org/abs/1805.00899) presents debate as a scalable-oversight proposal. Its promise depends on decomposability, judge competence, protocol incentives, and resistance to collusion or shared blind spots. Empirical multi-agent debate should not be treated as a completed demonstration of this stronger oversight claim. Multi-agent gains may arise from spending more tokens, not from meaningful specialization. Appropriate comparators include [*Self-Consistency Improves Chain of Thought Reasoning in Language Models*](https://arxiv.org/abs/2203.11171), [*Tree of Thoughts: Deliberate Problem Solving with Large Language Models*](https://arxiv.org/abs/2305.10601), best-of-\(N\) with a verifier, model cascades, a stronger single model, and deterministic workflows. [*Agentless: Demystifying LLM-based Software Engineering Agents*](https://arxiv.org/abs/2407.01489) is an especially important conceptual baseline: complex agent scaffolding must outperform a simpler domain pipeline before its complexity is justified. The proposed system adds an agent only when that call supplies a distinct model family, retrieval path, tool, permission boundary, or independently measured error profile. Multiple samples from one checkpoint supply sampling diversity, not model independence. Debate is an optional escalation mechanism below executable or authoritative evidence in the assurance hierarchy. ### 3.6 Oversight and verification [*Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision*](https://arxiv.org/abs/2312.09390) provides an empirical proxy for supervising stronger models with weaker labels. Its results indicate partial but incomplete recovery, with uneven performance across task families and particularly important limitations in reward modeling. This cautions against assuming that a smaller or related language-model judge can reliably supervise a substantially stronger generator. [*Let’s Verify Step by Step*](https://arxiv.org/abs/2305.20050) compares process and outcome reward models in mathematical problem solving and reports stronger selection performance from step-level process supervision in its studied setting. The paper supports separating generation from verification and evaluating intermediate units. Its results are bounded by a domain with checkable answers, expensive candidate sampling, learned-verifier fallibility, and questions about whether written reasoning faithfully reflects internal computation. The thesis generalizes the process-supervision lesson to an external structured trace: \[ \text{claim or action} \rightarrow \text{evidence or precondition} \rightarrow \text{verification} \rightarrow \text{decision} \rightarrow \text{observed state}. \] This is not a request for unrestricted hidden chain-of-thought. The production artifact is a concise, inspectable record of claims, citations, proposed operations, tool results, and state transitions. The proposed verification hierarchy is: 1. formal proof or deterministic invariant; 2. executable test against controlled state; 3. authoritative immutable primary evidence plus entailment; 4. independent specialist learned verifier; 5. heterogeneous critique or debate; 6. generator self-critique, confidence, or same-model consensus. The hierarchy is conditional on applicability. A compiler can test syntax and a test suite can test specified behavior, but neither proves that requirements are complete. An authoritative document may establish what a policy says without establishing whether the policy is ethical. Higher-ranked evidence is stronger only for the proposition it actually checks. Lower-level agreement should not override a failed applicable higher-level check. ### 3.7 Tool use, agent benchmarks, and security [*ReAct: Synergizing Reasoning and Acting in Language Models*](https://arxiv.org/abs/2210.03629) and [*Toolformer: Language Models Can Teach Themselves to Use Tools*](https://arxiv.org/abs/2302.04761) provide influential patterns for interleaving language-model generation and external tools. Function-calling and API-oriented work extends these interfaces. Yet conversational competence does not imply reliable state-changing action. Agent benchmarks in the artifacts include [*GAIA: a benchmark for General AI Assistants*](https://arxiv.org/abs/2311.12983), [*WebArena: A Realistic Web Environment for Building Autonomous Agents*](https://arxiv.org/abs/2307.13854), [*WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?*](https://arxiv.org/abs/2303.13187), [*OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments*](https://arxiv.org/abs/2404.07972), [*SWE-bench: Can Language Models Resolve Real-World GitHub Issues?*](https://arxiv.org/abs/2310.06770), [*τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains*](https://arxiv.org/abs/2406.12045), and [*ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities*](https://arxiv.org/abs/2408.04682). These environments motivate evaluating observed state completion, policy compliance, and interaction trajectories rather than judging final prose alone. Retrieval and tools also create security channels. [*Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection*](https://arxiv.org/abs/2302.12173) demonstrates the danger of untrusted content that contains instructions. [*AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents*](https://arxiv.org/abs/2406.13352), [*InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents*](https://arxiv.org/abs/2403.02691), [*StruQ: Defending Against Prompt Injection with Structured Queries*](https://arxiv.org/abs/2402.06363), and [*PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models*](https://arxiv.org/abs/2402.07867) motivate explicit trust channels, corpus controls, and attack evaluation. Prompt-only defenses are insufficient as the sole control for consequential actions. Required runtime mechanisms include typed schemas, argument validation, least privilege, secret isolation, read/write separation, sandboxing, allowlists, idempotency, dry runs, visible state diffs, confirmation thresholds, transaction boundaries, postcondition checks, and rollback. The language model proposes; an external gateway authorizes and executes. --- ## 4. Proposed Architecture ### 4.1 Design principles The architecture follows six principles: 1. **Authority is explicit.** Untrusted content cannot acquire policy status through natural-language persuasion. 2. **Evidence is governed.** Retrieved material retains source identity, version, access rights, authority, and time. 3. **Outputs are decomposable.** Claims and actions are represented as checkable units. 4. **Verification is object-specific.** The checker is selected according to what must be established. 5. **Compute is risk-adaptive.** Additional retrieval, agents, or debate are invoked only when expected value justifies them. 6. **Side effects are externally constrained.** Model outputs do not directly confer permission. A conceptual pipeline is: ```text Request ↓ Instruction-authority and risk gate ↓ Typed planner ↓ Governed retrieval → immutable evidence bundle ↓ Proposer → atomic claims and/or explicit action proposal ↓ Verification portfolio ↓ Decision controller ├─ approve ├─ revise and reverify ├─ retrieve more ├─ ask for clarification ├─ qualify ├─ abstain or refuse └─ escalate ↓ External action gateway, if an action is authorized ↓ Postcondition check, audit record, and rollback/compensation if needed ``` ### 4.2 Sparse backbone The backbone supplies language understanding and generation capacity. The thesis does not require a 200B model to validate the reliability mechanisms. A smaller open dense or sparse model will be used unless suitable compute exists. A 200B sparse configuration remains a scaling design. For every backbone, the study will report checkpoint identity, context length, inference precision, decoding settings, and post-training status. Sparse models additionally require total and active parameter counts, expert configuration, routing, capacity, overflow, communication, and routing telemetry. Backbone selection will not determine permissions. A stronger model may propose better plans but receives no greater authority merely because of benchmark performance or total parameters. ### 4.3 Policy and risk gate The gate classifies: - instruction authority; - task type; - evidence and freshness requirements; - action reversibility; - financial, privacy, security, and operational impact; - allowed data domains; - available tools; - required verification level; - confirmation or escalation threshold. The input is partitioned into typed channels such as policy, operator instruction, user request, retrieved evidence, tool observation, and prior state. The model may help classify ambiguous content, but an external policy engine enforces hard constraints. Retrieved text and tool output cannot modify capability grants. Risk levels may be ordinal—for example, informational, low-impact reversible, moderate-impact, high-impact reversible, and irreversible or safety-critical. The exact taxonomy will be pre-registered. Risk classification errors will be evaluated separately because underclassification can suppress necessary verification while overclassification can create excessive refusal and cost. ### 4.4 Typed planner The planner produces a structured plan with: - subgoal; - required evidence; - proposed tool; - expected observation schema; - preconditions; - validation rule; - predicted state change; - rollback or compensation method; - stop condition. Natural-language explanation may accompany these fields but cannot substitute for them. Invalid plans are rejected by schema validation. Plans that request unauthorized tools or domains are blocked before execution. The planner also labels which steps are purely epistemic and which can change state. This separation permits liberal use of read-only tools while maintaining stricter controls on writes. ### 4.5 Governed retrieval subsystem The retrieval subsystem combines lexical and dense retrieval where appropriate. Before retrieval, access-control filters restrict the searchable corpus by tenant, role, jurisdiction, or data domain. Results include: - source and passage identifiers; - immutable content hash or snapshot; - publication and retrieval times; - source-authority classification; - freshness or expiry status; - duplicate cluster; - contradiction indicators; - poisoning or anomaly signals; - retrieval score and rank; - query and query-rewrite provenance. The system supports an explicit `INSUFFICIENT_EVIDENCE` state. This state can trigger a clarifying question, broader retrieval, qualified response, or abstention. It must not be silently replaced by an unsupported answer. For heterogeneous retrieval, separate proposer agents may receive distinct search roles: primary-source retrieval, contradiction search, temporal verification, jurisdiction-specific search, or independent query rewriting. Diversity is created through different evidence paths, not merely different persona prompts. ### 4.6 Proposer and structured candidate The proposer produces either an answer candidate, an action candidate, or both. A factual candidate contains atomic claims. Each claim has: - claim ID; - normalized proposition; - evidence IDs; - support type: direct, inferred, disputed, or unsupported; - temporal scope; - confidence estimate; - requested verifier. An action candidate contains: - tool name and typed arguments; - authorization scope requested; - preconditions; - expected state diff; - reversibility; - idempotency key; - verification and rollback methods. The proposer is not permitted to mark its own claim as verified. It may recommend a verifier or express confidence, but support status is assigned by the verification and decision layers. ### 4.7 Verification portfolio Verification is dispatched by object type: - arithmetic to a calculator; - code to parsing, compilation, static analysis, and tests; - database mutation to schema checks, invariants, and transaction dry runs; - formal statements to a proof assistant when feasible; - current factual claims to authoritative-source retrieval and entailment; - policy interpretation to deterministic rules plus a specialist classifier; - open-domain factual claims to independent retrieval and a claim checker; - ambiguous requirements to user clarification; - high-impact actions to policy checks and human confirmation. A verifier records its inputs, evidence, result, confidence where relevant, and failure mode. Learned verifiers are calibrated on held-out and adversarial traces. Where possible, the verifier uses a different checkpoint or model family and does not receive irrelevant proposer identity. Executable checks are retained as artifacts. Debate may be invoked if sources conflict, a verifier and proposer disagree, or no deterministic checker applies. Debate outputs remain advisory. Minority reports and unresolved objections are preserved rather than erased through forced consensus. ### 4.8 Decision and abstention controller The controller selects among: - approve answer; - approve action; - revise and reverify; - retrieve additional evidence; - ask user for missing information; - provide a qualified or partial answer; - abstain because evidence is insufficient; - refuse because the request conflicts with policy; - escalate to a human reviewer. Decision thresholds optimize severity-weighted selective risk subject to coverage, latency, and cost constraints. Calibration data are separate from final evaluation data. The controller must distinguish refusal from epistemic abstention: the former concerns policy or authorization, while the latter concerns insufficient evidence or confidence. A conceptual loss is: \[ L = \lambda_f L_{\text{factual}} + \lambda_a L_{\text{action}} + \lambda_s L_{\text{security}} + \lambda_r L_{\text{unnecessary refusal}} + \lambda_c C_{\text{compute}} + \lambda_t C_{\text{latency}}, \] where weights are pre-registered by risk stratum rather than tuned after observing final results. This expression is a design abstraction, not a claim that all harms can be reduced to one universally valid scalar. ### 4.9 External action gateway All state-changing tool calls pass through a non-model authorization layer. The gateway enforces: - capability tokens; - least privilege; - typed tool schemas; - argument validation; - secret isolation; - network and filesystem allowlists; - read/write separation; - dry-run support; - visible state diffs; - rate, cost, and horizon limits; - idempotency; - transaction boundaries; - confirmation for irreversible or high-impact effects; - postcondition checks; - rollback or compensation where supported. Secrets are never inserted into model-visible context unless strictly necessary, and then only through narrowly scoped mediation. The gateway logs attempted unauthorized calls as safety events even when blocked. Post-action verification compares the observed state with the predicted state diff. A successful API return code is not sufficient if the intended state was not achieved. Conversely, a task may appear incomplete in conversation while the external state has already changed; audit and idempotency controls are therefore essential. ### 4.10 Audit trace Each run retains: - request and authority labels; - risk classification; - plan versions; - retrieval queries and immutable evidence IDs; - proposer claims and actions; - verifier results; - controller decisions; - authorized and blocked tool calls; - state diffs; - postconditions; - cost and latency; - human interventions. Audit records should minimize sensitive data and comply with retention requirements. Their scientific purpose is failure localization and reproducibility, not unrestricted surveillance. --- ## 5. Methodology ### 5.1 Research design The study will use a controlled comparative evaluation with pre-registered hypotheses, primary outcomes, stopping rules, and analysis procedures. It will include benchmark tasks, custom controlled tasks, adversarial security cases, and selected stateful agent environments. The independent variable is system configuration. The primary dependent variables are severe unsupported-claim rate, unsafe-action rate, severity-weighted expected loss, selective risk, task coverage, benign utility, and cost. Secondary variables include retrieval recall, citation quality, verifier error, latency, and routing telemetry. No experimental result is available at manuscript drafting time. All statements in this section describe planned methods. ### 5.2 System variants and baselines The principal variants are: 1. single base model without retrieval; 2. single model with standard top-\(k\) retrieval; 3. retrieval plus self-critique; 4. self-consistency or majority vote; 5. best-of-\(N\) with one learned verifier; 6. tree- or search-based reasoning; 7. homogeneous multi-agent debate; 8. published layered mixture-of-agents pattern; 9. deterministic or agentless domain workflow; 10. full proposed architecture. Where feasible, one stronger single model will be included to test whether orchestration merely compensates for a weaker backbone. Comparisons will use two regimes: - **Equal budget:** matched generated tokens and, where applicable, matched retrieval and tool-call budgets. - **Best achievable:** each method receives a predeclared practical resource ceiling and is tuned for its best performance. Equal-budget comparisons estimate architectural efficiency. Best-achievable comparisons estimate practical upper performance. Reporting only one regime would be incomplete. ### 5.3 Task sampling and stratification Tasks will be stratified by: - domain; - answerability; - evidence freshness; - evidence conflict; - risk level; - reversibility; - task horizon; - retrieval difficulty; - attack type; - prompt language where supported; - in-distribution versus shifted status. Randomization will assign tasks and prompt variants to system conditions where the environment permits. Stateful tasks will begin from reproducible snapshots. Model and retrieval caches will be controlled to avoid accidental information leakage between conditions. ### 5.4 Evidence corpora Factuality experiments will use frozen evidence snapshots. Each snapshot will include source metadata and stable passage identifiers. Controlled variants will manipulate: - whether decisive evidence exists; - retrieval rank; - distractor count; - evidence position; - freshness; - source authority; - contradictory sources; - poisoned documents; - malicious instructions embedded in otherwise relevant content. This factorial design helps localize failures. For example, if the relevant passage is retrieved but the answer remains unsupported, the failure lies downstream of retrieval recall. If a system cites a poisoned passage over a higher-authority source, authority scoring or contradiction handling has failed. ### 5.5 Atomic annotation Generated factual output will be decomposed into atomic claims. Automated decomposition may be used at scale, but a blinded human sample will assess split quality, omitted claims, and proposition validity. Each claim will be labeled as: - directly supported; - supported only by inference; - contradicted; - unsupported despite sufficient evidence; - unsupported because evidence is insufficient; - temporally invalid; - not externally verifiable; - decomposition error. Citation correctness asks whether the cited passage entails the claim. Citation completeness asks whether externally verifiable claims have adequate citations. Source quality and freshness are recorded separately. A real and entailing citation may still be low-authority or stale. ### 5.6 Action annotation Agent tasks will be evaluated against external state and policy. Events include: - authorized correct action; - authorized but incorrect action; - unauthorized attempt blocked by gateway; - unauthorized action executed; - irreversible effect; - recoverable near miss; - confirmation bypass; - secret-access attempt; - cross-tenant access attempt; - incorrect or missing postcondition; - unnecessary refusal or escalation. Blocked attempts remain important because they reveal model behavior and gateway effectiveness. Executed unauthorized actions are more severe and receive greater loss weights. ### 5.7 Calibration Uncertainty features may include model confidence, sample disagreement, semantic uncertainty, evidence sufficiency, verifier agreement, retrieval scores, contradiction signals, and risk class. A calibration model will be fitted on held-out data. Thresholds will be selected to achieve target coverage or expected-loss constraints. Evaluation will report risk at matched coverage rather than comparing systems at their naturally chosen answer rates. Calibration will be retested under domain and attack shift without threshold refitting, followed by a clearly separated adaptation experiment if desired. ### 5.8 Human evaluation Human reviewers will evaluate a statistically justified subset of factuality, policy, and safety cases. Reviewers will be blinded to system condition where presentation allows. Domain experts will be used for specialized claims when non-expert judgment is inadequate. Inter-rater agreement and adjudication procedures will be reported. Human preference will not substitute for factual support. Review forms will separate helpfulness, correctness, evidence support, citation completeness, uncertainty communication, policy compliance, and action appropriateness. ### 5.9 Automated judges Language-model judges may support large-scale screening but will be validated against human labels. Judge identity, prompt, temperature, evidence visibility, and ordering will be fixed and disclosed. Position, verbosity, and model-family affinity will be tested where feasible. Automated judgments will not be treated as independent if the judge is related to the generator. Generator–judge agreement and joint error will be estimated on the audited sample. ### 5.10 Statistical analysis Task-level bootstrap confidence intervals will be reported for principal metrics. Paired significance tests will be used when systems are evaluated on the same tasks. Effect sizes will accompany \(p\)-values. For binary outcomes such as attack success, intervals appropriate to proportions will be reported. For rare severe events, aggregate rates may be unstable; exact counts, exposure denominators, and severity categories will therefore be included. Multiplicity will be addressed by identifying primary and secondary outcomes before evaluation. Exploratory analyses will be labeled as such. Subgroup conclusions will require adequate sample sizes and will not be generalized beyond their tested strata. --- ## 6. Experimental Plan ### 6.1 Phase I: Evaluation harness and frozen evidence store The first phase will implement reproducible task execution, model-call accounting, retrieval logging, immutable evidence snapshots, and claim/action traces. This phase precedes the full architecture because instrumentation is necessary to distinguish genuine reliability improvement from scoring artifacts. Deliverables include: - versioned task registry; - frozen prompt and corpus snapshots; - cost and latency accounting; - state-reset mechanisms; - claim annotation interface; - attack-event logging; - human-audit workflow. ### 6.2 Phase II: Single-model and RAG baselines The second phase establishes base-model and standard retrieval performance. Retrieval evaluation will separate corpus availability, recall, utilization, and attribution. Initial suites include FEVER- or KILT-style evidence tasks, TruthfulQA, long-form FActScore-style evaluation, ALCE-style citation assessment, and custom time-sensitive or contradictory-source sets. The purpose is not to reproduce every published number. It is to establish controlled local baselines under frozen checkpoints, prompts, and evidence. ### 6.3 Phase III: Atomic decomposition and verification The third phase adds structured claims and object-specific checks. Experiments compare: - whole-answer evaluation versus atomic claims; - no decomposition versus decomposition; - proposer self-check versus independent claim checker; - learned entailment verifier versus human judgment; - learned checker versus executable checker in applicable domains. For code, arithmetic, and structured data, executable verification will be emphasized. The study will record when a test passes despite incomplete requirements, illustrating the bounded scope of executable evidence. ### 6.4 Phase IV: Selective abstention The fourth phase calibrates answer, qualification, clarification, and abstention decisions. Evaluation includes: - answerable versus unanswerable tasks; - missing evidence; - contradictory evidence; - stale evidence; - shifted domains; - poisoned retrieval; - high- versus low-risk requests. Primary plots will show risk–coverage curves and severity-weighted loss–coverage curves. Unnecessary refusal will be reported to prevent apparent safety gains from being purchased solely through non-participation. ### 6.5 Phase V: Policy and action gateway The fifth phase introduces typed trust channels and external permission enforcement. Security tasks include: - indirect prompt injection; - malicious retrieved documents; - malicious tool output or API documentation; - cross-tenant requests; - secret exfiltration attempts; - unsafe argument substitution; - irreversible actions; - confirmation bypass; - long-context or many-shot attacks. Conditions compare prompt-only defense, structured trust channels without external enforcement, and full gateway enforcement. Metrics include attack success, blocked unauthorized attempts, benign utility, latency, and false-positive blocking. ### 6.6 Phase VI: Heterogeneous verification The sixth phase introduces one genuinely heterogeneous verifier before testing larger multi-agent topologies. Heterogeneity conditions may include: - same checkpoint with different prompt; - same family with independent sampling; - different checkpoint; - different model family; - independent retrieval index; - specialized learned verifier; - executable tool. The central analysis estimates not only individual error but error correlation with the proposer. A weaker verifier with partially independent errors may provide more marginal value than a stronger but highly correlated critic. ### 6.7 Phase VII: MoA and debate Only after simpler mechanisms are established will the study add layered mixture-of-agents and debate. Conditions vary: - number of agents; - rounds; - model homogeneity; - evidence sharing; - retrieval-query diversity; - forced consensus versus preserved minority report; - final aggregator identity; - adaptive versus always-on invocation. Equal-token comparisons with self-consistency, best-of-\(N\), tree search, and a deterministic workflow are mandatory. The analysis will test whether improvement comes from conversational interaction, evidence diversity, candidate count, or merely increased computation. ### 6.8 Phase VIII: Sparse versus dense backbone The final phase, resources permitting, compares dense and sparse backbones at matched active compute. The analysis includes capability, factuality, safety, routing entropy, expert utilization, overflow, p50 and p95 latency, energy, and communication overhead. The expected finding is not presupposed. H7 will be supported only if sparse capacity improves capability without a residual reliability gain after controlling for capability and pipeline configuration. If particular experts correlate with safer or more grounded behavior, that observation will remain exploratory unless causal routing interventions support it. ### 6.9 Evaluation suites #### Instruction following and conflicts - [*Instruction-Following Evaluation for Large Language Models*](https://arxiv.org/abs/2311.07911) / IFEval-style constraints; - [*FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models*](https://arxiv.org/abs/2310.20410); - instruction-hierarchy conflict tests; - custom system/user/untrusted-content conflicts; - long-context and multilingual variants where supported. #### Retrieval and factuality - FEVER- and KILT-style evidence tasks; - TruthfulQA; - FActScore-style long-form atomic evaluation; - ALCE-style citation correctness and completeness; - RAGTruth and component diagnostics; - time-sensitive and contradictory-source tasks with frozen evidence. #### Agent and tool tasks - GAIA; - τ-bench; - ToolSandbox or MINT-style interaction; - SWE-bench Verified or CORE-Bench where operationally feasible; - a maintainable WebArena or WorkArena subset; - ScienceAgentBench-style research workflows. Benchmark use will depend on reproducibility, licensing, and available infrastructure. Substitution will be documented rather than presented as an exact replication. #### Security - AgentDojo- or InjecAgent-style indirect injection; - poisoned retrieval corpora; - malicious tool observations; - cross-tenant and secret-access attempts; - unsafe argument substitution; - confirmation bypass; - long-context and many-shot attacks; - adaptive attacks developed after observing fixed defenses, with a protected final holdout. ### 6.10 Metrics #### Capability - task success; - exact match or F1 where appropriate; - verified state completion; - instruction-constraint satisfaction. #### Retrieval and grounding - retrieval recall; - evidence sufficiency; - source authority and freshness compliance; - atomic supported-claim precision; - citation correctness; - citation completeness; - contradiction and omission rate; - unsupported claims per 1,000 claims. #### Selective prediction - coverage; - selective risk; - area under the risk–coverage curve; - Brier score; - expected calibration error; - abstention appropriateness; - unnecessary refusal. #### Safety and security - attack success rate; - benign utility under defense; - policy violation; - unauthorized tool attempts; - unauthorized actions executed; - irreversible effects and near misses; - privacy leakage; - severity-weighted expected loss. #### Systems - total and active parameters; - generated tokens; - retrieval and tool calls; - p50 and p95 latency; - GPU time; - energy and monetary cost; - retries and controller loops; - expert overflow and load imbalance. #### Oversight - verifier false-positive and false-negative rates; - generator–verifier joint error; - error correlation across agents; - human escalation rate; - human reversal rate; - cost per prevented severe error. ### 6.11 Required ablations The full architecture will be ablated by removing: - retrieval; - claim decomposition; - independent checker; - source authority and freshness scoring; - abstention; - policy gate; - external permission enforcement; - postcondition verification; - heterogeneous agent or verifier; - adaptive routing. Additional contrasts include: - self-check versus independent check; - same-model versus different-model verifier; - learned verifier versus executable tool; - one agent versus multiple agents; - prompt-only versus externally enforced permissions; - clean versus poisoned corpus; - low versus high distractor count; - early, middle, and late evidence position; - dense versus sparse backbone at matched active compute. ### 6.12 Success and failure criteria A strong result requires a statistically credible reduction in severe unsupported claims and unsafe actions at matched coverage and compute, with a practically acceptable loss in latency and benign utility, across held-out domains and adaptive attacks. The central claim is weakened or falsified if: - equal-compute simple baselines match the full architecture; - gains disappear under human audit; - verifiers mostly duplicate generator errors; - atomic decomposition introduces enough errors to offset its benefit; - abstention cannot remain calibrated under shift; - external controls block benign tasks without reducing adaptive attack success; - multi-agent gains are fully explained by additional token count; - sparse capacity is reported as a reliability gain without matched controls. Null or negative results will be reported. Such outcomes remain informative because they identify unnecessary architectural complexity and invalid assumptions about independence or verification. --- ## 7. Threats to Validity ### 7.1 Construct validity Reliability is multidimensional. Task success, factual support, policy compliance, security, and cost cannot be compressed into one universally meaningful score. Severity weights may reflect contestable judgments. The study therefore reports disaggregated outcomes and conducts sensitivity analysis over plausible severity schemes. Atomic supported-claim precision does not capture omissions, misleading framing, or harmful but technically true statements. Citation entailment does not establish source authority. Successful tests do not prove complete requirements. These metrics are useful only when interpreted within their scope. “Agent,” “expert,” “verification,” and “alignment” are especially overloaded terms. This manuscript reserves **expert** for a routed neural subnetwork and **agent** for a complete model invocation embedded in a workflow. Learned checking is described as probabilistic verification, not formal proof. Alignment claims are bounded to measured behavior under selected policies and distributions. ### 7.2 Internal validity Component interactions can confound ablations. Removing claim decomposition may also reduce token use; adding verification may change proposer behavior if feedback is visible. Each ablation will therefore report compute and prompt changes, and budget-matched controls will be used where possible. Model APIs and checkpoints can change. Frozen open checkpoints are preferred. If proprietary systems are included, model version and access date will be logged, and conclusions will be qualified. Adaptive routing may preferentially allocate more compute to harder tasks, making raw averages difficult to interpret. Task-level paired comparisons and difficulty strata will be used. ### 7.3 External validity Benchmarks are narrower than deployment. Static datasets may be contaminated by pretraining, while simulated environments may omit real organizational constraints. Results will not be generalized to all domains, languages, or risk settings. A system that works on short, verifiable tasks may fail on long-horizon objectives with delayed consequences. Web, software, and research tasks broaden scope but remain partial proxies. The manuscript will state the tested horizon and environment explicitly. The proposed full architecture has not been validated end to end at 200B scale in the supplied literature. Smaller-model findings may not transfer to larger sparse systems. Conversely, larger models may change both capability and attack surface. ### 7.4 Statistical conclusion validity Severe failures may be rare, producing wide confidence intervals. Adequate attack exposure and exact event counts are therefore necessary. Multiple benchmarks and ablations raise multiplicity concerns; primary outcomes will be pre-registered. Language-model judges can produce correlated measurement error. Human auditing reduces but does not eliminate subjectivity. Expert review is costly and may limit sample size. ### 7.5 Retrieval validity Frozen corpora improve reproducibility but reduce realism for continuously changing information. Live search improves freshness but makes exact replication difficult. The study will separate frozen-corpus experiments from explicitly labeled live-retrieval evaluations. Source-authority labels may encode researcher assumptions. Contradictory authoritative sources may not be resolvable through ranking. The architecture must support disclosed disagreement rather than forcing one answer. ### 7.6 Security validity Published attacks can overfit known defenses, while undisclosed adaptive attacks may discover new weaknesses. Security results establish resistance only to tested threat models. Prompt injection is one channel; compromised tools, credential theft, dependency attacks, and human social engineering may remain outside scope. Sandboxed environments can understate consequences. Conversely, synthetic attacks may overstate practical exposure if they assume unrealistic attacker access. Threat-model assumptions will be documented. ### 7.7 Systems validity Equal token budgets do not guarantee equal hardware cost. Different models, context lengths, retrieval systems, and tool calls have different latency and energy profiles. Both abstract budgets and measured end-to-end systems metrics are required. Sparse models are particularly sensitive to interconnect and batching. Results from one hardware topology may not transfer to another. --- ## 8. Ethics and Safety The thesis concerns systems that can retrieve sensitive information and change external state. Evaluation must avoid creating unnecessary harm. ### 8.1 Controlled environments State-changing experiments will use sandboxes, test accounts, synthetic data, or resettable benchmark environments. Real financial, medical, legal, employment, infrastructure, or production actions are excluded unless separately authorized under an appropriate review process. Irreversible-action tests will simulate consequences where possible. If a real service is required, capability scopes and spending limits will be minimal. ### 8.2 Data governance and privacy Corpora and logs may contain personal or confidential information. Retrieval will enforce tenant and domain access controls before ranking. Logs will minimize retained content, redact secrets, and apply retention limits. Human evaluators will receive only information necessary for the task. Sensitive examples will require additional handling procedures. The architecture’s provenance features must not become a mechanism for exposing private source material to unauthorized users. ### 8.3 Human oversight Human escalation is not treated as an unlimited safety resource. Reviewers can be overburdened, biased, or misled by persuasive model output. Escalations should include concise evidence, unresolved questions, and proposed state changes rather than long unstructured transcripts. High-impact confirmation must be meaningful. A user should see the specific action, arguments, expected effects, and reversibility, not a vague confirmation dialog designed to induce approval. ### 8.4 Dual-use security information Prompt-injection and exfiltration experiments may reveal exploitable patterns. Reporting should provide enough detail for reproducibility and defense while avoiding publication of live credentials, unpatched production vulnerabilities, or attack instructions targeted at identifiable systems. Responsible disclosure should precede public release when appropriate. ### 8.5 Labor and annotation Factuality and safety annotation can expose reviewers to harmful content. Reviewers should receive informed task descriptions, the ability to opt out, appropriate support, and fair compensation. Work should be allocated to qualified evaluators rather than assuming that any labeler can judge specialized evidence. ### 8.6 Value pluralism and refusal Policy and severity weights encode normative choices. The study will not equate one organization’s policy with universal ethics. Policy sources, conflict-resolution rules, and uncertainty will be documented. Refusal-heavy behavior will not automatically be labeled safer, because inappropriate refusal can deny legitimate assistance and distribute costs unevenly. ### 8.7 Environmental cost Always-on multi-agent inference can multiply token, latency, and energy use. Adaptive routing is therefore both a systems and ethical objective. The study will report resource consumption and avoid large-scale sparse experiments until smaller prototypes establish that the reliability mechanism is worth scaling. ### 8.8 Residual-risk communication The architecture must not be marketed as eliminating hallucination or guaranteeing safety. Results will identify the tested distribution, threat model, confidence intervals, and known residual failures. Blocked attacks should not be described as proof against all adaptive attacks. --- ## 9. Expected Contributions Because experiments have not yet been run, the following are expected rather than demonstrated contributions. ### 9.1 A precise conceptual separation of scaling mechanisms The thesis will distinguish sparse mixture-of-experts capacity from inference-time mixture-of-agents orchestration in architecture, metrics, and experiments. This separation prevents evidence about computational scaling from being misapplied to agent reliability. ### 9.2 A defense-in-depth reference architecture The work will specify an implementable pipeline joining instruction authority, governed retrieval, typed planning, atomic decomposition, heterogeneous verification, selective decision-making, and externally constrained action. Its value lies in explicit interfaces and falsifiable responsibilities rather than a claim of novelty for every component. ### 9.3 A unified claim-and-action trace The structured trace links claims and actions to evidence, preconditions, checks, decisions, and observed state. This extends atomic factual evaluation and process supervision into tool-using systems while avoiding reliance on unrestricted hidden reasoning. ### 9.4 An equal-budget evaluation of orchestration The study will compare multi-agent systems with self-consistency, best-of-\(N\), search, stronger single models, and deterministic workflows under matched inference budgets. This can clarify whether apparent MoA gains arise from architecture, evidence diversity, or raw test-time scaling. ### 9.5 Measurement of correlated verification error Rather than treating multiple model outputs as independent votes, the study will measure generator–verifier and agent–agent joint errors. This can operationalize heterogeneity as marginal error reduction rather than persona count. ### 9.6 Selective-reliability evaluation Risk–coverage and loss–coverage analysis will connect evidence sufficiency, uncertainty, abstention, and action risk. The resulting evaluation should reveal when the architecture appropriately declines or escalates and when it merely becomes overcautious. ### 9.7 Security evaluation of trust and action boundaries Prompt-only, structured-channel, and externally enforced permission conditions will be compared under both attack success and benign utility. This can quantify the contribution and cost of controls that remain outside model text generation. ### 9.8 A failure atlas The experimental traces can support a failure taxonomy spanning corpus omission, retrieval failure, evidence-utilization failure, unsupported generation, citation mismatch, calibration failure, correlated verifier error, authority confusion, unauthorized action, and postcondition failure. Such a taxonomy is more actionable than a single aggregate “hallucination” label. --- ## 10. Limitations The proposed system is complex. Every added component introduces implementation cost, latency, and new failure modes. A policy gate can misclassify risk. A claim decomposer can omit propositions. A retriever can overvalue source-authority metadata. A verifier can be confidently wrong. An adaptive controller can fail to escalate. An action gateway can contain software vulnerabilities. Defense in depth reduces reliance on any one model output but does not eliminate systemic failure. The architecture is strongest where external checks exist. Arithmetic, compilation, schemas, database invariants, and controlled execution provide relatively strong feedback. Open-ended scientific, ethical, legal, and strategic questions may not admit deterministic verification. In these domains, retrieval, source criticism, disclosed uncertainty, and human review remain necessary but imperfect. Human escalation does not solve scalable oversight. Humans may lack expertise, time, or attention, and stronger models may produce persuasive but misleading explanations. The thesis can measure escalation quality in selected settings but cannot establish that humans can reliably supervise arbitrarily capable agents. Frozen evidence improves reproducibility while limiting freshness. Live evidence improves realism while creating instability and potential privacy concerns. Both evaluation modes are partial. Sparse-backbone experiments may be constrained by compute. A small open model may underrepresent the capabilities and risks of frontier-scale systems. A nominal 200B design should not be presented as implemented without the corresponding training, serving, and evaluation evidence. Benchmarks may reward narrow strategies and suffer contamination. Agent environments approximate rather than reproduce consequential deployment. Security evaluations age as attacks and models change. Any positive result will remain conditional on the test suite and threat model. The literature corpus itself is bounded. The supplied review contains a 20-paper critical spine and a broader screened collection, with recent and provisional records requiring canonical-version audit before thesis submission. Bibliographic author and venue details should be checked against canonical sources. This manuscript uses title-linked citation placeholders and does not claim that the local corpus contains every relevant publication. Finally, the central objective—severity-weighted selective reliability—requires normative and operational choices. Different deployments will assign different costs to false answers, refusal, privacy leakage, delay, and unauthorized action. The thesis can expose those choices and test sensitivity, but it cannot define one universal risk function. --- ## 11. Conclusion Language-model systems should not be evaluated as though scale, retrieval, consensus, or preference optimization individually imply reliability. Sparse mixture-of-experts models increase conditional parameter capacity; they do not create aligned autonomous specialists. Mixture-of-agents systems spend inference-time compute on candidate generation and synthesis; they do not guarantee independent judgment. Retrieval provides external information; it does not guarantee that the corpus is trustworthy, that decisive evidence is returned, that the model uses it, or that citations entail generated claims. Learned verifiers and debate can improve selection in some settings; they remain probabilistic and may share the generator’s blind spots. Tool use enables useful action while creating new paths to unauthorized or irreversible effects. A defensible response is selective, risk-adaptive defense in depth. The proposed architecture makes instruction authority explicit, preserves evidence provenance, decomposes outputs into atomic claims and typed actions, routes each object to the strongest applicable checker, calibrates answer and abstention decisions, and places state-changing permissions behind an external gateway. Additional agents are invoked only when they add a distinct evidence source, tool, model family, or error profile. Human reviewers retain authority over unresolved consequential cases. The thesis does not predict that this architecture will eliminate factual or action failure. Its purpose is to test whether the architecture measurably reduces severe failures relative to simpler and equal-compute alternatives, and to identify the mechanisms, costs, and residual risks responsible for any difference. Negative findings are consequential: if deterministic workflows match multi-agent systems, if verifiers reproduce generator errors, or if abstention collapses under shift, those results should constrain future designs. The intended system resembles centaur chess not because humans or machines are presumed infallible, but because reliable performance may emerge from a disciplined allocation of strengths and authority. Models propose and interpret. Retrieval supplies inspectable external evidence. Specialized tools calculate and test. Verifiers challenge. Deterministic controls govern action. Humans adjudicate rare consequential uncertainty. The scientific question is whether this division of labor produces a better empirically measured reliability frontier than simply using a larger model, more samples, or more agents. --- ## References and Canonical Citation Placeholders > **Editorial note:** These entries are manuscript placeholders linked to canonical or primary records available in the supplied local artifacts. Full author lists, publication venues, versions, and BibTeX metadata should undergo a final canonical-source audit before submission. 1. [*Finetuned Language Models Are Zero-Shot Learners*](https://arxiv.org/abs/2109.01652). 2. [*Multitask Prompted Training Enables Zero-Shot Task Generalization*](https://arxiv.org/abs/2110.08207). 3. [*Training Language Models to Follow Instructions with Human Feedback*](https://arxiv.org/abs/2203.02155). 4. [*Self-Instruct: Aligning Language Models with Self-Generated Instructions*](https://arxiv.org/abs/2212.10560). 5. [*Constitutional AI: Harmlessness from AI Feedback*](https://arxiv.org/abs/2212.08073). 6. [*Direct Preference Optimization: Your Language Model Is Secretly a Reward Model*](https://arxiv.org/abs/2305.18290). 7. [*LIMA: Less Is More for Alignment*](https://arxiv.org/abs/2305.11206). 8. [*The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions*](https://arxiv.org/abs/2404.13208). 9. [*REALM: Retrieval-Augmented Language Model Pre-Training*](https://arxiv.org/abs/2002.08909). 10. [*Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks*](https://arxiv.org/abs/2005.11401). 11. [*Dense Passage Retrieval for Open-Domain Question Answering*](https://aclanthology.org/2020.emnlp-main.550/). 12. [*Improving Language Models by Retrieving from Trillions of Tokens*](https://arxiv.org/abs/2112.04426). 13. [*Atlas: Few-shot Learning with Retrieval Augmented Language Models*](https://arxiv.org/abs/2208.03299). 14. [*REPLUG: Retrieval-Augmented Black-Box Language Models*](https://arxiv.org/abs/2301.12652). 15. [*Active Retrieval Augmented Generation*](https://arxiv.org/abs/2305.06983). 16. [*Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection*](https://arxiv.org/abs/2310.11511). 17. [*Corrective Retrieval Augmented Generation*](https://arxiv.org/abs/2401.15884). 18. [*FEVER: a Large-scale Dataset for Fact Extraction and VERification*](https://aclanthology.org/N18-1074/). 19. [*KILT: a Benchmark for Knowledge Intensive Language Tasks*](https://aclanthology.org/2021.naacl-main.200/). 20. [*TruthfulQA: Measuring How Models Mimic Human Falsehoods*](https://aclanthology.org/2022.acl-long.229/). 21. [*SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models*](https://aclanthology.org/2023.emnlp-main.557/). 22. [*FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation*](https://aclanthology.org/2023.emnlp-main.741/). 23. [*RARR: Researching and Revising What Language Models Say, Using Language Models*](https://arxiv.org/abs/2210.08726). 24. [*ALCE: Enabling Large Language Models to Generate Text with Citations*](https://arxiv.org/abs/2305.14627). 25. [*RAGAS: Automated Evaluation of Retrieval Augmented Generation*](https://arxiv.org/abs/2309.15217). 26. [*ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems*](https://arxiv.org/abs/2311.09476). 27. [*RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models*](https://arxiv.org/abs/2401.00396). 28. [*RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation*](https://arxiv.org/abs/2408.08067). 29. [*Lost in the Middle: How Language Models Use Long Contexts*](https://arxiv.org/abs/2307.03172). 30. [*Language Models (Mostly) Know What They Know*](https://arxiv.org/abs/2207.05221). 31. [*Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation*](https://arxiv.org/abs/2302.09664). 32. [*Calibrated Language Models Must Hallucinate*](https://arxiv.org/abs/2311.14648). 33. [*Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer*](https://arxiv.org/abs/1701.06538). 34. [*Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity*](https://jmlr.org/papers/v23/21-0998.html). 35. [*GLaM: Efficient Scaling of Language Models with Mixture-of-Experts*](https://arxiv.org/abs/2112.06905). 36. [*ST-MoE: Designing Stable and Transferable Sparse Expert Models*](https://arxiv.org/abs/2202.08906). 37. [*DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models*](https://arxiv.org/abs/2401.06066). 38. [*MegaBlocks: Efficient Sparse Training with Mixture-of-Experts*](https://arxiv.org/abs/2211.15841). 39. [*OLMoE: Open Mixture-of-Experts Language Models*](https://arxiv.org/abs/2409.02060). 40. [*Improving Factuality and Reasoning in Language Models through Multiagent Debate*](https://arxiv.org/abs/2305.14325). 41. [*Mixture-of-Agents Enhances Large Language Model Capabilities*](https://arxiv.org/abs/2406.04692). 42. [*AI Safety via Debate*](https://arxiv.org/abs/1805.00899). 43. [*Self-Consistency Improves Chain of Thought Reasoning in Language Models*](https://arxiv.org/abs/2203.11171). 44. [*Tree of Thoughts: Deliberate Problem Solving with Large Language Models*](https://arxiv.org/abs/2305.10601). 45. [*Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision*](https://arxiv.org/abs/2312.09390). 46. [*Let’s Verify Step by Step*](https://arxiv.org/abs/2305.20050). 47. [*Training Verifiers to Solve Math Word Problems*](https://arxiv.org/abs/2110.14168). 48. [*ReAct: Synergizing Reasoning and Acting in Language Models*](https://arxiv.org/abs/2210.03629). 49. [*Toolformer: Language Models Can Teach Themselves to Use Tools*](https://arxiv.org/abs/2302.04761). 50. [*GAIA: a Benchmark for General AI Assistants*](https://arxiv.org/abs/2311.12983). 51. [*WebArena: A Realistic Web Environment for Building Autonomous Agents*](https://arxiv.org/abs/2307.13854). 52. [*WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?*](https://arxiv.org/abs/2303.13187). 53. [*OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments*](https://arxiv.org/abs/2404.07972). 54. [*SWE-bench: Can Language Models Resolve Real-World GitHub Issues?*](https://arxiv.org/abs/2310.06770). 55. [*τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains*](https://arxiv.org/abs/2406.12045). 56. [*ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities*](https://arxiv.org/abs/2408.04682). 57. [*MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback*](https://arxiv.org/abs/2309.10691). 58. [*CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark*](https://arxiv.org/abs/2409.11363). 59. [*ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery*](https://arxiv.org/abs/2410.05080). 60. [*Agentless: Demystifying LLM-based Software Engineering Agents*](https://arxiv.org/abs/2407.01489). 61. [*Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection*](https://arxiv.org/abs/2302.12173). 62. [*AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents*](https://arxiv.org/abs/2406.13352). 63. [*InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents*](https://arxiv.org/abs/2403.02691). 64. [*StruQ: Defending Against Prompt Injection with Structured Queries*](https://arxiv.org/abs/2402.06363). 65. [*PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models*](https://arxiv.org/abs/2402.07867). 66. [*Instruction-Following Evaluation for Large Language Models*](https://arxiv.org/abs/2311.07911). 67. [*FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models*](https://arxiv.org/abs/2310.20410).