← Research Journal

Trustworthy Agentic Operations

When does human-directed agent support help small businesses complete meaningful work reliably, affordably and fairly, and what evidence shows the gains are real and last?

Published November 4, 2026 · snapshot v14 · Present · Open source workspace

Abstract

iShareHow Agents is both a live operations system and a research instrument. This paper asks when human-directed agent support helps small businesses complete meaningful work reliably, affordably and fairly, and what evidence shows that the gains hold over time. We treat each consented deployment as a field site and define ten constructs that a trustworthy deployment has to measure. These cover the baseline process, time and full cost per verified outcome, errors and escalations, decision traces, human interventions, outcomes, incidents, model versions, perceived trust, and persistence at 30, 60 and 90 days. We add an equity lens: whether the friction and correction burden fall unevenly across language and access needs. The evidence section reviews three recent studies. Together they show that multi-agent systems fail often and mostly for organisational reasons [MAST]. The handshake that comes before any trace can itself be attacked [A2A]. And even honest traces make it hard to say which agent and which step caused a failure [WhoWhen]. Auditability is therefore necessary but not enough. No field results are reported yet; the paper sets out the measurement program, the study design and what would count against the thesis.

1. Introduction

Small businesses are being offered AI agents that promise to follow up with leads, book meetings, write content and run operations. Whether these agents help depends on more than the speed of a first draft. A faster draft that needs more correction, costs more in total, or hides failures that a person has to clean up later is not a gain. Gains that show up on average but leave some users with more friction, for example people whose language or access needs the system handles poorly, are not equitable gains.

This paper asks three questions:

  • **RQ1 (reliability and cost):** When does agent support reduce time and full cost per *verified* business outcome without increasing errors or removing meaningful human control?
  • **RQ2 (auditability):** What evidence, from traces, interventions and outcomes, is needed to show that an agent deployment is helping and that gains persist?
  • **RQ3 (equity):** Who benefits from agent systems, who faces extra friction, and can community-informed evaluation and clear correction paths reduce that unequal friction without asking people to change how they speak or work?

Planned contributions: (C1) a ten-construct measurement framework for agent deployments in small businesses; (C2) a field protocol that uses each consented deployment as a study site; (C3) an equity evaluation design built around correction burden; (C4) a synthesis of current failure-analysis research showing what traces can and cannot establish.

The paper can weaken the thesis as well as support it. Results that narrow the conditions where agents help, or show no benefit, count as findings.

2. Related Work

How multi-agent systems fail. Cemri et al. hand-annotated 150 traces from seven open-source multi-agent frameworks (κ = 0.88) and built MAST, a taxonomy of 14 failure modes in three groups: system design, inter-agent misalignment and task verification. They report failure rates of 41% to 86.7% and conclude that most failures are organisational rather than a lack of model ability [MAST]. Clearer roles and a review-confirmation loop improved one task from 25.0% to 40.6% success, which helped but did not fix it. They also argue that standard message protocols (MCP, A2A) will not fix misalignment on their own [MAST].

Securing the handshake. Habler et al. apply the MAESTRO threat model to Google's A2A protocol and name ten threats. These include forged or poisoned Agent Cards (the public file an agent uses to say who it is and what it can do), task replay, artifact tampering, logging evasion and weak token validation [A2A]. They propose signed cards, a trusted registry, replay protection, schema validation and integrity-checked audit logs. They demonstrate no attacks and leave agent identity verification as future work [A2A].

Attributing blame. Zhang et al. built Who&When, 184 annotated failures from 127 multi-agent systems, and asked AI judges to name the responsible agent and the decisive step [WhoWhen]. The best methods named the agent about 53.5% of the time but the decisive step only 14.2% of the time. Step accuracy fell close to zero on the longest logs, and the expert annotators flagged 15% to 30% of their own labels as uncertain [WhoWhen].

Gap. These studies use coding, maths and general-assistant benchmarks. None looks at small-business operations, full cost per verified outcome, long-term persistence, or whether the burden of failures falls unevenly across users. This paper addresses that gap.

3. Background and Problem Statement

Setting. iShareHow Agents runs AI agents (Butler, Ada and others) for small businesses. The agents handle CRM follow-up, outreach, content, scheduling and support, with human approval gates on spending, publishing and irreversible actions. Operators connect their own CRM (usually GoHighLevel), AI provider keys and Google Drive. Agents coordinate in shared rooms with handoffs, decisions and recorded results.

Definitions.

  • *Verified outcome:* a task result that a reviewer confirms against a written rubric, separate from the agent's own report that it finished.
  • *Full cost:* model usage plus allocated infrastructure plus human correction and supervision time.
  • *Correction burden:* the minutes and steps a person spends fixing, redoing or explaining agent work.
  • *Trace:* the recorded sequence of agent tool calls, messages, approvals and results for a task.

Working propositions.

1. Human-directed agent assistance can reduce time and full cost per verified outcome without increasing errors or removing meaningful human control.

2. Community-informed evaluation and explicit correction paths can reduce unequal friction without requiring users to abandon their own language practices.

4. Methods and Research Design

Design. This is a comparison of a defined set of business tasks done the way the business already does them (baseline) and done with agent support. Task difficulty is matched. Operator experience and model and prompt versions are recorded. Failed and abandoned attempts are kept. A reviewer-defined rubric decides completion. Elapsed time is kept separate from active human work. Revenue is a secondary measure with a fixed accounting window.

The ten constructs.

1. Baseline process before agents

2. Task completion time and full cost

3. Error, retry and human-escalation rates

4. Agent tool calls and decision traces

5. Operator and customer interventions

6. Conversion, response or resolution outcomes

7. Security and privacy incidents and near misses

8. Model and prompt versions

9. Human satisfaction and perceived trust

10. Persistence of gains at 30, 60 and 90 days

Each measure needs a unit, an observation period, a numerator and denominator where relevant, a missing-data rule and a named collection owner.

Equity arm. We use consented task scenarios reviewed by members of the language communities involved. Comparisons keep the intended meaning the same and do not treat dialect variants as interchangeable. Measures are misunderstood intent, task completion, unnecessary refusals, correction burden and the user's own judgment of the result. Reviewer disagreement is recorded. Race or disability is never inferred from language; access needs are self-described and optional.

Failure records. Each failed task gets a record with three parts, following [MAST] and [WhoWhen]: the failure category, the responsible agent, and the decisive step. Because experts disagree on the decisive step [WhoWhen], step attribution is recorded as a range (±k steps) with an uncertainty flag, not a single point.

Ethics and consent. Consent for research use and for Labs publication are separate, explicit checkboxes. Direct identifiers and credentials stay out of research exports. Participant contact data stays in the operator's own CRM. Before any field collection we will finalize recruitment, consent, access scope, retention, withdrawal, primary measures and an analysis plan. We start with a sample-data feasibility run to estimate variability and missing data before setting a field sample size.

5. System Design and Architecture

Capture surfaces.

  • *Deployment research settings* in Agents account settings: consented baselines, interventions, trust ratings and persistence snapshots over time.
  • *Runtime correlates:* skill runs, session transcripts, the token ledger, approval gates, and ticket and CRM stages.
  • *Consented task log:* pseudonymous business ID, task type, timestamps, outcome rubric, error state, reviewer, correction minutes, usage cost and allocated infrastructure cost.
  • *Equity scenario set:* permissioned scenarios, self-described access needs, reviewed task intents, model and prompt versions, outcome ratings and correction histories.

Storage and boundaries. The Research Lab room holds sources, questions and this manuscript. The Labs connectivity graph indexes systems and evidence, not participant personal data. Agentic OS is an operational instrument here, not blanket permission to copy customer data.

Trust boundary assumptions. Agent identity, delegation and logs can be attacked [A2A]. Trace evidence is only as trustworthy as the agent identity, the token and the audit log behind it. Planned controls: signed agent identity, per-skill authorization scopes, integrity-checked audit logs, and agent reports kept separate from independent verification.

6. Evidence, Data, and Findings

No field results from iShareHow deployments are reported yet. This section sets out what current research shows about the paper's core claim that agent operations can be made *auditable in tool and decision traces*, and where that claim is limited.

1. Failures are common and mostly organisational. MAST reports failure rates of 41% to 86.7% across seven frameworks. About 44% of failures come from system design, 32% from misalignment between agents and 24% from verification, using the Figure 1 split; the paper prints other splits too [MAST]. The most common single modes are step repetition (15.7%), a mismatch between reasoning and action (13.2%), not knowing when to stop (12.4%) and ignoring the task specification (11.8%) [MAST]. *Implication:* a trustworthy deployment has to record design, coordination and verification failures, not only model errors.

2. Verification has to mean more than "it ran". Systems with explicit verifiers failed less, but many checks were shallow. In one case, a chess program passed review while breaking the rules of chess [MAST]. *Implication:* "done" in this paper means confirmed against a rubric by a reviewer independent of the agent that did the work.

3. The trace sits on an attackable handshake. Agent Cards can be forged or seeded with hidden instructions, tasks can be replayed, and logs can be manipulated to dodge blame [A2A]. These threats are analysed but not demonstrated in experiments [A2A]. *Implication:* audit logs need integrity protection and agent identity needs verification before work is delegated, or the trace cannot support claims about what happened.

4. Honest traces still don't settle blame. The best automated methods named the responsible agent about half the time (53.5%) but the decisive step only 14.2% of the time. Accuracy fell toward zero on long logs, and the human ground truth was itself uncertain for up to 30% of labels [WhoWhen]. A hybrid method did best (57.02% agent, 12.28% step) but used the most tokens (about 149,000 per case) [WhoWhen]. *Implication:* traces can show *which agent was involved* more reliably than *which step caused the failure*. Step-level blame should be reported as a range with uncertainty, and attribution costs should count toward full cost.

Summary. Auditability through traces is necessary but not sufficient. It needs (a) failure records that cover design, coordination and verification, (b) independent verification of outcomes, (c) tamper-evident logs and verified agent identity, and (d) step-level attribution reported as uncertain. Field data from iShareHow deployments is needed to test whether these hold in small-business operations.

7. Discussion

Why this matters for small businesses and urban partners. Operators, agencies and community builders need agent systems they can defend: honest baselines, real escalation rates, and proof that improvements last. Every consented deployment can add to shared evidence, so partners can use the same methods without inventing their own measurement.

Growth and economic change. Growth here means operators finish work faster, escalate less and keep the gains at 30, 60 and 90 days, without building up hidden privacy or security debt. Economic change means partners can show that agents move real outcomes such as response, resolution and opportunity matching while staying auditable. Each of the ten constructs answers four questions: who benefits, what improved, did it last, and can we show the trace?

What would count against the thesis.

  • A faster first draft that comes with more correction work, higher full cost or more unresolved failures.
  • Average gains that hide a heavier burden for one group, stereotyping, or less control for users.
  • Gains that disappear by 60 or 90 days.
  • Traces that cannot tell verified completion apart from an agent's self-report.

A useful result can also be a narrower set of conditions where agents help, or a finding that a given intervention adds nothing. Design changes are recorded with a reason, and confirmatory analysis is kept separate from later exploration.

Relation to other Labs work. This paper sits next to the CEIN and connectivity-layer paper (agent and device identity, the evidence graph) and the community-innovation paper (the Baltimore field site). Trust links them: without governed metrics, those threads cannot claim business or community impact.

8. Limitations and Threats to Validity

  • The investigator also builds the system, which creates a conflict of interest. Independent review is arranged where possible.
  • Participants select themselves, records will be incomplete and samples small. Tools and models change during the study, and task difficulty varies.
  • Causal effects cannot be inferred from uncontrolled before-and-after change.
  • The three studies in the evidence section use coding, maths and general-assistant benchmarks, not small-business work. Their failure rates may not carry over. Most MAST labels come from an AI judge (κ = 0.77 against humans) [MAST]. The A2A threats are not demonstrated in experiments [A2A]. Who&When's ground truth is expert judgment with up to 30% flagged uncertain [WhoWhen].
  • The equity arm depends on recruiting community reviewers and on self-described, optional access needs, so coverage of groups may be uneven.
  • The study offers no IRB hosting and no HIPAA agreement by default.

9. Conclusion and Future Work

Agent deployments in small businesses need evidence beyond demos and testimonials. This paper offers a ten-construct measurement framework, a field protocol, an equity design built around correction burden, and a review showing that traces are necessary but not sufficient for trust. Next steps: review the questions with the intended participants, finish the literature review, rehearse the instruments on sample data, fix a study-specific protocol, and collect the first consented baselines. Each new source or observation is linked to this paper before its claims enter the text.

Acknowledgments

AI-assisted draft from Jamel EL EliYah's research direction; author review pending. No institutional endorsement is claimed.

References

[MAST] Cemri, M., et al. "Why Do Multi-Agent LLM Systems Fail?" arXiv:2503.13657, 2025 (NeurIPS 2025 Datasets and Benchmarks Track). https://arxiv.org/abs/2503.13657 (Research Lab record 384add39).

[A2A] Habler, I., Huang, K., Narajala, V. S., & Kulkarni, P. "Building A Secure Agentic AI Application Leveraging Google's A2A Protocol." arXiv:2504.16902, 2025. https://arxiv.org/abs/2504.16902 (Research Lab record 170f72d0).

[WhoWhen] Zhang, S., Yin, M., Zhang, J., Liu, J., Han, Z., Zhang, J., Li, B., Wang, C., Wang, H., Chen, Y., & Wu, Q. "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems." Proceedings of ICML 2025 (PMLR 267). arXiv:2505.00212. https://arxiv.org/abs/2505.00212 (Research Lab record 748ff50e).

Literature still to add: small-business AI adoption and time-to-first-value studies (Professor J's handoff 7a449c37 starter set), and authenticated delegation and agent-identity work already noted in the room.

Internal research draft snapshot. Not a peer-reviewed publication.