
Intelligence · R8
When AI Is the Wrong Tool
A decision framework for choosing between rules, AI, and human review
Organizations should choose the architecture of a workflow before choosing an AI model. Use deterministic rules when policy is explicit and the required data are authoritative. Use AI to interpret unstructured inputs or generate candidates when the result can be checked at an acceptable cost. Route material exceptions, disputed evidence, and consequential judgment to a qualified person. If the necessary authority, evidence, or controls are unavailable, hold the action rather than guessing. The evidence supports selective deployment. A field study of 5,172 customer-support agents found 15% more resolved issues per hour with an AI assistant. A randomized experiment with 758 consultants found faster work on a set of tasks within the tool’s capability, but correctness fell by about 19 percentage points on a deliberately challenging task outside it. A meta-analysis of 106 experiments found that human–AI combinations, on average, performed worse than the stronger of human-only and AI-only alternatives. Together, these findings argue for testing a specific workflow against its best alternative, not assuming that either AI or human review automatically improves it. This paper proposes a rules-first routing framework, a controlled evaluation protocol, and a financial test based on cost per correctly closed case. It recommends full-population shadow trials or representative sampling, independent adjudication, segment-level error limits, and rollback authority. Material failures are treated as hard constraints. Efficiency gains do not override authorization, admissible evidence, or an unacceptable residual risk.
Co-authored by Jonathan G. Browning, Founder & CEO of 48PEGASUS
Published November 19, 2026 · Version 1.0
Researched and drafted with AI assistance; reviewed and edited by Jonathan G. Browning.
Contents
- Overview
- Sources and notices
Chapters
Related service
Questions about this paper? Call 1-615-970-8316.
Browse: White Papers · Reports · Downloads
