An AI can copy a demo and still miss the rule
A smooth demo is not proof that an AI tool knows when to stop. Garg and coauthors release ShowTellArena, a set of narrated business tasks and questions about rules, exceptions, and bad proposed actions. Their selected pilot found both wrong answers and trouble completing lessons. It is an early test set, not a fair ranking of products.
What the source reports
Garg and coauthors built ShowTellArena to test what AI tools learn from narrated business demos. The first release has 50 tasks and 502 questions. It includes recordings, screenshots, narration, and starting data for business jobs in finance, hiring, buying, customer decisions, inventory, and shipping. The questions ask about rules, limits, exceptions, and errors in proposed actions. In a selected pilot, the team examined 218 attempts across 39 job cases. Some AI tools answered incorrectly; others failed to finish the teaching step. The authors say the pilot has uneven coverage and gaps in checking answers. It is not a controlled product ranking. The questions test whether a tool learned the rule behind a step, not whether it can repeat the screen action.
Original source
- Title
- ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations
- Author
- David Garg, Ritobrata Sarkar, Ehsan Azarnasab, Siddhartha Borah
- Publication
- arXiv
- Date
- Wednesday, September 23, 2026