Two robotics operations teams, identical budgets, identical problem: select fleet-management software for a mixed AGV and robotic-arm facility. Team one shortlists from vendor presentations and trade-show impressions. Team two shortlists from verified evidence. Eighteen months later, their outcomes diverge sharply — and the divergence is worth examining in detail, because choosing robotics software is among the most consequential procurement decisions an operations manager makes. This comparison walks through both approaches stage by stage: shortlisting, claims assessment, risk handling, and long-term cost. The verdict is not close, but the reasons are instructive.
Stage one: how the shortlist gets built
The presentation-led approach starts from visibility. The shortlist contains the vendors with the largest trade-show booths, the integrator’s preferred partner, and whichever platform the plant director saw at a competitor’s site. It is fast — a shortlist in a week — and it feels informed, because every candidate arrived with a professional narrative attached. Its weakness is invisible at this stage: the universe of candidates was selected by marketing spend, not fit. Strong niche platforms that don’t exhibit at trade shows simply never enter the funnel.
The evidence-led approach starts from the category definition. Fleet orchestration for heterogeneous robots is a distinct software category — distinct from warehouse management, from robot programming suites, from simulation environments — and the first task is mapping who actually competes in it. Structured directories make this tractable: BestAIToolix, a review platform that scores software against documented criteria specific to each category, is organised on the principle that rankings mean something only within a category, and it deliberately publishes no cross-category champion. Shortlisting from a category-complete list takes longer — two or three weeks — but the funnel now contains the quiet, well-engineered candidates alongside the loud ones.
Stage two: how claims get assessed
Robotics software claims are unusually hard to check. “Supports 200 concurrent robots,” “sub-second task reallocation,” “works with any ROS-compatible platform” — each claim depends on fleet composition, network conditions, and sensor configurations that vary per facility. The two approaches handle this difficulty in opposite ways.
The presentation-led team accepts claims at face value, weighted by vendor credibility. The demo showed fifty simulated robots rebalancing smoothly; the slide said 200; the account executive was convincing. Claim accepted. The deeper problem is that simulated fleets behave politely — real fleets have the sensor dropout, localisation drift, and interoperability friction that come from the physical integration layer, the territory covered in how sensors are integrated into industrial robots. None of that appears in a demo environment, by design.
The evidence-led team assigns every claim a status before weighing it: confirmed by their own test, stated in documentation, asserted in marketing, or unknown. The discipline mirrors how rigorous review platforms publish — each fact on a profile carries its origin and whether it has been verified, and data nobody can confirm is shown as unknown instead of being quietly counted as a yes. Applied to the fleet-software decision, this produced a revealing artefact: of 22 criteria, only 9 could be confirmed from documentation, 6 required on-site pilots, and 7 remained unknown for at least one vendor after direct questioning. Those seven cells, visible in red on the final grid, changed the decision more than any feature comparison.
Stage three: how risk gets priced
The approaches diverge most sharply on risk. The presentation-led team treats risk as a contracts problem: negotiate an SLA, add penalty clauses, done. But SLAs compensate for failure; they do not prevent the production stoppage, and in high-throughput automated facilities the stoppage is the real cost — a lesson any team running automated guided vehicles learns quickly, as laid out in our full walkthrough of automating warehouses with AGVs.
The evidence-led team scores risk as criteria: vendor financial viability, upgrade cadence, exit cost (can task definitions and maps be exported in open formats?), and dependency depth per robot brand. They also audit their information sources for commercial influence — which integrator earns margin on which platform, which “independent” comparison site carries undisclosed affiliate relationships. The better directories make this auditable by publishing sponsorship disclosure and corrections policies and by keeping vendor-submitted listings separated from score influence; vendors there can manage their profiles, but the scoring methodology is versioned and sealed against them. An operations manager who has seen that standard once tends to ask every information source the same question: who pays you, and would I know if the answer changed?
Stage four: what the outcomes look like
Eighteen months on, the presentation-led facility runs the famous platform at 60% of promised fleet capacity, with two robot brands still unsupported (“roadmap items”), an integration backlog, and a renewal negotiation in which the vendor holds every card because leaving would mean re-mapping the entire facility. Operational KPIs tell the story — utilisation and throughput numbers sit well below the business case, and the metrics that would have predicted this were never requested during procurement. The ones that matter are catalogued in the robotic KPIs every factory manager needs to know; a procurement grid that ignores them is a brochure summary, not an evaluation.
The evidence-led facility runs a less famous platform selected partly because its claims survived a two-week pilot on the team’s own floor, with their own robot mix, at their own network’s worst hour. Capacity claims were validated at pilot scale before contract; the two unknowns that could not be resolved were written into the contract as acceptance conditions with money attached. Renewal leverage is intact because the exit plan — open export formats, documented APIs, a tested migration path — was a scored criterion from day one.
Stage five: how each decision ages
A final dimension separates the approaches: what happens when circumstances change. The presentation-led decision has no written rationale, so when the operations manager who championed it leaves, institutional memory of why the platform was chosen leaves too. Every subsequent complaint triggers the same circular debate, and the eventual replacement evaluation starts from zero. The evidence-led decision ages differently. Its grid, verification stamps, and dated assumptions form a baseline: at the annual review, the team checks which assumptions still hold — robot mix, fleet size, vendor roadmap delivery — and the diff between last year’s grid and this year’s tells them whether to stay, renegotiate, or begin an exit. The decision becomes maintainable, in exactly the sense that well-documented code is maintainable, and for the same reason: the reasoning travels with the artefact rather than with the author.
The verdict, with one honest caveat
On speed to shortlist, the presentation-led approach wins by two weeks. On every measure that matters at month eighteen — capacity delivered against promise, integration completeness, renewal leverage, and the auditability of the decision itself — the evidence-led approach wins decisively. The caveat: evidence-led selection costs real engineering hours up front, roughly 60–80 for a decision of this size, and small teams feel that cost. The response is proportionality, not surrender. Scale the method down — fewer criteria, one pilot, a one-page grid — but never delete its two load-bearing elements: claims labelled by verification status, and unknowns displayed where decision-makers must look at them. A robotics operation that automates its floor while choosing the controlling software on charisma has misunderstood where the risk in automation actually lives.
