What NIST asks of an emergency response robot

·by Val

People in public safety robotics talk about "meeting the NIST requirements". Strictly, there are none. The National Institute of Standards and Technology (NIST) does not approve robots or set minimum scores. What it runs is a measurement program: a definition of what a response robot is, a suite of tests, and a way to use them when evaluating, buying and training. Together these work as the expectations that a robot sold to responders is held to.

This article describes that program from NIST's side. A companion article covers the ASTM test methods themselves: how a test is scored and what the individual standards contain. Some overlap is unavoidable. Everything here comes from NIST's public pages and its guide for evaluating, purchasing and training with response robots.

What NIST does and does not do

The program sits in the Intelligent Systems Division of NIST's Engineering Laboratory. It began in 2005 under the Department of Homeland Security's Science and Technology Directorate, which remains the main sponsor. Other support has come from the Department of Justice, the Army Research Laboratory, the military's counter-IED organization and DARPA.

NIST develops the test methods and shepherds them through ASTM International's Subcommittee E54.09 on Response Robots, where manufacturers, responders and test administrators are equally represented. It reports more than 50 test methods for ground robots, aerial drones and aquatic vehicles, of which more than 20 are published standards.

NIST is clear about the limits of this. The standards are not equipment standards, and it is not proposing a "standard robot". A standard robot would meet a specification for size, shape, capabilities and interfaces. That can lower cost and help interoperability, and it can also hold back innovation. A standard test method is only an agreed way to measure, and each user community decides for itself what result its missions need.

NIST's definition of a response robot

NIST's working definition is a remotely deployed device intended to perform operational tasks at operational tempos. It should act as an extension of the operator, improve the operator's awareness of a remote scene, make the mission more effective and reduce the operator's risk. NIST lists seven key features. They are the nearest thing the program has to a list of requirements.

The right-hand column is our reading of where each feature shows up in the test suite, not a mapping that NIST publishes.

The four parts of a test

NIST describes every test method as four elements:

• Apparatus. A repeatable, inexpensive stand-in for a task, with settings that get harder, built from materials available anywhere in the world.
• Procedure. A script for the operator and the test administrator. In NIST's words, the tests are not intended to surprise anybody. They are meant to be practised.
• Metric. A quantity, such as completing 10 continuous repetitions of a task or covering a distance, which together with elapsed time gives a rate.
• Fault condition. A failure that prevents completion, such as a stuck or disabled robot or a software problem at the operator's controls. Every fault is catalogued to show what recurs.

Trials run to 10, 20 or 30 repetitions depending on the number of failures. The usual objective is 80 % reliability with 80 % confidence, and a sponsor can change that according to how well the mission tolerates failure.

How a test method comes to exist

A test method starts from a requirement stated by responders. FEMA urban search and rescue teams gave NIST over 100 of them for 13 categories of robot. From there NIST works through five kinds of event.

NIST calls competitions a main driver of the process. The 2023 RoboCup Rescue championship used 20 of the ground robot test methods as its challenge tasks, and a quadruped robot challenge was run at a major robotics conference the same year.

Once responders and test administrators agree that an apparatus works, the method goes to ballot at ASTM.

A comprehensive capability assessment

This is the evaluation a manufacturer goes through, and the source of the data a buyer sees. NIST's pages and guide describe it this way:

• One named configuration is tested across every applicable method, usually 20 to 30 of them.
• Environmental variables are tested one at a time.
• The manufacturer supplies its own expert operator, so the result is the best the robot can do. NIST's guide says no developer should promise more.
• A test usually takes under an hour, so a reliable robot finishes in less than a week.
• Results are reviewed five times: by the administrator and operator together straight after the trial, twice within the team of administrators, once by the manufacturer, and finally by the test director.
• Four synchronized video feeds record the robot from two angles, the operator's screen and the operator's hands, so cause and effect can be checked afterwards.

In NIST's guide the results go into a Response Robot Capabilities Compendium, a database with bar charts that place each configuration within its class of robots. The guide, from 2014, reports over 100 robots tested to varying degrees of completeness. We have not confirmed how much of that data is published today.

Buying a robot the NIST way

NIST's guide argues against buying from a list of requirements. Requirements written in isolation compete with each other, and the guide says the result has repeatedly been disappointment, excessive cost or both. It recommends specifying combinations of capability that tested robots have already shown. The steps:

1. Select test methods that represent the mission. Usually 20 to 30, of which 5 to 10 are the priority.
2. Set an objective and a lower threshold for each. The gap between them gives makers room to trade cost against capability.
3. Filter the tested robots. Apply the thresholds and watch how many robots each one removes. The guide's advice is that where several robots cluster at an acceptable level, adopt that level and let the next test separate them. The first five or so priorities usually decide the outcome.
4. Down-select to about three and run them through operational scenarios.
5. Ask for quotes at fixed package prices. Each maker says what it can deliver for, say, 25, 50 or 100 thousand dollars. One may offer two chassis with a manipulator and tether, another a single bare robot. That puts cost and capability side by side.
6. Accept on delivery by repeating the tests. The maker's expert repeats the specified test methods on the delivered robot, which should score at least 80 % of the original result.

The guide gives two procurements as examples. A military counter-IED organization specified ultra-lightweight reconnaissance robots with 27 test methods and initially bought about 32 million dollars' worth. A working group of 13 federal, state and local bomb squads in the National Capital Region used 29 test methods to specify robots for suspicious packages on transport systems. NIST now reports that the ground robot tests have been used to specify more than 60 million dollars of procurements for counter-IED missions.

The guide also raises questions the tests cannot answer. Does the agency already own a robot, with spare parts and repair experience? Does it want to maintain and train on two operator interfaces?

Training and operator proficiency

NIST calls robot operation a very perishable skill, and the same apparatuses measure it.

• A training circuit is typically 10 basic skills tests covering maneuvering, dexterity and situational awareness.
• Each trial is limited to 5 or 10 minutes, so novices and experts work for the same time and are equally fatigued. Ten tests take about 100 minutes.
• The score is a rate of successful task completions.
• The maker's expert sets 100 %. Everyone else is scored as a percentage of that.

Each organization sets its own threshold for deployment. NIST offers example categories for an eventual certification of operators:

These are examples and not an adopted scheme. NIST reports that more than 250 civilian and military bomb technicians have used the ground robot tests this way.

After the baseline tests comes the realistic part. NIST's exercises embed the standard apparatuses in operational scenarios at responder training sites, so performance in a degraded environment can be compared with the baseline. The guide's rule of thumb is that a robot should complete all the standard terrains to statistical significance before it tries the rubble pile.

Where testing happens

• NIST's Robotics Test Facility is on the Gaithersburg, Maryland campus. It holds the test methods and the prototypes being validated, with isolation bays for cold, heat and darkness. Outside organizations may be able to use it, operated by NIST staff, on a cost-reimbursable basis.
• Evaluation exercises. By 2014 NIST had hosted eight at FEMA responder training facilities, including Disaster City in Texas, with more than 120 robots taking part.
• Collaborating facilities. NIST says the test methods have been replicated in dozens of locations worldwide. Its page for collaborating facilities did not name them when we looked.
• Your own shop. The apparatuses are meant to be built by developers and responders. NIST provides an assembly guide on request to its program email address.

Drones and aquatic vehicles

The program is not only about ground robots.

• Aerial. One standard is published, for endurance. Draft tests define five levels of system and pilot capability, from a basic proficiency trial through maneuvering to open, obstructed and confined test lanes, and are intended to support credentialing of remote pilots. NIST says they are referenced in NFPA 2400, the fire service standard for small unmanned aircraft in public safety.
• Aquatic. Tests for vehicles on the surface and on the bottom are in draft, beginning in clear tanks and pools and then adding lighting, water clarity and flow.

What this means for a robot maker

• The apparatus is the requirement. NIST describes the test apparatuses as tangible representations of what responders need. A maker who builds them has the specification in physical form.
• Practise, and bring your best operator. Nothing in the tests is secret, and the published result is the expert's.
• Expect to be compared. Results are shown against the whole class of similar robots, and an abstention is visible.
• Faults are data. Each failure and repair is recorded. NIST's example is a track that keeps coming off in different apparatuses, which tells the engineers where to look.
• Autonomy helps if it helps the operator. NIST encourages assistive behaviours such as self-righting and returning into radio range, and notes that a good interface and some onboard intelligence show up directly in operator performance.
• Deliver what you tested. Acceptance testing repeats the tests on the delivered unit.
• Help buyers train. The guide suggests makers structure the first hours of customer training around the standard apparatuses and use the scores to see where their interface frustrates new operators.

Where Ferradyne stands

We treat NIST's seven features as design inputs for our emergency response robots, and the apparatuses as the place to prove them. We intend to build the apparatuses, practise on them, and publish our results by configuration as we have them.

Sources and further reading

NIST pages are revised from time to time, and the guide dates from 2014. Figures are as NIST stated them when each source was last updated.

• Performance of Emergency Response Robots, NIST, updated April 2026
• Ground Robot Tests, NIST, updated October 2024
• Standard Test Methods for Response Robots, NIST, updated August 2024
• NIST's Robotics Test Facility, NIST, updated November 2023
• Guide for Evaluating, Purchasing, and Training with Response Robots Using DHS-NIST-ASTM International Standard Test Methods, NIST, 2014
• ASTM E54.09 committee update, NIST, January 2022, for the aerial tests and NFPA 2400