ASTM test methods for emergency response robots

·by Val

A fire department or search and rescue team that wants a robot faces a hard comparison. Every maker describes its machine in its own terms, and a datasheet that says "climbs stairs" does not say which stairs, how steep, wet or dry, or how often it fails. The ASTM E54.09 test methods exist to replace those claims with numbers measured the same way for every robot.

This article is an overview of those test methods and of how they connect to federal grant funding, written for responders who buy robots and for people who build them. It is a map of the territory and no substitute for the standards themselves.

Where the test methods come from

In 2005 the Department of Homeland Security's Science and Technology Directorate began funding the National Institute of Standards and Technology (NIST) to develop a suite of tests for robots used in emergency response. Responders set the requirements. FEMA urban search and rescue teams alone supplied over 100 of them across 13 categories of robot.

NIST builds a prototype apparatus for each requirement and validates it with responders, manufacturers and test administrators at evaluation exercises and robot competitions. It then goes to ballot at ASTM International, in Subcommittee E54.09 on Response Robots. The subcommittee has equal representation from those same three groups. Older documents call it E54.08.01, its former name.

As of April 2026 NIST reports more than 50 test methods for ground robots, aerial drones and aquatic vehicles, of which more than 20 are published ASTM standards. The rest are still being validated.

Test methods, not equipment specifications

This is the point most often misunderstood. The standards describe how to measure a robot. They do not say how a robot must be built or how well it must perform.

• There is no pass or fail. A test produces a measurement, such as the steepest stairs the robot climbed reliably.
• There is no certification. A robot is not "ASTM certified". It has test results, for a stated configuration, in the test methods it attempted.
• The buyer sets the bar. Each responder organization picks the test methods that represent its missions and decides what result is good enough. The standards say so directly: different user communities can set their own thresholds of acceptable performance.

NIST's own guide makes the case for buying this way. Agencies that write their own list of requirements often ask for things that compete with each other, and end up disappointed, overcharged or both. Choosing among combinations of capability that robots have already demonstrated avoids that.

What gets measured

No single test describes a robot. A mission is represented by a set of them, typically 20 to 30, of which 5 to 10 matter most. The published ground robot standards cover these capabilities:

A terminology standard, E2521, defines the vocabulary they share. For aerial drones, an endurance standard (E3426) is published and a set of piloting tests is in draft. Aquatic vehicle tests are in draft too. Methods still in development include door opening, two-way audio, video latency and terrains for legged robots.

How a test works

Every test method defines three things: an apparatus that represents a task, a procedure that the test administrator and operator follow, and metrics. The standards were written over many years and the older ones are worded differently from the newer, but the rules below are common to them.

The rules

• A sponsor commissions the test. The test sponsor is whoever orders the testing and receives the results: a buyer, a programme or the maker. The sponsor chooses the test methods, the apparatus size, the environmental conditions and the statistical level.
• The configuration is fixed and recorded. The robot is identified by make, model and a named configuration, photographed against a 20 cm grid, and weighed and measured along with its control unit, batteries, spares and shipping cases. The configuration stays the same for every test. Any variation makes a new configuration, to be tested again across all the applicable methods.
• The operator works remotely. The operator is out of sight and sound of the robot, so everything the operator knows comes through the robot's own interface. An operator who has to sit near the apparatus wears ear protection. Once a trial starts, the administrator tells the operator when to start and when a fault has occurred, and nothing else.
• Practice is allowed. The operator may practise in the apparatus beforehand, within sight of the robot.
• Autonomy is allowed. Assistive and autonomous behaviours that make the system more effective are encouraged. The result is scored the same way.
• Capability or proficiency. The same test measures two things. Run by the maker's expert operator, it shows what the robot can do. Run by anyone else, under a time limit, it shows how proficient that operator is.

Repetitions, not a race

The current test methods rank their metrics in this order:

1. Completeness. Was a full set of repetitions completed? A trial is 10 to 30 repetitions.
2. Reliability. Successful repetitions as a share of attempts.
3. Efficiency. The rate of successful repetitions over the elapsed time.

The standards give a worked example for reliability. To show at least 80 % reliability with at least 80 % confidence, a trial needs one of these:

The sponsor may ask for a different level, and the older test methods quote 85 % confidence for the same counts. If more than one trial is run, the last 30 consecutive repetitions are judged together, so a bad start can be overcome by a clean run afterwards.

Apparatus settings get harder in steps until the robot's limit is found. Once a robot fails at a setting, all harder settings are treated as beyond it.

When the robot gets stuck

A robot that is stuck or disabled, or that damages the apparatus, has a fault. The administrator pauses the timer, records the time, place and nature of the incident, and scores a fault for that repetition. The robot may then be checked and maintained, and the trial continues. Any help to the robot or the operator at another time, whether visual, audible or physical, is itself a fault.

NIST's 2014 guide describes the event practice in more detail: about 10 minutes at the start point, using only the tools photographed with the robot, and no spare parts beyond tape and cable ties. Each intervention is written up with the symptom, the remedy and the tools used, so a week of testing also produces a record of what breaks and what it takes to fix it in the field.

Abstaining

The terminology standard defines abstaining as the maker or operator declaring that they will not perform a test, or will not have its result released. The test form is then marked "abstained". That is different from "not tested", which means the test was unavailable or does not apply to that kind of robot. A buyer who considers a test critical should treat an abstention as a failure until shown otherwise.

The apparatus

The apparatuses are deliberately cheap and abstract. They are built from lumber and oriented strand board, which grips much like dusty concrete, so that anyone can build them and results can be compared between places and dates. The standards say they are meant to be easy to fabricate so that robot developers can evaluate themselves.

Each apparatus scales with one dimension, the clearance width, chosen to match the mission and not the size of the robot:

Results from different sizes are not comparable with each other. A trial on a terrain takes 10 to 30 minutes and a radio range trial 30 to 60, so a reliable robot can get through 20 to 30 test methods in less than a week.

Four tests up close

Stairs (E2804)

The stairs are five steps with 20 cm risers, set at inclines of 30, 35, 40 and 45 degrees. Treads are wood or steel diamond plate, and each is tested dry and wet. One repetition is a climb from the floor to the top landing and back. The robot's capability is the steepest stair it climbs reliably on every surface selected. The sponsor can add conditions such as darkness, defined as 0.1 lux or less. A safety belay is available. If it has to intervene during a repetition, that is a fault.

Radio range (E2854, E2855)

The line-of-sight test needs a flat paved strip at least 6 m wide with no radio-reflective metal within 50 m of the centreline, such as a road or airstrip. At each distance the robot has to do real work: follow a circle of 3 m radius while straddling the line, stop precisely and read visual acuity targets. The test starts near where control or video is lost and moves closer in 6 m steps. The result is the greatest distance at which every task is completed. The same test run at an operational site gives a practical range with real interference.

In the non-line-of-sight test the robot turns off the centreline behind a stack of six shipping containers, 24 m wide and 7 m tall, and performs its tasks in their radio shadow.

Endurance (E3380)

The robot drives laps of a standard terrain until its energy source is exhausted. The metrics, in order, are distance travelled, runtime and efficiency. Distance comes first because two robots with the same runtime can cover very different ground. After every ten repetitions there is a one minute dwell with everything left switched on. The age and condition of the batteries are recorded where known. So is any software setting that affects the result, such as maximum speed, and it may not change during the test.

Logistics (E2592)

This is a practice and not a test method, and it is the closest the suite comes to hard numbers. It assumes a deployment of up to ten days with no resupply in the first 72 hours, and it measures:

• Packaging. How many cases the whole system needs, including control unit, batteries, chargers, spares and tools, and what each weighs. For FEMA urban search and rescue task forces a packed case should weigh no more than 68 kg (150 lb), with 90 kg (200 lb) as the absolute limit for two people to carry. Their two standard cases measure roughly 0.94 by 0.53 by 0.44 m and 0.78 by 0.50 by 0.43 m.
• Setup time. From packed in its cases to fully functional, in minutes, by a user with moderate experience.
• Routine tasks. The time to change a battery, a track or a manipulator.
• Tools. Whether field service needs no tools, tools already in a responder's toolbox, or specialized tools. Specialized tools have to be packed and are discouraged, since losing one can put the robot out of use.
• Downrange weight. The robot and its control unit as carried to the work site.

The other tests follow the same pattern. Sustained speed (E2829) is a 100 m figure-eight around two pylons 50 m apart, with the robot straddling a painted line. Search (E2853) sends the robot through a labyrinth of hallways, rooms and alcoves to find and read at least forty visual targets at different heights and angles.

How this connects to FEMA funding

Many public safety agencies buy equipment with federal preparedness grants. Those grants pay only for equipment types on FEMA's Authorized Equipment List. Robots are item 03OE-07-ROBT, described as robotic platforms for explosive device remediation, hazardous materials operations, tactical law enforcement, search and rescue, and surveillance and detection. FEMA lists it under seven grant programs:

• State Homeland Security Program
• Urban Area Security Initiative
• Tribal Homeland Security Grant Program
• Port Security Grant Program
• Transit Security Grant Program
• Intercity Bus Security Grant Program
• Intercity Passenger Rail Grant Program

The link to ASTM is in the grant notes on that item. They say that comparing manufacturers' claims is difficult, and that results from standard test methods can be trusted wherever and whenever the testing took place. They then say that purchasing organizations "should require" candidate manufacturers to have their robots evaluated by the applicable ASTM test methods and to provide the completed test charts.

Two things follow.

1. It is advice to the buyer, not a condition of the grant. The item does not make test results a requirement for eligibility. An agency can buy a robot with grant money without them.
2. It still shapes the sale. The federal guidance tells the buyer what to ask for. A maker who can hand over test charts answers the question, and a maker who cannot has to explain why.

The same page points to DHS market surveys of robots under the SAVER program, and to the Standardized Equipment List for related standards and training.

What the methods ask of a robot

The test methods do not constrain the design, but they reward some properties and expose others.

• Reliability over peak performance. Climbing a 45 degree stair once proves little. Doing it 30 times with no more than 3 failures is the result that counts.
• A complete operator interface. With the robot out of sight and sound, cameras, lighting, audio and feedback about the robot's own posture decide the outcome as much as the drivetrain does.
• Field repair with ordinary tools. Whatever goes wrong is fixed in the field and written down, and every specialized tool counts against the robot in the logistics measurements.
• Trade-offs in the open. Every configuration is tested across the whole suite, so the cost of an added capability shows up. NIST's example is a manipulator arm that made stair climbing worse.
• Stable configurations. A new battery, radio or payload is a new configuration with its own results. Even battery age and a speed setting are recorded for the endurance test.
• Packaging and setup. A robot that needs a case heavier than two people can carry, or a long assembly on site, shows it in the logistics results before it ever reaches a test lane.

What the results do not tell you

The apparatuses are abstract on purpose. A result says the robot and operator can perform a baseline task reliably. It does not say they will succeed in a collapsed building. NIST's position is that failing the baseline predicts failing the mission, and that passing it makes success much more likely, with training in realistic scenarios as the next step.

Results also go stale. They belong to one configuration, and robots change quickly, which is why the approach favours broad, fast testing that can be repeated.

Training operators

Because the expert's result is on record, the same apparatus measures anyone else's skill against it. An agency can build a few of the apparatuses, run its operators through them and track proficiency as a percentage of expert performance. Operating skills fade without practice, and a repeatable test shows when they have.

Questions to ask a supplier

1. Which ASTM E54.09 test methods has the robot been tested in, and can we see the completed test forms?
2. Which exact configuration was tested? Is it the one being quoted?
3. Who administered the tests, and where?
4. How many repetitions were run at each setting, how many failed, and what reliability and confidence level was the testing run to?
5. Which test methods did you abstain from, and why?
6. What field maintenance was needed during testing, and with which tools?
7. Which apparatus size were the mobility tests run in?
8. What are the packaged weight, case sizes and setup time, and does field service need any specialized tools?
9. For endurance results: how old were the batteries, and at what speed setting?

Where Ferradyne stands

We design our robots for emergency response against these test methods from the start, as a set of design inputs and not as a test to pass at the end. The apparatuses are public and inexpensive to build, so we intend to test ourselves on them early and often, and to publish our results by configuration as we have them.

Sources and further reading

The procedures and figures here are drawn from the current editions of twelve of the ASTM standards: E2521, E2566, E2592, E2804, E2828, E2829, E2853, E2854, E2855, E3132, E3380 and E3408. The history, the counts and the advice on purchasing come from the NIST and FEMA pages below. Editions change, so work from the current text for any purchasing or design decision.

• Performance of Emergency Response Robots, NIST, with the current list of published test methods
• Standard Test Methods for Response Robots, NIST
• Guide for Evaluating, Purchasing, and Training with Response Robots Using DHS-NIST-ASTM International Standard Test Methods, NIST, 2014
• Authorized Equipment List item 03OE-07-ROBT, FEMA
• Subcommittee E54.09 on Response Robots, ASTM International