AI is learning to ‘see’ the color of smoke, flames, and differences in temperature
By Jared Shelly
Fire breaks out inside an office building. As first responders mobilize, they’re already accessing 3D blueprints of the building. Sensors inside deliver real-time data about fire locations, materials burning, and possible dangers like toxic gases. When firefighters get inside, they won’t have to rely on instinct and experience alone. Instead, advanced computing systems navigate them through — warning them to avoid the bathroom on the second level because the floor has caved in, and to rescue the dog cowering in the corner.
Such a vision isn’t too far off. That’s because of AUDREY, the Assistant for Understanding Data through Reasoning, Extraction and sYnthesis being developed by the Department of Homeland Security Science and Technology Directorate and the NASA Jet Propulsion Laboratory.
AUDREY uses artificial intelligence to learn how fire behaves and how temperatures vary. Machine learning helps the system “watch” a fire and forecast how it might continue to burn and what resources will be most effective in fighting it. It uses GPS to identify the locations of firefighters and warns of dangerous flashovers before they happen. It uses object recognition to determine which types of objects are on fire and identify their burning patterns. After all, mattresses burn differently than household items like furniture. AUDREY delivers all that information to first responders without them having to frantically search for it during an emergency.
“Growing up, people learn what fire and heat mean because we have the ability to touch and feel things. We don’t stick our hand in it,” said John Merrill, director, first responders and detection at Department of Homeland Security. “Through AI, AUDREY is learning to distinguish the color of flames and color of smoke to determine temperature variances and offer first responders important data very quickly.”
AUDREY isn’t yet deployed for live action, but Merrill’s team has been experimenting with the technology in controlled test burns — and teaching AUDREY valuable lessons — with fire departments like Cosumnes Fire Department in California. Capt. Kirk McKinzie of Cosumnes said he’s impressed with the technology. He hopes that AUDREY will one day help teams like his communicate critical details to authorities and the public.
“When I’m working the hose line, I may not have time to get on the radio and paint a picture with words,” said McKinzie. “If I need help, I need it now.”
He’s also hoping that AUDREY can help give fire command critical information about firefighters working together.
“If we have 100 or 1,000 or even 10,000 firefighters on an incident, we want to be empowered with their locations, their heart rates, and real-time video feeds from each of our personnel — and we want AI that uses multiphysics computational fluid dynamic models to tell us how fast a fire will progress from A to B,” he said.
McKinzie hopes AUDREY will lead to a future where first responders gain critical information about a location and the players involved before getting to a scene.
“Rideshare companies know where everyone is. They know their credit card number, their next of kin and their cell phone number,” he said. “You call 911, a first responder is dispatched and when seconds count, we often know little more than a street name.”
Both Merrill and McKinzie are working toward a future where AI technology is a vital and common part of first response. Merrill explained that in the future, the technology could scour a fully distributed, cloud-based service of databases nationwide to see if similar incidents have already occurred — and use them to deliver instructions on how to respond. If a similar incident happened in Florida, for example, and focusing fire hoses on a certain spot helped save a home, replicating that process can help fight a similar blaze in Baltimore.
Meanwhile, McKinzie envisions a suite of digital solutions designed to provide community risk reduction by merging the Internet of Things (devices and sensors talking to one another) AI, fire telemetry, and other technologies to give first responders critical information and give citizens updates on emergencies.
“Citizens should get notifications on their phones. Just like they know when their rideshare is two minutes away they should know the layout of a building they’re trying to escape in 3D or weather and fire hazard that’s headed their way,” he said.
The biggest hurdle in getting AUDREY and technologies like it to go mainstream isn’t tech advancements, it’s cooperation between jurisdictions and private citizens. Privacy will undoubtedly be a major concern. Asking people to share blueprints of homes, offices, schools, and churches with the government will certainly not be palatable for everyone. There will likely be pushback against IoT sensors inside buildings as well.
“The tech is already there,” said Merrill. “Each jurisdiction has its own rules and regulations they have to overcome and be willing to share their data with other jurisdictions that are possibly thousands of miles apart.”
Merrill pointed to one important clarification that could set some minds at ease — every AI solution requires a human being to operate it.
“AUDREY does not take the human out of the decision chain,” said Merrill. “It’s another tool for the firefighters to use but the human makes the final decision.”