Using large language models and image analysis for disaster damage assessment

• NEC Corporation has developed a technology for disaster damage assessment using LLM and image analysis.
• The images can be narrowed down from a multitude of field images to include only those aligned with a user’s intent.
• This makes it possible to quickly respond to disasters.

Image analysis tools have been around for some time. Also known as image recognition or computer vision, image analysis tools are pieces of software that process images to extract details using artificial intelligence (AI). The details are analyzed with machine learning algorithms for identification.

Today, image analysis is the basis of most recognition applications. For example, for facial recognition tools, image analysis scans and recognizes a user’s facial features. For more advanced use cases, image analysis is used to scan and identify shapes and objects in motion, such as those applied to autonomous machines and even autonomous vehicles.

Combined with large language modeling (LLM), image analysis is enabling more use cases. In Japan, NEC Corporation has developed a technology for disaster damage assessment using LLM and image analysis. Using this technology, users can immediately and accurately assess the extent and location of damage from the multitude of images collected when a disaster strikes.

Given the number of natural disasters around the world, there is a growing need to quickly and accurately assess the damage to expedite initial response activities. This includes evacuation procedures and even rescue efforts for disaster victims.

Unfortunately, many countries still struggle with initial response speed because of the quality and amount of data available to them. Most precipitation distribution maps and seismic intensity distribution maps, as well as textual information received from residents, do not include sufficient details on the extent and location of damage.

In most cases, images offer the best solution to plan disaster management. This not only includes pictures from social media by the general public but also images captured by CCTVs in an area. For example, in the recent bushfires in Maui, the extent of the damage and victims involved to date is not fully confirmed due to the lack of data from the disaster area.

While satellite images can provide some assistance when it comes to mapping disaster zones and such, rescue personnel on the ground would need to get information much faster as they need to plan initial response activities.

For NEC, this area is where it feels the technology developed will contribute best. By providing this technology to government ministries and municipal authorities in charge of disaster responses, NEC hopes it can lead to the acceleration of evacuation guidance, rescue efforts, and other initial response activities in the event of a disaster.

Speeding up disaster management with image analysis

NEC’s technology has two key features that make it a reliable tool for disaster management. Firstly, the technology narrows down field images in accordance with a user’s intent. For context, the images can be narrowed down from a multitude of field images to include only those aligned with a user’s intent by utilizing semantic interpretation of words in an LLM and image analysis to determine image similarity.

Generally, image recognition technologies are only capable of recognizing pre-trained images, thereby limiting which images can be narrowed down. This has made it difficult to conduct surveys per a user’s intent, which varies depending on the type and scale of a disaster, the affected areas, and the extent of the situation.

NEC utilizes LLM to narrow down field images using keywords. Moreover, employing image analysis and specifying the scene a user would like to search for makes it possible to narrow down scenes that are difficult to express in words. As such, by combining LLM and image analysis, users can accurately narrow down the images to only those matching their intentions. This in turn makes it possible to quickly respond to disasters as they unfold.

Secondly, the tool uses field images to display the extent and location of the damage on a map at the street address level. The location can be estimated with street address-level accuracy and then displayed on a map by comparing the images with aerial imagery and map data covering extensive areas.

With the help of satellite images and aerial photographs, this technology has achieved the world’s highest matching accuracy, enabling location estimation with a high degree of precision, even for field images captured during a disaster.

For context, the technology automatically extracts areas such as roads, buildings, and traffic signals from field images and then matches them with map layout information (the shape and layout of roads, buildings, and such). By doing this, a highly accurate estimation of a photographed location is made possible. This is also applied to images of partially collapsed buildings or partially flooded roads.

While the technology is still being tested, NEC hopes to put it to practical use in 2025. In addition, NEC plans to expand the use of LLM and image analysis technology to other applications.

Imaging and AI can be crucial in disaster management.

---

---

---

---