Limitations of ChatGPT for Briefing Preparation
ChatGPT (a chatbot released by OpenAI in 2022) has captured industry and layman attention alike of late.
Capable of producing human-like text and handling a wide variety of general queries, the viral enthusiasm for the new tool has also motivated pushes for further funding and innovation from major companies like Microsoft. Google plans to release a similar tool.
Risk Frontiers is no stranger to deploying new technology in our analysis and products, and we were excited to see if it could handle some of our internal use-cases (with minimal modification). One area we pride ourselves in is our ability to rapidly aggregate information for an ongoing major event, and then report it in our briefings, paired with commentary from one of our own experts.
So, can ChatGPT write a briefing as well as one of our veteran staff members?
We tested the capability of ChatGPT to prepare a briefing on the Mw 7.8 earthquake that occurred in southeastern Turkey on 6 February 2023.
We conclude that ChatGPT is not inherently knowledgeable on this topic and does not demonstrate what we would consider a true understanding.
Furthermore, there are flagrant errors (the date of the event) as well as odd value judgements (like claiming a seismograph is “advanced technology”), which are perhaps the algorithm’s misunderstanding of what makes an informative article appealing to a reader.
Risk Frontiers does not plan to use ChatGPT as a default method for writing Briefings.
However, it’s worth remembering the intention of ChatGPT and its design. For one, the database used to teach it only goes to 2021, so one should expect it to be at a disadvantage with recent information.
To write the article, ChatGPT was provided with textual information from the USGS website regarding the earthquake that hit Turkey. The source input article was: https://www.usgs.gov/news/featured-story/magnitude-78-earthquake-nurdagi-turkey. This allowed the AI to be “aware” of a more recent event outside of its training dataset.
It was then asked to write a short essay on the matter using scientific wordings and including references where possible. Some interaction was required to obtain a sensible output and allowed the AI to refine its work.
Fundamentally, ChatGPT is chatbot and is meant to appear conversational; all should be familiar with a conversational individual who could chat plenty without saying much. The ability to be flexible and general nearly always means an AI will struggle once you try to make it excel at specific problems, even if they seem simple to us, like differentiating cats from dogs.
Using ChatGPT in this way also has challenging ethical considerations: who is the author of works produced this way? The expert supervising the input and output, the AI’s developer, the bot itself? The assignment of attribution seems open for discussion. Not to mention that there still tends to be some collective mistrust when algorithms, especially those in the uncanny valley, encroach too much on society.
Using a chatbot to efficiently summarise an event with a catastrophic loss of human life may be seen as lacking compassion or tact.
Much of modern AI should be considered a set of tools – they have utility for the areas they’ve been designed for, with respect and understanding of their limitations.
While ChatGPT won’t be writing briefings, the valuable methods and learnings that went into its creation will find their way into our tool kit in other forms in the near future.
The following is the briefing, with commentary from Paul Somerville tracked by Microsoft Word.
The Magnitude 7.8 Earthquake in Nurdagi, Turkey
On January 24th [Correct date is February 6th], 2023, the city of Nurdagi [Not just the city of Nurdagi; the earthquake ruptured a 120 km long segment of the East Anatolia fault and damaged at least 10 cities], Turkey was struck by a magnitude 7.8 earthquake, causing significant damage and loss of life. This devastating event highlights the importance of earthquake preparedness and the impact that these natural disasters can have on communities. The earthquake, also known as the mainshock [Misunderstands the generic nature of “mainshock;” any new event that is not an aftershock is a mainshock], had its epicenter [The epicentre is on the ground surface above the hypocentre; the latter is where the earthquake begins and spreads out over the fault] at a depth of 18 kilometers below the Earth’s surface.
In addition to the mainshock, the region was also affected by several aftershocks. An aftershock is a smaller earthquake that occurs after the main shock, and it is common for several aftershocks to follow a large earthquake. The aftershocks in the Nurdagi region had magnitudes ranging from 4.5 to 6.0 and had epicenters at depths ranging from 10 to 15 kilometers below the Earth’s surface. The occurrence of aftershocks can cause further damage to already weakened buildings and infrastructure and can also trigger additional landslides, making the recovery process more challenging.
Recent earthquakes in the European [More relevant to note the Mw 7.4 Kocaeli eq of 17 August 1999 in Turkey that killed about 18,000 people and caused massive destruction] continent have shown the destructive power that these natural disasters can have on communities. For example, the earthquake in L’Aquila, Italy in 2009 had a magnitude of 6.3 and resulted in more than 300 deaths and extensive damage to the city’s buildings and infrastructure (Scarf, et.al, 2009). Similarly, the August 2012 earthquake in Emilia Romagna, Italy, had a magnitude of 5.9 and resulted in at least 26 deaths and significant damage to the region’s buildings and infrastructure (Conte et al., 2013).
The USGS (United States Geological Survey) was able to quickly determine the magnitude and location of the earthquake in Nurdagi, Turkey, using advanced technology such as seismographs [Seismographs were invented in 1875 and so are not really advanced technology] and GPS [USGS did not use GPS to estimate the magnitude or location of the earthquake]. The USGS also utilized its ShakeMap tool to assess the extent of ground shaking, which is a critical factor in determining the level of damage caused by the earthquake.
One of the key factors that affects the level of damage caused by an earthquake is the building code and the earthquake resilience of the structures in the affected area. In some regions, buildings are constructed to meet specific standards designed to improve their ability to withstand earthquakes. In other regions, building codes may be less strict or not enforced, resulting in structures that are more vulnerable to damage during an earthquake. In the case of Nurdagi, Turkey, it is unclear [it is straightforward to identify the current building code standards and those at the time of construction of the affected buildings] what the building code standards were and how well the structures in the region were able to withstand [Plentiful field data on building damage on the internet tells you this] the earthquake.
In conclusion, earthquakes have the potential to cause widespread damage, especially in areas with inadequate building codes and construction practices. The magnitude 7.8 earthquake in Nurdagi, Turkey, and its aftershocks serve as a reminder of the destructive power of earthquakes and the importance of being prepared. The depths of the mainshock [This is a favourite topic in media reports, but it is largely irrelevant in earthquakes like this that rupture the entire thickness of the seismogenic crust, from the ground surface to depths 20 km or more] and aftershocks played a role in the extent of the damage caused by the earthquakes, as well as the building code and earthquake resilience of the structures in the region. While earthquakes are a natural phenomenon and cannot be prevented, their impact can be reduced through education, preparation, and investment in technology and infrastructure, as well as the implementation of strict building codes and earthquake-resistant construction practices.
Conte, J.F., Malagnini, L., Boschi, E., and Giardini, D. (2013) Seismic hazard and seismic risk in the Emilia Romagna region (Northern Italy) after the May 2012 earthquake. Bulletin of Earthquake Engineering, 11(2), 627-643.
Scarf, F., Mariucci, M. T., Valensise, G., & Boschi, E. (2009). Seismic hazard assessment for the city of L’Aquila. Engineering Geology, 104(3-4), 107-121.