Use of large language models to classify failure modes in damage notifications from transmission systems

Abstract:
Text classification is a fundamental task in Natural Language Processing (NLP), and leveraging Large Language Models (LLMs) has shown great promise to perform this task across various domains. Failure Mode Classification (FMC) is an auspicious approach in this context, that refers to the automatic assignment of failure modes to corresponding incidents. FMC is relevant in the field of maintenance, as it can reduce the need for technical workers to spend their time manually analyzing reports. However, the performance of LLMs used for FMC depends heavily on the quality of their prompts and the input data used to fine-tune the model. In practice, the data quality is often insufficient. A few-shot learning approach without the dependency on extensive fine tuning is a promising approach for FMC, which this article examines. The zero and few-shot learning strategies of LLMs, which reduce the dependency on labeled high quality datasets and complex model retraining, serve as the basis for this. Among the investigated LLMs Mistral 7B Instruct and LLaMA 3 8B Instruct the LLaMA 3 consistently delivered the highest results, underscoring its efficiency and compatibility for FMC. With a Macro F1-score of 66% this model is able to correctly assign 57 out of 86 failure modes. This investigation highlights the possibility of performing FMC using LLMs with the few-shot learning approach.