Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Trusted Knowledge Extraction for Operations and Maintenance Intelligence

Created by
  • Haebom

Author

Kathleen P. Mealey, Jonathan A. Karr Jr., Priscila Saboia Moreira, Paul R. Brenner, Charles F. Vardeman II

Outline

This paper addresses the dichotomy between data confidentiality and data integration goals in deriving operational intelligence from organizational data repositories, as well as the challenges faced by Natural Language Processing (NLP) tools for the knowledge structures of specific domains such as operations and maintenance (O&M). We decompose the knowledge graph construction and knowledge extraction processes into functional components, including named entity recognition, coreference resolution, named entity association, and relationship extraction. We evaluate 16 NLP tools and conduct a comparative analysis with rapidly evolving large-scale language models (LLMs). We focus on O&M intelligence use cases for trusted applications in the aviation industry, developing a baseline dataset based on the public-domain datasets of the U.S. Federal Aviation Administration (FAA). We evaluate the zero-shot performance of NLP and LLM tools operating within a controlled, confidential environment that does not transmit data to third parties. Based on our observations of significant performance limitations, we discuss the challenges associated with trusted NLP and LLM tools and their readiness for widespread use in mission-critical industries such as aviation. We provide recommendations for improving reliability and provide open, managed datasets to support additional baseline testing and evaluation.

Takeaways, Limitations

•
Takeaways:
◦
We present a practical approach to building knowledge graphs and leveraging NLP/LLM tools to enhance operations and maintenance intelligence in critical industries such as aviation.
◦
By comparing and analyzing the performance of existing NLP tools and LLM, we identify the strengths and weaknesses of each tool and suggest future research directions.
◦
We support further research by releasing a baseline dataset for evaluating the performance of reliable NLP/LLM tools operating in constrained environments.
◦
We address technical and ethical challenges and explore solutions for developing and applying reliable NLP/LLM tools.
•
Limitations:
◦
The dataset used for evaluation is limited to publicly available data from the Federal Aviation Administration, which may limit generalizability to other industries or data characteristics.
◦
Focusing on zero-shot performance evaluation, there is a lack of discussion on the potential for performance improvement through fine-tuning or additional learning.
◦
Assessing the level of technical readiness for reliable NLP/LLM tools can be relatively subjective, and more objective metrics are needed.
◦
The lack of a comprehensive comparative analysis of the different types of NLP/LLM tools necessitates an in-depth discussion of the pros and cons of specific tools.
👍
Made with Slashpage