Automating the Assessment of Japanese Cyber-Security Technical Assessment Requirements Using Large Language Models

Several countries, including the U.S. and European nations, are implementing security assessment programs for IoT devices. Reducing human effort in security assessment has great importance in terms of increasing the efficiency of the assessment process. In this paper, we propose a method of automating the conformance assessment of security requirements based on Japanese program called JC-STAR. The proposed method performs document analysis and device testing. In document analysis, the use of rewrite-retrieve-read and chain of thought within retrieval-augmented generation (RAG) increases the assessment accuracy for documents that have limited detailed descriptions related to security requirements. In device testing, conformance with security requirements is assessed by applying tools and interpreting the results with a large language model. The experimental results show that the proposed method assesses conformance with security requirements with an accuracy of 95% in the best case.