LLM-based Functional Coverage Generation and Auto-Evaluation Framework

Abstract:
To ensure that a Design Under Verification (DUV) is thoroughly examined during the simulation-based verification process, one of the metrics verification engineers may rely on is functional coverage. This metric is manually implemented in the testbench and tracks which functionalities have been exercised in DUV during testing. Large language models (LLMs) have recently shown potential in automating code generation across various domains. This paper investigates their capabilities to transform verification requirements written in natural language into syntactically and semantically correct functional coverage code, as this domain remains underexplored. To accomplish this, an automated evaluation framework was developed to assess several openweight LLMs’ performance on this task for a reference computational design. The goal was to have a highly controllable evaluation environment for these initial experiments. The results reveal promising capabilities of LLMs in this context, while also identifying challenges and limitations where their performance fell short. The authors provide insights into the underlying reasons for these difficulties, contributing to the understanding of LLMs’ potential and limitations in verification tasks.