A study on the use of LLMs for generating a capability ontology. This repository contains the complete data set, including all prompts and results, of the paper "On the Use of Large Language Models to Generate Capability Ontologies" by Luis Miguel Vieira da Silva, Aljosha Köcher, Felix Gehlhoff and Alexander Fay. A preprint of the paper is available on arXiv: https://arxiv.org/abs/2404.17524
capability-models: This folder contains both the manually modeled as well as the LLM-generated capability models. The manually modeled ones serve as a benchmark to validate all generated models against. Inside capability-models/generated, there is a substructure of folders for the LLMs used, the temperature values tested as well as for the different prompt techniques used. To give one example: capability-models/generated/claude/temp-0/0_zero-shot contains all models generated by Claude with a temperature of 0 and the zero-shot prompt. All other subfolders are analogous. Please note that we only tested a restricted set of prompts with temp = 1 and these were not used in the paper.
examples: This folder contains examples used in the one-shot and few-shot prompts. For every example, there is a natural-language task description as well as an ontology.
metaprompts: Contains the three meta prompts used, i.e., zero-shot, one-shot and few-shot. Please note that the meta prompts contain placeholders that are filled to obtain the actual prompts.
prompts: Contains the actual prompts that were sent to ChatGPT and Claude. These prompts are obtained by using the python script inside the prompts folder. It takes the three meta prompts and adds the examples as well as task descriptions to create individual prompts for each meta prompt and task
shapes: Contains the SHACL shapes used to evaluate the generated results against certain criteria that were defined beforehand
task-descriptions: This folder contains the seven task descriptions that were used to generate capabilities. For each task description, meta prompt and LLM used, there is a result inside of capability-models/generated.