Write Recipe files during data mixing #185
Assignees
Milestone
Comments
This was referenced Jul 23, 2024
bbrowning
added a commit
to bbrowning/instructlab-sdg
that referenced
this issue
Jul 25, 2024
This introduces Recipe yaml files, which are used both as an input
into the data mixing process and as an output of the process.
As an input, we have some default recipe files that specify any
precomputed datasets that should be mixed with data from new skills
when generating the overall mix of samples that will be sent to the
training process.
If a downstream user/packager wants to add default recipes (and
datasets), they should install them to a path like
`/usr/share/instructlab/sdg` (varies by platform, uses Python's
`platformdirs.PlatformDirs` to respect platform conventions).
Recipes should be in sdg/default_data_recipes/{knowledge,skills}.yaml
Datasets should be in sdg/datasets but this location is not enforced.
Currently we are not shipping any default recipe files in the upstream,
but there is a unit test in place to ensure the functionality to load
default recipes from disk works once we decide how we want to ship a
precomputed dataset to our upstream users.
As an output of the data generation process, we write recipe yamls to
document which datasets were mixed together and in what proportions
along with the system prompt that was used during the
generation. Here's an example of a recipe yaml put into the output
directory after running data generation:
```yaml
datasets:
- path: node_datasets_2024-07-25T17_49_46/knowledge_tonsils_overview_e2e-tonsils_p10.jsonl
sampling_size: 1.0
metadata:
sys_prompt: "I am, Red Hat\xAE Instruct Model based on Granite 7B, an AI language\
\ model developed by Red Hat and IBM Research, based on the Granite-7b-base language\
\ model. My primary function is to be a chat assistant."
```
Datasets may be referenced by relative paths, which are relative to the
recipe's own directory. Or, they may use absolute filesystem paths.
Anything written out under the metadata section (currently just
sys_prompt) is purely informational for the user and ignored when
loading recipes.
Parts of this are extracted and rebased from
aakankshaduggal#4
aakankshaduggal#20
Refs instructlab#162, instructlab#171, instructlab#185, instructlab#201.
Co-authored-by: shivchander <shivchander.s30@gmail.com>
Co-authored-by: Khaled Sulayman <khaled@thesulaymans.com>
Co-authored-by: abhi1092 <abhi1092@gmail.com>
Co-authored-by: Aakanksha Duggal <aduggal@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Ben Browning <bbrownin@redhat.com>
|
Resolved by #203 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The writing of Recipe files was removed from #163 to reduce scope, but it's a requirement to resolve #162. We need to add back the functionality that writes the recipe yaml files to disk within the generated data output directory, as that becomes one of the necessary artifacts to document what went into the mixed dataset.
This may also be needed for #171, depending on how that gets solved.
The text was updated successfully, but these errors were encountered: