Making Copier generate a complete project

[rafalkrupinski]

Hi Copier Community,

I'm the author of Lapidary, an API client library and generator.
The generator generates function stubs and a complete model (actual function calls are handled by the library at run-time) required to talk to a JSON API.

I was wondering, would it be hard to change Copier in such way, to be able to generate a complete project, like Lapidary does? If it's possible without large architectural changes, I'd make a PR or just fork the project.

The difference is that, AFAICT, Copier accepts only a single value for a key used to generate package or module, while Lapidary accepts a model structured as a tree. It generates a variable number of packages, each containing a set of modules, each containing a variable number of classes. All that based on an OpenAPI document.

[pawamoy]

Copier accepts arbitrary data as answers (JSON, YAML), so users could definitely paste their OpenAPI schema there.

However, you won't be able to create complex file trees as you cannot use Jinja loops for directories/files creation.

Still, Copier has the concept of tasks, which can be Python scripts, and which you could use to create the complex file tree, but you'd probably lose some of the update system cleverness. Definitely worth a try though ^^

[rafalkrupinski]

Thank you for your answer!

you cannot use Jinja loops for directories/files creation.

So Copier would need to magically know that the variable(s) used in directory/file is a list and loop over it?

you'd probably lose some of the update system cleverness

The cleverness is exactly why I want to use Copier :)

[pawamoy]

What I mean is that currently, without the help of tasks, the dir/file structure can only be static, i.e. you pre-create dir/files with templated names, and they get created and their names rendered when generating a project. There's no way to, for example, create nested directories by iterating/recursing on a JSON/YAML answer.

But! There's a possibility that we implement partial support for arbitrarily nested directory templating in Copier (I say "we" because I believe @yajo agrees with the idea). The goal is to make Copier able to create multiple nested directories given a single directory with a templated name containing simili-slashes. There's a feature request for it, let me find it and update my comment.

With such a feature, you could use a context hook to pre-compute all your nested dir/file paths (based on the user's OpenAPI contract) and name your templated dir/files accordingly to let Copier generate them all.

It might not be easy to picture it in the mind only, so I'll try to give more concrete examples when I get access to a computer (currently on mobile) :)

[pawamoy]

GitHub doesn't let me edit my previous comment so here is the link to the feature request: #692

[yajo]

In the worst scenario, I don't think you need to fork copier. You can use it as a library to render projects. That way you can add any custom automation on top of it while benefiting from all its good things. It's not copier's main target use case, but it's supported.

You can use that to download the API spec and pass it to copier as an answer, like @pawamoy said. Or to apply different templates on different directories in loop.

[rafalkrupinski]

Thank you @pawamoy and @yajo for your answers.

You can use it as a library to render projects. That way you can add any custom automation on top of it while benefiting from all its good things

Yes, that was the idea. I already have a program to do all those things, just wanted to replace the rendering phase.

I think my issue boils down to a couple of questions.

1. How can I render multiple files from a single template? I missed such case in the examples and documentation.
2. Assuming I don't have arbitrarily nested directory structure, can I still use variables in to render paths?

[yajo]

How can I render multiple files from a single template? I missed such case in the examples and documentation.

All files are rendered, so just add more files to the template.

Assuming I don't have arbitrarily nested directory structure, can I still use variables in to render paths?

Of course. File names can be templated too.

[rafalkrupinski]

All files are rendered, so just add more files to the template.

I'm not sure what you're saying.

I want to generate an arbitrary number of files, based on some input model. Surely you're not suggesting I should generate template file for each item before I can use Copier...

[rafalkrupinski]

Turns out it wasn't that difficult.
I did a little experiment here: https://github.com/copier-org/copier/compare/master...python-lapidary:copier:feat/items-mode?expand=1

it adds two new special variables: _items and _item
if _items is set, Copier uses it as the items key, iterates over it, assigns each item to answers.local['_item'] variable and renders the whole model.
I also changed the way the root directory is handled, so it's rendered outside of the items loop.
Generally, my intention was to keep it compatible.

I'm just about to try it in my project, but I thought I'd get some early feedback from you.

Let me know if you like to merge it at some point, and if so, if I should change anything before that. Otherwise I keep it as a fork.

[yajo]

I can spot a couple of issues in the implementation but I like it overall. It seems a very pristine approach that can give a lot of possibilities to Copier. If you want to open the PR I can review it better.

[rafalkrupinski]

Singular and Looped templates

This version of Copier renders the entire template once against the model without [item], and then again for each item, against the model extended with the [item].
This means that each template file is rendered multiple times, even if the output file name is the same, so I need to add a check for that.

To counter it, each template file needs to have a conditional in their name, either {% if item %} or {% if not item %} and that's how I do it now, but perhaps having separate directories for singular and loop templates would make it more readable? Wouldn't spreading a template over multiple directories make it harder to use?

Kinds of items

I have a couple of kinds of items, but only one loop variable (items/item). Allowing multiple looped variables in a single template would make it more complex, it would need to be a nested loop for each looped variable, so instead, I handle it with a dispatcher template file:

{% include 'includes/kind_{}.py.jinja'.format(item.kind) %}

Having a separate directory for looped variable would allow having multiple looped variables, without adding computational complexity.

example:
copier.yaml

_items:
- parameters
- request_bodies
- response_bodies
_item: item

📁 template
├── 📄 copier.yml
├── 📁 common
│   └── 📁 pyproject.toml.jinja
├── 📁 parameters
│       └── 📄 {{ item.name }}.py.yml
└── 📁 request_bodies
...

Multiple prompts

I use Copier as a library, so I'm not so much concerned about how the questions are asked, but it starts looking like #750

Perhaps adding an array type to questions, similar to JSON Schema, would make sense?
Instead of _items:

request_bodies:
   type: array
   items:
    type: yaml
request_bodies:
   type: array
   items:
    type: yaml
...

[rafalkrupinski]

I'll just add check for overriding files and directories in loops and send a patch. The rest are just ideas.

[yajo]

This means that each template file is rendered multiple times, even if the output file name is the same, so I need to add a check for that.

If you keep a record of rendered files, you can skip them if the full file name is already in the record, for example. It shouldn't be too hard.

In any case, this might never become a real problem. I expect that files that get rendered multiple times produce the same output, so the only problem will be a few CPU cycles wasted.

I have a couple of kinds of items, but only one loop variable (items/item). Allowing multiple looped variables in a single template would make it more complex, it would need to be a nested loop for each looped variable

It seems to me that, if _items is a template, that should be enough for any case.

# copier.yaml
_items: "{{ [var1, var2] }}"
var1: a
var2: b

Possibly we'd need to use https://jinja.palletsprojects.com/en/3.0.x/nativetypes/ in this case.

Multiple prompts

If _items is a template, then we don't need any change in prompts. Users already can define validators, so they can make the form as complex as they want. Then, in _items, things would be combined as needed. Most use cases would be doable with just that IMHO.

[rafalkrupinski]

This means that each template file is rendered multiple times, even if the output file name is the same, so I need to add a check for that.

If you keep a record of rendered files, you can skip them if the full file name is already in the record, for example. It shouldn't be too hard.

That's why I intend to do with directories, but not for files. Read on.

In any case, this might never become a real problem. I expect that files that get rendered multiple times produce the same output, so the only problem will be a few CPU cycles wasted.

Not if the template contents depends on item, in which case the resulting file is rendered using the last one. It's would be confusing to users, so it should fail.

I have a couple of kinds of items, but only one loop variable (items/item). Allowing multiple looped variables in a single template would make it more complex, it would need to be a nested loop for each looped variable

It seems to me that, if _items is a template, that should be enough for any case.

# copier.yaml
_items: "{{ [var1, var2] }}"
var1: a
var2: b

Possibly we'd need to use https://jinja.palletsprojects.com/en/3.0.x/nativetypes/ in this case.

So, in this solution you're actually saying

_items:
- a
- b
a:
- item of kind a
- ...
b:
- item of kind b
- ...

I must be a missing something bc I don't see a point of using a template here.

Multiple prompts

If _items is a template, then we don't need any change in prompts. Users already can define validators, so they can make the form as complex as they want. Then, in _items, things would be combined as needed. Most use cases would be doable with just that IMHO.

Oh, that's why you want to use a template. Cool.

[sisp]

I have another idea how to implement "looped templates" which IMO generalizes to a broader range of use cases. It will allow to handle nested loops and offers granular control over whether a new loop context should be created or whether an item in the current loop context should be accessed. Let me elaborate.

A loop context can be created in three places and this context is propagated down in the filesystem hierarchy:

1. Folder: One folder for each item in the looped iterable is created. Any folders, files, and file contents within that folder are typically associated with the item used to create that folder. Any folders, files, and file contents within that folder can create new loop contexts themselves.
2. File: One file for each item in the looped iterable is created. The contents of these files are typically associated with the item used to create that file. New loop contexts can be created in file contents as well.
3. File content: A regular Jinja loop is used to loop over the iterable in the content of a file.

This may be a bit abstract, so let's get more concrete.

With the following variables available in the Jinja context

strings = ['a', 'b', 'c']

integers = [1, 2, 3]

dicts = [
    {
        'folder_name': 'folder_1',
        'file_name': 'file_1',
    },
    {
        'folder_name': 'folder_2',
        'file_name': 'file_2',
    },
    {
        'folder_name': 'folder_3',
        'file_name': 'file_3',
    },
]

a Copier template

📁 my_copier_template
├── 📄 copier.yml
├── 📁 folder_loop
│   └── 📁 {{ strings | foreach('item') }}
│       └── 📄 {{ item }}.txt.jinja
├── 📁 nested_folder_loop
│   └── 📁 {{ strings | foreach('string_item') }}
│       └── 📁 {{ integers | foreach('integer_item') }}
│           └── 📄 {{ string_item }}_{{ integer_item }}.txt.jinja
├── 📁 file_loop
│   └── 📄 {{ strings | foreach('string_item) }}.txt.jinja
├── 📁 folder_loop_dict_items
│   └── 📁 {{ dicts | foreach('dict_item') | map(attribute='folder_name') }}
│       └── 📄 {{ dict_item.file_name }}.txt.jinja
└── 📁 file_contents_loop
    └── 📄 my_file.txt.jinja

would generate the project

📁 my_project
├── 📁 folder_loop
│   ├── 📁 a
│   │   └── 📄 a.txt
│   ├── 📁 b
│   │   └── 📄 b.txt
│   └── 📁 c
│       └── 📄 c.txt
├── 📁 nested_folder_loop
│   ├── 📁 a
│   │   ├── 📁 1
│   │   │   └── 📄 a_1.txt
│   │   ├── 📁 2
│   │   │   └── 📄 a_2.txt
│   │   └── 📁 3
│   │       └── 📄 a_3.txt
│   ├── 📁 b
│   │   ├── 📁 1
│   │   │   ├── 📄 b_1.txt
│   │   │   └── 📄 ...
│   │   └── 📄 ...
│   └── 📄 ...
├── 📁 file_loop
│   ├── 📄 a.txt
│   ├── 📄 b.txt
│   └── 📄 c.txt
├── 📁 folder_loop_dict_items
│   ├── 📁 folder_a
│   │   └── 📄 file_a.txt
│   ├── 📁 folder_b
│   │   └── 📄 file_b.txt
│   └── 📁 folder_c
│       └── 📄 file_c.txt
└── 📁 file_contents_loop
    └── 📄 my_file.txt

where in the file content of file_contents_loop/my_file.txt a regular Jinja loop could be used to iterate over any of those iterable variables.

In addition, if a zip function were available in Jinja (which we can add, of course), then several iterable variables (with the same length) could be combined, e.g. the Copier template

📁 my_copier_template
├── 📄 copier.yml
└── 📁 folder_loop_zipped
    └── 📁 {{ zip(strings, integers) | foreach('zipped_item') | map(attribute='0') }}
        └── 📄 {{ zipped_item[1] }}.txt.jinja

would generate the project:

📁 my_project
└── 📁 folder_loop_zipped
    ├── 📁 a
    │   └── 📄 1.txt
    ├── 📁 b
    │   └── 📄 2.txt
    └── 📁 c
        └── 📄 3.txt

And similar to list-based iterables, looping over dict items is also possible thanks to Jinja's items() filter.

So what's happening here? foreach(var: str) is a custom filter we would need to implement which converts each item of the iterable input into a new "loop item" object (that contains the original item value and the var string). Copier's folder/file name rendering would need to be extended to recognize the "loop item" object to generate the folder/file name from the original value and create a Jinja context where a variable name according to var contains the item value. Copier would also need to manage the lifecycle of that context. Of course, var name collision would need to be detected and should raise an error. And there may be edge cases like mixing regular looped folder/file names with generated directory structures. But those are details in the implementation that need to be addressed properly and tested.

And just like with regular Jinja for loops, the special variable loop (see the table in that section) could be added to the context as well, if that's something people would like to have.

Where these iterables come from is another possibly separate topic for discussion. Above, the examples by @rafalkrupinski showed a new special variable _items which wouldn't allow multiple iterables to co-exist. If statically defined iterables are required for some use cases, we could think about adding a variable in copier.yml like this:

_items: # or `_data` or ...
  strings:
    - a
    - b
    - c
  integers:
    - 1
    - 2
    - 3
  dicts:
    - folder_name: folder_1
      file_name: file_1
    # ...

And as also shown above, these lists could be populated dynamically with rendered answers from the questionnaire. Answers to YAML questions could be lists as well, or once I've implemented #750, list answers would also be more native to Copier.

I haven't done a proof of concept implementation, but I imagine this should be possible.

WDYT, @rafalkrupinski @yajo @pawamoy?

[yajo]

I don't think you need to combine for and yield. Just: {% yield single_var from looped_var %}{{ single_var }}{% endyield %}

The implementation could be similar to https://github.com/orgs/copier-org/discussions/908#discussioncomment-6505536:

Does single_var exist in context?

• yes: do nothing special
• no: raise YieldException("single_var", looped_var)

OK, we got the exception. Now, are you rendering a filename?

• Yes: start loop. Inject single_var in the context and keep rendering the contents (of the dir/file, recursively).
• No: just let the exception raise and print a nice message "Using yield outside of file names isn't supported".

Do you think yield is too generic? Maybe fileyield, iterfiles, filefor or something that obviously tells you it will only work in file/dir names?

[sisp]

{% yield single_var from looped_var %}{{ single_var }}{% endyield %}

looks nice! 😍 And this way there's no problem with rendering with a singleton list anymore as single_var is replaced and not looped_var. 👌

I think yield isn't too generic. Using another name that contains "file" sounds a bit misleading to me, as if iterating worked only on files but not directories. Wouldn't you agree? Somehow, "yield" sounds catchy to me. 😆

[yajo]

Haha ok 😁👍

Well, it looks we have a plan. Let me create the issue.

[yajo]

See #1271.

I've put it in the "later" milestone, but if someone wants to contribute it, we can move it to "soon". It should be easier now that the design is clear.

[rafalkrupinski]

Do I understand correctly that you've been discussing here how to generate names for output files in a loop?
If so, how would you match such generated file name with the data element to feed to the template?

[rafalkrupinski]

I've written a POC in a side project inspired by @sisp idea. It turned out quite simple really (at least as a side project, not sure about integration with copier).

1. when rendering a file (or directory) name pass a function loop_over that takes an iterable and raises it in an exception.
2. when the filename template calls it, the engine catches the exception and switches to the loop mode
3. in loop mode, instead of rendering the file (or dir) content, it iterates over the passed iterable, and renders name again, only this time loop_over is a lambda returning the current item
4. when rendering file contents, engine adds a variable item with the current value

The name template might look like this;
{{ loop_over(mydict.items())[0]}} <- use key, the first element of tuple from .items()
The contents:
{{item[1].property}}

I guess loop_over might also take a name arg, so nested loops can name their items differently.

The project is here and currently uses Mako templates.