Distributing the fpm model with fpm

[awvwgk]

I'm opening this thread for a bit of brainstorming and coordination on refining fpm model.

We have been talking about creating a standalone tarball (#123) and using other build backends (#69) already. @certik started working on a way to represent the internal model (#291) and I'm currently trying to get a prototype of an fpm-dist command together (e2a837b).

There is some overlap between the two projects and it turns out to be not that simple and therefore might require more coordination before we repeat too much work in this direction.

A viable strategy on this might be:

1. Refine the fpm_model_t to provide the required information
2. Create a distribution outside of the source tree of fpm containing foreign build files
3. Generate foreign build files in the source tree of the fpm project

Let me know what you think on this.

[certik]

I don't think there is any overlap with #291, is it?

There might be some overlap with adding a cmake backend. Once I get there, after #291 is done, I will simply work with you on this together. Do you have any things you want to discuss right now?

[awvwgk]

Not actual overlap from the implementation, but from the goal.

[awvwgk]

To elaborate a bit on the issue, it seems like the fpm model is missing some information, which are necessary to implement a high-level build backend (like cmake and meson). The mapping from the package manifest to the list of source files and build targets loses the information of the original project on the way. But exactly this information is crucial to generate build files for build systems that are working in project scopes.

Currently this means the fpm source files are mixed together with the toml-f and M_CLI2 source files, but it would be beneficial for the backend to actually separate the three targets.

So the main question is, am I overlooking something in our model or do we need a better model first before starting with a new backend?

[LKedward]

One way to do this would be changing the current model and backend to create a separate static library for each dependency, ...

This would actually require very minimal changes to the model and no changes to the backend. We only need to change how we construct the build target list, see fpm_targets.

[certik]

Well, I would like to keep the option open to be able to let it up to the compiler how to build it --- not necessarily force things via the static library option, so that compilers have an option to optimize things the way they like. To rephrase it, I would not encode anything about "static" libraries into the model itself. Rather, leave it to the backend to decide how it wants to actually build the project.

However, we need the information about packages. So I see two ways to do it:

• Lump all Fortran files together, as it is now, but for each file also note which package it comes from, and have those package_t derived types for each package.
• Have a high level package_t for the current project, which depends on other package_t (even recursively) and each package_t has a list of files.

[certik]

The second option above might be preferable to make it easier for the backend to build packages separately (in parallel). If all files are lumped together, then one has to painfully retrieve all files belonging to a given package.

[awvwgk]

Adding some abstraction layer around the current model might also be useful. This could allow to query the model in a highlevel way, like:

• return a list of all active packages in this project
• return all source files associated with a program
• return all source files in a target (including source files from other packages)

This would be a natural follow-up on the show-model functionality and allow to develop the model somewhat independent from possible backends.

[certik]

To keep things simple, I was thinking of designing the model so that it is easy to write functions to deliver the above. If we keep the model as a set of packages and a set of files per package, then it's easy to write a function to recursively go over all packages and collect files, so it becomes an O(n) operation where n is the total number of files. So the current backend would simply call this function internally.

If on the other hand we use a list of files with pointers to a package for each, then it's a lot harder to separate them ---- you have to go over the whole list and extract files per package, so an O(n*m) operations where n is the number of files and m is the number of packages. Unless you recreate the data structure in the previous paragraph, then it should be an O(n) operation. But in that case I feel it's better to simply start with such a data structure in the first place.

[ivan-pi]

Is lumping all the files a prerequisite for using the interprocedural optimization flags available in some compilers (e.g. ifort IPO, xlf -qipa)? The way I understand it, all object files compiled with "ipa" should be present at linking in order for the compiler to do all the possible inlining, transformations, etc. I recall a thread containing some anecdotal evidence that Fortran compilers would produce faster binaries if all sources were in a single file.

[LKedward]

Good point, but I don't think this will be a problem. You can definitely generate lto-compatible static archives using gcc-ar and I assume Intel can do something similar. Worst case, we can iterate over all packages and then over all files within each package to reconstruct the full list.