large project for testing

[urbanjost]

For testing I put GPF in the form of an fpm project; as it has a lot of files; so it tests things typical packages might not. Thought I would mention it in case anyone else wants an atypical test package to play with.

I have tried that a few times before but this time it worked, and worked well with a few limitations.
In the past there were problems with building C, with the time it took to scan for a rebuild with a run or test command, and so on when I tried that. Terrific improvement in speed since the last test It requires having several packages installed to build but if anyone wants a test case with hundreds of files in it to test with there it is. Requires a POSIX system. Requires several commonly available libraries (see the apt-get list on the github page if interested). With those restrictions it worked really well!

Just half-formed ideas from using GPF as a test so far ...

• should there be an equivalent of "make -k" that keeps going on a build and builds as much as possible?

• gfortran will pass C code as well as Fortran, but a lot of compilers will not. Do we need to specify a C compiler and defaults for the Fortran compilers we have listed as well,
  or maybe a list of compilers with associated file suffixes? That is, if I specify "-compiler ifort" should it automatically select the C C++ coprocessors for ifort and options; or should there be a --c_compiler option; or something like
  --compiler "ifort:.f,f90 icc:.c bash:.sh" or some other scheme? Some of that has come up before; but right now I have to
  hack some odd scripts together to use fpm to build Fortran/C with compilers that will not cross-compile multiple languages.

Should we be thinking of making a generic "compile" command that handles different compilers? I mention that because in my own environment I use a ccall(1) command that looks to determine what system it is on via uname and some other commands and what compilers are available (which can all be overridden by environment variables) and then using the file suffix builds or installs a file (*.f, *.f90, *.man, *.[1-8], *.c, *.cc, *.md, ...). that has worked well for me for a long time. A generic compile command as a separate utility would free the core fpm machine up from having to know programming-environment-specific information to a large degree.

That works
now in a fashion for fpm if you create a command called "mycompiler" in your path and use "fpm --compiler mycompiler", assuming your script can handle some common compiler switches. Sort of like the Cygwin "cygstart" command or (depending on the system) "start", "open", ... except just for compiling, loading, installing and translating.

• should there be an option on run and test to not do a rebuild (in earlier versions only an explicit build did a build anyway; then I was convinced it was needed due to speed; now with a decent sized build it seems amazingly good at doing the check all the time -- nice).
• should we ultimately depend on pre-processing or directories that are only built for a specific OS conditionally? It would be nice if I could tell fpm easily to only build certain files or only build from a certain directory if my OS type were MSWindows or Solaris, for example. If using preprocessing it would be nice if the compilers were passed the equivalent of

 -D$COMPILER -D$OS_TYPE 

I also tried hookingthis test package up to ford but I have a few non-ANSI and/or unicode characters in there that seem to crash it; and some non-ford non-doxygen comments that are coincidentally already in "ford-context" so I did not get too far; but plan on taking a second swipe at that.

[LKedward]

Great stuff @urbanjost! I'll be using this as a performance metric for fpm among other things. Over 2000 objects to build!

~/git/fpm_packages/general-purpose-fortran$ fpm build --list  2>&1|wc -l
2165

I ran into a few problems when building in parallel which I think are down to there being multiple module definitions called DATA. EDIT: Also multiple m_testsuite modules as well.
Was a bit awkward to track down; we should have a check in fpm for duplicate module definitions (#396).

Also git clone --depth 1 helped speed up the checkout a lot — looks like you've got some big objects in your git history!

[urbanjost]

I trimmed the visible commits. The check for duplicate modules seems like a good idea; but not sure what visible duplicates there are that would cause a problem. I did a parallel build with make(1) without a problem. Does the last master branch allow parallel builds? Did not know that!

Definitely duplicates in the tests; been refactoring the tests. Usually had the tests right in the modules so you could call the test_suite routine in each module. In my own system a script gets the names of all the routines in a module and builds a skeleton with an empty test for each routine to insert into the module that defaults to saying "UNTESTED". But that requires being built in my environment because the tests require everything I started to move to M_msg/M_verify and a bunch of routines for comparing floats and doing basic statistical analysis of arrays and comparing files numerically; which makes them hard to make public in repositories, as there are so many dependencies for the tests. Still moving those and cut and pasted some module names and got duplicates; hence my vote for your duplicate detection being a good idea; but it worked as long as each test was built individually as the modules were just for use with each test.

So wondering if you would make duplicates an error or a warning if they occurred in test/ or example/ or app? Definitely an issue with src/ files, especially if using a lot of external dependencies, so would be really good to detect the duplicates.

[LKedward]

Does the last master branch allow parallel builds? Did not know that!

Yep! But only when built with openmp; I recently updated the install.sh script to make it easier to build with openmp when bootstrapping with the Haskell implementation.

So wondering if you would make duplicates an error or a warning if they occurred in test/ or example/ or app?

Good point, I had originally considered any duplicate module to be a fatal problem, but you're right; since fpm has a special restricted scope for test/executable/example modules, it is feasible that duplicates could be tolerated as long as they 'belong' to different programs. As I've shown, this doesn't strictly work at the moment since all .mod files are put in the same place which causes problems when building in parallel. A straightforward solution is to place program scope .mod files in their own directory to avoid clashes between different programs. As you say any duplication from the src/ directories is a fatal error however.