Auto-install

[ghost]

In MikTeX I can write

\usepackage{amsmath}
\usepackage{geometry}

and MikTeX will download and install the required packages for me.

In Fortran, it would be nice to be able to write

use stdlib_linalg

and fpm would download and install stdlib_linalg for me.

What we need

• what-provides webservice
• what-provides client (fpm itself)

what-provides webservice

Example request: https://fortran-lang.org/what-provides?module=stdlib_linalg
Example response: {git = "https://github.com/LKedward/stdlib-fpm.git"}

what-provides client (fpm itself)

In a newly created fpm project, one would write

program main
use stdlib_linalg
end program main

and run fpm run; fpm would query the what-provides webservice and update fpm.toml for me.

The message is currently
Unable to find source for module dependency: "stdlib_linalg" used by "app\main.f90"

I find this inconvenient, because in this example I know exactly what the problem is, and we can automate the fix.

For security, users should be able to configure their allow-auto-list in $HOME/fpm.toml:

[allow-auto-install]
my-repos, univesity-repos, fortran-lang-org-repos, other-trusted-repos, etc

What do you think?

[ghost]

When a build fails with Unable to find source for module dependency: "stdlib_linalg", fpm can call an fpm plugin

fpm-handle-unable-to-find-module "stdlib_linalg"

If the plugin exits with SUCCESS, then fpm can automatically attempt a rebuild.

[arjenmarkus]

That looks attractive, indeed. Perhaps some caution is required: - Different compilers will differ in the exact message - Can we guarantee that we get the right module? (Perhaps limit this to stdlib modules?) Op do 15 jul. 2021 om 06:03 schreef Carlos Une ***@***.***>:

…

When a build fails with Unable to find source for module dependency: "stdlib_linalg", fpm can call an fpm plugin fpm-handle-unable-to-find-module "stdlib_linalg" If the plugin exits with SUCCESS, then fpm can automatically attempt a rebuild. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#519 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR2GCBG6NNOUTWMP5MLTXZMYJANCNFSM5AMUWVBQ> .

[ghost]

Good points, thanks @arjenmarkus.

The idea is to use an "allow-list" only, the user can add to "allow-list" only the modules he commonly uses (stdlib, common libs in their field, etc). I think this is safe.

[LKedward]

Yes this can be automated and would certainly be a nice convenience. Presumably the plugin will add the new dependency to the cache file? One downside is that this could take several fetch-scan iterations for deep dependency hierarchies. I am also concerned about getting the right module since Fortran doesn't have namespaces and there might be a lot of clashes in the future perhaps (see also #153). I'd probably prefer a new command such as fpm add stdlib which acts similar to fpm-search and adds to the toml or cache file.

[ghost]

@LKedward, I didn't think about the scenario with deep dependency hierarchies. Yes, this feature/idea needs more work.

I don't see a case with lots of clashes. When I write use stdlib_xxx I want to use stdlib from fortran-lang.org in 100% of cases. I think this won't cause harm, because the users should configure in $HOME/fpm.toml what modules they allow to use this feature.

[ghost]

The idea is to configure $HOME/fpm.toml what repositories the user trusts, not the modules... need to develop this idea

[urbanjost]

I use a preprocessor for almost code (mostly for documentation) and I made $INCLUDE call curl for URL syntax,( but that is specifically a particular URL so it did not cause many issues with clashes) and was thinking along somewhat similiar lines about a $USE, but I can see that doing it via a scan of the Fortran directly is very appealing, but if I had something like "use lapack" I could picture that really meaning a local package or one of several fpm packages. Of course if it was supposed to be local hopefully I had put that in my external libraries to load; but that would mean you have to check a name to make sure it it not specified in some other way via the fpm.toml file; but I like the idea. I could just write a program acting like all registered fpm packages were standard intrinsics if they were in modules and "automagically" get them; but I think there should be a flag to do it automatically that is not the default, and that you get prompted by default as to whether to add it or not.

Maybe start with just doing this for stdlib? Might encourage stdlib usage and make it feel more "standard" too when using fpm. Maybe do it without a prompt if followed by a comment with the URL?

[ghost]

Hi @urbanjost. I agree, doing this for stdlib is a good start.

The user should configure it in $HOME/fpm.toml:

enable-auto-install-plugin = true
prompt-whether-to-add-it-or-not = true

[auto-install]
allow-list = [
"https://git.domain/me",
"https://git.domain/org1",
"https://git.domain/org2"
]
# deny everything else by default

When I write

use plato
use kant

If plato and kant don't exist in my app source tree then fpm can ask the plugin. The plugin should search for the modules, using its own database or (web)service. If plato module is not found in a trusted repository in allow-list, then the plugin should return NOT_FOUND. If there are multiple repositories providing the requested module, the plugin should exit with CONFLICT.

Maybe do it without a prompt if followed by a comment with the URL?

If the URL is found in the allow-list, yes it is do-able, but I think it's a bit ugly. I would rather configure the trusted repositories in $HOME/fpm.toml once and "forget" about it.

[zoziha]

Background

Maybe my next discussion is not directly related to this topic.

I learned that the process of fpm running is to first parse the source code files and generate fpm_model, and then parse and generate fpm_backend.
Almost all these processes complete these conversions in RAM memory, and what is left is the .digest files.

RAM has no memory if once fpm ends its running, but hard disk storage has memory.
The use of .digest files is to use the memory function of the hard disk storage, so that we can ignore the unmodified files and not be compiled multiple times, which is great!

Description (Here comes my question:)

Is it possible for fpm or its plugin to convert file dependencies, that is, fpm_model and fpm_backend, into toml files, similar to fpm_model.toml and fpm_backend.toml, they are generated in the build/cache folder.
This behavior will likely become the basis for the complex build process and the extension of the fpm plug-in function, because toml is designed to be a more complete file-system with a strong structure than .digest files or something.

Back to #519 discussion

#519 discusses the establishment of a global fpm.toml to automatically install module dependencies.
Of course, the ideas I put forward may also be beneficial to the function of automatically installing modules. We need more ideas and discussions? (Hope fpm will be better~)

[urbanjost]

using a bash shell script and fpm-search I made a script that simulates part of this, in that just given a program it looks for USE statements and then finds packages of the same name. As limited as that is, since a good number of packages just contain one module of the same name it is very nice for building a quick program. it makes a scratch directory, starts an fpm package setup with the named dependencies and copies the program into and builds it and does install back into the current directory; making it seem like all those packages are Fortran intrinsics. It is primitive in that it just changes the toml and cache file and reused the scratch directory and leaves it there, but it it just for testing the idea. It does quickly show that deep-diving for dependencies gets complicated quickly when a package contains many modules and they have nothing to do with the name of the package. So would a registered package be required to state all of the modules it builds; or how would you handle the case where something like module A and B are in package BIG?

For the cases where it works, it is almost like there was a giant stdlib that already existed. Just write a program calling a command line parser, colored messages, some statisics and linear algebra routines and "compile" it. No searching for packages, setting up dependencies, creating toml files. I like the idea but it certainly will not be trivial to implement unless there is some kind of catalog that identifies what modules (and what about non-module routines?) are in what packages.

[ghost]

Thanks for testing the idea, @urbanjost, that is exactly what I had in mind. I will write an fpm-catalog plugin and see if it works.

[ghost]

An unrelated issue, I find using stdlib with fpm is not a very pleasant experience for my uses (small programs and homework assignments). Consider 10 homework assignments: I will have to build stdlib 10 times unless I put unrelated exercises on a single fpm project. It looks this problem has no solution.

[zoziha]

Consider 10 homework assignments: I will have to build stdlib 10 times unless I put unrelated exercises on a single fpm project. It looks this problem has no solution.

I have used the R language, and I like the idea of introducing a local compiled package like load package. Here comes my ideas:

Set up fpm local spaces

.local/bin
.local/lib
.local/src
.local/fpm.toml

Allow fpm to install library

Currently fpm supports installing apps, and it would be great to install libraries, such as stdlib.
(Always install the latest version of libraries, if we need a specific version, we can set it in our fpm project fpm.toml)

fpm install stdlib
fpm install https://github.com/fortran-lang/stdlib/tree/stdlib-fpm

Generate tags

When fpm installs stdlib, toml files are generated according to the compiler:

- .local/lib
          |- stdlib(*compiled code)
		|- cache.toml 🔰
		|- gfortran-debug
			|- libstdlib.so
			|- libstdlib.a
			|- stdlib_io.mod
			|- stdlib_strings.mod
			|- stdlib_strings@to_string.smod
			|- ...
		|- gfortran-release
		|- ifort-debug
		|- ifort-release
		|- ...
- .local/src
	 |- stdlib(*source code)
		  
- .local/fpm.toml 🔰

# .local/fpm.toml
libraries = [
"stdlib", 
"toml-f",
"fpm", 
...
]

[stdlib]
git = ""
tag = ""

[toml-f]
git = ""
tag = ""

# .local/~/cache.toml
fpm-package = true
modules = [
"stdlib_io",
"stdlib_strings",
...
]

Introduce local packages in the workspace

- ~workspace
	|- app
	|- build
		|- cache.toml
		|- loaded.toml 🔰
	|- src
	|- test
	|- fpm.toml 🔰

fpm load stdlib

# fpm.toml
[dependencies]
stdlib = { git="", tag="" }

# loaded.toml
libraries = ["stdlib", ... ]

Uninstall library

fpm uninstall stdlib

Unload library in the workspace

We don’t need to uninstall the package, but if you import an unnecessary package by mistake, you do need to uninstall the package.

fpm unload stdlib

Inquire

fpm install --list  # Query local package
fpm search --list   # Query remote package
fpm load --list     # Query workspace loaded packages

For users

If we often need to use a package, we will be happy to:

fpm install stdlib
fpm load stdlib

# Coding ...
fpm run

If users forget fpm load stdlib and use stdlib_io, when fpm run, fpm can remind users whether they need to load stdlib.

Support non-fpm packages

Non-fpm packages are not supported. Non-fpm packages can only be used with links and modules.

The fpm-local package can only be built artificially:

- lib/fgsl
	|- libfgsl.so
	|- libfgsl.a
	|- fgsl.mod

cd lib/fgsl
fpm install fgsl
# - - -
fpm load fgsl

# fpm.toml
link = ["fgsl"]
modules = ["fgsl.mod"]

These functions seem to be an fpm plug-in, but it can also be that fpm itself supports such functions.
There are still many technical details that need to be considered. I don't know what you want to do. If I have the ability, I hope to help you and participate in this process.

[urbanjost]

Well, it should be easier that this, but this bash(1) script illustrates the steps. Remember the drawback to not building it in your packages
is that different compilers and compiler options will not apply.

It is easiest to illustrate with the fpm package version of stdlib, but this would work with any library whether built with fpm or not.

1. clone and build fpm-stdlib in its own directory

2. modify the fpm.toml file to allow for installing a library

3. install it, making sure you specify the output directory you want

4. Make some new packages called A,B,C.

5. change their manifest file to include the external library and the names of the external modules you want to use

6. change the example code in each src directory to use something from stdlib -- in this case to_lower, and to_upper

7. build and run everything using fpm(1) without having to rebuild libstdlib.a, as long as you keep using a compatible build.

For illustrations the script makes a directory called $HOME/github/test/
and builds and installs fpm-stdlib and three seeds for examples there.

Even if not a bash(1) user, I think the script is readable enough to show the steps; but maybe this should be annotated and written up somewhere; as a few users of libGPF.a library, which also takes a while to build, were dealing with the same issue.

 #!/bin/bash
WHERE=$HOME/github/test
mkdir -p $WHERE
cd $WHERE
#######################################################
STDLIB(){
git clone https://github.com/LKedward/stdlib-fpm
(cd stdlib-fpm;fpm build)
cat >>stdlib-fpm <<\EOF
[install]
library = true
EOF
(cd stdlib-fpm;install --prefix $WHERE )
}
#######################################################
# make a manifest file that uses external stdlib module
MAKETOML(){
cat <<EOF
name = "$NAME"
version = "0.1.0"
license = "MIT"
author = "$MY_NAME"
maintainer = "$USER@$(hostname)"
copyright = "Copyright $(date +%Y), $MY_NAME"
[build]
#####################################
link = ["stdlib"]
external-modules = ["stdlib_ascii"]
#####################################
auto-executables = true
auto-tests = true
auto-examples = true
[install]
library = false
EOF
}
#######################################################
MAKESRC(){
# make a source file that uses stdlib module
cat <<EOF
module $NAME
use stdlib_ascii, only : to_lower, to_upper
  implicit none
  private

  public :: say_hello
contains
  subroutine say_hello
    print *, "Hello, $NAME!"
    print *, to_lower("Hello, $NAME!")
    print *, to_upper("Hello, $NAME!")
  end subroutine say_hello
end module $NAME
EOF
}
#######################################################
#STDLIB
export NAME MY_NAME='John S. Urban'
for NAME in A B C
do
   # make a seed package
   fpm new $NAME
   # overwrite with modified manifest file that includes external
   # link and external module example
   MAKETOML >$NAME/fpm.toml
   # make a source file that uses stdlib module
   MAKESRC >$NAME/src/$NAME.f90
   (
   cd $NAME
   fpm run -flag '-I../include -L../lib -lstdlib'
   )
done
exit

Typing the long list of -flag options can get tedious. You do not have to use -L if you add the directory to LD_LIBRARY_PATH on ULS; some compilers like ifort(1) let you set up files of options or environment variables of options; and generically you can set up a fpm.rsp file and then just use fpm @run instead of fpm run; which is really only described on the M_CLI2 site.

[urbanjost]

Another unrelated related issue I have is git submodules are really awkward as far as I am concerned, but github does not allow for folders of git repositories so it is really hard to lay out a bunch of example packages in a single github repository such that people can access each one individually. I have several dozen little mini-packages I would make public except I would have to create several dozen new repos, which gets ugly pretty quickly. gitlab lets you have folders of repositories, which is really much nicer for setting up little examples or class exercises or for a way to distribute individual programs.

[urbanjost]

Also, if you put each exercise in a subdirectory you can add to the fpm.toml file the specific files and their dependencies, which would let you build the dependencies once, and even have different dependencies for each file while still having them feel separate.