Calling SML compiled with MLton from C #456

klartext · 2021-10-30T03:16:29Z

klartext
Oct 30, 2021

I want to have a C program calling SML (used as library).
I looked at this:
http://www.mlton.org/CallingFromCToSML
but that is not, what I was looking for.

The example C code contains functions that call SML functions,
but there is no main() in the C file.

What I want to do is: have C functions including the main(), calling the SML-functions.
But my experiments to somehow achieve that did not work - multiple definition of main was one error I got.

Is it possible at all, to call the output of MLton like a static or shared library from a C program (including main())?

If it's possible: where can I find documentation on that?
If it is not possible, will that be possible one day? I would like to ask for that possibility.

Answered by MatthewFluet

Nov 4, 2021

[Consolidating discussion to a single answer.]

Yes, it is possible to create an SML library that is called by a C program. The documentation is at http://mlton.org/LibrarySupport.

The essential process is to create your SML library as an SML program that uses _export "func" public as described in http://www.mlton.org/CallingFromCToSML; note the public symbol scope to make the symbol visible by other DSOs. Compile that program with -export-header foo.h -format library -libname foo (to create a dynamic shared library (i.e., a .so)) or with -export-header foo.h -format archive -libname foo (to create an archive (i.e., a .a)). Then your C program should #include "foo.h" (the header created wh…

View full answer

MatthewFluet · 2021-10-31T01:24:30Z

MatthewFluet
Oct 31, 2021
Maintainer

Yes, it is possible to create an SML library that is called by a C program. The documentation is at http://mlton.org/LibrarySupport.

The essential process is to create your SML library as an SML program that uses _export "func" public as described in http://www.mlton.org/CallingFromCToSML; note the public symbol scope to make the symbol visible by other DSOs. Compile that program with -export-header foo.h -format library -libname foo (to create a dynamic shared library (i.e., a .so)) or with -export-header foo.h -format archive -libname foo (to create an archive (i.e., a .a)). Then your C program should #include "foo.h" (the header created when compiling the SML library) and should call libfoo_open(int argc, const char **argv) (to initialize the MLton runtime, passing any @MLton runtime arguments) before calling any of the exported SML functions and should call libfoo_close() (to shutdown the MLton runtime).

0 replies

klartext · 2021-10-31T14:05:26Z

klartext
Oct 31, 2021
Author

I used

mlton -export-header foo.h -format library -libname foo  ex2.sml

and got:

_export disallowed, compile with -default-ann 'allowFFI true'.

Then I used:

mlton -export-header foo.h -format library -libname foo -default-ann 'allowFFI true' ex2.sml

and got:

/usr/bin/ld: cannot find -lmlton-pic
/usr/bin/ld: cannot find -lgdtoa-pic

I have mlton 20210117 here.

Is there somehere a working example that shows, that it's not only a hypothetical assumption that the compilation as lib should work?

0 replies

MatthewFluet · 2021-10-31T19:43:06Z

MatthewFluet
Oct 31, 2021
Maintainer

Here is a simple, working example: mlton-lib-example.zip

[matthew@shadow mlton-lib-example]$ mlton
MLton 20210117
[matthew@shadow mlton-lib-example]$ make
mlton -default-ann 'allowFFI true' -format library -export-header fib.h -libname fib libfib.mlb
cc -o prog -I. -L. -lfib prog.c
[matthew@shadow mlton-lib-example]$ ls
fib.h  fib.sml  libfib.mlb  libfib.so  Makefile  prog  prog.c
[matthew@shadow mlton-lib-example]$ export LD_LIBRARY_PATH=.
[matthew@shadow mlton-lib-example]$ ./prog 
fib(20) = 10946

Both libmlton-pic.a and libgdtoa-pic.a should be part of the binary release of MLton. They are for those distributed here on GitHub. What platform and package manager are you using to install MLton? It may be that they failed to correctly build and distribute the full suite of libraries.

18 replies

fweimer Nov 1, 2021

I would suspect that the reverse is supported (a -fPIC compiled object file/archive used in a -fPIE executable), else it would be difficult to distribute libraries as just one .a archive.

Yes, linking position-independent code into executables is supported (whether it's PIE or PIC).

klartext Nov 1, 2021
Author

What do you recommend me to get the pic-files I need?
Should I open a separate issue for it?
The "working code" provided above is not working, but could be used as a test case for that issue.

MatthewFluet Nov 1, 2021
Maintainer

You can use the binary release for MLton here on GitHub (https://github.com/MLton/mlton/releases/download/on-20210117-release/mlton-20210117-1.amd64-linux-glibc2.23.tgz), which has all of the requisite files. Else you need to bring the issue up with the package maintainer on Arch -- that's why the libraries are not present on your system.

xyproto Nov 1, 2021

Hi, maintainer of mlton on Arch here. I would like to fix the mlton package if it is missing any files.

When compiling without these two lines:

export CFLAGS="$CFLAGS -fPIC"
export COMPILE_ARGS="-codegen c -cc-opt '-fPIC'"

The files installed with make install with either pie or pic in the name are still these:

/usr/lib/mlton/sml/mlrisc-lib/flowgraph/cfgCountCopies.sml
/usr/lib/mlton/sml/mlrisc-lib/flowgraph/cfgExpandCopies.sml
/usr/lib/mlton/sml/mlrisc-lib/instructions/expandCopies.sig
/usr/lib/mlton/targets/self/libgdtoa-dbg-pie.a
/usr/lib/mlton/targets/self/libgdtoa-pie.a
/usr/lib/mlton/targets/self/libmlton-dbg-pie.a
/usr/lib/mlton/targets/self/libmlton-pie.a

Here is the PKGBUILD file I'm currently working on:

# Maintainer: Alexander F. Rødseth <xyproto@archlinux.org>
# Contributor: Michael Koloberdin <koloberdin@gmail.com>
# Contributor: tochiro@no.spam.mail.berlios.de
# Contributor: Andreas W. Hauser <andy-aur@splashground.de>
# Contributor: Brian De Wolf <arch@bldewolf.com>

pkgname=mlton
pkgver=20210117
pkgrel=2
pkgdesc='Whole-program, optimizing Standard ML compiler'
arch=(x86_64)
# no HTTPS available
url='http://mlton.org/'
license=(BSD LGPL MIT)
depends=(gmp)
makedepends=(git mlton)
options=(staticlibs)
source=("git+https://github.com/MLton/mlton#commit=b1f1f0f0916d28c0d183fba85549d5bf96f1fa41") # tag: on-20210117-release
b2sums=(SKIP)

# For bootstrapping mlton on x86_64:
#makedepends_x86_64=('smlnj' 'setconf')

prepare() {
  cd $pkgname
  sed 's/_BSD_SOURCE/_DEFAULT_SOURCE/g' -i runtime/cenv.h

  # For bootstrapping mlton on x86_64
  #if [ $CARCH = 'x86_64' ]; then
  #  find . -name Makefile -type f -exec setconf {} CFLAGS "$CFLAGS -fPIC" \;
  #fi
}

build() {
  #export CFLAGS="$CFLAGS -fPIC"
  #export COMPILE_ARGS="-codegen c -cc-opt '-fPIC'"

  # For bootstrapping mlton on x86_64
  #if [ "$CARCH" = 'x86_64' ]; then
  #  # Bootstrap with SML-NJ on x86_64
  #  make -C "$pkgname" -j1 smlnj-mlton all-no-docs
  #else
  #  make -C "$pkgname" -j1 all-no-docs
  #fi

  make -C $pkgname -j1
}

package() {
  make -C $pkgname PREFIX=/usr DESTDIR="$pkgdir" install
  for f in $pkgname/doc/license/*-LICENSE; do
    install -Dm644 "$f" "$pkgdir/usr/share/licenses/$pkgname/$f"
  done
}

Should different compilation flags be exported when compiling mlton? Any input is appreciated.

MatthewFluet Nov 2, 2021
Maintainer

@xyproto The default Makefile behavior is to only build libraries in the C compiler's default mode (which, is typically -fPIE on recent Linux systems), but when building a binary release, we should build libraries in all the different modes (-fno-pic -fno-pie, -fPIC, and -fPIE). See:

mlton/Makefile.config

Line 78 in b1f1f0f

RELEASE := false

and:

mlton/runtime/Makefile

Lines 36 to 38 in b1f1f0f

    
           WITH_NPI_RUNTIME := $(or $(findstring true,$(RELEASE) $(WITH_ALL_RUNTIME)),false) 
        
           WITH_PIC_RUNTIME := $(or $(findstring true,$(RELEASE) $(WITH_ALL_RUNTIME)),false) 
        
           WITH_PIE_RUNTIME := $(or $(findstring true,$(RELEASE) $(WITH_ALL_RUNTIME)),false)

When building a source release (like https://github.com/MLton/mlton/releases/download/on-20210117-release/mlton-20210117.src.tgz), we set the RELEASE variable to true:

mlton/Makefile

Lines 336 to 344 in b1f1f0f

    
           .PHONY: version 
        
           version: 
        
           	@echo 'Instantiating version numbers.' 
        
           	$(SED) \ 
        
           		-e "s/^MLTON_VERSION := .*/MLTON_VERSION := $(MLTON_VERSION)/" \ 
        
           		-e "s/^RELEASE := .*/RELEASE := true/" \ 
        
           		< "$(SRC)/Makefile.config" \ 
        
           		> z && \ 
        
           		$(MV) z "$(SRC)/Makefile.config";

mlton/Makefile

Lines 448 to 456 in b1f1f0f

    
           .PHONY: source-release 
        
           source-release: 
        
           	$(MAKE) clean 
        
           	$(MAKE) MLTON_VERSION=$(MLTON_VERSION) version 
        
           	$(MAKE) -C doc/guide 
        
           	$(TAR) cvzf ../mlton-$(MLTON_VERSION).src.tgz \ 
        
           		--exclude .git --exclude package \ 
        
           		--transform "s@^@mlton-$(MLTON_VERSION)/@S" \ 
        
           		*

If you would like to build from the git sources (rather than from the packaged source release), then I would suggest adding

make MLTON_VERSION="$pkgver" version

to the prepare step.

You can drop the replacement of _BSD_SOURCE with _DEFAULT_SOURCE, since it no longer applies (see cabaa6e).

The export CFLAGS isn't actually having any effect, because that Makefile variable is set explicitly:

mlton/runtime/Makefile

Line 41 in b1f1f0f

CCFLAGS :=

You could achieve the behavior by using

export CFLAGS="$CFLAGS -fPIC"
make -C $pkgname CFLAGS=${CFLAGS} -j1

but I wouldn't suggest doing so, since that would build all variants of the runtime with -fPIC

The export COMPILE_ARGS also isn't having any effect, because that Makefile variable is no longer used (bf96a96), and the new variables are set explicitly:

mlton/Makefile.config

Line 34 in 4a86efc

MLTON_COMPILE_ARGS :=

mlton/Makefile.config

Line 42 in 4a86efc

OLD_MLTON_COMPILE_ARGS :=

All in all, I think the PKGBUILD could be simply

prepare() {
  cd $pkgname
  make MLTON_VERSION=$pkgver version
}
build() {
  make -C $pkgname
}

If the -j1 was there because Arch defaults to parallel builds and MLton's old Makefile broke with parallel builds, that has since been fixed (#363).

xyproto · 2021-11-03T20:18:07Z

xyproto
Nov 3, 2021

@MatthewFluet Thanks for the detailed answer! The new prepare, build and package functions now look like this:

prepare() {
  cd $pkgname
  setconf Makefile.config RELEASE true
  make MLTON_VERSION="$pkgver" version
}

build() {
  make -C $pkgname
}

package() {
  make -C $pkgname PREFIX=/usr DESTDIR="$pkgdir" install
  for f in $pkgname/doc/license/*-LICENSE; do
    install -Dm644 "$f" "$pkgdir/usr/share/licenses/$pkgname/$f"
  done
}

This works fine, but after building the package, the package checking utility namcap reports the following:

mlton W: ELF file ('usr/bin/mllex') lacks FULL RELRO, check LDFLAGS.
mlton W: ELF file ('usr/bin/mlnlffigen') lacks FULL RELRO, check LDFLAGS.
mlton W: ELF file ('usr/bin/mlprof') lacks FULL RELRO, check LDFLAGS.
mlton W: ELF file ('usr/bin/mlyacc') lacks FULL RELRO, check LDFLAGS.
mlton W: ELF file ('usr/lib/mlton/mlton-compile') lacks FULL RELRO, check LDFLAGS.

Is adding export CFLAGS="$CFLAGS -Wl,-z,relro,-z,now" before make an acceptable solution to this, or would that disturb the compilation of executables or libraries that should not have those flags?

1 reply

MatthewFluet Nov 4, 2021
Maintainer

@xyproto This looks much simpler. In fact, you don't need to do setconf Makefile.config RELEASE true in the prepare function, because one of the actions of the version target is to rewrite Makefile.config with RELEASE set to true.

As for obtaining FULL RELRO, that will depend on exactly what the Arch policy that you are trying to achieve is stating. Do you want just the executables (built by mlton (the compiler) in the process of building mlton (the package)) in the distributed mlton package to have FULL RELRO or should, in addition, mlton (the compiler) distributed by the mlton package default to generating FULL RELRO executables?

I would hope that the policy is the former, because if it is the latter, then it would seem to make more sense for Arch to distribute a binutils that defaults to generating FULL RELRO. Indeed, we prefer to invoke cc (when used as an assembler, C compiler, or linker) with as few options as possible, so as to typically follow the default behavior of cc on the host platform.

To achieve the former (make the executables of the mlton package FULL RELRO), then you would want to change the build function to

build() {
  make -C $pkgname MLTON_COMPILE_ARGS="-link-opt -Wl,-z,relro,-z,now"
}

which will pass -Wl,-z,relro,-z,now to cc when invoked as a linker when (the to be built) mlton is compiling a program. With a little more work, you could convert an LDFLAGS environment variable to an appropriate sequence of -link-opt ... options, which would make things robust against future changes to LDFLAGS.

To achieve the latter (the to be built mlton compiler defaults to generating FULL RELRO executables, including the executables distributed with the mlton package), then you would want to patch the ./bin/mlton-script file to include the -link-opt -Wl,-z,relro,-z,now option. Of course, the difficulty with that is that it becomes more difficult to disable that behavior. A user could pass -link-opt -Wl,-z,norelro, but there doesn't seem to be a nonow ld option (though, perhaps it doesn't matter if norelro is in effect).

In any case, setting CFLAGS or LDFLAGS as environment variables won't work for a couple of reasons. First, is that those are also explicitly set as Makefile variables, so will mask the environment variables:

mlton/runtime/Makefile

Lines 44 to 45 in b1f1f0f

    
           CFLAGS := 
        
           LDFLAGS :=

Second, mlton does not inspect the environment variables for CFLAGS or LDFLAGS when invoking cc as a C compiler or as a linker; instead, you must explicitly pass such options to mlton as -cc-opt or -link-opt options --- which seems to be acceptable: cc doesn't implicitly honor CFLAGS or LDFLAGS; the idiom is to invoke cc $CFLAGS or cc $LDFLAGS when one wants to compile or link while honoring the environment variables.

klartext · 2021-11-03T23:18:30Z

klartext
Nov 3, 2021
Author

Thanks to all who helped to fix the problem (mlton and Arch).
The working example works now :-)

2 replies

xyproto Nov 4, 2021

@MatthewFluet I ended up using make -C $pkgname MLTON_COMPILE_ARGS="-link-opt -Wl,-z,relro,-z,now", as you suggested, and updated the mlton package to use these prepare/build/package functions:

prepare() {
  cd $pkgname
  make MLTON_VERSION="$pkgver" version
}

build() {
  make -C $pkgname MLTON_COMPILE_ARGS="-link-opt -Wl,-z,relro,-z,now"
}

package() {
  make -C $pkgname PREFIX=/usr DESTDIR="$pkgdir" install
  for f in $pkgname/doc/license/*-LICENSE; do
    install -Dm644 "$f" "$pkgdir/usr/share/licenses/$pkgname/$f"
  done
}

namcap reports no issues, and local tests works fine here.

For now, I think just letting the compiler executables have FULL RELRO is fine.

Thanks for the help!

@klartext I'm glad it's working for you now. 👍

MatthewFluet Nov 4, 2021
Maintainer

Excellent!

MatthewFluet · 2021-11-04T12:52:03Z

MatthewFluet
Nov 4, 2021
Maintainer

[Consolidating discussion to a single answer.]

Yes, it is possible to create an SML library that is called by a C program. The documentation is at http://mlton.org/LibrarySupport.

The essential process is to create your SML library as an SML program that uses _export "func" public as described in http://www.mlton.org/CallingFromCToSML; note the public symbol scope to make the symbol visible by other DSOs. Compile that program with -export-header foo.h -format library -libname foo (to create a dynamic shared library (i.e., a .so)) or with -export-header foo.h -format archive -libname foo (to create an archive (i.e., a .a)). Then your C program should #include "foo.h" (the header created when compiling the SML library) and should call libfoo_open(int argc, const char **argv) (to initialize the MLton runtime, passing any @MLton runtime arguments) before calling any of the exported SML functions and should call libfoo_close() (to shutdown the MLton runtime).

Here is a simple, working example: mlton-lib-example.zip

[matthew@shadow mlton-lib-example]$ mlton
MLton 20210117
[matthew@shadow mlton-lib-example]$ make
mlton -default-ann 'allowFFI true' -format library -export-header fib.h -libname fib libfib.mlb
cc -o prog -I. -L. -lfib prog.c
[matthew@shadow mlton-lib-example]$ ls
fib.h  fib.sml  libfib.mlb  libfib.so  Makefile  prog  prog.c
[matthew@shadow mlton-lib-example]$ export LD_LIBRARY_PATH=.
[matthew@shadow mlton-lib-example]$ ./prog 
fib(20) = 10946

If there are linking errors (e.g., /usr/bin/ld: cannot find -lmlton-pic), then it may be that there are packaging errors with the MLton package being used (e.g., they failed to correctly build and distribute the full suite of libraries).

1 reply

xyproto Nov 4, 2021

I can confirm that the example code in mlton-lib-example.zip and the above commands works on Arch Linux, using the latest mlton package.

bikallem · 2024-03-05T20:34:36Z

bikallem
Mar 5, 2024

I am playing around with the provided sample code from this discussion(i.e. calling sml library from C (main). One thing I curious/confused about is what happens to the Mlton garbage collection. Is the Mlton GC not invoked in this scenario?

1 reply

MatthewFluet Mar 6, 2024
Maintainer

The MLton garabage collector can be invoked anytime the code is executing in SML. The call to libfoo_open initializes the runtime (i.e., creates the initial heap).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Calling SML compiled with MLton from C #456

{{title}}

Replies: 7 comments 23 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Calling SML compiled with MLton from C #456

klartext Oct 30, 2021

Replies: 7 comments · 23 replies

MatthewFluet Oct 31, 2021 Maintainer

klartext Oct 31, 2021 Author

MatthewFluet Oct 31, 2021 Maintainer

fweimer Nov 1, 2021

klartext Nov 1, 2021 Author

MatthewFluet Nov 1, 2021 Maintainer

xyproto Nov 1, 2021

MatthewFluet Nov 2, 2021 Maintainer

xyproto Nov 3, 2021

MatthewFluet Nov 4, 2021 Maintainer

klartext Nov 3, 2021 Author

xyproto Nov 4, 2021

MatthewFluet Nov 4, 2021 Maintainer

MatthewFluet Nov 4, 2021 Maintainer

xyproto Nov 4, 2021

bikallem Mar 5, 2024

MatthewFluet Mar 6, 2024 Maintainer

klartext
Oct 30, 2021

Replies: 7 comments 23 replies

MatthewFluet
Oct 31, 2021
Maintainer

klartext
Oct 31, 2021
Author

MatthewFluet
Oct 31, 2021
Maintainer

klartext Nov 1, 2021
Author

MatthewFluet Nov 1, 2021
Maintainer

MatthewFluet Nov 2, 2021
Maintainer

xyproto
Nov 3, 2021

MatthewFluet Nov 4, 2021
Maintainer

klartext
Nov 3, 2021
Author

MatthewFluet Nov 4, 2021
Maintainer

MatthewFluet
Nov 4, 2021
Maintainer

bikallem
Mar 5, 2024

MatthewFluet Mar 6, 2024
Maintainer