Linux support

[tafia]

Is linux supported out of the box?
If yes, what is the recommended way to compile on linux?

[MarcusKlik]

Hi @tafia, fstlib is verified to compile correctly with the VS, Clang, GCC and Intel compilers. In the project I've currently only included a build setup using Visual Studio 2017, but it would be nice to include also a build setup on linux, what are your tools of choice on linux?

greetings

[tafia]

Thanks for answering!
GCC or CLang would be nice!

[MarcusKlik]

Great, the compiler options that are used by R on Linux are:

-O3  -fopenmp -Wall -pipe -Wno-unused -pedantic  -mtune=generic -DNDEBUG

The library uses OpenMP for multi-threading, so that one is very important. O3 helps making the compressor- and bit-shifter code faster...

[tafia]

I'm still struggling building it on linux. Do you have per chance the whole commands used with gcc?

[MarcusKlik]

Hi @tafia, thanks for spending time on that! The Makevars file used by R can be found here. This file uses macro's that are defined by the R framework (such as SHLIB_OPENMP_CXXFLAGS), but the general structure should be comparable to a normal makefile. The actual build output on Linux can be seen on the Travis build machine that compiles the package for Linux. The actual build log is located under the tab R CMD check out logs. The complete compiler options from that log are (here for the code file compression.o):

g++ -std=gnu++11 -I/home/travis/R-bin/lib/R/include -DNDEBUG -fopenmp -I. -Ifstcore
 -Ifstcore_v1 -Ifstcore/LZ4 -Ifstcore/ZSTD -Ifstcore/ZSTD/common -Ifstcore/ZSTD/decompress
 -Ifstcore/ZSTD/compress -I"/home/travis/R/Library/Rcpp/include" -I/home/travis/R-bin/include
 -fpic  -g -O2 -O0 --coverage -c fstcore/compression/compression.cpp
 -o fstcore/compression/compression.o

You don't need the R libraries obviously :-). Also, Travis uses -O2 which is not ideal.

I apologize for not being able to spend more time on creating a decent build setup for Linux at the moment (I'm finalizing the release of the fst package for R).

But I will get back to doing that as soon as possible!

[damondd]

Here is a working script to create libfst.so in Linux.

compile_fstlib_github.txt

It is generated and modified based on the output of command 'Rscript -e "devtools::document();' executed in fstpackage/fst.

BTW, suggest we re-organize file structure of fstpackage/fstlib to have something similar to fstpackage/fst, e.g. rename lib to core and move files in programs/fstcpp to the parent directory of core.

[MarcusKlik]

Hi @damondd , thanks for supplying the script!

Yes, the current structure is not very obvious. The code in the programs/fstcpp directory is a simple C++ implementation of the fstlib library which is currently only used by the googletest unit tests defined in tests\fstlibtests. The rational for putting it in a separate programs directory is to allow it to grow into a command line tool for manipulation of fst files (that would be one of several example programs that use the fstlib library).

But you're right, that could be more clearly structured and the tests should be able to run without a dependency on the programs directory.

A better approach might be to structure the repository in a way comparable to the ZSTD library and with the googletests run automatically on Travis like in this repository (and with a proper makefile supplied to the user as well)

Thanks for sharing your ideas!

[damondd]

@MarcusKlik, I have created a branch at https://github.com/damondd/fstlib/tree/cmake, which shows how I build fstlib on local Linux environment, just FYI:

cd fstpackage/fstlib
mkdir build
cd build
# build release version (by default)
cmake -DCMAKE_INSTALL_PREFIX=/var/tmp/fstlib ..
# build debug version
# cmake -DCMAKE_INSTALL_PREFIX=/var/tmp/fstlib -DCMAKE_BUILD_TYPE=Debug ..
make
make install

And here are the primary changes in the initial commit:
Copy 'programs/fstcpp/' to new directory 'fstlib' and 'lib/' to 'fstlib/fstcore'.
Unify the way include internal headers like <fstlib/XX.h> <fstlib/fstcore/XX.h>.
Internal gtest is broken for Linux, so copied the latest googletest/include and googletest/src from github.

[MarcusKlik]

Hi @damondd, thanks a lot, that's great!

I will study your setup with much interest when I can. Currently I am travelling and don't really have the means to do so, but I will get back to you in a few weeks (hope you don't mind).

Thanks for sharing your code!

[tafia]

Thanks both of you! It is definitely not as straightforward as I imagined, I'll have a try asap.

[tafia]

The cmake branch worked fine (I mean, I haven't tested the lib yet but it seems that the whole compilation went fine). THANK YOU!

On the other hand, the script didn't work on @MarcusKlik master branch. I am on ubuntu 18.04. I have run the script from the root directory.

$ sh build.sh 
In file included from programs/fstcpp/fsttable.cpp:6:0:
programs/fstcpp/fsttable.h: In member function ‘virtual void BlockWriter::SetBuffersFromVec(long long unsigned int, long long unsigned int)’:
programs/fstcpp/fsttable.h:514:3: error: ‘memset’ was not declared in this scope
   memset(naInts, 0, nrOfNAInts * 4);  // clear NA bit metadata block (neccessary?)
   ^~~~~~
programs/fstcpp/fsttable.h:514:3: note: suggested alternative: ‘wmemset’
   memset(naInts, 0, nrOfNAInts * 4);  // clear NA bit metadata block (neccessary?)
   ^~~~~~
   wmemset
programs/fstcpp/fsttable.h:565:4: error: ‘strncpy’ was not declared in this scope
    strncpy(activeBuf + lastPos, str, pos - lastPos);
    ^~~~~~~
programs/fstcpp/fsttable.h:565:4: note: suggested alternative: ‘wcpncpy’
    strncpy(activeBuf + lastPos, str, pos - lastPos);
    ^~~~~~~
    wcpncpy
programs/fstcpp/fsttable.h: In member function ‘FstTable* FstTable::SubSet(std::vector<std::__cxx11::basic_string<char> >&, long long unsigned int, long long unsigned int) const’:
programs/fstcpp/fsttable.h:618:19: error: ‘find’ is not a member of ‘std’
    auto it = std::find(colNames->begin(), colNames->end(), *colIt);
                   ^~~~
programs/fstcpp/fsttable.h:618:19: note: suggested alternative: ‘end’
    auto it = std::find(colNames->begin(), colNames->end(), *colIt);
                   ^~~~
                   end
g++: error: ./fst_table.o: No such file or directory

[damondd]

@tafia please pull the branch again, I've uploaded FstRowReader and FstColReader, as well as the sample c++ usage code in readme.