Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git repo failing when hashing via http://git2nixsha.dev #209

Closed
eliocamp opened this issue Jun 24, 2024 · 76 comments
Closed

Git repo failing when hashing via http://git2nixsha.dev #209

eliocamp opened this issue Jun 24, 2024 · 76 comments

Comments

@eliocamp
Copy link
Contributor

I'm getting a 500 error for a github package.

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             branch_name = "master", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = ".",
    overwrite = TRUE,
    print = FALSE)
#> Error: Request `curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=1dd5d391d5da6a80fde03301671aea5582643914')` failed:
#>  The requested URL returned error: 500
#> If it's a Github repo, check the url, branch name and commit.
#> Are these correct? If it's an archived CRAN package, check the name
#> of the package and the version number.

The information of repo, branch and commit sha is correct as far as I can tell: eliocamp/metR@1dd5d39

Other similar remotes seem to work correctly.

ggperiodic <- list(package_name = "ggperiodic", 
                   repo_url = "https://github.com/eliocamp/ggperiodic", 
                   branch_name = "master",
                   commit = "74f41571b0f82cf8a46e4f782a0dad148d09df29")
rix(r_ver = "4.3.2",
    git_pkgs = ggperiodic,
    ide = "rstudio",
    project_path = ".",
    overwrite = TRUE,
    print = FALSE)

This seems to be an issue on the git2nisha thing, though. For some reason it fails with the first repo.

curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=1dd5d391d5da6a80fde03301671aea5582643914')$status_code
#> [1] 500

Any commit seems to fail, too.

curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=ff01f509513706facef846887f17a506c0df3563')$status_code
#> [1] 500

Am I missing something in the specification of the remote?

@b-rodrigues
Copy link
Contributor

b-rodrigues commented Jun 24, 2024

This is probably related: #159

I don’t know what’s going on exactly, there seems to be an issue with certain repos. In my tests (for that other issue), I ran the api locally and was able to get the results. So I don’t really know what’s going on yet...

We are working on making this work without using the remote server if Nix is available. So this would be done locally and should work more consistently.

@philipp-baumann
Copy link
Collaborator

Let's also investigate while implementing linked whishlist for 0.8.0: #143

@philipp-baumann
Copy link
Collaborator

Hey @b-rodrigues and @eliocamp This should be solved when hashing locally (the new feature added: ).

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             branch_name = "master", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = "_test",
    overwrite = TRUE,
    print = FALSE)

This generates the following expression in default.nix:

# This file was generated by the {rix} R package v0.8.0 on 2024-07-04
# with following call:
# >rix(r_ver = "219f896bdf192721446db4fedc338eebf732057d",
#  > git_pkgs = metR,
#  > ide = "rstudio",
#  > project_path = ".",
#  > overwrite = TRUE)
# It uses nixpkgs' revision 219f896bdf192721446db4fedc338eebf732057d for reproducibility purposes
# which will install R version 4.3.2.
# Report any issues to https://github.com/b-rodrigues/rix
let
 pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/219f896bdf192721446db4fedc338eebf732057d.tar.gz") {};
  
  git_archive_pkgs = [
    (pkgs.rPackages.buildRPackage {
      name = "metR";
      src = pkgs.fetchgit {
        url = "https://github.com/eliocamp/metR";
        branchName = "master";
        rev = "1dd5d391d5da6a80fde03301671aea5582643914";
        sha256 = "sha256-CLTX347KwwsNyuX84hw2/n/9HwQHBYQrGDu7jFctGO4=";
      };
      propagatedBuildInputs = builtins.attrValues {
        inherit (pkgs.rPackages) 
          checkmate
          data_table
          digest
          Formula
          formula_tools
          ggplot2
          gtable
          memoise
          plyr
          scales
          sf
          stringr
          purrr
          isoband
          lubridate;
      };
    })
   ];
   
  system_packages = builtins.attrValues {
    inherit (pkgs) 
      R
      glibcLocales
      nix;
  };
  
in

pkgs.mkShell {
  LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then  "${pkgs.glibcLocales}/lib/locale/locale-archive" else "";
  LANG = "en_US.UTF-8";
   LC_ALL = "en_US.UTF-8";
   LC_TIME = "en_US.UTF-8";
   LC_MONETARY = "en_US.UTF-8";
   LC_PAPER = "en_US.UTF-8";
   LC_MEASUREMENT = "en_US.UTF-8";

  buildInputs = [ git_archive_pkgs   system_packages   ];
  
}
nix_build(project_path = "_test")
# ...takes a while to build since we cannot get it from the official nix cachix cache ;-)
[100%] Generating gdal.jar, build/maven/gdal-3.8.4.jar, build/maven/gdal-3.8.4-sources.jar, build/maven/gdal-3.8.4.pom, build/maven/gdal-3.8.4-javadoc.jar
Buildfile: /tmp/nix-build-gdal-3.8.4.drv-0/source/swig/java/build.xml

init:
     [echo] version = 3.8.4

compile:
    [mkdir] Created dir: /tmp/nix-build-gdal-3.8.4.drv-0/source/build/swig/java/build/classes
    [javac] Compiling 88 source files to /tmp/nix-build-gdal-3.8.4.drv-0/source/build/swig/java/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 7
    [javac] error: Source option 7 is no longer supported. Use 8 or later.
    [javac] error: Target option 7 is no longer supported. Use 8 or later.

BUILD FAILED
/tmp/nix-build-gdal-3.8.4.drv-0/source/swig/java/build.xml:25: Compile failed; see the compiler error output for details.

Total time: 0 seconds
make[2]: *** [swig/java/CMakeFiles/java_binding.dir/build.make:107: swig/java/gdal.jar] Error 1
make[1]: *** [CMakeFiles/Makefile2:9841: swig/java/CMakeFiles/java_binding.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
error: builder for '/nix/store/ilcpzhq2jpva5pkwvk5r5fwd9gnx7zcj-gdal-3.8.4.drv' failed with exit code 2;
       last 10 log lines:
       >     [javac] error: Source option 7 is no longer supported. Use 8 or later.
       >     [javac] error: Target option 7 is no longer supported. Use 8 or later.
       >
       > BUILD FAILED
       > /tmp/nix-build-gdal-3.8.4.drv-0/source/swig/java/build.xml:25: Compile failed; see the compiler error output for details.
       >
       > Total time: 0 seconds
       > make[2]: *** [swig/java/CMakeFiles/java_binding.dir/build.make:107: swig/java/gdal.jar] Error 1
       > make[1]: *** [CMakeFiles/Makefile2:9841: swig/java/CMakeFiles/java_binding.dir/all] Error 2
       > make: *** [Makefile:146: all] Error 2
       For full logs, run 'nix log /nix/store/ilcpzhq2jpva5pkwvk5r5fwd9gnx7zcj-gdal-3.8.4.drv'.
error: 1 dependencies of derivation '/nix/store/58s1d8wqnl901qma286clvwyxk8m0fri-r-sf-1.0-15.drv' failed to build
error: 1 dependencies of derivation '/nix/store/c35djnxrbn1apdndmv6jf7bmy1bary88-r-metR.drv' failed to build
error: 1 dependencies of derivation '/nix/store/37bgkq0akrcppybv8cgi7wl2gmwpg9d2-nix-shell.drv' failed to build

@b-rodrigues The question I have is why are online hashing vs. local hashing not the same? Can you experimentally change the newly locally implementednix-hash vs. nix hash approach to compute the NAR SRI SHA-256 hashes of the guntared downloaded package directories? I really don't know why could be the case the API request on git2nixsha.dev on port 1506 is failing.

To sum up, the Git repo is not failing now anymore with the newest version (see changelog), however the issue with building seems to be a darwin-aarch64 related Nix issue (currently tested on Mac); guess linux does work. @eliocamp can you briefly confirm? Thanks a lot for reporting and generally your detailed feedback on Nix :-)

@philipp-baumann
Copy link
Collaborator

philipp-baumann commented Jul 4, 2024

We get an internal server error when doing it online; my gut feeling tells me it takes quite long to download (what we do locally) and process the hashing in this case. Maybe quite a bit of (reverse) deps involved. So it maybe something with response timing?

> req <- curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/eliocamp/metR&branchName=master&commit=1dd5d391d5da6a80fde03301671aea5582643914')
> jsonlite::fromJSON(rawToChar(req$content))
$error
[1] "500 - Internal server error"

@philipp-baumann philipp-baumann changed the title Git repo failing Git repo failing when hashing via http://git2nixsha.dev Jul 4, 2024
@eliocamp
Copy link
Contributor Author

eliocamp commented Jul 5, 2024

I'm getting

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

@b-rodrigues
Copy link
Contributor

That’s very surprising, could you do which nix-hash in a terminal?

@eliocamp
Copy link
Contributor Author

eliocamp commented Jul 5, 2024

/nix/var/nix/profiles/default/bin/nix-hash

@b-rodrigues
Copy link
Contributor

b-rodrigues commented Jul 5, 2024

@b-rodrigues The question I have is why are online hashing vs. local hashing not the same? Can you experimentally change the newly locally implementednix-hash vs. nix hash approach to compute the NAR SRI SHA-256 hashes of the guntared downloaded package directories? I really don't know why could be the case the API request on git2nixsha.dev on port 1506 is failing.

@philipp-baumann I’ll work on it this weekend; also I think I’ll get completely rid of the branchName argument, as per our Matrix discussion. After some thinking, I think that what you propose makes the most sense: since the commit correctly identifies the code at a point in time, no need for it, and we might as well get rid of it before submitting to CRAN.

/nix/var/nix/profiles/default/bin/nix-hash

@eliocamp Are you trying to build the same expression as Philipp, above? From an R session running from your system, or a Nix R session that contains rix? I wonder if perhaps (but I don’t know why/how) you local library of packages might interfere?

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             branch_name = "master", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = "_test",
    overwrite = TRUE,
    print = FALSE)

@b-rodrigues
Copy link
Contributor

@eliocamp could you try with latest master? You can now remove the 'branchName':

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = ".",
    overwrite = FALSE,
    print = TRUE)

@philipp-baumann
Copy link
Collaborator

I'm getting

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

@eliocamp that should be solved with #244 for more detailed diagnostics why i implemented this approach, see #243 does it work for you linux too?

@eliocamp
Copy link
Contributor Author

eliocamp commented Jul 16, 2024

Can confirm that

library(rix)

metR <- list(package_name = "metR", 
             repo_url = "https://github.com/eliocamp/metR", 
             commit = "1dd5d391d5da6a80fde03301671aea5582643914")

rix(r_ver = "4.3.2",
    git_pkgs = metR,
    ide = "rstudio",
    project_path = ".",
    overwrite = FALSE,
    print = TRUE)

creates the default.nix file correctly.

nix-build fails with

[100%] Built target python_binding
make: *** [Makefile:146: all] Error 2
error: builder for '/nix/store/vss6vy5px8fw1wh8rq5zrbpj9xx563vk-gdal-3.8.4.drv' failed with exit code 2;
       last 25 log lines:
       >
       > compile:
       >     [mkdir] Created dir: /build/source/build/swig/java/build/classes
       >     [javac] Compiling 88 source files to /build/source/build/swig/java/build/classes
       >     [javac] warning: [options] bootstrap class path not set in conjunction with -source 7
       >     [javac] error: Source option 7 is no longer supported. Use 8 or later.
       >     [javac] error: Target option 7 is no longer supported. Use 8 or later.
       >
       > BUILD FAILED
       > /build/source/swig/java/build.xml:25: Compile failed; see the compiler error output for details.
       >
       > Total time: 0 seconds
       > make[2]: *** [swig/java/CMakeFiles/java_binding.dir/build.make:107: swig/java/gdal.jar] Error 1
       > make[1]: *** [CMakeFiles/Makefile2:10036: swig/java/CMakeFiles/java_binding.dir/all] Error 2
       > make[1]: *** Waiting for unfinished jobs....
       > g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/iygsjhqpc4s47y0xar4pi5pi6pmfigad-libxcrypt-4.4.36/include -shared build/temp.linux-x86_64-cpython-311/extensions/ogr_wrap.o -L/build/source/build -L/nix/store/3v2ch16fkl50i85n05h5ckss8pxx6836-python3-3.11.8/lib -lgdal -o build/lib.linux-x86_64-cpython-311/osgeo/_ogr.cpython-311-x86_64-linux-gnu.so
       > g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/iygsjhqpc4s47y0xar4pi5pi6pmfigad-libxcrypt-4.4.36/include -shared build/temp.linux-x86_64-cpython-311/extensions/gdal_wrap.o -L/build/source/build -L/nix/store/3v2ch16fkl50i85n05h5ckss8pxx6836-python3-3.11.8/lib -lgdal -o build/lib.linux-x86_64-cpython-311/osgeo/_gdal.cpython-311-x86_64-linux-gnu.so
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gdal.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gdalconst.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_osr.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_ogr.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gnm.cpython-311-x86_64-linux-gnu.so -> osgeo
       > copying build/lib.linux-x86_64-cpython-311/osgeo/_gdal_array.cpython-311-x86_64-linux-gnu.so -> osgeo
       > [100%] Built target python_binding
       > make: *** [Makefile:146: all] Error 2
       For full logs, run 'nix log /nix/store/vss6vy5px8fw1wh8rq5zrbpj9xx563vk-gdal-3.8.4.drv'.
error: 1 dependencies of derivation '/nix/store/2rdsad5bw8sx94h5g0cz2wvpa1c8m9yz-r-sf-1.0-15.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kkdyb5g529dac6ysng7mfn9q4jbndidy-r-metR.drv' failed to build
error: 1 dependencies of derivation '/nix/store/b31nmwqrwavgb2b2rcka5sa468fylm8n-nix-shell.drv' failed to build

Not sure if a nix/rix bug or just that that particular commit doesn't work. It seems to be a compilation error and metR doesn't have any source code that needs compiling, though.

@b-rodrigues
Copy link
Contributor

it's sf that fails, in particular gdal apparently, what happens with a more recent commit ?

@eliocamp
Copy link
Contributor Author

Not even

rix::rix(r_ver = "4.3.2",
    r_pkgs = "metR",
    ide = "rstudio",
    project_path = ".",
    overwrite = TRUE,
    print = TRUE)

works. 😱

I don't know if something changed with my system or with rix, but before environments would build in a few seconds and now it's taking ages.

@MGousseff
Copy link

Hello, still trying to use NIX, and I had the same error message. I used the code from your tutorial :

rix(
  r_ver = "latest",
  r_pkgs = c("dplyr", "janitor"),
  git_pkgs = list(
    list(
      package_name = "housing",
      repo_url = "https://github.com/rap4all/housing/",
      commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"
    ),
    list(
      package_name = "fusen",
      repo_url = "https://github.com/ThinkR-open/fusen",
      commit = "d617172447d2947efb20ad6a4463742b8a5d79dc"
    )
  ),
  ide = "code",
  project_path = path_default_nix,
  overwrite = TRUE
)

And get Error in sys::exec_internal(cmd = cmd, args = args) : Executing 'nix-hash' failed with status 1

I'm afraid that for an applied statistician quite non devops savvy, the use of NIX / RIX ; the learning curve is too steep.

@b-rodrigues
Copy link
Contributor

that's weird, are you the latest version of rix?

@MGousseff
Copy link

I installed it like this :

utils::remove.packages("rix")
install.packages("rix", repos = c(
  "https://b-rodrigues.r-universe.dev",
  "https://cloud.r-project.org"
))
require(rix)

Then ran this :

path_default_nix <- tempdir()


rix(
  r_ver = "latest",
  r_pkgs = c("dplyr", "janitor"),
  git_pkgs = list(
    list(
      package_name = "housing",
      repo_url = "https://github.com/rap4all/housing/",
      commit = "1c860959310b80e67c41f7bbdc3e84cef00df18e"
    ),
    list(
      package_name = "fusen",
      repo_url = "https://github.com/ThinkR-open/fusen",
      commit = "d617172447d2947efb20ad6a4463742b8a5d79dc"
    )
  ),
  ide = "code",
  project_path = path_default_nix,
  overwrite = TRUE
)

And got this error message :

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

For information : I use r installed with rig, so if some lib paths are hardcoded it could be a problem.

@b-rodrigues
Copy link
Contributor

instead of tempdir() could you try any other valid path on your computer ? tempdir() is weird sometimes

@MGousseff
Copy link

I got the same result with a "normal" directory. The error only occurs when I set some remote installations with git_pkgs.

@b-rodrigues
Copy link
Contributor

can you then successfully build the generated expression using nix-build?

@b-rodrigues
Copy link
Contributor

b-rodrigues commented Sep 9, 2024

Could you try options(rix.sri_hash="api_server") and then try to generate the expression with Github packages again ?

@MGousseff
Copy link

I did, and the error message is :

Error: Request `curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/rap4all/housing/&commit=1c860959310b80e67c41f7bbdc3e84cef00df18e')` failed:
 Failed to connect to git2nixsha.dev port 1506 after 4193 ms: No route to host
If it's a Github repo, check the url and commit.
Are these correct? If it's an archived CRAN package, check the name
of the package and the version number.

I checked that a normal remotes::install_github works (it does). From what I try to understand git2nixsha means there could be different ways to create the hash and it would cause this to fail ?

@b-rodrigues
Copy link
Contributor

yes there are two ways to compute the hash and both are failing for you. What operating system are you using? does your installation of Nix work? can you run nix-shell --expr "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)" successfully ?

@MGousseff
Copy link

So I typed

nix-shell --expr --show-trace "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)"

(I added --show-trace to have a better error message since the expression alone give few informations).

I get the following error

error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': Connection refused

       … while realising the context of a path

       at «string»:14:9:

           13| let
           14|  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/976fa3369d722e76f37c77493d99829540d43845.tar.gz") {};
             |         ^
           15|

The cannot connect to socket made me think of a permission problem so I ran the same command preceded by sudo. Only then did it downloaded the default.nix and started to build things (it still is as I'm writing this).
Do you think it could be something tied to R not being launched as root ? I'll follow up as soon as the nix-shell command is done executing.

@b-rodrigues
Copy link
Contributor

there's something wrong with your Nix installation, you don't need root to use Nix. How did you install Nix? did you use the determinate systems installer?

@b-rodrigues
Copy link
Contributor

b-rodrigues commented Sep 9, 2024

@philipp-baumann this convo makes me think we need a rix::check_install() that checks if everything is alright and provide some guidance if not

@MGousseff
Copy link

My current version is Ubuntu 22.04.4 LTS.

The determinate systems installer never worked, I didn't save all the error messages, but it did not install seamlessly. I had to :

nix-channel --add https://nixos.org/channels/nixpkgs-unstable
sudo nix-channel --update

and then an apt install nix.
I f the determinate systems is the only Nix installation that works with Rix, I think I'll have to let this aside for a while, because I spent quite some time trying to install Nix, and I'm alas getting late on what I need to deliver.
I appreciate your work but I lack system skills to use it for now.

@MGousseff
Copy link

The result of sudo nix-shell --expr --show-trace "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)"

was the following error message

building '/nix/store/hi03px6v99iri5invmkqmi47caz82bdb-r-curl-5.0.1.drv'...
unpacking sources
unpacking source archive /nix/store/wxldyqldqvs02jafv6pgp31z7qy2a9v2-curl_5.0.1.tar.gz
source root is curl
setting SOURCE_DATE_EPOCH to timestamp 1686179412 of file curl/MD5
patching sources
updateAutotoolsGnuConfigScriptsPhase
configuring
patching script interpreter paths in configure
configure: interpreter directive changed from "#!/bin/sh" to "/nix/store/ir0j7zqlw9dc49grmwplppc7gh0s40yf-bash-5.2-p15/bin/sh"
building
running tests
installing
During startup - Warning messages:
1: package 'utils' in options("defaultPackages") was not found 
2: package 'stats' in options("defaultPackages") was not found 
Error: .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: cannot popen '/nix/store/m1kcpc0n70sam6c1x8fdx6c5vwxjacia-which-2.21/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'
* removing '/nix/store/8yq377d5hgg0hjk93w5na89fnfbq4wdg-r-curl-5.0.1/library/curl'
error: builder for '/nix/store/hi03px6v99iri5invmkqmi47caz82bdb-r-curl-5.0.1.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/v0b2663c9s4kwz0hibqshq94x6z9dwch-r-rix.drv' failed to build

Is this message useful in anyway ?

@b-rodrigues
Copy link
Contributor

yeah I think that something is not quite right with your installation. It's possible to install Nix using the determinate systems installer but also the official Nix installer but it requires more work. I don't think using the ubuntu repos to install Nix will work, because it'll be missing configuration steps that other installers perform. The best approach really is to the determinate systems installer and ideally not have a system wide install of R either

@MGousseff
Copy link

My installation of R is through rig, and it should not be a problem.
Would you like me to remove my nix install and try again the determinate systems nix install and update you with errors I encounter ?

@philipp-baumann
Copy link
Collaborator

@philipp-baumann Thank you for your patience and perseverance. I added the path manually. I had to run nix-shell without sudo as it would not find nix-shell if "sudoed". It is currently copying. I keep you updated once it's done.

that's a good sign, happy to hear :-)

@MGousseff
Copy link

It seems it worked ? Here is the end of the message :

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix
checking for references to /build/ in /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix...
patching script interpreter paths in /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix
rewriting symlink /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix/nix-support/propagated-user-env-packages to be relative to /nix/store/929fx96hhwpbqwcr75c4jwn73kgam79f-r-rix

Now, what would be the safest next steps to try and buil a rix environment using VSCode (or RStudio if you feel it's more stable). Do you think I have to kill my rig R installation and go for a new R installation with the default directory I don't like so much ? Or can I try and launch VSCode with the R plugin and specify the R interpreter I want and launch rix from there ?

@b-rodrigues
Copy link
Contributor

b-rodrigues commented Sep 9, 2024 via email

@MGousseff
Copy link

Well, I think I reached my limit for today :

  • the nix path had disappeared after the last nix-shell --pure --expr --show-trace ...
  • I set it again.

The rix() returned the same error when I tried to install github packages :
Error: Request `curl::curl_fetch_memory(url = 'http://git2nixsha.dev:1506/hash?repo_url=https://github.com/rap4all/housing/&commit=1c860959310b80e67c41f7bbdc3e84cef00df18e')` failed: Failed to connect to git2nixsha.dev port 1506 after 2039 ms: No route to host If it's a Github repo, check the url and commit. Are these correct? If it's an archived CRAN package, check the name of the package and the version number.

Could it be that nix uses a port that my system administrators do not allow ? If so, how could I check this ?

In order to still try to have a small victory, I tried to generate a nix environment with only cran packages and to launch vscode from there. So I ran nix-build and nix-shell in the directory where my default.nix was created. It worked. From there I launched code. And then I had to set, in code, the path to the R interpreter that which R in nix shell returned (here it was /nix/store/fnl3bp7k0qw8cayqz2lciq8sq0xmib2c-R-4.4.1/bin/R).
I checked the library janitor was already installed and library(janitor) returned no error.

So for now I'd say :

  • The nix installation is almost OK : it is probably well installed but the path seems to be destroyed every time I reboot
  • The rix installation seems OK but...
  • No github package can be installed through rix(git_pkgs = list()) as I get Failed to connect to git2nixsha.dev port 1506 after 2039 ms: No route to host

I'd be grateful if one of you could tell me how to check if the problem comes from my network permissions.

I'll stop for today, my brain is "stack overflowed" for today, thank you for your patience and will to help.

@MGousseff
Copy link

Here's the morning update. I worked from home today to test : it seems the problem with github packages comes from the way my university network is configured. From home, the packages seem to install.
I then can run R in terminal mode inside the nix shell (after nix_build and nix-shell) and housing and fusen packages are installed. AT LAST A SMALL VICTORY. Thanks for your help and sorry that our restrictive network made you(us) lose time.

So I have to ask my sysadmins if they would allow the proper ports. Do you know why ports are different when downloading from CRAN and from github ? Is there a way to specify this in the rix function ? At least, I can create the nix environments from home and use them at work.

One thing not resolved, though, is that I have to add path to nix manually after each reboot. I guess there is a way to include a way to add it in some kind of profile, but I admit there are lots of ".bashrc", ".profile" and such things and I don't know where to specify it.

And last but not least, even though I created the nix.default with the ide="code" argument, when I launch VSCode and specify the Rterm path corresponding to the nix install, when I ctr+shift an instruction, it is sent to the R term with weird formatting characters. For instance ls() in my script is passed as 0~ls()1~. I launched VScode from the nix shell with code & Hav you ever encountered this behaviour ?

PS : the bold font is just to help separate issues, not a way to show any impatience, as I'm on the contrary grateful for the help.

@philipp-baumann
Copy link
Collaborator

philipp-baumann commented Sep 10, 2024

Here's the morning update. I worked from home today to test : it seems the problem with github packages comes from the way my university network is configured. From home, the packages seem to install. I then can run R in terminal mode inside the nix shell (after nix_build and nix-shell) and housing and fusen packages are installed. AT LAST A SMALL VICTORY. Thanks for your help and sorry that our restrictive network made you(us) lose time.

Thanks for the update. I think those diagnostics help to strengthen debugging, although not all is directly related to {rix} but rather general nix tooling. It is however stuff that other users could experience. If we can make debugging smoother, e.g. also how @b-rodrigues has suggested add some diagnostics functions that summarize what we already adjust in {rix} to make it work in different contexts. Especially in mixed computer systems where users have R tooling and libraries and linked system dependences at host system and software cleanly managed at nix level in the Nix store.

Ideally you would achieve to adjust your setup at work so that you can also achieve to hash the github sources of R packages locally. I *think you should have nix-hash enabled in your nix setup. So it seems a matter of why and what shell environmental variable linked to nix is not available.

So I have to ask my sysadmins if they would allow the proper ports. Do you know why ports are different when downloading from CRAN and from github ? Is there a way to specify this in the rix function ? At least, I can create the nix environments from home and use them at work.

One thing not resolved, though, is that I have to add path to nix manually after each reboot. I guess there is a way to include a way to add it in some kind of profile, but I admit there are lots of ".bashrc", ".profile" and such things and I don't know where to specify it.

There is an easy way to do it. You can use your .bashrc or .profile in your $HOME dir. So you just need to add those lines. I don't know your IT settings and maybe the IT has some custom stuff doing with your linux instance, but when you open bash the one in your home should be sourced.

# add these lines in your $HOME/.bashrc or $HOME/.profile substituting your actual username
export PATH=$PATH:/home/<your-username>/.nix-profile/bin:/nix/var/nix/profiles/default/bin

And last but not least, even though I created the nix.default with the ide="code" argument, when I launch VSCode and specify the Rterm path corresponding to the nix install, when I ctr+shift an instruction, it is sent to the R term with weird formatting characters. For instance ls() in my script is passed as 0~ls()1~. I launched VScode from the nix shell with code & Hav you ever encountered this behaviour ?

If you use the R extension which i would recommend, then you need to enable bracketed paste.

https://francojc.github.io/posts/r-in-vscode/ There is a section with the lines to add here.

I think its a matter of preference, but i usually open vscode/vscodium first on my host and then just use nix-shell and then prompt R in a vscode bash terminal to send code to specific code in scripts to nix-R sessions. That works well. You can also manually add languageserver to r_pkgs and this does work as well if you default to ide = "rstudio" and somebody else wants to work in vscode etc.

PS : the bold font is just to help separate issues, not a way to show any impatience, as I'm on the contrary grateful for the help.

No offence taken.

@MGousseff
Copy link

Thank you. I added the path in my .profile ut added a line to check if it was already in and this didn't seem to work.

I will go back to my workplace and see what the error message is concerning git packages and see what blocks non cran packages and see if I can get a proxy set.

I still encounter small problems using VSCode. I'm not sure if it is just standard use of VSCode with R or a Rix problem, as it only appears after using rix(), and this would be a different issue.
But for your curiosity, when I use rix() it seems to create an .Rprofile that seems to create a duplicate space in VSCode, which returns a warning :
```Keeping .Rprofile generated by `rix::rix_init()`###

Cannot open the connectionwarning messages from top-level task callback 'vsc.workspace'
Warning message:
In file(con, "w") :
cannot open file '/tmp/RtmpVIm7Xe/vscode-R/workspace.json': No such file or directory


But I will explore and ask on stock overflow in case I just wouldn't understand well how to use the R extension. 

About this issue, I will go back to my workplace as soon as possible and try to see with ma admins what proxy is needed to get rix() to work with remotes packages. 

Thank you for your help @philipp-baumann and @b-rodrigues 

@b-rodrigues
Copy link
Contributor

I think you might be having the same issue than here #281

I will be changing the port serving the api after I'm back from holidays it should work then

@philipp-baumann
Copy link
Collaborator

Thank you. I added the path in my .profile ut added a line to check if it was already in and this didn't seem to work.

That is unusual. Have you also tried with .bashrc. To make the source happen, you either need to reopen a new bash terminal our do

source $HOME/.bashrc

Then check the updated path that should include the bin folder of the nix profile.

echo $PATH

I will go back to my workplace and see what the error message is concerning git packages and see what blocks non cran packages and see if I can get a proxy set.

I still encounter small problems using VSCode. I'm not sure if it is just standard use of VSCode with R or a Rix problem, as it only appears after using rix(), and this would be a different issue. But for your curiosity, when I use rix() it seems to create an .Rprofile that seems to create a duplicate space in VSCode, which returns a warning : ```Keeping .Rprofile generated by `rix::rix_init()`###

That is how the function should behave, i.e. create this custom .Rprofile as a side-effect, to make sure the user library is excluded in the library path (what .libPaths() reports). The R built with nix will otherwise inherit R_LIBS_USER, which points to the default installation from the host system R, which is not at all what we want. This is an extra step needed to garantee pure set of nix R libraries from the nix store in the library path. Otherwise, in case you have the path specified in R_LIBS_USER, it would put those entries first in the library search path and you would end up with a polluted nix-R library setup, i.e. on macOS you would even get segmentation faults or sometimes on linux it would just use R packages from the system.

Cannot open the connectionwarning messages from top-level task callback 'vsc.workspace' Warning message: In file(con, "w") : cannot open file '/tmp/RtmpVIm7Xe/vscode-R/workspace.json': No such file or directory

The problem is that you are loading workspace.json from the /tmp subfolder. Why are you using a tmp folder and not a home subdir to host your vscode project?


But I will explore and ask on stock overflow in case I just wouldn't understand well how to use the R extension. 

About this issue, I will go back to my workplace as soon as possible and try to see with ma admins what proxy is needed to get rix() to work with remotes packages. 

Thank you for your help @philipp-baumann and @b-rodrigues 

@philipp-baumann
Copy link
Collaborator

philipp-baumann commented Sep 10, 2024

I have been working with both vscode (packaged in nixpkgs and "natively" via package manager or binary install) and rstudio, Ubuntu, on WSL2 Ubuntu, macOS, it has all been running very smoothly.

@MGousseff there is no duplicate space created by the custom .Rprofile. If you call rix::rix(), it also calls rix_init(), which is a utility that constructs the .Rprofile. As you can see in the code that generates the file, and also the .Rprofile in your project_path dir. Codebase: https://github.com/b-rodrigues/rix/blob/5a95641f03bb9fc4941e7d420adcdcde754adfc2/R/rix_init.R#L394-L442 , which is effectively boilerplated and written here: https://github.com/b-rodrigues/rix/blob/5a95641f03bb9fc4941e7d420adcdcde754adfc2/R/rix_init.R#L138-L148

### File generated by `rix::rix_init()` ###
# 1. Currently, system RStudio does not inherit environmental variables
#   defined in `$HOME/.zshrc`, `$HOME/.bashrc` and alike. This is workaround to
#   make the path of the nix store and hence basic nix commands available
#   in an RStudio session
# 2. For nix-R session, remove `R_LIBS_USER`, system's R user library.`.
#   This guarantees no user libraries from the system are loaded and only
#   R packages in the Nix store are used. This makes Nix-R behave in pure manner
#   at run-time.
{
  is_rstudio <- Sys.getenv("RSTUDIO") == "1"
  is_nix_r <- nzchar(Sys.getenv("NIX_STORE"))
  if (isFALSE(is_nix_r) && isTRUE(is_rstudio)) {
    cat("{rix} detected RStudio R session")
    old_path <- Sys.getenv("PATH")
    nix_path <- "/nix/var/nix/profiles/default/bin"
    has_nix_path <- any(grepl(nix_path, old_path))
    if (isFALSE(has_nix_path)) {
      Sys.setenv(PATH = paste(old_path, nix_path, sep = ":"))
    }
    rm(old_path, nix_path)
  }
  if (isTRUE(is_nix_r)) {
    current_paths <- .libPaths()
    userlib_paths <- Sys.getenv("R_LIBS_USER")
    user_dir <- grep(paste(userlib_paths, collapse = "|"), current_paths, fixed = TRUE)
    new_paths <- current_paths[-user_dir]
    .libPaths(new_paths)
    rm(current_paths, userlib_paths, user_dir, new_paths)
  }
  rm(is_rstudio, is_nix_r)
}

options(dplyr.width = Inf)

So it does all what is explained in ?rix::rix_init.

Once cause of failure, because of your setup, that you do not have proper nix environment variables that are inherited from the shell set up, is that currently we only have adjustments in rstudio, that this is added according to the PATH in the custom .Rprofile.

/nix/var/nix/profiles/default/bin

So when not using rstudio, i see a hidden behavior that i have missed when programming the tweaks.

https://github.com/b-rodrigues/rix/blob/5a95641f03bb9fc4941e7d420adcdcde754adfc2/R/rix_init.R#L275-L295 set_message_session_PATH(), which also calls set_nix_path(), is also not effective when rprofile_action = "overwrite" or rprofile_action = "append". In your rare case, where nix profiles are not correctly in the shell env, this should be also called there, that your in-session PATH is adjusted to included the required nix path.

@MGousseff
Copy link

I think you might be having the same issue than here #281

I will be changing the port serving the api after I'm back from holidays it should work then

Thank you very much. I'm sorry you had all these notifications while on vacations. I'll wait for you to modify this before I reach to my sys admins, as it is not always very straightforward to add a proxy or modify port opening here (I don't blame them, attacks are very frequent).

@MGousseff
Copy link

In your rare case, where nix profiles are not correctly in the shell env, this should be also called there, that your in-session PATH is adjusted to included the required nix path.

I think I'm going to have a stroke :-) Seriously, thank you for explaining me all this. I am not very aware of how profils work.
I went and searched for the .nix-profile in my home, and I found an interesting thing : the files exists but links to something that doesn't exist : "/nix/var/nix/profiles/per-user//profile". And this file doesn't exist.
The only directory in /nix/var/nix/profiles/per-user/ is root, and it is empty. Is there something I can do to create this myusername nix profile file ?

@philipp-baumann
Copy link
Collaborator

philipp-baumann commented Sep 10, 2024

/nix/var/nix/profiles/per-user/

In your rare case, where nix profiles are not correctly in the shell env, this should be also called there, that your in-session PATH is adjusted to included the required nix path.

I think I'm going to have a stroke :-) Seriously, thank you for explaining me all this. I am not very aware of how profils work. I went and searched for the .nix-profile in my home, and I found an interesting thing : the files exists but links to something that doesn't exist : "/nix/var/nix/profiles/per-user//profile". And this file doesn't exist. The only directory in /nix/var/nix/profiles/per-user/ is root, and it is empty. Is there something I can do to create this myusername nix profile file ?

I have exactly the same situation. only root in /nix/var/nix/profiles/per-user/profile, but it is not empty.

/nix/var/nix/profiles/per-user/root$ ll
total 16
drwxr-xr-x 2 root root 4096 Jun 10 10:47 ./
drwxr-xr-x 3 root root 4096 Jun 10 10:47 ../
lrwxrwxrwx 1 root root   14 Jun 10 10:47 profile -> profile-2-link/
lrwxrwxrwx 1 root root   60 Jun 10 10:47 profile-1-link -> /nix/store/9s6w0vyxcbvvw52bgc97vk2bl082xmhi-user-environment/
lrwxrwxrwx 1 root root   60 Jun 10 10:47 profile-2-link -> /nix/store/xr51aqblxflgkh84kvasbmzi1cfi6ih8-user-environment/

This seems a similar issue, and apparently newer nix versions link profiles in $HOME/.nix-profile. -> DeterminateSystems/nix-installer#477 Just checked on my macOS at home, which was created by the Determinate Systems installer.

# on ubuntu it's also some `.../.local/...` folder
➜ ls -l $HOME/.nix-profile
lrwxr-xr-x  1 philipp  staff  48 Dec 16  2023 /Users/philipp/.nix-profile -> /Users/philipp/.local/state/nix/profiles/profile

This explains a bit more about profiles: https://nixos.wiki/wiki/User_Environment

ls -l /nix/var/nix/profiles/ 
total 4
lrwxrwxrwx 1 root root   43 Jun 10 10:47 default -> /nix/var/nix/profiles/per-user/root/profile
drwxr-xr-x 3 root root 4096 Jun 10 10:47 per-user

I'd also be interested to resolve which i consider your main problem, respectively, what might be improved to signal in {rix}.

Error in sys::exec_internal(cmd = cmd, args = args) : 
  Executing 'nix-hash' failed with status 1

Did the .bashrc PATH like suggested work? This would mean if you restart your host R (guess using the version managed by rig in /opt...,), you will have this in your PATH in the R (assuming rig does not do black magic with env vars when starting R; who knows).

# in bash terminal
echo $PATH

Somehow should include in the ":"-separated list:
/nix/var/nix/profiles/default/bin

the same in the host rig R session

Sys.getenv("PATH")

Then there is a second hint i'd like to know:

It would be great if you can report the output of these commands in your work ubuntu rig R session

Sys.getenv("LD_LIBRARY_PATH")
Sys.getenv("NIX_STORE")

You seem to have anaconda installed, don't wanna draw premature conclusions but it can fuck envvars up as well.

To get the actual error message of nix-hash (not just the exit status 1 for unsuccessful, it would greatly help if you can report the output of this command you call in the project folder e.g., the current directory

nix-hash --type sha256 --sri .

This is what internal hashing (not via nixsha ...server API) should do with a local nix install you have. What rig does is actually another thing. Stuff like this doesn't convince me that it leaves important standard R mechanisms unchanged: r-lib/rig#203

@MGousseff
Copy link

Hello,
I changed the export PATH in my .profile rather than in my .bashrc and it seems to stick.

But I tried to follow the workaround of the issue you pointed to : DeterminateSystems/nix-installer#477... And now my nix install is broken again. I try to reinstall things. I am afraid that so many tedious installation problems will not help the adoption of rix (and to make it clear, I get that it is a nix problem and not a rix problem).

About the nix-hash, it worked from my home network, so I think it's more a proxy thing.

@philipp-baumann
Copy link
Collaborator

Hello, I changed the export PATH in my .profile rather than in my .bashrc and it seems to stick.

But I tried to follow the workaround of the issue you pointed to : DeterminateSystems/nix-installer#477... And now my nix install is broken again. I try to reinstall things. I am afraid that so many tedious installation problems will not help the adoption of rix (and to make it clear, I get that it is a nix problem and not a rix problem).

About the nix-hash, it worked from my home network, so I think it's more a proxy thing.

Thanks for your patience. I assume you have systemd and control over you Ubuntu installation. I cannot tell if it is really a problem of the installer, or rather a permission issue. There is a fork of the Determinate Systems installer, called Lix, which is "Lix, like Nix", so all setup and the command line tools are equivalent. Its a fork, and it could be that there are some improvements: https://git.lix.systems/lix-project/lix-installer and https://lix.systems/install/

on my macOS laptop at home i have just updated from NixCpp to Lix, was really smooth.

sudo --preserve-env=PATH nix run \
     --experimental-features "nix-command flakes" \
     --extra-substituters https://cache.lix.systems --extra-trusted-public-keys "cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o=" \
     'git+https://git.lix.systems/lix-project/lix?ref=refs/tags/2.91.0' -- \
     upgrade-nix \
     --extra-substituters https://cache.lix.systems --extra-trusted-public-keys "cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o="

Have you tried uninstalling the determinate system installer via (see https://zero-to-nix.com/start/uninstall )

/nix/nix-installer uninstall

?

On top, you might also try uninstalling stuff like described in the docs. Probably, the determinate systems nix uninstaller does the same, but it might be worth doing below manually. Also, have you managed to remove your nix installed via apt completely.

https://nix.dev/manual/nix/2.18/installation/uninstall

# remove nix daemon service
sudo systemctl stop nix-daemon.service
sudo systemctl disable nix-daemon.socket nix-daemon.service
sudo systemctl daemon-reload
# remove files created by nix
sudo rm -rf /etc/nix /etc/profile.d/nix.sh /etc/tmpfiles.d/nix-daemon.conf /nix ~root/.nix-channels ~root/.nix-defexpr ~root/.nix-profile
# remove build users and their groups
for i in $(seq 1 32); do
  sudo userdel nixbld$i
done
sudo groupdel nixbld
```

@MGousseff
Copy link

I uninstalled NIX, and reinstalled it with lix. I'm testing it right now.

@MGousseff
Copy link

I uninstalled lix, reinstalled lix, and also ran the recommended instruction (although I did not understand what this was for)

Then open R in VS code, st the Rterm at the local install so to run the rix() function. It builds the nix.
default but I still get, after any subsequent R instruction I send to the terminal :

Keeping `.Rprofile` generated by `rix::rix_init()`###
cannot open the connectionwarning messages from top-level task callback 'vsc.workspace'
Warning message:
In file(con, "w") :
  cannot open file '/tmp/RtmpdT7BnF/vscode-R/workspace.json': No such file or directory

I thought: well, I add complexity from using rix and using vscode, I'll try to build with ide="rstudio". And when I try to launch rstudio from the nix shell, I get :

TTY detected. Printing informational message about logging configuration. Logging configuration loaded from '/etc/xdg/xdg-ubuntu:/etc/xdg/rstudio/logging.conf'. Logging to '/home/gousseff/.local/share/rstudio/log/rdesktop.log'.
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize 1, greenBufferSize 1, blueBufferSize 1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize 1, greenBufferSize 1, blueBufferSize 1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
Could not initialize GLX
Aborted (core dumped)

Rstudio is installed on my computer and works ok aoutside of nix shell.

I reached my limit for now. I am convinced building reproducible environments to build and distribute packages with R is important, I am sure the nix/rix solution is useful for those for whom ti works, but I just can't spend more time trying to set it up as I am getting behind on my work. I will for a while just code with the up to date CRAN versions hoping my code will be OK, and I'l come back to all this later.

I think NIX should provide a .deb package to make it easy to install and uninstall in the ubuntu/debian community, I think a step-by-step tutorial could be useful (that is first in my general session I do this, then I quit, enter my nix-shell session and launch this and this and then I can run my R scripts into the nix R environment).

@philipp-baumann I hope I'll find some time later to try again. Sorry for the time we lost both without having the release to make it all work, it also must be frustrating for you.

@philipp-baumann
Copy link
Collaborator

philipp-baumann commented Sep 11, 2024

I uninstalled lix, reinstalled lix, and also ran the recommended instruction (although I did not understand what this was for)

Then open R in VS code, st the Rterm at the local install so to run the rix() function. It builds the nix. default but I still get, after any subsequent R instruction I send to the terminal :

Keeping `.Rprofile` generated by `rix::rix_init()`###
cannot open the connectionwarning messages from top-level task callback 'vsc.workspace'
Warning message:
In file(con, "w") :
  cannot open file '/tmp/RtmpdT7BnF/vscode-R/workspace.json': No such file or directory

I thought: well, I add complexity from using rix and using vscode, I'll try to build with ide="rstudio". And when I try to launch rstudio from the nix shell, I get :

TTY detected. Printing informational message about logging configuration. Logging configuration loaded from '/etc/xdg/xdg-ubuntu:/etc/xdg/rstudio/logging.conf'. Logging to '/home/gousseff/.local/share/rstudio/log/rdesktop.log'.
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize 1, greenBufferSize 1, blueBufferSize 1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize 1, greenBufferSize 1, blueBufferSize 1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
Could not initialize GLX
Aborted (core dumped)

Rstudio is installed on my computer and works ok aoutside of nix shell.

I reached my limit for now. I am convinced building reproducible environments to build and distribute packages with R is important, I am sure the nix/rix solution is useful for those for whom ti works, but I just can't spend more time trying to set it up as I am getting behind on my work. I will for a while just code with the up to date CRAN versions hoping my code will be OK, and I'l come back to all this later.

I think NIX should provide a .deb package to make it easy to install and uninstall in the ubuntu/debian community, I think a step-by-step tutorial could be useful (that is first in my general session I do this, then I quit, enter my nix-shell session and launch this and this and then I can run my R scripts into the nix R environment).

@philipp-baumann I hope I'll find some time later to try again. Sorry for the time we lost both without having the release to make it all work, it also must be frustrating for you.

i get it no prob. I think it's really something peculiar in your setup. I have managed to get it working on Ubuntu on two systems, so i'm still positive. @b-rodrigues has also tested it on a latest Ubuntu container and all works smoothly there, too, i think. I would generally advise for diagnostics to start simple. Use a plain R session (not rig R, just the one in apt) in a bash terminal, then install newest rix and see if it works.

The other hints are envvars, which depending on the shell environment and IDE might not optimally be loaded for a smooth nix environment. My feeling is we have already done a lot there, there is small things to improve and a diagnostics helper is one of it. Issues like yours are not getting in the way, but are there to learn something. So in that spirit feel free to reiterate over this issue once you have time. We all sometimes want to make it work as simply as possibles and there are trade-offs with deliverables.

Cheers!

@philipp-baumann
Copy link
Collaborator

for a project now i for example use ide = "code", but i work with vscode and rstudio using the same setup.

library("rix")

rix_init()

r_pkgs <- c(
  "targets", "tarchetypes", "dplyr", "data.table", "dm", "DiagrammeR", "quarto",
  "R.utils", "here", "yaml", "readxl", "reactable", "writexl",
  # logging framework
  "lgr",
  # development tools
  "devtools", "styler", "lintr"
)

git_pkgs <- list(
  list(
    package_name = "rix",
    repo_url = "https://github.com/b-rodrigues/rix",
    commit = "c64ccaa34eb48ae4dcd78148ace211f72eebcb16"
  ),
  list(
    package_name = "fnmate",
    repo_url = "https://github.com/MilesMcBain/fnmate/",
    commit = "18a4c00536e0deafeb6b3f91e48b53b641f47e6a"
  ),
  list(
    package_name = "rscodeio",
    repo_url = "https://github.com/anthonynorth/rscodeio",
    commit = "88a1a373d64f0bd81b6730eb6fa005fd0baa9078"
  )
)

# Define expressions with software environment
rix::rix(
  r_ver = "9355fa86e6f27422963132c2c9aeedb0fb963d93", # "3c0be6ff38ccf079cb7976a571c5c865a19230a6",
  r_pkgs = r_pkgs,
  system_pkgs = c("git", "quarto"),
  git_pkgs = git_pkgs,
  ide = "rstudio",
  project_path = ".",
  overwrite = TRUE,
  shell_hook = "export QT_XCB_GL_INTEGRATION=none"
)

@b-rodrigues
Copy link
Contributor

@MGousseff for rstudio you need to add

shell_hook = "export QT_XCB_GL_INTEGRATION=none"

in your rix call, as documented (scenario 1).

Too bad it's not working well for you, as @philipp-baumann mentioned I was able to get everything running on an Ubuntu vm and on my rpi5 running ubuntu as well. Really wonder what went wrong for you

@MGousseff
Copy link

My guess is that rig and nix don't go along too well, with symbolic links from rig messing with things ?
I first discovered rig then tried rix. I may uninstall my rix R version and try from scratch.

@b-rodrigues
Copy link
Contributor

I'll test this setup in a vm, thanks!

to be honest, the best way is to not have a system wide R version, and rely solely on project specific versions of R managed by Nix. But I know this is not really how most people like to work

@philipp-baumann
Copy link
Collaborator

I'll test this setup in a vm, thanks!

to be honest, the best way is to not have a system wide R version, and rely solely on project specific versions of R managed by Nix. But I know this is not really how most people like to work

Yeah have to admit me sometimes, too 😅 no reason for it. I think the critical phase is adoption. The transition needs to be frictionless, and i think we are good on track.

@MGousseff
Copy link

What ? But you need a system wide R version to create your nix environment with rix. Or you are in a infinite loop : you need nix to launch R to create the rix file which allows you to create the nix environment to launch R to... etc.

I'll try to uninstall rig and my versions of R, install the latest Rbase, install RIX and create a setup. But I need to find a couple of hours, and I'm behind on my job.

By the way, have you tried rix with positron ? The future replacement of Rstudio, or at least the new IDE provided by posit (Rstudio mother company) is in beta public version. It deals with R and python in a fork of VScode. I'm trying it (after the weird messages I get with rix in vscode). It feels not bad, even if debugging capabilites are still behind what I had with intellij (which I loved so much until the plugin broke and we realized only one man was on it and didn't answer anymore).

@b-rodrigues
Copy link
Contributor

b-rodrigues commented Sep 12, 2024

as described in the Readme, you can simply run a temporary shell to generate a default.nix by running

nix-shell --expr "$(curl -sl https://raw.githubusercontent.com/b-rodrigues/rix/master/inst/extdata/default.nix)" 

as for Positron: I've been trying to package it for NixOS 😁

@MGousseff
Copy link

So here is what I did

  • Fully uninstalled rig and the r versions it installed
  • Uninstalled nix and reinstalled it thanks to the lix installer
  • Installed a CRAN Base R version. I know you advise against it, but I'm not sure yet I will always work in a dedicated nix environment. I want to be able to install packages seamlessly and work freely until I'm OK with my code and reproduce it in a fixed and stable environment
  • Launched my Rsession with codium, installed rix package, ran the rix() function with the proper parameters
  • once the nix.default was created, I ran nix-build and nix-shell
  • from the nix shell checked which R and copied the nix environment R path
  • launched codium again, set the Rpath to the result of which R

What doesn't work:

  • the .Rprofile rix() creates mess with my codium session. I have the warning message. Yet, I now have a profile, profile-1-link and profile-2-link in /nix/var/nix/profiles/per-user/root (no other directory in per-user, though), and they point to nix user environments in nix/store It is not critical, as I won't spend my time creating rix environments, and as it is a warning which doesn't kill the rsession.
  • if I don't launch codium from the nix shell,setting rpath in the settings has no effect. I need to launch codium from nix shel then set the proper rpath

What works:

  • once I have done all this and launched codium from nix-shell and set rpath in codium settings, it works. WE DID IT !

What SURPRISINGLY WORKS:

  • I don't have the nix-hash error anymore and I can add github packages even from my work network. Maybe it had nothing to do with working from home, and it was just the shutting down from work and booting up again from home which solved the problem ? Or the lix install ? I'm at a loss
  • INCREDIBLE : if I launch positron from the nix shell it finds the proper R interpreter. Which is good because there is not rpath explicit argument (AFAIK) in positron and I tried to set in from outside the nix-shell to no avail.

What I'd love:
I don't get how positron finds the proper R path when ran from the nix-shell. I'd love a function in the R extension of codium who would do that, or maybe a launcher inside nix which would check nix-shell R path, and launch codium after setting the rpath in the settings parameter of codium. Sorry I'm unable to write such a thing.

Thank you both @b-rodrigues @philipp-baumann. I hope you don't feel this was a waste of time.

I'd like to write a quick summary of this and how to install and use nix/lix and rix, but I don't have any "blogging platform anymore". Maybe some people will find this issue helpful.

@b-rodrigues
Copy link
Contributor

if I launch positron from the nix shell it finds the proper R interpreter.

Yes, this was my experience as well. FYI, this seems to be the case with many other editors, such as emacs and neovim.

In any case, glad it works for you now!

@b-rodrigues
Copy link
Contributor

I think that we can now close this. @MGousseff should you have another problem, don't hesitate to open an issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants