losoto error #31

HannoSpreeuw · 2018-10-17T11:57:15Z

[hspreeuw@fs5 ~]$ git clone git@github.com:EOSC-LOFAR/prefactor-cwl.git
Cloning into 'prefactor-cwl'...
remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 714 (delta 25), reused 40 (delta 19), pack-reused 652
Receiving objects: 100% (714/714), 7.47 MiB | 1.27 MiB/s, done.
Resolving deltas: 100% (449/449), done.
[hspreeuw@fs5 ~]$ cd prefactor-cwl/
[hspreeuw@fs5 prefactor-cwl]$ make run-singularity
virtualenv -p python2 .virtualenv
Running virtualenv with interpreter /home/hspreeuw/miniconda2/bin/python2
New python executable in .virtualenv/bin/python2
Also creating executable in .virtualenv/bin/python
ERROR: The executable .virtualenv/bin/python2 is not functioning
ERROR: It thinks sys.prefix is u'/usr' (should be u'/home/hspreeuw/prefactor-cwl/.virtualenv')
ERROR: virtualenv is not compatible with this system or executable
make: *** [.virtualenv/] Error 100
[hspreeuw@fs5 prefactor-cwl]$ which pip
~/miniconda3/bin/pip
[hspreeuw@fs5 prefactor-cwl]$ which pip2
~/miniconda2/bin/pip2
[hspreeuw@fs5 prefactor-cwl]$ which pip3
~/miniconda3/bin/pip3
[hspreeuw@fs5 prefactor-cwl]$ which python
~/miniconda3/bin/python
[hspreeuw@fs5 prefactor-cwl]$ which python2
~/miniconda2/bin/python2
[hspreeuw@fs5 prefactor-cwl]$ which python3
~/miniconda3/bin/python3

mr-c · 2018-10-17T12:15:58Z

pip show virtualenv ?

Can you turn off your conda installation?

If that doesn't work, I would try the following modification, to use the system python.

rm -Rf .virtualenv

diff --git a/Makefile b/Makefile
index eeb9d90..d16d650 100644
--- a/Makefile
+++ b/Makefile
@@ -13,7 +13,7 @@ SMALL_ARCHIVE=L570745_uv_first10.MS.tar.xz
 
 
 .virtualenv/:
-       virtualenv -p python2 .virtualenv
+       /usr/bin/virtualenv -p /usr/bin/python2 .virtualenv
  
 .virtualenv/bin/cwltool: .virtualenv/
        .virtualenv/bin/pip install -r requirements.txt

HannoSpreeuw · 2018-10-17T12:17:48Z

[hspreeuw@fs5 prefactor-cwl]$ pip show virtualenv
Traceback (most recent call last):
File "/home/hspreeuw/miniconda3/bin/pip", line 7, in
from pip._internal import main
File "/usr/lib/python2.7/site-packages/pip/init.py", line 15, in
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/lib/python2.7/site-packages/pip/vcs/subversion.py", line 9, in
from pip.index import Link
File "/usr/lib/python2.7/site-packages/pip/index.py", line 30, in
from pip.wheel import Wheel, wheel_ext
File "/usr/lib/python2.7/site-packages/pip/wheel.py", line 6, in
import compileall
File "/home/hspreeuw/miniconda3/lib/python3.7/compileall.py", line 20, in
from concurrent.futures import ProcessPoolExecutor
File "/usr/lib/python2.7/site-packages/concurrent/futures/init.py", line 8, in
from concurrent.futures._base import (FIRST_COMPLETED,
File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 357
raise type(self._exception), self._exception, self._traceback
^
SyntaxError: invalid syntax

HannoSpreeuw · 2018-10-17T12:21:21Z

I removed both miniconda2 and miniconda3 from my PATH.
Now I get:
[hspreeuw@fs5 ~]$ git clone git@github.com:EOSC-LOFAR/prefactor-cwl.git
Cloning into 'prefactor-cwl'...
remote: Enumerating objects: 62, done.
remote: Counting objects: 100% (62/62), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 714 (delta 25), reused 40 (delta 19), pack-reused 652
Receiving objects: 100% (714/714), 7.47 MiB | 903.00 KiB/s, done.
Resolving deltas: 100% (449/449), done.
[hspreeuw@fs5 ~]$ cd prefactor-cwl/
[hspreeuw@fs5 prefactor-cwl]$ make run-singularity
virtualenv -p python2 .virtualenv
Running virtualenv with interpreter /usr/bin/python2
New python executable in .virtualenv/bin/python2
Also creating executable in .virtualenv/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
.virtualenv/bin/pip install -r requirements.txt
Downloading/unpacking html5lib (from -r requirements.txt (line 1))
Downloading html5lib-1.0.1.tar.gz (252kB): 252kB downloaded
Running setup.py egg_info for package html5lib
Traceback (most recent call last):
File "", line 16, in
File "/home/hspreeuw/prefactor-cwl/.virtualenv/build/html5lib/setup.py", line 54, in
MarkerEvaluation = pkg_resources.MarkerEvaluation
AttributeError: 'module' object has no attribute 'MarkerEvaluation'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 16, in

File "/home/hspreeuw/prefactor-cwl/.virtualenv/build/html5lib/setup.py", line 54, in

MarkerEvaluation = pkg_resources.MarkerEvaluation

AttributeError: 'module' object has no attribute 'MarkerEvaluation'

Cleaning up...
Command python setup.py egg_info failed with error code 1 in /home/hspreeuw/prefactor-cwl/.virtualenv/build/html5lib
Storing complete log in /home/hspreeuw/.pip/pip.log
make: *** [.virtualenv/bin/cwltool] Error 1

mr-c · 2018-10-17T12:22:21Z

.virtualenv/bin/pip install -U pip
.virtualenv/bin/pip install -U setuptools wheel

HannoSpreeuw · 2018-10-17T13:00:39Z

Excellent. Thanks.
It ran some tens of minutes and I expected it to run into a known issue like #23.
This is with module load singularity/2.5.2
However, it ended with

Output is missing expected field file:///home/hspreeuw/prefactor-cwl/prefactor.cwl#h5imp_cal/losoto_h5

More verbose ouptput:

DEBUG - Writing visibilities to column: CORRECTED_DATA
DEBUG - Read time (meta data): timer: avg = 29.1 us, total =  146 us, count =         5
DEBUG - Write time: timer: avg = 1.59 ms, total = 9.57 ms, count =         6
DEBUG - Executing a Finalize command:
Command: Finalize
INFO - bbs-reducer terminated successfully.
INFO - clone CA                 : avg =  116 ns, total =  250 ms, count =   2158792
INFO - new CA                   : avg = 47.3 ns, total =  109 ms, count =   2299856
[job calib_cal_20] completed success
[step calib_cal] completed success
[workflow ] starting step h5imp_cal
[step h5imp_cal] start
[job h5imp_cal] Output of job will be cached in /home/hspreeuw/prefactor-cwl/cache/387889929697b4a8f8a74c8d9c98e3f8
Using local copy of Singularity image found in /home/hspreeuw/prefactor-cwl
[job h5imp_cal] /home/hspreeuw/prefactor-cwl/cache/387889929697b4a8f8a74c8d9c98e3f8$ singularity \
    --quiet \
    exec \
    --contain \
    --pid \
    --ipc \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/387889929697b4a8f8a74c8d9c98e3f8:/var/spool/cwl:rw \
    --bind \
    /tmp/tmps8BRRE:/tmp:rw \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/3624b3f46f1ad175ef15dc5eba8a8f4e/calibrated.MS:/var/lib/cwl/stg11a2c476-603e-4ed2-834e-40e24bbde677/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/b1093dc6bb04e71d4bad0af1ee108706/calibrated.MS:/var/lib/cwl/stg0ed8e3ba-7f9f-40d6-91e5-d81beffc54ea/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/3cd85532f2f244d2d5191f5ab15628c5/calibrated.MS:/var/lib/cwl/stg6efb7e22-4da8-4dbc-820e-47548ca0b38c/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/20bac55ad3273acd3ce5b7dabf81abde/calibrated.MS:/var/lib/cwl/stg8e162382-352c-493e-bc3d-c88c2aa64928/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/bcee14698a197ead0d0c41d881ce4ea7/calibrated.MS:/var/lib/cwl/stg95a515c6-dadd-4943-8b18-6b1a426cf863/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/95502154f927525857708faef4477b85/calibrated.MS:/var/lib/cwl/stge1d501bf-f860-489b-a122-8ef85d5927ce/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/aea5c783e503950d7b2b834cc147648d/calibrated.MS:/var/lib/cwl/stge3c394af-8ca2-4416-9f47-a23ffbe0692e/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/c3cf20bd9ddb627f75c586d7da8035d8/calibrated.MS:/var/lib/cwl/stg11cdf87f-e9f6-4c4d-8bb4-feff442eede1/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/2ef9ae2e321c31377d428907f2b5a8d4/calibrated.MS:/var/lib/cwl/stg0ab7482d-b892-46b8-855f-a06643a74279/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/91524ca1bd4c45c354838529554b5edf/calibrated.MS:/var/lib/cwl/stgd54b5aec-b4d7-4581-901b-4e259f997dfe/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/bcc318d76575432116bec4a8ea99a747/calibrated.MS:/var/lib/cwl/stg0356c231-a49e-4016-8b13-c509fc854987/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/64f066e64c96fb48e28ae523dbeb4bbc/calibrated.MS:/var/lib/cwl/stg4333efde-286a-4078-9fe9-30ebf189de9c/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/1cef29b2f4600c7a9c0b8cc90507126e/calibrated.MS:/var/lib/cwl/stg8aaaacfb-b8eb-414d-8446-9f06cb80bb04/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/fa5b9b4c52081dc0e3bffae6056f4370/calibrated.MS:/var/lib/cwl/stg68f86e9a-b504-4cfa-a710-60c1f6f0c5a2/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/0ad9c6962ffdf5565c81a1969b5f3d82/calibrated.MS:/var/lib/cwl/stg58d192ef-145d-44c8-bb2c-fbe7a0ab8ae5/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/24f250d71a06f473bedcaa31222e4359/calibrated.MS:/var/lib/cwl/stg5f7d8bd7-c898-47b0-8ac7-2a00d4ae1438/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/31ac1f24d373fc2220cecc7a7d7a8947/calibrated.MS:/var/lib/cwl/stg4e17ff09-2f0f-42b5-bec9-e239e8f2586e/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/72f5ef74dc0fbb161e5dec73019f144d/calibrated.MS:/var/lib/cwl/stg365999d8-1036-485d-b67f-3dd264d93991/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/5c642df6490546980cf6ff08b767f036/calibrated.MS:/var/lib/cwl/stg4d8f9ea6-d802-4129-9e61-f83645eb5788/calibrated.MS:ro \
    --bind \
    /home/hspreeuw/prefactor-cwl/cache/a80f038dbebd6bf9d681bad15157b547/calibrated.MS:/var/lib/cwl/stg7add5ebb-ba24-4f36-b93a-63f9df208b55/calibrated.MS:ro \
    --pwd \
    /var/spool/cwl \
    /home/hspreeuw/prefactor-cwl/kernsuite-prefactor.img \
    python \
    /usr/lib/prefactor/scripts/losotoImporter.py \
    losoto.h5 \
    -c \
    7 \
    -s \
    sol000 \
    /var/lib/cwl/stg8aaaacfb-b8eb-414d-8446-9f06cb80bb04/calibrated.MS \
    /var/lib/cwl/stg0ab7482d-b892-46b8-855f-a06643a74279/calibrated.MS \
    /var/lib/cwl/stg8e162382-352c-493e-bc3d-c88c2aa64928/calibrated.MS \
    /var/lib/cwl/stg4333efde-286a-4078-9fe9-30ebf189de9c/calibrated.MS \
    /var/lib/cwl/stg68f86e9a-b504-4cfa-a710-60c1f6f0c5a2/calibrated.MS \
    /var/lib/cwl/stg11cdf87f-e9f6-4c4d-8bb4-feff442eede1/calibrated.MS \
    /var/lib/cwl/stg7add5ebb-ba24-4f36-b93a-63f9df208b55/calibrated.MS \
    /var/lib/cwl/stg4d8f9ea6-d802-4129-9e61-f83645eb5788/calibrated.MS \
    /var/lib/cwl/stg0ed8e3ba-7f9f-40d6-91e5-d81beffc54ea/calibrated.MS \
    /var/lib/cwl/stg0356c231-a49e-4016-8b13-c509fc854987/calibrated.MS \
    /var/lib/cwl/stg6efb7e22-4da8-4dbc-820e-47548ca0b38c/calibrated.MS \
    /var/lib/cwl/stg4e17ff09-2f0f-42b5-bec9-e239e8f2586e/calibrated.MS \
    /var/lib/cwl/stg365999d8-1036-485d-b67f-3dd264d93991/calibrated.MS \
    /var/lib/cwl/stg5f7d8bd7-c898-47b0-8ac7-2a00d4ae1438/calibrated.MS \
    /var/lib/cwl/stg11a2c476-603e-4ed2-834e-40e24bbde677/calibrated.MS \
    /var/lib/cwl/stg58d192ef-145d-44c8-bb2c-fbe7a0ab8ae5/calibrated.MS \
    /var/lib/cwl/stgd54b5aec-b4d7-4581-901b-4e259f997dfe/calibrated.MS \
    /var/lib/cwl/stge1d501bf-f860-489b-a122-8ef85d5927ce/calibrated.MS \
    /var/lib/cwl/stg95a515c6-dadd-4943-8b18-6b1a426cf863/calibrated.MS \
    /var/lib/cwl/stge3c394af-8ca2-4416-9f47-a23ffbe0692e/calibrated.MS
/usr/lib/python2.7/dist-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
/usr/lib/python2.7/dist-packages/lofar/parmdb/__init__.py:22: RuntimeWarning: to-Python converter for std::vector<double, std::allocator<double> > already registered; second conversion method ignored.
  from _parmdb import ParmDB
/usr/lib/python2.7/dist-packages/lofar/parmdb/__init__.py:22: RuntimeWarning: to-Python converter for std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > already registered; second conversion method ignored.
  from _parmdb import ParmDB
Traceback (most recent call last):
  File "/usr/lib/prefactor/scripts/losotoImporter.py", line 16, in <module>
    from losoto.importer import create_h5parm
ImportError: No module named importer
[job h5imp_cal] Job error:
Error collecting output for parameter 'losoto_h5':
steps/h5imp_cal.cwl:48:7: Did not find output file with glob pattern: '[u'losoto.h5']'
[job h5imp_cal] completed permanentFail
[step h5imp_cal] Output is missing expected field file:///home/hspreeuw/prefactor-cwl/prefactor.cwl#h5imp_cal/losoto_h5
[step h5imp_cal] completed permanentFail
[workflow ] completed permanentFail
{
    "psf": null, 
    "losoto_h5": null, 
    "dtec_allsols": null, 
    "dclock_1st": null, 
    "phase_array": null, 
    "amplitude_array": null, 
    "phase_xx_yy_offset": null, 
    "amp_allsols": null, 
    "dclock_allsols": null, 
    "freqs_for_phase_array": null, 
    "dclock_1st_sm": null, 
    "residual": null, 
    "polYY_dirpointing": null, 
    "dirty": null, 
    "model": null, 
    "station_names": null, 
    "dTEC_1st": null, 
    "polXX_dirpointing": null, 
    "dTEC_1st_sm": null
}
Final process status is permanentFail
make: *** [run-singularity] Error 1

mr-c · 2018-10-17T13:05:31Z

@HannoSpreeuw I'm glad that worked.
I think the key error is

from losoto.importer import create_h5parm
ImportError: No module named importer

HannoSpreeuw · 2018-10-17T13:13:44Z

Is that someting we can fix using pip?

mr-c · 2018-10-17T13:19:00Z

@HannoSpreeuw that a problem with the container or something within it

mr-c · 2018-10-17T13:19:53Z

You can request a fix at https://github.com/kernsuite/packaging/issues

HannoSpreeuw · 2018-10-17T13:26:07Z

But why kernsuite? Couldn't it be prefactor?
Since we have

File "/usr/lib/prefactor/scripts/losotoImporter.py", line 16, in

mr-c · 2018-10-17T13:30:15Z

That's a path within the container. It is true, could be a prefactor bug.

HannoSpreeuw · 2018-10-17T13:31:52Z

@tammojan please comment.

gijzelaerr · 2018-10-17T13:47:01Z

KERN-dev uses losoto 2.0, it looks like prefactor is not compatible with this release?

gijzelaerr · 2018-10-19T11:35:18Z

KERN-3 still contains the old losoto, trying that one out now.

mr-c changed the title ~~ERROR: The executable .virtualenv/bin/python2 is not functioning~~ losoto error Oct 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

losoto error #31

losoto error #31

HannoSpreeuw commented Oct 17, 2018

mr-c commented Oct 17, 2018 •

edited

Loading

HannoSpreeuw commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018 •

edited

Loading

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018 •

edited by mr-c

Loading

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018

mr-c commented Oct 17, 2018

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018

gijzelaerr commented Oct 17, 2018

gijzelaerr commented Oct 19, 2018

losoto error #31

losoto error #31

Comments

HannoSpreeuw commented Oct 17, 2018

mr-c commented Oct 17, 2018 • edited Loading

HannoSpreeuw commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018 • edited Loading

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018 • edited by mr-c Loading

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018

mr-c commented Oct 17, 2018

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018

mr-c commented Oct 17, 2018

HannoSpreeuw commented Oct 17, 2018

gijzelaerr commented Oct 17, 2018

gijzelaerr commented Oct 19, 2018

mr-c commented Oct 17, 2018 •

edited

Loading

HannoSpreeuw commented Oct 17, 2018 •

edited

Loading

HannoSpreeuw commented Oct 17, 2018 •

edited by mr-c

Loading