Problems with owncloud/webdav file source

[abretaud]

Hi!
I'm testing the new pluggable file sources introduced in #9888 with our owncloud server.

The owncloud server is online at https://data-access.cesgo.org/, I had to change a little the configuration to make it use the correct webdav url:

- type: webdav
  id: owncloud1
  label: OwnCloud
  doc: External OwnCloud files (configure access in user preferences)
  url: ${user.preferences['owncloud|url']}
  root: ${user.preferences['owncloud|root']}
  login: ${user.preferences['owncloud|username']}
  password: ${user.preferences['owncloud|password']}

= I added the root param, and it kind of works by setting url=https://data-access.cesgo.org and root=/remote.php/webdav. Setting only the url to https://data-access.cesgo.org/remote.php/webdavdidn't work, it ended up with a duplicate webdav at the end.

"kind of works" means galaxy start listing the content, I see it in the logs, but it fails at some point with this exception:

Traceback (most recent call last):
  File "lib/galaxy/web/framework/decorators.py", line 294, in decorator
    rval = func(self, trans, *args, **kwargs)
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 71, in index
    index = file_source.list(file_source_path.path, recursive=recursive, user_context=user_context)
  File "lib/galaxy/files/sources/_pyfilesystem2.py", line 39, in list
    return list(map(to_dict, res))
  File "lib/galaxy/files/sources/_pyfilesystem2.py", line 51, in _resource_info_to_dict
    path = os.path.join(dir_path, name)
  File "/opt/galaxy/.venv/lib/python3.6/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/opt/galaxy/.venv/lib/python3.6/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

or this other one after modifying https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/files/sources/_pyfilesystem2.py#L50 with something like if name is None: return {}

Traceback (most recent call last):
  File "/opt/galaxy/.venv/lib/python3.6/site-packages/webdavfs/webdavfs.py", line 248, in getinfo
    info = self.client.info(_path.encode('utf-8'))
  File "/opt/galaxy/.venv/lib/python3.6/site-packages/webdav2/client.py", line 58, in _wrapper
    res = fn(self, *args, **kw)
  File "/opt/galaxy/.venv/lib/python3.6/site-packages/webdav2/client.py", line 629, in info
    return parse(response, path)
  File "/opt/galaxy/.venv/lib/python3.6/site-packages/webdav2/client.py", line 606, in parse
    raise RemoteResourceNotFound(path)
webdav2.exceptions.RemoteResourceNotFound: Remote resource: /remote.php/webdav/Some_file not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lib/galaxy/web/framework/decorators.py", line 294, in decorator
    rval = func(self, trans, *args, **kwargs)
  File "lib/galaxy/webapps/galaxy/api/remote_files.py", line 71, in index
    index = file_source.list(file_source_path.path, recursive=recursive, user_context=user_context)
  File "lib/galaxy/files/sources/_pyfilesystem2.py", line 39, in list
    return list(map(to_dict, res))
  File "/opt/galaxy/.venv/lib/python3.6/site-packages/fs/base.py", line 1266, in <genexpr>
    for name in self.listdir(path)
  File "/opt/galaxy/.venv/lib/python3.6/site-packages/webdavfs/webdavfs.py", line 254, in getinfo
    raise errors.ResourceNotFound(path, exc=exc)
fs.errors.ResourceNotFound: resource '/Some_file' not found

This last exception is raised when listing a specific file in owncloud that was probably buggy on the owncloud side, but I guess galaxy should handle the error more gracefully.

Also I find that listing files is quite slow, I see in the logs 1 webdav query for each file/dir. I wonder if maybe we're asking for too much info on each file?

Anyway, that's a super cool new feature, very excited about it!

[abretaud]

Just found PyFilesystem/webdavfs#25 which fixes the first error above (TypeError: join() argument must be str or bytes, not 'NoneType')

[abretaud]

And PyFilesystem/webdavfs#28 for the slowness