Skip to content

Commit

Permalink
fs: transfer: don't raise if link failed with ENOSYS
Browse files Browse the repository at this point in the history
Discovered this while playing with databricks. E.g. `dvc import-url` resulted in:

```
2023-07-27 01:02:33,446 DEBUG: v3.8.1 (pip), CPython 3.10.6 on Linux-5.15.0-1040-azure-x86_64-with-glibc2.35
2023-07-27 01:02:33,446 DEBUG: command: /local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/bin/dvc import-url https://archive.ics.uci.edu/static/public/186/wine+quality.zip -v
2023-07-27 01:02:36,628 DEBUG: Removing output 'wine+quality.zip' of stage: 'wine+quality.zip.dvc'.
2023-07-27 01:02:36,629 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip'
Importing 'https://archive.ics.uci.edu/static/public/186/wine+quality.zip' -> 'wine+quality.zip'
2023-07-27 01:02:37,874 DEBUG: Computed stage: 'wine+quality.zip.dvc' md5: 'bec84f2324d59f520ab3460709b68f87'
2023-07-27 01:02:37,874 DEBUG: 'md5' of stage: 'wine+quality.zip.dvc' changed.
2023-07-27 01:02:44,428 DEBUG: Computed stage: 'wine+quality.zip.dvc' md5: 'ef727310d96dc9022717839b298235be'
2023-07-27 01:02:45,086 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/99fd51b05b33c5f95ca322e09d4465732bac0c61226db94ea7322cb29a00110f' to '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5'
2023-07-27 01:02:45,086 DEBUG: Preparing to collect status from '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5'
2023-07-27 01:02:45,086 DEBUG: Collecting status from '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5'
2023-07-27 01:02:45,386 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/99fd51b05b33c5f95ca322e09d4465732bac0c61226db94ea7322cb29a00110f'
2023-07-27 01:02:45,984 ERROR: failed to transfer '0ddfa7a9379510fe7ff88b9930e3c332' - [Errno 38] Function not implemented: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' -> '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332'
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 306, in transfer
    _try_links(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 250, in _try_links
    _link(link, from_fs, from_path, to_fs, to_path)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 62, in _link
    func(from_path, to_path)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 381, in link
    return self.fs.link(from_info, to_info)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 166, in link
    return system.hardlink(path1, path2)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_objects/fs/system.py", line 32, in hardlink
    os.link(src, link_name)
OSError: [Errno 38] Function not implemented: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' -> '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332'

2023-07-27 01:02:45,986 DEBUG: failed to protect '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332' - [Errno 2] No such file or directory: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332'
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/db/local.py", line 119, in protect
    os.chmod(path, self.CACHE_MODE)
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/0d/dfa7a9379510fe7ff88b9930e3c332'

2023-07-27 01:02:46,584 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-07-27 01:02:46,585 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.bF8oLrv5ibS4ysuLVnMuhm.tmp'
2023-07-27 01:02:47,446 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.bF8oLrv5ibS4ysuLVnMuhm.tmp'
2023-07-27 01:02:47,499 DEBUG: Removing '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/.dvc/cache/files/md5/.29mHH5XQQ6unJZpZSkL9qD.tmp'
2023-07-27 01:02:49,079 ERROR: failed to import https://archive.ics.uci.edu/static/public/186/wine+quality.zip. You could also try downloading it manually, and adding it with `dvc add`. - unable to remove '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. Use `-f` to force.
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 759, in _checkout
    return checkout(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 276, in checkout
    _checkout(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 223, in _checkout
    _checkout_file(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 88, in _checkout_file
    _relink(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 59, in _relink
    _remove(path, fs, in_cache, force=force, prompt=prompt)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc_data/hashfile/checkout.py", line 53, in _remove
    raise PromptError(path)
dvc_data.hashfile.checkout.PromptError: unable to remove '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/commands/imp_url.py", line 15, in run
    self.repo.imp_url(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 151, in run
    return method(repo, *args, **kw)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/repo/imp_url.py", line 85, in imp_url
    stage.run(jobs=jobs, no_download=no_download)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
    return call()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/__init__.py", line 626, in run
    self.commit(allow_missing=allow_missing)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/decorators.py", line 43, in rwlocked
    return call()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/stage/__init__.py", line 549, in commit
    out.commit(filter_info=filter_info, **kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 800, in commit
    self._checkout(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-5a432df4-b771-45c9-ad4e-de37cca00570/lib/python3.10/site-packages/dvc/output.py", line 761, in _checkout
    raise ConfirmRemoveError(exc.path)  # noqa: B904
dvc.exceptions.ConfirmRemoveError: unable to remove '/Workspace/Repos/ruslan@iterative.ai/databricks-mlflow-end-to-end-example/wine+quality.zip' without a confirmation. Use `-f` to force.

2023-07-27 01:02:49,111 DEBUG: Analytics is enabled.
2023-07-27 01:02:49,314 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpnej7wva2']'
2023-07-27 01:02:49,318 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpnej7wva2']'
```
  • Loading branch information
efiop committed Jul 27, 2023
1 parent c76ad54 commit f48f77c
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions src/dvc_objects/fs/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@ def _try_links(
errno.ENOTSUP,
errno.EXDEV,
errno.ENOTTY,
errno.ENOSYS,
):
raise
error = exc
Expand Down

0 comments on commit f48f77c

Please sign in to comment.