Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix large SFTP reads #638

Open
wants to merge 7 commits into
base: devel
Choose a base branch
from
Open

Fix large SFTP reads #638

wants to merge 7 commits into from

Conversation

Jakuje
Copy link
Contributor

@Jakuje Jakuje commented Aug 30, 2024

SUMMARY

Fixes and reproducers for large SFTP reads that truncate file with the second chunk.

Fixes: #341

ISSUE TYPE
  • Bugfix Pull Request

@psf-chronographer psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Aug 30, 2024
@Jakuje Jakuje marked this pull request as ready for review August 30, 2024 11:23
Copy link

sonarcloud bot commented Aug 30, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarCloud

Copy link

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo yum install -y dnf-plugins-core on RHEL 8
  • sudo dnf install -y dnf-plugins-core on Fedora
  • dnf copr enable packit/ansible-pylibssh-638
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

@Jakuje Jakuje changed the title Sftp large Fix large SFTP reads Aug 30, 2024
@Qalthos Qalthos self-requested a review September 9, 2024 12:53
with open(local_file, 'wb+') as f:
with open(local_file, 'ab') as f:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will append content to the file which already exists. I've tested this diff with ansible-libssh v1.2.2 and only a60a6cb applied. I see multiple executions on the same destination file growing above original size:

remote-machine# ls -l /tmp/foo?
-rw-r--r--  1 root  wheel  1048576 Nov 15 12:31 /tmp/foo1
-rwx------  1 root  wheel  3145728 Nov 15 12:37 /tmp/foo3
local-machine$ ls -l /tmp/foo?
-rw-r--r--  1 mkucharski  wheel  3145728 Nov 15 12:35 /tmp/foo2

Above we see size 3x after 3 executions of:

t1 = sftp_session.get(remote_file, local_file)
t2 = sftp_session.put(local_file, new_remote_file)

I think this needs more work. Also mode doesn't match after put(), but that is another issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for throughful testing! Indeed, you are right. I thought this small change will make it working, but it looks like its more complicated. Updated the PR.

For the modes, I think this would be a different issue or rfe. Right now, the code opens the file with normal python open, which will probably set modes based on your umask, which will result in sane permissions. We could update the permissions based on what is on the server (if the server uses unix/linux ACL), but given that they will likely have different usernames and groups, it might result in unwanted permissions.

Fixes: ansible#341
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Thanks @kucharskim for the report and testing!

Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Copy link

sonarcloud bot commented Nov 15, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bot:chronographer:provided There is a change note present in this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sftp.get downloads only the last chunk of file
2 participants