Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't write the ref file unless necessary #2657

Merged
merged 5 commits into from
Nov 14, 2024
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions src/huggingface_hub/_snapshot_download.py
Original file line number Diff line number Diff line change
Expand Up @@ -254,11 +254,26 @@ def snapshot_download(
# if passed revision is not identical to commit_hash
# then revision has to be a branch name or tag name.
# In that case store a ref.
if revision != commit_hash:
ref_path = os.path.join(storage_folder, "refs", revision)
os.makedirs(os.path.dirname(ref_path), exist_ok=True)
with open(ref_path, "w") as f:
f.write(commit_hash)
ref_path = os.path.join(storage_folder, "refs", revision)
# If the file exists, read the commit hash from it
write_commit_hash = True
if os.path.exists(ref_path):
try:
with open(ref_path) as f:
existing_commit_hash = f.read()
if existing_commit_hash == commit_hash:
write_commit_hash = False
except Exception as e:
logger.warning(f"Error reading commit hash from {ref_path}: {e}")

try:
# Write the commit hash to the ref file if it doesn't exist or is different
if revision != commit_hash and write_commit_hash:
os.makedirs(os.path.dirname(ref_path), exist_ok=True)
with open(ref_path, "w") as f:
f.write(commit_hash)
except Exception as e:
logger.warning(f"Error writing commit hash to {ref_path}: {e}")
d8ahazard marked this conversation as resolved.
Show resolved Hide resolved

# we pass the commit_hash to hf_hub_download
# so no network call happens if we already
Expand Down