Parallelize artifact downloads. Update progress bar styling.

[IanButterworth]

This PR

% JULIA_DEPOT_PATH=$(mktemp -d): JULIA_PKG_PRECOMPILE_AUTO=0 ./julia --project=../Pkg.jl -t1 -e 'import Pkg; Pkg.activate(temp=true); @time Pkg.add("FFMPEG")'
...
 12.999424 seconds (9.81 M allocations: 915.741 MiB, 2.30% gc time, 73 lock conflicts, 11.69% compilation time: 16% of which was recompilation)

Master

% JULIA_DEPOT_PATH=$(mktemp -d): JULIA_PKG_PRECOMPILE_AUTO=0 ./julia-t1 -e 'import Pkg; Pkg.activate(temp=true); @time Pkg.add("FFMPEG")'
...
 23.540455 seconds (9.37 M allocations: 887.289 MiB, 0.82% gc time, 123 lock conflicts, 5.85% compilation time: 18% of which was recompilation)

Screen.Recording.2024-08-15.at.12.49.12.AM.mov

Todo

• parallel progress printing
• effective interrupt handling (proven to be hard to do because of Downloads.jl being so async)
  • needs testing
• error handling & reporting
  • needs testing
• code tidy and remove duplication (will implement a new MiniMultiProgressBar type in a follow-on)
• simple printing for CI etc.
• show progress in bytes
  • rate/s ?
  • total across all artifacts?

[KristofferC]

Pkg.jl/src/Operations.jl

Lines 918 to 931 in a717900

	# Check if the current package is available in one of the registries being tracked by the pkg server
	# In that case, download from the package server
	if server_registry_info !== nothing
	server, registry_info = server_registry_info
	for reg in ctx.registries
	if reg.uuid in keys(registry_info)
	if haskey(reg, pkg.uuid)
	url = "$server/package/$(pkg.uuid)/$(pkg.tree_hash)"
	push!(archive_urls, url => true)
	break
	end
	end
	end
	end

could be extracted into a function and used here.

[IanButterworth]

Sounds good, but we'd have to feed the pkg info into this point. I'm unsure of the relationship between packages and artifacts and the pkgserver. Probably best as another PR.

[KristofferC]

Ideally this ANSI stuff could be moved to the progress bar code where there is something like:

struct MultiProgressBar
    progress_bars::Vector{MiniProgressBar}
end

which takes care of the printing?

[IanButterworth]

Yeah. Will handle this in a follow-on. Added a TODO

[KristofferC]

I think the number of significant figures in Base.format_bytes is a bit high.

Edit: JuliaLang/julia#55495

[KristofferC]

I updated / modernized the progress bar styling a bit. It now looks like:

Screencast.from.08-15-2024.12.19.04.PM.webm

[IanButterworth]

[IanButterworth]

Lets give this a go. @KristofferC and I have both been testing it out.