Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throttle httpclient requests in Pip and SimplePip #847

Closed
wants to merge 1 commit into from

Conversation

Omotola
Copy link
Contributor

@Omotola Omotola commented Oct 10, 2023

This pr uses semaphore to reduce the amount of http requests to 10 concurrent requests at a time in both pip and simplepip. This is to avoid httpclient timing out due to too many requests at a time.

The fix also seems to have made the cd scan run faster. It was initially taking over 30 minutes to run on the vienna repo locally, but after throttling the requests the scan is taking about 23 minutes.

@Omotola Omotola requested a review from a team as a code owner October 10, 2023 17:02
@Omotola Omotola requested a review from chsalgado October 10, 2023 17:02
@codecov
Copy link

codecov bot commented Oct 10, 2023

Codecov Report

Merging #847 (b168a9e) into main (ae50a41) will increase coverage by 0.0%.
Report is 8 commits behind head on main.
The diff coverage is 75.0%.

@@          Coverage Diff          @@
##            main    #847   +/-   ##
=====================================
  Coverage   77.4%   77.5%           
=====================================
  Files        235     235           
  Lines       9933    9956   +23     
=====================================
+ Hits        7696    7720   +24     
+ Misses      2237    2236    -1     
Files Coverage Δ
...ft.ComponentDetection.Detectors/pip/IPyPiClient.cs 64.3% <75.0%> (+0.2%) ⬆️
...mponentDetection.Detectors/pip/SimplePypiClient.cs 89.4% <75.0%> (-0.4%) ⬇️

... and 2 files with indirect coverage changes

@Omotola Omotola closed this Oct 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant