Add progress log to HTTP downloads

[leonardehrenfried]

Summary

This adds a progress log for HTTP downloads which give you a bit of visibillity when OTP does large downloads.

Issue

❌

Unit tests

❌

[codecov]

Codecov Report

Attention: Patch coverage is 14.28571% with 6 lines in your changes missing coverage. Please review.

Project coverage is 69.79%. Comparing base (738969f) to head (2928fbd).

Files with missing lines	Patch %	Lines
...ipplanner/datastore/https/HttpsFileDataSource.java	0.00%	4 Missing ⚠️
...anner/datastore/https/HttpsDataSourceMetadata.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             dev-2.x    #6065      +/-   ##
=============================================
- Coverage      69.79%   69.79%   -0.01%     
  Complexity     17368    17368              
=============================================
  Files           1964     1964              
  Lines          74427    74430       +3     
  Branches        7632     7632              
=============================================
  Hits           51947    51947              
- Misses         19831    19834       +3     
  Partials        2649     2649              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[habrahamsson-skanetrafiken]

good catch

[t2gran]

Currently progress tracking is done one abstraction level up. E.g. this will result in double tracking of OSM loading data. The benefit of adding this one level up, is that we have better/more detailed control of the context like use-case, expected speed and source. If added to the data-source it should be added as a generic feature that is opt-in and with configurable log message. Adding it to one DataSource and not the others is inconsistent. If I load it from HTTP I get tracking, but not if I load it from Google Cloud. Since we have few data-sources I think keeping this responsibility out of the DataSource component is a good thing.

I suggests searching for the use of the asInputStream() and asOutputStream() and add progress tracking those locations instead - where the content is expected to take more than 5 seconds for a big source. In places where it is small, I would not add it - since it has a small overhead(synchronization - ignorable if we are fetching data outside mem).

Another reason for not adding it the the data-source is that the progress tracker always log, so for things that are so small that we do not want to track them, the tracker will create noice.

Another way to identify places to add tracking is to look through the logs and look for gaps in the logging that is more than 5 sec. If found, it is likely that there is a big task in the gap that shuld be tracked.

[t2gran]

When I looked at this I identified on place witch is missing logging - reading the graph:

@SuppressWarnings("Convert2MethodRef")
  public static SerializedGraphObject load(DataSource source) {
    return load(source.asInputStream(), source.path());
    InputStream in = ProgressTracker.track("Reading graph '" + source.name() + "'",
      64*1024,
      source.size(),
      source.asInputStream(),
      // Keep lambda, do not convert to method-ref. Make the logger
      // report the correct class & line number.
      m -> LOG.info(m)
      );
    return load(in, source.path());
  }

(SerializedGraphObject:124)