Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Drenv cache test #7

Closed
wants to merge 7 commits into from
Closed

Drenv cache test #7

wants to merge 7 commits into from

Commits on May 15, 2024

  1. Tweak e2e workflow permissions 

    So it can run on my fork.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed May 15, 2024
    Configuration menu
    Copy the full SHA
    930ea42 View commit details
    Browse the repository at this point in the history

  2. Extract drenv.cache.clear() 

    And remove the timing of the operation, deleting a directory is fast
local operation.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed May 15, 2024
    Configuration menu
    Copy the full SHA
    d3bc0c5 View commit details
    Browse the repository at this point in the history

  3. Extract skip_addon helper 

    Called to warn about a missing addon.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed May 15, 2024
    Configuration menu
    Copy the full SHA
    8ae063c View commit details
    Browse the repository at this point in the history

  4. Streamline cache interface 

    Previously we had to do:

    path = cache.patch(key)
    cache.fetch(kustomziation_dir, key)

Now we do both operations in one call:

    path = cache.fetch(kustomziation_dir, key)

Thanks: Raghavendra Talur <rtalur@redhat.com>
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed May 15, 2024
    Configuration menu
    Copy the full SHA
    172ff6d View commit details
    Browse the repository at this point in the history

  5. Prepare for timed based cache refresh 

    Extract the a _fetch() helper to be used by new public APIs.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed May 15, 2024
    Configuration menu
    Copy the full SHA
    a83dfac View commit details
    Browse the repository at this point in the history

  6. Add do_ prefix to command functions 

    Preparing for renaming "fetch" to "cache" - the cache() function will
shadow the cache module. We can rename the imports or the function to
avoid this. Renaming the import is bad since it makes it harder to
understand the code. Renaming the functions is good since it helps to
understand what is a command function.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed May 15, 2024
    Configuration menu
    Copy the full SHA
    2c82e03 View commit details
    Browse the repository at this point in the history

  7. Time based cache refresh 

    Previously to update the cache you had to clear the cache and fetch.
This leaves a window where a build may use uncached value and fail.  A
better way is to refresh the cache periodically *without* clearing the
cache, ensuring that builds will always use cached values.

To support time based refresh, we fetch data only if the cached item it
too old or missing. Upon successful fetch, the item age is set to zero,
so the next access to the cache will skip the fetch.

We use 2 threshold for refreshing the cache:

- REFRESH_THRSHOLD: 12 hours
- FETCH_THRESHOLD: 48 hours

This allows refreshing the cache once per day (e.g. from a cron job). On
errors, we can rerun the refresh until the refresh succeeds. Fresh items
are skipped during the refresh.

If refreshing the cache failed, jobs will trigger a refresh after 48
hours. The longer threshold ensures that this will never happen unless
the refresh job is broken.

The `fetch` command was renamed to `cache`. This describe better the
purpose of the command. The hook was update as well to match the command
name.

The `drenv.cache` module provides now 2 operations:

- cache.refresh() - rebuild the cache if older than
  cache.REFRESH_THRESHOLD (12 hours). Will is used in the `cache` hook.

- cache.get() - rebuild the cache if older than cache.FETCH_THRESHOLD
  (48 hours) and return the path to the cached value. Will is used in
  `start` hooks to get a cached value.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
    @nirs
    nirs committed May 15, 2024
    Configuration menu
    Copy the full SHA
    7cf6e90 View commit details
    Browse the repository at this point in the history