Shared filesystem cache - Multiple PHP process in same time : Wait/Restart problems

[kdefives]

Hello,

Context:
I am using Custom stores because i have to execute two different PHP process in parallel. This two PHP process have to call the same API and use the same client API key for which i have API limit rate. My application using Symfony 5.2.

Problem:
Seems the store is correctly shared between my two process, because i have a rate limit of 50 requests per minute and when i execute my two program, the addition of API calls of my two processes equals 50 before to block and wait for next calls, ok. Then after wait some seconds, my two process continue but after few calls, i got the error message API 429 Too Many Requests because the limit is reached.
Do you think the problem could be because my two process are waiting in the same time to do an API call?

Possible rootcause, but not tested/confirmed:
As my two process are waiting in the same time, because they are doing a lot of API call in same time, do you think the problem could be because they try to continue to call API in same time? I really don't know. :/

Below the custom rate limiter store that I implemented:

<?php


namespace App\Store;


use Psr\Cache\InvalidArgumentException;
use Spatie\GuzzleRateLimiterMiddleware\Store;
use Symfony\Component\Cache\Adapter\FilesystemAdapter;
use Symfony\Component\Cache\CacheItem;
use Throwable;

class RateLimiterStore implements Store
{

    /**
     * @var FilesystemAdapter $cache
     */
    private $cache;

    /**
     * @var string $itemKey
     */
    private $itemKey;

    /**
     * RateLimiterStore constructor.
     * @param FilesystemAdapter $cache
     * @param string $itemKey
     */
    public function __construct(FilesystemAdapter $cache, string $itemKey)
    {
        $this->cache = $cache;
        $this->itemKey = $itemKey;
    }

    /**
     * @return array<int>
     * @throws InvalidArgumentException&Throwable
     * @throws Throwable
     */
    public function get(): array
    {
        return $this->cache->get($this->itemKey, function () {
            return [];
        });
    }

    /**
     * @param int $timestamp
     * @param int $limit
     * @throws InvalidArgumentException&Throwable
     */
    public function push(int $timestamp, int $limit): void
    {
        /**
         * @var CacheItem $cacheItem
         */
        $cacheItem = $this->cache->getItem($this->itemKey);
        $cacheItem->set(array_merge($this->get(), [$timestamp]));
        $this->cache->save($cacheItem);


    }
}

Below how i declare Guzzle http client in services.yaml:

# This file is the entry point to configure your own services.
# Files in the packages/ subdirectory configure your dependencies.

# Put parameters here that don't need to change on each machine where the app is deployed
# https://symfony.com/doc/current/best_practices/configuration.html#application-related-configuration
parameters:
    #locale: 'en'
    app.supported_locales: 'en|fr'

services:
    # default configuration for services in *this* file
    _defaults:
        autowire: true      # Automatically injects dependencies in your services.
        autoconfigure: true # Automatically registers your services as commands, event subscribers, etc.
        bind:
            $httpClientApi: '@http_client_api'
            $rateLimiterStorePerMinute: '@App\Store\RateLimiterStorePerMinute'
            $rateLimiterStorePerSecond: '@App\Store\RateLimiterStorePerSecond'
            $supportedLocales: '%app.supported_locales%'

    # makes classes in src/ available to be used as services
    # this creates a service per class whose id is the fully-qualified class name
    App\:
        resource: '../src/'
        exclude:
            - '../src/DependencyInjection/'
            - '../src/Entity/'
            - '../src/Kernel.php'
            - '../src/Tests/'

    # controllers are imported separately to make sure services can be injected
    # as action arguments even if you don't extend any base controller class
    App\Controller\:
        resource: '../src/Controller/'
        tags: ['controller.service_arguments']

    # add more service definitions when explicit configuration is needed
    # please note that last definitions always *replace* previous ones

    GuzzleHttp\MessageFormatter:
        class: GuzzleHttp\MessageFormatter
        arguments:
            - '{request} {response} {error}'

    log_middleware:
        class: GuzzleHttp\Middleware
        factory: [ 'GuzzleHttp\Middleware', log ]
        arguments: [ '@logger', '@GuzzleHttp\MessageFormatter' ]

    Symfony\Component\Cache\Adapter\FilesystemAdapter:
        class: Symfony\Component\Cache\Adapter\FilesystemAdapter
        arguments: ['fs-adapter-cache']

    App\Store\RateLimiterStorePerMinute:
        class: App\Store\RateLimiterStore
        arguments: [ '@Symfony\Component\Cache\Adapter\FilesystemAdapter', 'rate-limiter-per-minute' ]

    App\Store\RateLimiterStorePerSecond:
        class: App\Store\RateLimiterStore
        arguments: [ '@Symfony\Component\Cache\Adapter\FilesystemAdapter', 'rate-limiter-per-second' ]

    Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddlewarePerMinute:
        class: Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddleware
        factory: [ 'Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddleware', perMinute ]
        arguments: [ '%env(int:API_RATE_LIMIT_PER_MINUTE)%', '@App\Store\RateLimiterStorePerMinute' ]

    Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddlewarePerSecond:
        class: Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddleware
        factory: [ 'Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddleware', perSecond ]
        arguments: [ '%env(int:API_RATE_LIMIT_PER_SECOND)%', '@App\Store\RateLimiterStorePerSecond' ]

    GuzzleHttp\HandlerStack:
        class: GuzzleHttp\HandlerStack
        factory: [ 'GuzzleHttp\HandlerStack', create ]
        calls:
            - [ push, [ '@log_middleware', 'log' ] ]
            - [ push, [ '@Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddlewarePerMinute', 'rate-limit-minute' ] ]
            - [ push, [ '@Spatie\GuzzleRateLimiterMiddleware\RateLimiterMiddlewarePerSecond', 'rate-limit-second' ] ]

    http_client_api:
        class: GuzzleHttp\Client
        arguments:
            - handler: '@GuzzleHttp\HandlerStack'
              headers:
                  'X-App-Token': '%env(API_KEY)%'

Workaround solved for the moment:
My real API limit rate is 100 requests every 2 minutes.
Initially i set the value to 50 per minutes but i the problem explained above occurs. Then as a workaround a set the value to 40 per minutes to be sure to not reach the real limit.

I hope someone could help me.
Thank you.

[kdefives]

Hello,

Unfortunately my workaround is not enough because now I have to execute my jobs in parallel multiple times. Now I can have 10+ executions of the same job in parallel, and i still have a lot of 429 Too many request. :(

I will try to debug but for the moment I do not know if it is a problem about cache or if the problem come from the guzzle-rate-limiter.
IMO the cache is ok because when I execute jobs one by one I can see the cache file is generated and increasing... I conclude the file system adapter for cache is correctly set.

What I plan to test:

1. Test with a Redis cache to check if the problem is due to usage of a shared file system as a cache...
2. Continue debugging Guzzle rate limiter to understand how it is continuing to execute request after waiting the end of rate limit. I have the feeling that when multiple processes are waiting in same time (due to rate limit), guzzle rate limiter is not managing correctly the "restart" to continue to execute requests between multiple process in same time. :/

Anyway, If anybody can help me I really appreciate 🙂

[kdefives]

Issue similar as #11

[kdefives]

Finally for my use case, i changed my strategy by using retry strategy (using https://github.com/caseyamcl/guzzle_retry_middleware) instead of using a rate limiter.