perf(gateway): poor performance when usingtower-http 0.4.0

[kckeiks]

Description

Upgradingtower-http #420 introduced a regression.

Running with tower-http = "0.4.0"

Summary:
  Success rate:    1.0000
  Total:    67.1743 secs
  Slowest:    7.6177 secs
  Fastest:    0.2376 secs
  Average:    3.2824 secs
  Requests/sec:    14.8866

  Total data:    25.61 MiB
  Size/request:    26.22 KiB
  Size/sec:    390.35 KiB

Response time histogram:
  0.238 [1]   |
  0.976 [5]   |
  1.714 [42]  |■■■
  2.452 [99]  |■■■■■■■■■
  3.190 [319] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  3.928 [346] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  4.666 [125] |■■■■■■■■■■■
  5.404 [38]  |■■■
  6.142 [14]  |■
  6.880 [5]   |
  7.618 [6]   |

Latency distribution:
  10% in 2.2874 secs
  25% in 2.7222 secs
  50% in 3.2680 secs
  75% in 3.7787 secs
  90% in 4.3584 secs
  95% in 4.8468 secs
  99% in 6.3910 secs

Details (average, fastest, slowest):
  DNS+dialup:    3.3122 secs, 0.9046 secs, 3.4577 secs
  DNS-lookup:    0.0000 secs, 0.0000 secs, 0.0000 secs

Status code distribution:
  [200] 1000 responses

Running with tower-http = "0.3.5"

Summary:
  Success rate:    1.0000
  Total:    5.3613 secs
  Slowest:    3.5667 secs
  Fastest:    0.0480 secs
  Average:    0.2648 secs
  Requests/sec:    186.5223

  Total data:    25.67 MiB
  Size/request:    26.28 KiB
  Size/sec:    4.79 MiB

Response time histogram:
  0.048 [1]   |
  0.400 [949] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.752 [0]   |
  1.104 [0]   |
  1.455 [0]   |
  1.807 [0]   |
  2.159 [0]   |
  2.511 [0]   |
  2.863 [0]   |
  3.215 [0]   |
  3.567 [50]  |■

Latency distribution:
  10% in 0.0727 secs
  25% in 0.0839 secs
  50% in 0.0937 secs
  75% in 0.1120 secs
  90% in 0.1326 secs
  95% in 3.3333 secs
  99% in 3.5269 secs

Details (average, fastest, slowest):
  DNS+dialup:    3.3217 secs, 1.0806 secs, 3.4888 secs
  DNS-lookup:    0.0000 secs, 0.0000 secs, 0.0000 secs

Status code distribution:
  [200] 1000 responses

Notes

Checklist

• I have ensured that my version is up-to-date
• I have ensured that my issue is reproducible
• I have ensured that my issue is not a duplicate

[kckeiks]

@ozwaldorf this one would be a good one to tackle to do more research. First I would talk to the team though to discuss before picking it up.