[question] How does this benchmark against Sozu?

[lukehsiao]

Recently I was browsing rust-based reverse proxies and happened upon this one as well as https://www.sozu.io/. I'm curious how the performance compares.

[paulocoghi]

(I'm not a maintainer, just a user)

If Sozu provides comparisons with Nginx, it will be easier to know which is faster, since rpxy is currently ~1.5x faster than Nginx.

But it's also important to emphasize that Sozu still doesn't support HTTP/3 (QUIC) sozu-proxy/sozu#565, as rpxy does.

[lukehsiao]

If Sozu provides comparisons with Nginx, it will be easier to know which is faster, since
rpxy is currently ~1.5x faster than Nginx.

Unfortunately, they provide no benchmarks that I could find. They do have a Dockerfile, so it may be fairly straightforward to set up a new benchmark from the code in the bench/ directory of this repo.

But it's also important to emphasize that Sozu still doesn't support HTTP/3 (QUIC) sozu-
proxy/sozu#565, as rpxy does.

Great point.

[paulocoghi]

They do have a Dockerfile, so it may be fairly straightforward to set up a new benchmark from the code in the bench/ directory of this repo.

It would be awesome if you could help the project with this update!

[junkurihara]

Hi Luke and Paulo,

At this moment, I have never tried sozu and not sure if rpxy works faster than it. But yes, adding benchmarking result on sozu sounds nice. I will take a look on sozu's Dockerfile!

[junkurihara]

P.S.: As you may know, I added the latest benchmarking result on my laptop (MBP'14 2021). Surprisingly, the rpxy became faster than before in my environment. I am not sure why, but this might be due to the optimization of Rust?

[paulocoghi]

Just wow! Not only the latency with rpxy remains lower than Nginx (22% to 42% lower latency), but now the total req/s is 1.7x faster!

[junkurihara]

I tried to build Sozu docker image on my laptop (Macbook Pro with Apple M1), but unfortunately failed. This is due to its implementation of kawa, an internal http parser made by the Sozu developers. kawa currently supports only amd64 (x86_64) architecture (as shown in its src/protocol/utils.rs#L123 and other parts), and cannot be built on arm64 like my laptop.

So, to benchmark rpxy against Sozu, we should consider other ways. I will first try to build on my VPS machine and benchmark it.

[paulocoghi]

Would be interesting if @lukehsiao could help us doing the test in his computer. What do you think, Luke?

[lukehsiao]

Would be interesting if @lukehsiao could help us doing the test in his computer. What do you think, Luke?

Sure, I can help when I have some time.

One thing that isn't clear to me from the Sozu docs is what the configuration should be for the benchmark. If you've already figured one out what you can share, that would be helpful.

[lukehsiao]

I also see there is pre-built image available here: https://hub.docker.com/r/clevercloud/sozu/ this tracks the master branch. So you don't need to build it yourself.

[junkurihara]

Thanks Paulo and Luke for your comments!

I just put the latest benchmark result into the bench dir examined on the amd64 architecture, which was done with our office's computer ;-)

Due to the limitation/flaw (?) of sozu's message parser, sozu cannot handle request messages dispatched fromrewrk. So I used the legacy wrk instead of rewrk for amd64 test. From the benchmarking result, /the performance of rpxy and sozu seems to be comparable. But sozu performs somewhat faster than rpxy?

[lukehsiao]

Thanks Paulo and Luke for your comments!

I just put the latest benchmark result into the bench dir examined on the amd64 architecture, which was done with our office's computer ;-)

Due to the limitation/flaw (?) of sozu's message parser, sozu cannot handle request messages dispatched fromrewrk. So I used the legacy wrk instead of rewrk for amd64 test. From the benchmarking result, /the performance of rpxy and sozu seems to be comparable. But sozu performs somewhat faster than rpxy?

Thanks! Very interesting results, and cool to see two different rust implementations doing so well.

[paulocoghi]

We thank you, Jun! The new benchmarks are top notch and show a promising future for rpxy!

Despite a good performance on Sozu's side (congrats to the devs), rpxy sill manages to outperform it both on req/s and on average latency.

Also, because rpxy is already # 1, even on its current form and without the optimizations discussed on #57, it gives us promising perspectives regarding production performance after the updates with an even faster HTTP/3 strategy (a "league" where Sozu unfortunately still doesn't play).

When using it on HTTP/3, we could expect even better results.

[futurist]

Now there's another player called pingora-proxy.
Could you add it to the benchmark too?
Choose between rpxy and pingora-proxy is a good option for me since Sozu is under AGPL.

[junkurihara]

Sounds nice but pingora is a library and does not seem to work “as-is”. To compare it with rpxy, we need to implement a reference reverse proxy using pingora. i think it would not be “fair” actually.

[futurist]

pingora can run in a spawned std::thead, and communicate with outside? Is that make sense?

[paulocoghi]

@futurist There is a lot of work to do between a proxy library and a fully-featured proxy software, and benching Pingora vs rpxy would be an apples-to-orange comparison, since Pingora is just a rust library, as @junkurihara already pointed out.

There is a project called River, leaded by the Memory Safety / Prossimo initiative, which plans to create a production ready standalone reverse proxy, based on Pingora from Cloudflare.

When River becomes in fact usable and testable (and it seems this will take a lot of time), then it would be the correct way to bench against rpxy. 🙂