Add Accept-Encoding (data compression) when pulling in documents

[mboelen]

When NextCloud-News pulls in our feed, it does this with a HTTP/1.1 and data encoding active. However, directly after that, it looks to be using a different HTTP client that does NOT provide the Accept-Encoding header (but uses HTTP/2.0), resulting in unneeded bandwidth usage (on both the client as the server).

Also, I'm wondering why it needs to retrieve the pages, instead of pulling the information from the feed itself. Again, it looks to be a waste of resources on both ends and room for improvement making the client a better net citizen.

So my feature request is:

• Provide Accept-Encoding when pulling in resources and allow modern compression methods like Brotli, Gzip, Zstd
• Retrieve the information from the feed instead of from the individual pages.

2024-04-18T11:11:04+00:00 200 1.2.3.4 "GET /feed/ HTTP/1.1" 17777 "-" "NextCloud-News/1.0" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:04+00:00 426 1.2.3.4 "GET /viewing-available-test-categories-in-lynis/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:04+00:00 426 1.2.3.4 "GET /cheat-sheets/awk/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:04+00:00 426 1.2.3.4 "GET /contact/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:04+00:00 426 1.2.3.4 "GET /web/nginx-show-all-configured-virtual-hosts/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:04+00:00 426 1.2.3.4 "GET /alternative-netstat-ss-tool/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:04+00:00 426 1.2.3.4 "GET /yum-plugins-available-plugins-and-built-in-security-support/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:04+00:00 426 1.2.3.4 "GET /ubuntu-server-hardening-guide-quick-and-secure/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:05+00:00 426 1.2.3.4 "GET /why-linux-security-hardening-scripts-might-backfire/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:05+00:00 426 1.2.3.4 "GET /what-is-a-security-audit/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-18T11:11:05+00:00 426 1.2.3.4 "GET /how-to-clear-the-arp-cache-on-linux/ HTTP/2.0" 16 "-" "-" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .

The IP address has been masked with 1.2.3.4, but as you can see it is all the same, in just a matter of a second of time.

[SMillerDev]

While I'm generally in favour of an implementation of this, I doubt I'll be making a pull request for it since I don't really have the motivation for such a complex problem. This is mostly because:

Provide Accept-Encoding when pulling in resources and allow modern compression methods like Brotli, Gzip, Zstd

It'll be a bunch of checks we need to add to identify what the user system actually supports.

[mboelen]

Thanks for the quick reply!

I have not enough knowledge about this app yet to know what HTTP client is used here. Typically, if a modern HTTP client is used, the compression support should already be available. It "only" needs to be activated by telling which types of compression it supports. And if the HTTP client supports just Gzip, then that is already a good start. After all, the first request is already doing it, even though the HTTP protocol is slightly older than the more modern 2.0. So it looks like it are two different tools that pull in the data?

What are your thought about those 10 extra requests that come after retrieving the feed? As a webmaster, I don't understand it why the app is doing this (especially with the main contents already in the feed).

[mboelen]

Just checked, and see cURL and Guzzle in the code, including already with some references to the Accept-Encoding header. So it looks like some parts are already there. If Guzzle is in the mix, then this recent issue might be relevant: guzzle/guzzle#3215 (it was created as part of a similar issue I reported to the Selfoss project).

[Grotax]

There is already an older issue or discussion about this. News already does what is possible. But we also use a faction library to fetch an icon for the feed in case the feed does not provide a logo.

And I think that library does not set the required headers and development is not very active and I'm not aware of a more actively maintained alternative.

[Grotax]

See #2551

Realistic suggestions of course welcome.

[mboelen]

Thanks @Grotax - I did a search for a few keywords, but that issue didn't come up.

So, after reading that issue, is it really just a matter of users that have to upgrade? Because if it is, I don't mind contacting those admins that I see in the log and reach out to them and give them a hint about their outdated version.

Does it also resolve the "issue" where URLs are pulled in instead of gathered from the feed itself?