SS4 loadtimes significantly slower than SS3 version of exact same site

[micschk]

Affected Version

Silverstripe 4 (on Apache/PHP 7.3.5 with Nginx proxy)
Silverstripe 3 (on Apache/PHP 7.1.x with Nginx proxy)

Description

We've just finished the SS4 update of a quite large website of one of our clients.
The update process has been resource intensive and the result... well, frankly quite disappointing honestly.
Upon switching the site to the new version, even the non-technical client immediately noticed it 'felt' a bit slower.
After a few hours/days, this was confirmed by our monitoring (and turned out to be a lot worse than 'a bit').

Switched to SS4 version around 10:40...

Figured maybe it was due to lots of images being resampled on new pageloads or so but still after a few days:

Comparing the exact same (simple) page on both versions:

Version	TTFB	DOMContentLoaded	Load
SS3	243ms	558 ms	604 ms
SS4	543ms	885 ms	1.92 s

(SSv3 website runs on a subdomain on the same server as SSv4).

Steps to Reproduce

Well, basically update an existing site to SS4 and check the loadtimes.
Curious to hear from others about their experience.

[dhensby]

Hi @micschk - thanks for taking the time to write up some thorough findings. Whilst it's clear that there has been a degredation in performance for your instance I will say that my experience of SS4 has been that it can be quite performant but this can depend on server set-up (for example php with opcache enabled, etc).

The other thing to look at would be the number of DB requests that are being made per page-load and to see if there's a way to reduce those (template caching, will help here). Have you done any profiling to see where the processing time is happening?

I'm also quite surprised at how big a difference there is between DomContentLoaded and Load events for what you're implying is an identical front-end; this implies there's been a serious degradation in the serving of assets from the server. Are static assets responding slower as well?

[micschk]

@dhensby Good points, thx for looking into this!
As a test I've created two 'vanilla' installs, one SS3 and one SS4 on this same server (just CMS + Simple theme via composer).
They're in two separate directories on the same subdomain, running PHP 7.3.5 (opcache enabled) on Apache with Nginx proxy (for static assets).

The results initially appear similar:

Version	TTFB	DOMContentLoaded	Load	Finish
3.7.6	70 ms	190 ms	197 ms	217 ms
4.7.3	138 ms	239 ms	263 ms	399 ms

• Based on simple in-browser testing using Chrome network inspector with cache disabled & refreshing a ~10-20 times until the timing/figures were stable
• Serving the v4 version with public as webroot or via .htaccess rewrite didn't seem to make a difference speedwise

Regarding your other remarks:

• There are indeed slight differences in the front-end code (templates & JS), this is something we're looking at optimizing. But these differences are marginal (and mainly js/scripting time), not enough to warrant such a speed degradation imo (plus the above tests show similar speed ratios between 3 & 4 on a plain/identical install).
  
  Looking at the monitoring, an average total load time of around 1 second seems more representative (than my manual tests at least :-)).
• We will further investigate db queries and probably add additional caching, but this will be on top of/in addition to what was necessary for loading speedily on SS3 (meaning that's all patching up, there seems to actually be a quite significant performance degradation in the base/core).

[dhensby]

I'd also say this is probably related: #8603

[micschk]

I'd also say this is probably related: #8603

Quick initial showqueries (as logged in user to put the sites in dev mode -> results in additional 'Member' queries and probably also in deactivating any partial caching):

Site	Homepage query-count	Note
Vanilla SS3	19 DB queries	Slowest: 0.0052s UPDATE Member.LastVisited, all under 11ms, most under 5ms
Vanilla SS4	14 DB queries	Slowest: 0.0011s Member related (LastVisited removed in SS4 I think), all under 10ms, most under 5ms
Client-site SS3	103 DB queries	Slowest (3) around 50ms (Member & security related UPDATEs), all under 15ms, most under 5ms
Client-site SS4	114 DB queries	Slowest (2) around 25ms, one 15ms, all others under 10 most under 5ms

[dhensby]

This is interesting because it appears to show that the problem is not caused by the database layer and so coming from a different bottleneck. SS4 on vanilla looks like it should be faster when looking at the DB layer, but we are seeing double the TTFB and finish events...

[lekoala]

hi @micschk ! i highly recommend using https://github.com/lekoala/silverstripe-debugbar to troubleshoot these kind of issues. I recently added a couple of things like template rendering times that should allow you to see what's taking so long. Actually you could install it on your ss3 and ss4 version and see if you can pinpoint where the extra load comes from.
In my experience, rendering template can be quite slow, and database queries can quickly pile up due to the orm not caching things as it should (Member::get()->byId() is not cached while DataObject::get_by_id(Member::class) is ...).

[micschk]

Thomas (@lekoala), thanks for the suggestion. Your module is on my radar and todo list. And while it's probably the tool to dig deeper into this issue, I'd like to stress that the issue doesn't seem to be isolated to this specific custom system as the 'vanilla' installs show similar behaviour.

[sminnee]

A few diagnostic steps:

• It would be worth using ApacheBench or similar to test the speed of just the dynamic server render, to confirm it's a server-side issue.

• it might be worth running the 2 codebases on the same version of PHP

One change that might be relevant is SilverStripe's caching: in Silverstripe 4 it makes use of Symfony's cache system and there are a number of cache backends that it can use - its possible that proper optimisation of these is more important for getting good performance.

[sminnee]

In particular you probably want APC-based caching enabled.

The code that configures this is in https://github.com/silverstripe/silverstripe-framework/tree/4/src/Core/Cache

[brettt89]

Basic performance testing

Siege running against vanilla Silverstripe installations comparing latest Silverstripe 4 vs latest Silverstripe 3.

NOTE: These tests were performed without OPCache on either configuration

Silverstripe 4 was running on port 8080 for the test, Silverstripe 3 was running on port 8090. Each environment had a /dev/build executed and request to the Homepage made before testing began to pre-warm database and caches.

Siege was run with a default 10 concurrent users and 10 reps (-r 10) for each test execution.

Each test was run twice, with a 5 second delay between tests. The best results returned between the two, for each Silverstripe installation, were used for reporting below.

$ siege -r 10 http://localhost:8080/
**
** SilverStripe 4 install (composer create-project silverstripe/install ./project)
**
** SIEGE 4.0.9
** Preparing 25 concurrent users for battle.
The server is now under siege...
Transactions:                   1750 hits
Availability:               100.0000 %
Elapsed time:                 7.0500 secs
Data transferred:            10.7255 MB
Response time:                0.0918 secs
Transaction rate:           248.2269 trans/sec
Throughput:                   1.5213 MB/sec
Concurrency:                 22.7972
Successful transactions:        1750
Failed transactions:               0
Longest transaction:          0.5200
Shortest transaction:         0.0000




$ siege -r 10 http://localhost:8090/
**
** SilverStripe 3 install (composer create-project silverstripe/install ./project ^3)
**
** SIEGE 4.0.9
** Preparing 25 concurrent users for battle.
The server is now under siege...
Transactions:                   1750 hits
Availability:               100.0000 %
Elapsed time:                 6.2500 secs
Data transferred:            20.9770 MB
Response time:                0.0769 secs
Transaction rate:           280.0000 trans/sec
Throughput:                   3.3563 MB/sec
Concurrency:                 21.5456
Successful transactions:        1750
Failed transactions:               0
Longest transaction:          0.7600
Shortest transaction:         0.0000

Results

Silverstripe 3 vanilla installation on average was approximately %20 faster than a Silverstripe 4 vanilla installation when comparing the average "Response time" from a basic siege test.

Replicating Tests

This test used a vanilla installation of Silverstripe using composer to facilitate the install, and Docker to run the emulated environment.

Installing Silverstripe

Silverstripe 4

$ composer create-project silverstripe/installer TestWebsitev4

Silverstripe 3

$ composer create-project silverstripe/installer TestWebsite ^3

Docker environment

Both Silverstripe 4 and Silverstripe 3 use the same configuration with the only difference being public/ as the DOCUMENT_ROOT for Silverstripe 4.

Add the docker-compose.yml file to the project directory for each installation.

docker-compose.yml

version: "3.8"
services:
  silverstripe:
    image: brettt89/silverstripe-web:7.4-apache
    ports:
      - "8080:80"
    volumes:
       - .:/var/www/html
    depends_on:
       - database
    #
    # For Silverstripe 4 installations use /var/www/html/public for DOCUMENT_ROOT
    #
    environment:
       - DOCUMENT_ROOT=/var/www/html
       - SS_TRUSTED_PROXY_IPS=*
       - SS_ENVIRONMENT_TYPE=dev
       - SS_DATABASE_SERVER=database
       - SS_DATABASE_NAME=SS_mysite
       - SS_DATABASE_USERNAME=root
       - SS_DATABASE_PASSWORD=
       - SS_DEFAULT_ADMIN_USERNAME=admin
       - SS_DEFAULT_ADMIN_PASSWORD=password

  database:
    image: mysql:5.7
    environment:
       - MYSQL_ALLOW_EMPTY_PASSWORD=yes
    volumes:
       - db-data:/var/lib/mysql
volumes:
     db-data:

NOTE: For Silverstripe 4 installations use DOCUMENT_ROOT=/var/www/html/public instead

Prepping Environment

Docker Compose is used to manage the environment for testing. Execute the following command at the project root to start the environment in the background.

$ docker-compose up -d

Getting the environments ready requires a couple of extra steps to ensure the database is accessible and can be built, and the cache is pre-warmed.

Silverstripe 3 Installations only

Silverstripe 3 installations require the usage of _ss_environment.php instead of using Environment Variables. So copy the following to the project root for _ss_environment.php.

<?php

$_FILE_TO_URL_MAPPING['/var/www/html'] = 'localhost:8080';

define('SS_TRUSTED_PROXY_IPS', '*');
define('SS_ENVIRONMENT_TYPE', 'dev');
define('SS_DATABASE_SERVER', 'database');
define('SS_DATABASE_NAME', 'SS_mysite');
define('SS_DATABASE_USERNAME', 'root');
define('SS_DATABASE_PASSWORD', '');
define('SS_DEFAULT_ADMIN_USERNAME', 'admin');
define('SS_DEFAULT_ADMIN_PASSWORD', 'password');

All Installations from here

Load the environment using your browser via the remote port defined in docker-composer.yml (e.g. http://localhost:8080/). This should trigger a /dev/build on the environment and any errors with the environment will be identified here.

NOTE: If you get an chown error when loading the website, execute the following command to fix it.
docker-compose exec silverstripe chown -R www-data:www-data /var/www/html/public

Once Database building has completed successfully, reload the homepage to pre-warm the cache. (E.g. http://localhost:8080/). This should set your environment up ready for testing.

Executing Tests

Siege 4.0.9 was used to execute the tests against the environment. You can replicate the same tests results using the following execution command.

$ siege -r 10 http://localhost:8080/

NOTE: Siege used in the testing for results above was compiled from source with small customization to output 4 decimal places in the results to give more details.

Test will be executed against the environment and the results will be output shortly.

Cleanup

Use the following docker command from your project root to destroy the environment created for testing. This may be required between testing Silverstripe installations, or you can use different ports (as with results) for each installation to run them in parallel.

docker-compose kill

[sb-relaxt-at]

We have gone through some extensive performance testing and optimization as we encountered similar problems. Due to lots of involved DataObjects we do heavily use caching (using a self written cache module), this improves the performance considerably. I just add some of our findings along the way (note that this may differ and may depend on our specific setup):

• Having XDebug enabled (or even the extension just loaded) has an extreme performance impact
• A big performance impact has been caused by CachedConfigCollection is written to disk on every page load silverstripe-config#34, you might want to check the size of the config cache file. Having a big config file and lots of concurrent writes might cause issues.
• Having open_basedir set can have quite a performance impact (not SilverStripe specific)
• Using different cache backends had only very little impact (note that the php files backend of symfony utilizes opcache which we have enabled)
• The database queries are not the bottleneck in most cases but the ORM is (creating DataObjects, applying Extensions, …), so looking at query execution times is rarely insightful. Of course if there are large tables and missing indices or an overuse of indices this might causes performance issues.
• The view layer seems to be slow when using lots of template files

The debugbar is definitely a great tool for (performance) debugging :-)

Do you have data whether there is a high load or iowait?

edit: Just seen that this was posted 2021, so I am one year late to the party ;) Would be interested to hear if you @micschk have made any progress?

[lekoala]

Small update regarding SS4 performance, you might be interested in these:
#10836
#10834
silverstripe/silverstripe-assets#558
silverstripe/silverstripe-assets#560
silverstripe/silverstripe-session-manager#150