High CPU, RAM and I/O usage

[SoWhy]

Describe the bug
Okay, since a few days, even after the upgrade to the 2024 version, I noticed the VM on which Remotely is running will quickly start using a lot of CPU and RAM and I can literally watch the free space being eaten by the "overlay" device:

root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  2.2G   28G   8% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  180K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  2.2G   28G   8% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  6.1G   24G  21% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  6.1G   24G  21% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  7.1G   23G  24% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  7.1G   23G  24% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  8.0G   22G  27% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  8.0G   22G  27% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  9.0G   21G  31% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  9.0G   21G  31% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G   10G   20G  34% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G   10G   20G  34% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
[...]
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G   18G   13G  59% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G   18G   13G  59% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged

For a while, it will only spike in intervals, filling up a lot of space before returning to 2.2G but after that, it will fill up everything and the CPU will keep being at 99% until I kill the server and restart which will only give me a short reprieve before the circle repeats
htop will show (when it works) that "dotnet Remotely_Server.dll" is the culprit I assume that the problem is some runaway setting for logging but I cannot find it.

Trying to analyze the mount directory with ncdu shows no additional files, especially not of that size

I set up a new VM with a manual Debian install (instead of the Debian CT), installed Remotely fresh and copied the settings from the old install but the problem persists. It also persists when installing Debian and Docker on a fresh PC and copying over the configuration files from the old install

The problem does not seem to appear before accessing the web interface for the first time. So the service will run with 0% CPU for hours but once you access the web interface, it will start acting up.

Assigning more CPU cores will mitigate the problem but it's not a fix because it will still create spikes in usage, just not as high.

I found some information online that indicates that it might be related to dotnet 8.x but the problem started with my dotnet 7.x based version before the upgrade. Unfortunately, I cannot remember what changed on that day and I cannot find anything related in the log files.

To Reproduce
Steps to reproduce the behavior:

1. Install fresh Debian and Remotely
2. Import Remotely.db etc. from old install
3. Access web interface

Remotely Version
Server (can be found on about page): 2024.02.23.1927
Agent (can be found in device card): 2024.02.23.1927

Expected Behavior
No runaway CPU, memory and I/O usage

Screenshots
See https://www.reddit.com/r/remotely_app/comments/1ccyhs8/docker_overlay_eating_up_all_my_space_quickly_and/

Desktop (please complete the following information):

• OS: Windows 10/11 Pro
• Browser Firefox
• Version 126

Additional Context
Add any other context about the problem here.

• I am running Remotely in Docker and not on my QNAP, Synology, or Internet Connected Toaster

[bitbound]

What's in the logs on the server?

[SoWhy]

If you tell me which logs I should share, I will gladly do so

[bitbound]

If you tell me which logs I should share, I will gladly do so

Sorry. If you're using Docker, they will be in the mounted volume under the logs directory (relevant readme section).

If you're hosting "bare metal" with systemd, it will be in the content root directory (where the binaries are located), under /AppData/logs.

I'm basically wondering if there's an error that's spamming rapidly.

[SoWhy]

Sorry for the delay in replaying. The log files in /var/www/remotely/logs contain no information apart from what I did (so like six lines after a reboot). I ran ncdu on / while it was happening but it didn't find any additional files

[SoWhy]

If you're hosting "bare metal" with systemd, it will be in the content root directory (where the binaries are located), under /AppData/logs.

Is it still possible to run Remotely without docker, so I can test whether it's docker-related? if so, how?

[bitbound]

How many agents are connected to the server? Remotely won't scale well into the thousands or anything.

Is it still possible to run Remotely without docker

It's still possible. There's a ZIP with each release that contains linux-x64 binaries. You'd extract them, then set up a systemd service that executes the Remotely_Server binary. Then put a reverse proxy in front of it. I prefer caddy, since the default configuration has everything needed for Remotely to work correctly.

Below is an example systemd service and caddy configuration. Official docs can be found here.

/etc/systemd/system/remotely-server.service:

[Unit]
Description=Remotely Server

[Service]
WorkingDirectory=/var/www/remotely
ExecStart=/var/www/remotely/Remotely_Server --urls "http://*:5020"
Restart=always
RestartSec=10
SyslogIdentifier=remotely
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false

[Install]
WantedBy=multi-user.target

/etc/caddy/Caddyfile:

remotely.example.com {
  reverse_proxy 127.0.0.1:5020
}

[SoWhy]

How many agents are connected to the server? Remotely won't scale well into the thousands or anything.

Like 30 or so. I'll try the barebones and report back

[SoWhy]

Set up a new VM with barebones with the same DB. Same problem unfortunately. Still nothing in the app log

[SoWhy]

Huge disk reads / writes going on and df -h shows a lot of space used but ncdu can't find any additional files. Now sure what Remotely is doing there. Any idea how to attach a debugger to that?

[bitbound]

I don't know how to help in this situation. I've never seen this before. I would try starting with a basic out-of-the-box configuration and change one thing at a time until you find the configuration item that's setting it off.

[SoWhy]

Out of the box works, so I'm assuming it's connected to the database or settings somehow. But how do I import them one at a time?

[gravasio]

@SoWhy Have you run some scheduled scripts?

[bitbound]

As @gravasio asked, what have you done since "taking it out of the box"? Scheduled scripts could indeed cause this if they're sending a lot of output.

To be fair, though, the whole process of handling script output was put together hastily and could use some optimization.

Out of the box works, so I'm assuming it's connected to the database or settings somehow. But how do I import them one at a time?

By default, it uses SQLite, and the database file will be created in /app/AppData. The example docker-compose.yml has this mapped to /var/www/remotely.

Did you change the defaults? Are you using an external Postgres or SQL Server database?

[gravasio]

In the past, I encountered significant CPU usage issues with scheduled scripts, which required clearing the DeviceScriptRun table to resolve. I intended to conduct a detailed investigation but am currently short on time.
Your experience seems similar to the issue I encountered.

[SoWhy]

@bitbound: I only copied the Remotely.Db* and appsettings.json from the previous install to a fresh install, both docker and bare-bones (for the latter, I did change the path in the appsettings.json since it contained the docker-path). I made no other changes and the OS was a freshly installed Debian 12 each time without an external SQL server

@gravasio: Good idea but there are no scheduled scripts in my Db unfortunately, the table is empty