SIGSEGV: segmentation violation. Invalid memory address or nil pointer dereference

[SpartakusMd]

What happened?

The container started to crash after recent deploy and I'm receiveng SIGSEGV: segmentation violation today. I believe it may be related to the fact that I wanted to increase memory_limit to 3.5G or 3584M. It may also be linked to the fact that there is an env variable which is very long 7700 chars. Or maybe it's related to the new PHP version? Locally it works ok but fails when deployed to Azure Container Apps.

Build Type

Docker (Debian Bookworm)

Worker Mode

Yes

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

memory_limit=3.5G

or

memory_limit=3584M

Relevant log output

*Note: may not be in corect order

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1558081]
goroutine 17 [running, locked to thread]:
github.com/dunglas/frankenphp.(*phpThread).pinString(...)
/go/src/app/phpthread.go:77
github.com/dunglas/frankenphp.go_getfullenv(0x0?)
/go/src/app/env.go:33 +0xe1

[dunglas]

Are you using version 1.3.4? If yes could you try downgrading to v1.3.3? Regarding the backtrace, I fear it's an issue introduced with the refactoring of worker threads (#1137). cc @AlliBalliBaba

[SpartakusMd]

You are right @dunglas. I have downgraded to v1.3.3 and it works ok. Will wait for v1.3.5 to switch back to v1. Thank you.

[dunglas]

Do you have any environment variable or HTTP header containing an empty string? I think my patch will fix the issue, but I don't manage to reproduce it currently (I would like to add a test ensuring that the patch works).

[SpartakusMd]

No, there are no empty variables if I use env or printenv.

Maybe there is some issue with some characters if the env value contains some special symbols. I have a variable containing JSON with special characters, but probably not as locally it runs ok.

Here is the output of printenv (redacted):

KUBERNETES_SERVICE_PORT_HTTPS=443
SERVER_NAME=:80
KUBERNETES_SERVICE_PORT=443
BUILD_CI_PIPELINE_ID=12463
HOSTNAME=api--0000039-6997640000-qstds
PHP_VERSION=8.3.14
PHP_INI_DIR=/usr/local/etc/php
GPG_KEYS=1198C0117...76CA
XDG_DATA_HOME=/data
GODEBUG=cgocheck=0
XDG_CONFIG_HOME=/config
PHP_LDFLAGS=-Wl,-O1 -pie
MSI_SECRET=1cd70000-0000-0000-0000-1d8562e6e0c4
PWD=/app
BUILD_DATE=20241220
HOME=/root
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
BUILD_BRANCH=production
APP_ENV=prod
APP_ENVIRONMENT=production
APP_CONTAINER=api
PHP_SHA256=58b4cb9019bf70c0cbcdb814c7df79b9065059d14cf7dbf48d971f8e56ae9be7
IDENTITY_HEADER=1cd76bbe-0000-0000-0000-1d8562e6e0c4
APP_SECRETS_HASH=70fc69a853600000000340a3543987e9
PHPIZE_DEPS=autoconf            dpkg-dev                file            g++             gcc             libc-dev                make            pkg-config re2c
IDENTITY_ENDPOINT=http://localhost:42356/msi/token
TERM=xterm
PHP_URL=https://www.php.net/distributions/php-8.3.14.tar.xz
CONTAINER_APP_HOSTNAME=api--0000039.wonderfulsand-c47d0000.francecentral.azurecontainerapps.io
CONTAINER_APP_NAME=api
SHLVL=1
COMPOSER_ALLOW_SUPERUSER=1
KUBERNETES_PORT_443_TCP_PROTO=tcp
PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
APP_SECRETS=[{"name":"allow-users","value":"1"}]
CONTAINER_APP_REPLICA_NAME=api--0000039-6997640000-qstds
FRANKENPHP_CONFIG=import worker.Caddyfile
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_PORT=tcp://10.0.0.1:443
CONTAINER_APP_ENV_DNS_SUFFIX=wonderfulsand-c47d0000.francecentral.azurecontainerapps.io
KUBERNETES_PORT_443_TCP_PORT=443
BUILD_GIT_SHA=3cda72622c5873000081ba3ddef667b95f25161
CONTAINER_APP_PORT=80
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
CONTAINER_APP_REVISION=api--0000039
MSI_ENDPOINT=http://localhost:42356/msi/token
PHP_ASC_URL=https://www.php.net/distributions/php-8.3.14.tar.xz.asc
PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
_=/usr/bin/env

[dunglas]

I published a new release that will soon be available: https://github.com/dunglas/frankenphp/actions/runs/12433328417

I hope it will fix the issue.

[vdwijngaert]

``Still get it with FrankenPHP v1.3.5 PHP 8.2.26 Caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=, unfortunately! I don't see any empty env vars either at first glance. Any way we can help debug?

Edit: Here's the output from the logs.

{"level":"info","ts":1734713530.230431,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}                                                                                                                                                                                                                                                                                                                                        
{"level":"info","ts":1734713530.2359915,"msg":"adapted config to JSON","adapter":"caddyfile"}                                                                                                                                                                                                                                                                                                                                               
{"level":"warn","ts":1734713530.2360206,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}                                                                                                                                                                                                                                          
{"level":"info","ts":1734713530.2379818,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}                                                                                                                                                                                                                                 
{"level":"warn","ts":1734713530.23839,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}                                                                                                                                                                                                                               
{"level":"info","ts":1734713530.238434,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000709700"}                                                                                                                                                                                                                                                                                          
{"level":"info","ts":1734713530.2402396,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}                                                                                                                                                                                                                                                                                                              
{"level":"info","ts":1734713530.2403092,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}                                                                                                                                                                                                                                                                                                              
panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                                                                                                                                                                                                                                     
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x155cafe]                                                                                                                                                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                                                                                                                                            
goroutine 17 [running, locked to thread]:                                                                                                                                                                                                                                                                                                                                                                                                   
github.com/dunglas/frankenphp.(*phpThread).pinString(...)                                                                                                                                                                                                                                                                                                                                                                                   
    /go/src/app/phpthread.go:81                                                                                                                                                                                                                                                                                                                                                                                                             
github.com/dunglas/frankenphp.(*phpThread).pinCString(0x0, {0xc000000101?, 0xc00009ae70?})                                                                                                                                                                                                                                                                                                                                                  
    /go/src/app/phpthread.go:88 +0x3e                                                                                                                                                                                                                                                                                                                                                                                                       
github.com/dunglas/frankenphp.go_sapi_getenv(0x0, 0x480901?)                                                                                                                                                                                                                                                                                                                                                                                
    /go/src/app/env.go:73 +0x58

[AlliBalliBaba]

This happens immediately after starting the server? Are you using the worker with symfony or laravel? Are you explicitly using getenv() somewhere in your code?

[vdwijngaert]

This happens immediately after starting the server?

Yes.

Are you using the worker with symfony or laravel?

Symfony.

Are you explicitly using getenv() somewhere in your code?

Yes!

[dunglas]

Ok that's thread that is null 😅

[AlliBalliBaba]

Oh I think you're right @dunglas
Can you share your Caddy configuration @vdwijngaert? How many threads are you starting?

[vdwijngaert]

32 threads and 30 workers. Those numbers are provided with two env vars... With a default to 3 and 2.

Was just in the process of doing performance testing and trying out different configs and resource allocation.

Will check for the file when I get back home :)

[dunglas]

Closed by #1278

[SherinBloemendaal]

For me the issue still persists after upgrading to FrankenPHP v1.3.5 PHP 8.3.14 Caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

In my case it seems related to opcache preloading, when its enabled the server immediately crashes:

php-1                    | panic: runtime error: invalid memory address or nil pointer dereference
php-1                    | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x142db0c]

Framework: Symfony
Mode: Worker
Calling getenv(): Yes

[dunglas]

#1278 is not in 1.3.5. It will be in the upcoming 1.3.6 version.

[vdwijngaert]

I think a build job failed so no changes were pushed for 1.3.6 to dockerhub 🙈

Edit: I tested it with the dev build that ran shortly after, dunglas/frankenphp-dev:sha-5a14834-php8.2 in my case, which works perfectly!

[andersonpem]

The 1.3 tag wasn't updated yet to 1.3.6 :)
I'm pulling docker.io/dunglas/frankenphp:1.3-php8.3-bookworm
Using the patch version for now.

[SpartakusMd]

Tested with 1.3.6 version, works as expected.

[SherinBloemendaal]

Same for me. Thanks everyone. happy holidays :)