Symlink to bref/lib64/libgpg-error.so broken in some layers

[logicalor]

Description:

Development Environment is Ubuntu 20.04, Serverless 2.28

Sample layers:

• arn:aws:lambda:ap-southeast-2:209497400698:layer:php-74:17
• arn:aws:lambda:ap-southeast-2:209497400698:layer:php-73:31

How to reproduce:

Create a serverless config referring to these layers:

e.g.

  layers:
    - ${bref:layer.php-73}

Attempt to locally invoke serverless.

Serverless will fail to execute with the error

Error: ENOENT: no such file or directory, stat '/home/{username}/.cache/serverless/invokeLocal/layers/php-73/31/bref/lib64/libgpg-error.so'

Upon examining /home/{username}/.cache/serverless/invokeLocal/layers/php-73/31/bref/lib64/ (eg. ls -la) one will see the symlink is broken: libgpg-error.so -> ../../lib64/libgpg-error.so.0.10.0

Manually download and examine the layer in question - eg. .arn:aws:lambda:ap-southeast-2:209497400698:layer:php-73:31 and examine the contents. One will see that the symlink is broken in the layer package as well.

Solution

Fix the symlink in the layers - change from ../../lib64/libgpg-error.so.0.10.0 to ../lib64/libgpg-error.so.0.10.0 or libgpg-error.so.0.10.0

[mnapoli]

Hi! I've since published a new release with new layers, could you update Bref, test and let us know if you still have the problem?

[logicalor]

Hi again @mnapoli

Sorry about the delayed response, and thanks for the update. I tried again against the layer arn:aws:lambda:ap-southeast-2:209497400698:layer:php-73:32.

In this layer it appears the symlink doesn't exist at all.

ls on 32/bref/lib64 gives me

drwxrwxr-x 2 shaunj shaunj   4096 Apr 22 11:29 .
drwxrwxr-x 5 shaunj shaunj   4096 Apr 22 11:29 ..
lrwxrwxrwx 1 shaunj shaunj     22 Apr 22 11:29 libgpg-error.so.0 -> libgpg-error.so.0.10.0
-rwxr-xr-x 1 shaunj shaunj  19296 Apr 22 11:29 libgpg-error.so.0.10.0
lrwxrwxrwx 1 shaunj shaunj     18 Apr 22 11:29 libreadline.so.6 -> libreadline.so.6.2
-rwxr-xr-x 1 shaunj shaunj 285232 Apr 22 11:29 libreadline.so.6.2
lrwxrwxrwx 1 shaunj shaunj     19 Apr 22 11:29 libsqlite3.so -> libsqlite3.so.0.8.6
lrwxrwxrwx 1 shaunj shaunj     19 Apr 22 11:29 libsqlite3.so.0 -> libsqlite3.so.0.8.6
-rwxr-xr-x 1 shaunj shaunj 732528 Apr 22 11:29 libsqlite3.so.0.8.6
lrwxrwxrwx 1 shaunj shaunj     12 Apr 22 11:29 libssh2.so -> libssh2.so.1
lrwxrwxrwx 1 shaunj shaunj     16 Apr 22 11:29 libssh2.so.1 -> libssh2.so.1.0.1
-rwxr-xr-x 1 shaunj shaunj 184128 Apr 22 11:29 libssh2.so.1.0.1
lrwxrwxrwx 1 shaunj shaunj     13 Apr 22 11:29 libssl.so -> libssl.so.1.1
-rwxr-xr-x 1 shaunj shaunj 712072 Apr 22 11:29 libssl.so.1.1
lrwxrwxrwx 1 shaunj shaunj     17 Apr 22 11:29 libxslt.so -> libxslt.so.1.1.28
lrwxrwxrwx 1 shaunj shaunj     17 Apr 22 11:29 libxslt.so.1 -> libxslt.so.1.1.28
-rwxr-xr-x 1 shaunj shaunj 250104 Apr 22 11:29 libxslt.so.1.1.28
lrwxrwxrwx 1 shaunj shaunj     11 Apr 22 11:29 libzip.so -> libzip.so.5
lrwxrwxrwx 1 shaunj shaunj     13 Apr 22 11:29 libzip.so.5 -> libzip.so.5.3
-rwxr-xr-x 1 shaunj shaunj 101600 Apr 22 11:29 libzip.so.5.3
-rwxr-xr-x 1 shaunj shaunj  41024 Apr 22 11:29 preloadable_libintl.so

[mnapoli]

Attempt to locally invoke serverless.

Oh I just realized that, I would recommend using bref local instead: https://bref.sh/docs/function/local-development.html

There is no official support for local invoke with serverless as it doesn't work in many cases with bref.

Let us know if there are issues running on Lambda.

[logicalor]

Attempt to locally invoke serverless.

Oh I just realized that, I would recommend using bref local instead: https://bref.sh/docs/function/local-development.html

There is no official support for local invoke with serverless as it doesn't work in many cases with bref.

Let us know if there are issues running on Lambda.

I ended up forking the Bref repo and figuring out where the missing libs are, and solving it by modifying the base dockerfile for PHP compilation. It was a case of fixing the broken symlinks and copying some additional lib files. If you would like me to create a PR with my fixes I'm happy to do this.

[mnapoli]

Hey, if you have a link to the diff why not take a look. If you don't know how to create such a link, it will be easier for you to open a PR then. I don't know if we will want to merge it, but let's take a look 👍

[logicalor]

Hey, if you have a link to the diff why not take a look. If you don't know how to create such a link, it will be easier for you to open a PR then. I don't know if we will want to merge it, but let's take a look +1

No worries, I have submitted a PR for this issue.

[mnapoli]

Thanks, I've written a comment about all that here: #928 (comment)

I would suggest closing this issue since there isn't anything actionable we can do here, what do you think?

[Maxwell2022]

@mnapoli bref local cannot simulate functions triggered via SQS message (offline), hence why we are using serverless-offline + serverless-offline-sqs in our case. Bref even recommends using serverless for to deployment so I don't really understand the logic here? Why not making it compatible with serverless-offline?

[deleugpn]

@Maxwell2022 Napoli's comment isn't against supporting serverless-offline, it's about protecting the production environment. It's not ideal to include modifications to 1B of Lambda executions using Bref without knowing if they will have a bad reaction just because of serverless-offline. Bref tries on a best-effort to help running workloads locally for development purposes, but the primary commitment is to run PHP in production.

We need a solid understanding of which modifications are necessary, why they are necessary and the confidence to know they won't break production workloads.

[mnapoli]

+1, also on this:

bref local cannot simulate functions triggered via SQS message (offline)

bref local can trigger SQS messages locally, you have to run bref local with a JSON payload that looks like what SQS does when invoking lambda.

[Maxwell2022]

I meant having serverless running offline and having the function being triggers automatically (listening for message) without having to invoke it manually, something that serverless-offline + serverless-offline-sqs + elasticmq is doing.

[logicalor]

There is no official support for local invoke with serverless as it doesn't work in many cases with bref.

Are there any other issues (open or closed) where examples of serverless invoke local not working with Bref are expanded upon? It would be good to understand these issues better in order to hopefully contribute to solving them.

The team I'm part of has been running serverless invoke local against functions based on Bref for over a year now both in local development and in test / deployment pipelines, and the only issue we've run into is that which I've outlined above (missing or broken libraries / symlinks in the Bref layers), which we ended up solving by forking, applying the patch I submitted, and building / deploying our own layers based on the patched build script.

[Maxwell2022]

Just thinking about this, could it be because these libs are missing in the lambci image? It would make sense if this is working in lambda. lambci/docker-lambda#341