Exception(runtime_meta) in custom runtime on Openwhisk #1229

nadeshseen · 2024-01-15T11:07:15Z

nadeshseen
Jan 15, 2024

Hi,
I apologize in advance if my question appears to be basic.
I am trying to deploy a simple hello world action on Openwhisk. My ultimate goal is to run the lithops map reduce task on my own setup of Openwhisk + Minio.
But right now, I am getting this error message. The custom runtime that I have registered RUNS (confirmed on my openwhisk cluster) when I run the following command and I also get the correct output. But the issue is that the output I get is returned as an Exception.

❯ lithops runtime deploy 10.129.26.3:5000/hello_testing
2024-01-15 16:24:34,978 [INFO] config.py:139 -- Lithops v3.1.0 - Python3.10
2024-01-15 16:24:34,978 [DEBUG] config.py:101 -- Loading configuration from /home/nadesh/thesis/.lithops_config
2024-01-15 16:24:34,990 [DEBUG] config.py:191 -- Loading Serverless backend module: openwhisk
2024-01-15 16:24:34,991 [DEBUG] config.py:233 -- Loading Storage backend module: minio
2024-01-15 16:24:35,051 [DEBUG] minio.py:37 -- Creating MinIO client
2024-01-15 16:24:35,051 [DEBUG] minio.py:42 -- Setting MinIO endpoint to http://10.129.26.118:31043
2024-01-15 16:24:35,141 [INFO] minio.py:61 -- MinIO client created - Endpoint: http://10.129.26.118:31043
2024-01-15 16:24:35,156 [DEBUG] openwhisk.py:36 -- Creating OpenWhisk client
2024-01-15 16:24:35,156 [DEBUG] openwhisk.py:49 -- Set OpenWhisk Endpoint to https://10.129.26.3:31001
2024-01-15 16:24:35,156 [DEBUG] openwhisk.py:50 -- Set OpenWhisk Namespace to guest
2024-01-15 16:24:35,156 [DEBUG] openwhisk.py:51 -- Set OpenWhisk Insecure to True
2024-01-15 16:24:35,156 [INFO] openwhisk.py:65 -- OpenWhisk client created - Namespace: guest
2024-01-15 16:24:35,157 [INFO] openwhisk.py:121 -- Deploying runtime: 10.129.26.3:5000/hello_testing - Memory: 256 Timeout: 300
2024-01-15 16:24:35,157 [DEBUG] client.py:238 -- Creating functions package lithops_23bc4
2024-01-15 16:24:35,212 [DEBUG] client.py:246 -- Package lithops_23bc4: resource already exists
2024-01-15 16:24:35,212 [DEBUG] utils.py:233 -- Creating function handler zip in /home/nadesh/thesis/lithops_openwhisk.zip
2024-01-15 16:24:35,262 [DEBUG] client.py:147 -- Creating function action: 10.129.26.3_5000_hello_testing_256MB_3.1.0
2024-01-15 16:24:35,660 [DEBUG] client.py:155 -- OK --> Created action 10.129.26.3_5000_hello_testing_256MB_3.1.0
2024-01-15 16:24:35,660 [DEBUG] openwhisk.py:225 -- Extracting runtime metadata from: 10.129.26.3:5000/hello_testing
Traceback (most recent call last):
  File "/home/nadesh/.local/bin/lithops", line 8, in <module>
    sys.exit(lithops_cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/nadesh/.local/lib/python3.10/site-packages/lithops/scripts/cli.py", line 518, in deploy
    runtime_meta = compute_handler.deploy_runtime(runtime_name, runtime_memory, runtime_timeout)
  File "/home/nadesh/.local/lib/python3.10/site-packages/lithops/serverless/serverless.py", line 84, in deploy_runtime
    return self.backend.deploy_runtime(runtime_name, memory, timeout=timeout)
  File "/home/nadesh/.local/lib/python3.10/site-packages/lithops/serverless/backends/openwhisk/openwhisk.py", line 140, in deploy_runtime
    return self._generate_runtime_meta(docker_image_name, memory)
  File "/home/nadesh/.local/lib/python3.10/site-packages/lithops/serverless/backends/openwhisk/openwhisk.py", line 239, in _generate_runtime_meta
    raise Exception(runtime_meta)
Exception: {'greeting': 'hello2'}

openwhisk.py file Line 239

 238      if not runtime_meta or 'preinstalls' not in runtime_meta:
 239           raise Exception(runtime_meta)

Basically, I don't have preinstalls keyword in the runtime_meta. Any idea where the problem is?

Answered by JosepSampe

Jan 26, 2024

For some reason you runtime does not contain flask:

$ docker run docker.io/nadeseen/custom_runtime-3.10:0.2
Traceback (most recent call last):
  File "/pythonAction/pythonrunner.py", line 26, in <module>
    from actionproxy import ActionRunner, main, setRunner
  File "/pythonAction/../actionProxy/actionproxy.py", line 35, in <module>
    import flask
ModuleNotFoundError: No module named 'flask'

To build a custom runtime you can use either this Dockerfile or the Dockerfile.slim

View full answer

JosepSampe · 2024-01-16T09:46:32Z

JosepSampe
Jan 16, 2024
Maintainer

Hi @nadeshsee, I understand from the first post logs that the runtime 10.129.26.3:5000/hello_testing is not properly built for lithops.
How did you build it?

12 replies

nadeshseen Jan 24, 2024
Author

I am still not able to run a customized runtime. Pods keeps on crashing if I use this Dockerfile
Additionally, the command lithops hello is currently not functioning as expected. Now, it seems to be unable to trigger an action on the OpenWhisk cluster.

❯ lithops hello -d
2024-01-24 12:36:27,493 [INFO] config.py:139 -- Lithops v3.1.0 - Python3.10
2024-01-24 12:36:27,493 [DEBUG] config.py:101 -- Loading configuration from /home/nadesh/thesis/.lithops_config
2024-01-24 12:36:27,506 [DEBUG] config.py:191 -- Loading Serverless backend module: openwhisk
2024-01-24 12:36:27,507 [DEBUG] config.py:233 -- Loading Storage backend module: minio
2024-01-24 12:36:27,555 [DEBUG] minio.py:37 -- Creating MinIO client
2024-01-24 12:36:27,555 [DEBUG] minio.py:42 -- Setting MinIO endpoint to http://10.129.26.118:31043
2024-01-24 12:36:27,636 [INFO] minio.py:61 -- MinIO client created - Endpoint: http://10.129.26.118:31043
2024-01-24 12:36:27,655 [DEBUG] openwhisk.py:36 -- Creating OpenWhisk client
2024-01-24 12:36:27,655 [DEBUG] openwhisk.py:49 -- Set OpenWhisk Endpoint to https://10.129.26.3:31001
2024-01-24 12:36:27,655 [DEBUG] openwhisk.py:50 -- Set OpenWhisk Namespace to guest
2024-01-24 12:36:27,655 [DEBUG] openwhisk.py:51 -- Set OpenWhisk Insecure to True
2024-01-24 12:36:27,655 [INFO] openwhisk.py:65 -- OpenWhisk client created - Namespace: guest
2024-01-24 12:36:27,656 [DEBUG] invokers.py:93 -- ExecutorID ad5515-0 - Invoker initialized. Max workers: 100
2024-01-24 12:36:27,656 [DEBUG] invokers.py:288 -- ExecutorID ad5515-0 - Serverless invoker created
2024-01-24 12:36:27,656 [DEBUG] executors.py:148 -- Function executor for openwhisk created with ID: ad5515-0
2024-01-24 12:36:27,656 [INFO] invokers.py:107 -- ExecutorID ad5515-0 | JobID A000 - Selected Runtime: docker.io/lithopscloud/ibmcf-python-v310 - 256MB
2024-01-24 12:36:27,656 [DEBUG] storage.py:474 -- Runtime metadata found in local disk cache
2024-01-24 12:36:27,657 [DEBUG] job.py:242 -- ExecutorID ad5515-0 | JobID A000 - Serializing function and data
2024-01-24 12:36:27,657 [DEBUG] serialize.py:73 -- Referenced Modules: lithops
2024-01-24 12:36:27,657 [DEBUG] module_dependency.py:109 -- Module 'lithops' is already installed in the runtime, skipping
2024-01-24 12:36:27,658 [DEBUG] serialize.py:118 -- Modules to transmit: None
2024-01-24 12:36:27,658 [DEBUG] job.py:276 -- ExecutorID ad5515-0 | JobID A000 - Uploading function and modules to the storage backend
2024-01-24 12:36:27,739 [DEBUG] minio.py:99 -- PUT Object lithops.jobs/ad5515-0/7f8c46108b0f9689da91f7ce4845afed.func.pickle - Size: 667.0B - OK
2024-01-24 12:36:27,739 [DEBUG] job.py:316 -- ExecutorID ad5515-0 | JobID A000 - Data per activation is < 8.0KiB. Passing data through invocation payload
2024-01-24 12:36:27,739 [INFO] invokers.py:173 -- ExecutorID ad5515-0 | JobID A000 - Starting function invocation: hello() - Total: 1 activations
2024-01-24 12:36:27,739 [DEBUG] invokers.py:178 -- ExecutorID ad5515-0 | JobID A000 - Worker processes: 1 - Chunksize: 1
2024-01-24 12:36:27,739 [DEBUG] invokers.py:297 -- ExecutorID ad5515-0 - Async invoker 0 started
2024-01-24 12:36:27,740 [DEBUG] invokers.py:297 -- ExecutorID ad5515-0 - Async invoker 1 started
2024-01-24 12:36:27,740 [DEBUG] invokers.py:426 -- ExecutorID ad5515-0 | JobID A000 - Free workers: 100 - Going to run 1 activations in 1 workers
2024-01-24 12:36:27,740 [INFO] invokers.py:209 -- ExecutorID ad5515-0 | JobID A000 - View execution logs at /tmp/lithops-nadesh/logs/ad5515-0-A000.log
2024-01-24 12:36:27,740 [DEBUG] monitor.py:381 -- ExecutorID ad5515-0 - Starting Storage job monitor
2024-01-24 12:36:27,740 [INFO] wait.py:98 -- ExecutorID ad5515-0 - Getting results from 1 function activations
2024-01-24 12:36:27,907 [DEBUG] invokers.py:372 -- ExecutorID ad5515-0 | JobID A000 - Calls 00000 invoked (0.167s) - Activation ID: 33c58e2c73224753858e2c73226753fe
2024-01-24 12:36:29,788 [DEBUG] monitor.py:127 -- ExecutorID ad5515-0 - Pending: 1 - Running: 0 - Done: 0

EDIT:
can you share the dockerfile used to run the lithops hello command and the contents of hello.py python file?

JosepSampe Jan 24, 2024
Maintainer

The changes I made are on master branch, not yet released, so its normal you experience the same behavior as the past week if you didn't update Lithops.
To update your local Lithops installation to use the latest changes you must run python3.10 -m pip install -U git+https://github.com/lithops-cloud/lithops.git@master

nadeshseen Jan 25, 2024
Author

after updating, with default runtime the test function is working fine.
But custom runtime is still not working with this Dockerfile
tried with both python 3.10 (nadeseen/custom_runtime-3.10:0.2) & 3.11 (nadeseen/custom_runtime-3.11:0.1)
Both give the same errors as below

> kubectl describe pod -n openwhisk wskowdev-invoker-01-28-guest-dockerio-nadeseen-custom-runtime31
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  60s                default-scheduler  Successfully assigned openwhisk/wskowdev-invoker-01-28-guest-dockerio-nadeseen-custom-runtime31 to smareo-worker2
  Normal   Pulling    59s                kubelet            Pulling image "docker.io/nadeseen/custom_runtime-3.10:0.2"
  Normal   Pulled     57s                kubelet            Successfully pulled image "docker.io/nadeseen/custom_runtime-3.10:0.2" in 1.964529132s
  Normal   Created    13s (x4 over 57s)  kubelet            Created container user-action
  Normal   Pulled     13s (x3 over 56s)  kubelet            Container image "docker.io/nadeseen/custom_runtime-3.10:0.2" already present on machine
  Normal   Started    12s (x4 over 56s)  kubelet            Started container user-action
  Warning  BackOff    12s (x5 over 55s)  kubelet            Back-off restarting failed container

> wsk -i activation get 01b06e81aaca45fab06e81aacaa5face                                
ok: got activation 01b06e81aaca45fab06e81aacaa5face
{
    "namespace": "guest",
    "name": "docker.io_nadeseen_custom_runtime-3.10_0.2_256MB_3.1.1.dev0",
    "version": "0.0.1",
    "subject": "guest",
    "activationId": "01b06e81aaca45fab06e81aacaa5face",
    "start": 1706208971577,
    "end": 1706208971577,
    "duration": 0,
    "statusCode": 3,
    "response": {
        "status": "whisk internal error",
        "statusCode": 0,
        "success": false,
        "result": {
            "error": "Failed to run container with image 'docker.io/nadeseen/custom_runtime-3.10:0.2'."
        }
    },
    "logs": [],
}

import lithops

def my_function(x):
    return x + 7

fexec = lithops.FunctionExecutor(runtime="nadeseen/custom_runtime-3.10:0.2")
fexec.call_async(my_function, 3)
result = fexec.get_result()
print(result)

JosepSampe Jan 26, 2024
Maintainer

For some reason you runtime does not contain flask:

$ docker run docker.io/nadeseen/custom_runtime-3.10:0.2
Traceback (most recent call last):
  File "/pythonAction/pythonrunner.py", line 26, in <module>
    from actionproxy import ActionRunner, main, setRunner
  File "/pythonAction/../actionProxy/actionproxy.py", line 35, in <module>
    import flask
ModuleNotFoundError: No module named 'flask'

To build a custom runtime you can use either this Dockerfile or the Dockerfile.slim

Answer selected by nadeshseen

nadeshseen Jan 27, 2024
Author

thanks @JosepSampe. somehow I missed importing mandatory packages mentioned here requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception(runtime_meta) in custom runtime on Openwhisk #1229

{{title}}

Replies: 1 comment 12 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Exception(runtime_meta) in custom runtime on Openwhisk #1229

nadeshseen Jan 15, 2024

Replies: 1 comment · 12 replies

JosepSampe Jan 16, 2024 Maintainer

nadeshseen Jan 24, 2024 Author

JosepSampe Jan 24, 2024 Maintainer

nadeshseen Jan 25, 2024 Author

JosepSampe Jan 26, 2024 Maintainer

nadeshseen Jan 27, 2024 Author

nadeshseen
Jan 15, 2024

Replies: 1 comment 12 replies

JosepSampe
Jan 16, 2024
Maintainer

nadeshseen Jan 24, 2024
Author

JosepSampe Jan 24, 2024
Maintainer

nadeshseen Jan 25, 2024
Author

JosepSampe Jan 26, 2024
Maintainer

nadeshseen Jan 27, 2024
Author