Extism Python PDK

This project contains a tool that can be used to create Extism Plug-ins in Python.

Overview

This PDK uses PyO3 and wizer to run Python code as an Extism Plug-in.

Install the compiler

We release the compiler as native binaries you can download and run. Check the releases page for the latest.

Install Script

Linux, macOS

curl -Ls https://raw.githubusercontent.com/extism/python-pdk/main/install.sh | bash

This will install extism-py (and wasm-merge/wasm-opt if not already installed) to $HOME/.local/bin and create $HOME/.local/share/extism-py

Testing the Install

Note: Binaryen, specifically the wasm-merge and wasm-opt tools are required as a dependency. We will try to package this up eventually but for now it must be reachable on your machine. You can install on mac with brew install binaryen or see their releases page.

Then run command with no args to see the help:

extism-py
error: The following required arguments were not provided:
    <input-py>

USAGE:
    extism-py <input-py> -o <output>

For more information try --help

Note: If you are using mac, you may need to tell your security system this unsigned binary is fine. If you think this is dangerous, or can't get it to work, see the "compile from source" section below.

Getting Started

The goal of writing an Extism plug-in is to compile your Python code to a Wasm module with exported functions that the host application can invoke. The first thing you should understand is creating an export.

Exports

Let's write a simple program that exports a greet function which will take a name as a string and return a greeting string. Paste this into a file plugin.py:

import extism

@extism.plugin_fn
def greet():
  name = extism.input_str()
  extism.output_str(f"Hello, {name}")

Some things to note about this code:

1. We can export functions by name using the extism.plugin_fn decorator. This allows the host to invoke this function.
2. In this PDK we code directly to the ABI. We get input from the using using extism.input* functions and we return data back with the extism.output* functions.

Let's compile this to Wasm now using the extism-py tool:

extism-py plugin.py -o plugin.wasm

We can now test plugin.wasm using the Extism CLI's run command:

extism call plugin.wasm greet --input="Benjamin" --wasi
# => Hello, Benjamin!

Note: Currently wasi must be provided for all Python plug-ins even if they don't need system access.

Note: We also have a web-based, plug-in tester called the Extism Playground

More Exports: Error Handling

We catch any exceptions thrown and return them as errors to the host. Suppose we want to re-write our greeting module to never greet Benjamins:

import extism

@extism.plugin_fn
def greet():
  name = extism.input_str()
  if name == "Benjamin":
    raise Exception("Sorry, we don't greet Benjamins!")
  extism.output_str(f"Hello, {name}")

Now compile and run:

extism-py plugin.py -o plugin.wasm
extism call plugin.wasm greet --input="Benjamin" --wasi
# => Error: Sorry, we don't greet Benjamins!:
# =>  File "<source>", line 17, in __invoke
# =>  File "<source>", line 9, in greet
echo $? # print last status code
# => 1
extism call plugin.wasm greet --input="Zach" --wasi
# => Hello, Zach!
echo $?
# => 0

JSON

import extism

@extism.plugin_fn
def sum():
  params = extism.input_json()
  extism.output_json({"sum": params['a'] + params['b']})

extism call plugin.wasm sum --input='{"a": 20, "b": 21}' --wasi
# => {"sum":41}

You can also specify your input and output types as dataclasses using extism.Json:

from typing import Optional, List 
from dataclasses import dataclass

# ...

@dataclass
class User(extism.Json):
  admin: bool
  name: Optional[str]
  email: str
  addresses: List[Address]


@extism.plugin_fn
def reflect_user():
  input = extism.input(User)
  extism.output(input)

Configs

Configs are key-value pairs that can be passed in by the host when creating a plug-in. These can be useful to statically configure the plug-in with some data that exists across every function call. Here is a trivial example using Config.get:

import extism

@extism.plugin_fn
def greet():
  user = extism.Config.get("user")
  extism.output_str(f"Hello, {user}!")

To test it, the Extism CLI has a --config option that lets you pass in key=value pairs:

extism call plugin.wasm greet --config user=Benjamin --wasi
# => Hello, Benjamin!

Logging

At the current time, calling console.log emits an info log. Please file an issue or PR if you want to expose the raw logging interface:

import extism

@extism.plugin_fn
def log_stuff():
  extism.log(Extism.LogLevel.Info, "Hello, world!")

Running it, you need to pass a log-level flag:

extism call plugin.wasm logStuff --wasi --log-level=info
# => 2023/10/17 14:25:00 Hello, World!

Compiling the compiler from source

Prerequisites

Before compiling the compiler, you need to install prerequisites.

1. Install Rust using rustup
2. Install the WASI target platform via rustup target add --toolchain stable wasm32-wasi
3. Install CMake (on macOS with homebrew, brew install cmake)
4. Install Binaryen and add it's install location to your PATH (only wasm-opt is required for build process)
5. Install 7zip(only for Windows)

Compiling from source

Run make to compile the core crate (the engine) and the cli:

./build.py

To test the built compiler (ensure you have Extism installed):

./extism-py examples/count-vowels.py -o count-vowels.wasm
extism call out.wasm count_vowels --wasi --input='Hello World Test!'
# => "{\"count\":4}"

How it works

This works a little differently than other PDKs. You cannot compile Python to Wasm because it doesn't have an appropriate type system to do this. The extism-py command we have provided here is a little compiler / wrapper that does a series of things for you:

1. It loads an "engine" Wasm program containing the Python runtime
2. It initializes the Python runtime
3. It loads your Python source code into memory
4. It parses the Python source code for exports and generates 1-to-1 proxy export functions in Wasm
5. It freezes and emits the machine state as a new Wasm file at this post-initialized point in time

This new Wasm file can be used just like any other Extism plugin.