SeBS supports three commercial serverless platforms: AWS Lambda, Azure Functions, and Google Cloud Functions. Furthermore, we support the open source FaaS system OpenWhisk.
The file configs/example.json contains all parameters that users can change
to customize the deployment.
Some of these parameters, such as cloud credentials or storage instance address,
are required.
In the following subsections, we discuss the mandatory and optional customization
points for each platform.
Note
Most of the cloud providers offer free usage allowances and promotional credits, subject to account eligibility, service, plan, and region. Function execution and supporting services, such as storage and builds, are usually included with certain free allocations. Review the current terms before deploying benchmarks.
Warning
On many platforms, credentials can be provided as environment variables or through the SeBS configuration. SeBS will not store your credentials in the cache. When saving results, SeBS stores user benchmark and experiment configuration for documentation and reproducibility, except for credentials that are erased. If you provide the credentials through JSON input configuration, do not commit or publish these files anywhere.
Warning
Depending on the platform, SeBS might deploy resources that are publicly accessible, such as Lambda functions with function URLs that do not use authentication; details can be found in each platform's description. It is recommended to deallocate such resources after testing.
Supported platforms:
SeBS benchmarks rely on persistent object and NoSQL storage for input and output data. For configuration instructions regarding both object storage and NoSQL databases, please refer to the storage documentation. Storage configuration is particularly important for local deployments, OpenWhisk, and other open-source FaaS platforms.
By default, SeBS builds functions for the x64 (x86_64) architecture. On AWS, functions can also be built and deployed for ARM CPUs to benefit from Graviton CPUs available on Lambda.
This change primarily affects functions that make use of dependencies with native builds, such as torch, numpy or ffmpeg.
Such functions can be built as code packages on any platform, as we rely on package managers like pip and npm to provide binary dependencies. However, special care is needed to build Docker containers: since installation of packages is a part of the Docker build, we cannot natively execute binaries based on ARM containers on x86 CPUs. To build multi-platform images, we recommend following official Docker guidelines and providing a static QEMU installation. On Ubuntu-based distributions, this requires installing an OS package and executing a single Docker command to provide seamless emulation of ARM containers.
Build images, which encapsulate package building, are available as both x64 and arm64 for Python and Node.js on AWS Lambda.
The local deployment's build and run images for Python and Node.js are multi-platform too: local functions run on the host, so x86 and ARM machines (e.g., Apple Silicon, Graviton) pull the image of their own architecture.
Set the experiment's architecture to the host's (x64 or arm64) for local deployments; SeBS warns about a mismatch, since dependencies are installed for the configured architecture.
To rebuild multi-platform images, an additional flag is needed to enable the internal docker buildx command:
sebs docker build --image-type build --language python --deployment aws --language-version 3.11 --multi-platform
sebs docker build --deployment local --language python --language-version 3.11 --multi-platformA multi-platform build pushes the images for both architectures, and the combined manifest, to the repository. sebs docker push --multi-platform skips these images; without the flag, it pushes the locally built single-platform image.
When rebuilding build images (not necessary for regular users, only for developers), make sure that your Docker installation supports multi-platform images, e.g., you use containerd image store - old Docker installations might not change the storage type after an upgrade to Docker 29.0, where containerd is the default.
SeBS ensures that all locally cached cloud resources are valid by storing a unique identifier associated with each cloud account. Furthermore, we store this identifier in experiment results to easily match results with the cloud account or subscription that was used to obtain them. We use non-sensitive identifiers such as account IDs on AWS, subscription IDs on Azure, and Google Cloud project IDs.
If you have JSON result files, such as experiment.json from a benchmark run or '/*.json' from an experiment, you can remove all identifying information by removing the JSON object .config.deployment.credentials. This can be achieved easily with the CLI tool jq:
jq 'del(.config.deployment.credentials)' <file.json> | sponge <file.json>
AWS Lambda provides an ongoing monthly free allowance for Lambda requests and compute time. Supporting services, such as S3, have separate pricing and free allowances - see "always free" on AWS website. Additionally, new AWS accounts receive credits for additional usage.
To work with AWS, you need to provide access and secret keys to a role with permissions
sufficient to manage functions and S3 resources.
Additionally, the account must have AmazonAPIGatewayAdministrator permission to set up
automatically an AWS HTTP trigger.
You can provide a role
with permissions to access AWS Lambda, S3, and DynamoDB; otherwise, one will be created automatically.
The automatically created role sebs-lambda-role has the following managed policies attached:
AWSLambdaBasicExecutionRole (CloudWatch logs), AmazonS3FullAccess (benchmark input and output data),
and AmazonDynamoDBFullAccess (NoSQL storage used by benchmarks such as 130.crud-api).
To use a user-defined Lambda role, set its ARN in the config JSON - see an example in configs/example.json.
You can pass the credentials either using the default AWS-specific environment variables:
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=XXXX
or in the JSON input configuration:
"deployment": {
"name": "aws",
"aws": {
"region": "us-east-1",
"lambda-role": "",
"credentials": {
"access_key": "YOUR AWS ACCESS KEY",
"secret_key": "YOUR AWS SECRET KEY"
}
}
}SeBS supports two methods for HTTP-based function invocation on AWS Lambda:
- Lambda Function URLs (default) - Direct Lambda invocations.
- API Gateway HTTP API (optional) - Traditional approach using AWS API Gateway.
SeBS used API Gateway to trigger Lambda functions. However, API Gateway has a hard timeout limit of 29 seconds, which can be restrictive for long-running benchmarks. To overcome this limitation and simplify the architecture, we added support for Lambda Function URLs, which allow direct invocation of Lambda functions without the need for API Gateway. Since we do not rely on more complex API management features, function URLs are now the default version.
However, an API gateway can still be used for benchmarking. The switch between the two options is configured in the deployment settings:
"deployment": {
"name": "aws",
"aws": {
"region": "us-east-1",
"resources": {
"use-function-url": true,
"function-url-auth-type": "NONE"
}
}
}Warning
SeBS implements the "NONE" authentication mode for function URLs, making Lambda functions publicly accessible without any authentication.
To make a function URL public, SeBS adds two statements to the function's resource-based policy,
as required by AWS for function URLs created since October 2025:
lambda:InvokeFunctionUrl (statement FunctionURLAllowPublicAccess) and lambda:InvokeFunction
restricted to invocations through the function URL (statement FunctionURLAllowInvokeAccess).
Both statements are removed when the function URL is deleted.
These permissions are attached to the function, not to the Lambda execution role.
Azure provides a free tier for 12 months, with certain services providing free allocations beyond that. You need to create an account and add a service principal to enable non-interactive login through CLI. Since this process has an easy, one-step CLI solution, we added a small tool tools/create_azure_credentials that uses the interactive web-browser authentication to login into Azure CLI and create a service principal.
Please provide the intended principal name
XXXXX
Please follow the login instructions to generate credentials...
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code YYYYYYY to authenticate.
Login successful with user {'name': 'ZZZZZZ', 'type': 'user'}
Created service principal http://XXXXX
AZURE_SECRET_APPLICATION_ID = XXXXXXXXXXXXXXXX
AZURE_SECRET_TENANT = XXXXXXXXXXXX
AZURE_SECRET_PASSWORD = XXXXXXXXXXXXXSave these credentials - the password is non-retrievable! Provide them to SeBS, and we will create additional resources (storage account, resource group) to deploy functions. We will create a storage account and the resource group and handle access keys.
You can pass the credentials either using the environment variables:
export AZURE_SECRET_APPLICATION_ID=XXXXXXXXXXXXXXXX
export AZURE_SECRET_TENANT=XXXXXXXXXXXX
export AZURE_SECRET_PASSWORD=XXXXXXXXXXXXX
or in the JSON input configuration:
"deployment": {
"name": "azure",
"azure": {
"region": "westeurope",
"credentials": {
"appID": "YOUR SECRET APPLICATION ID",
"tenant": "YOUR SECRET TENANT",
"password": "YOUR SECRET PASSWORD"
}
}
}Warning
The tool assumes there is only one subscription active on the account. If you want to bind the newly created service principal to a specific subscription, or the created credentials do not work with SeBS and you see errors such as "No subscriptions found for X", then you must specify a subscription when creating the service principal. Check your subscription ID on in the Azure portal, and use the CLI option tools/create_azure_credentials.py --subscription <SUBSCRIPTION_ID>.
Warning
Sometimes there's a delay within Azure platform that causes properties like subscription assignment to not be propagated immediately across systems. If you keep seeing errors such "No subscription found", then wait for a few minutes before trying again.
Warning
When you log in for the first time on a device, Microsoft might require authenticating your login with Multi-Factor Authentication (MFA). In this case, we will return an error such as: "The following tenants require Multi-Factor Authentication (MFA). Use 'az login --tenant TENANT_ID' to explicitly login to a tenant.". Then, you can pass the tenant ID by using the --tenant <tenant-id> flag.
- By default, all functions are allocated in the single resource group.
- Each function has a separate storage account allocated, following Azure guidelines.
- All benchmark data is stored in the same storage account.
The Google Cloud Free Tier gives free access to certain cloud resources. It has two parts:
- A free trial for new customers with credit to use with eligible Google Cloud services (time-limited).
- A free tier, which provides limited access to many common Google Cloud resources, free of charge.
A valid Cloud Billing account is required, including for Free Tier usage. Function execution, builds, artifact storage, and other supporting services have their own pricing and free allowances.
You need to create an account and add service account to permit operating on storage and functions. From the Google Cloud Console, download the cloud credentials saved as a JSON file.
You should have at least write access to Cloud Functions (Cloud Functions Admin) and Logging. Furthermore, SeBS needs the permissions to create Firestore databases through
Google Cloud CLI tool; the Firestore Service Agent role allows for that.
You can pass the credentials either using the default GCP-specific environment variable:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/project-credentials.json
using the SeBS environment variable:
export GCP_SECRET_APPLICATION_CREDENTIALS=/path/to/project-credentials.json
or in the JSON input configuration:
"deployment": {
"name": "gcp",
"gcp": {
"region": "europe-west1",
"project_name": "your-gcp-project-id",
"credentials": "/path/to/project-credentials.json"
}
}SeBS models three GCP deployment targets:
function-gen1: the first Google Cloud Functions Gen1 path.function-gen2: Google Cloud Functions Gen2 package deployment.container: direct container deployment to Cloud Run. These deployment types intentionally share a single GCP backend in SeBS, but they are not identical in packaging, naming, scaling, or performance behavior.
On GCP, there are two different concurrency layers that should not be confused:
- platform concurrency: how many requests GCP may send to one instance (
gcp-concurrency) - runtime concurrency: how many requests the language server is prepared to process internally (
worker-concurrency,worker-threads) This design is intentional. A single Cloud Run concurrency number is not enough to reason about performance if the application server is underprovisioned or overprovisioned relative to the platform.
Warning
SeBS implements the "allUsers" permission mode for invocations, making GCP functions and containers publicly accessible without any authentication.
Gen1 is the currently implemented Google-managed function deployment path in SeBS. The packaging flow is ZIP-based:
- benchmark sources are moved into a
function/subdirectory, - the language wrapper file is renamed to the GCP-required entrypoint name (
main.pyfor Python,index.jsfor Node.js), - the whole directory is archived and uploaded to Cloud Storage,
- Cloud Functions Gen1 is then updated from the uploaded archive.
Gen1 configuration currently exposes instance-scaling controls:
"deployment": {
"name": "gcp",
"gcp": {
"region": "europe-west1",
"project_name": "your-gcp-project-id",
"credentials": "/path/to/project-credentials.json",
"configuration": {
"function-gen1": {
"min-instances": 0,
"max-instances": 20
}
}
}
}Use Gen1 when you want the most established GCP path in SeBS and do not need container-level runtime tuning.
Gen2 reuses the same local ZIP packaging flow as Gen1, but deploys through the Cloud Functions v2 API. It is selected directly through the experiment-level system_variant:
"deployment": {
"name": "gcp",
"gcp": {
"region": "europe-west1",
"project_name": "your-gcp-project-id",
"credentials": "/path/to/project-credentials.json",
"configuration": {
"function-gen2": {
"vcpus": 1,
"gcp-concurrency": 80,
"worker-concurrency": 1,
"worker-threads": 8,
"min-instances": 0,
"max-instances": 20,
"cpu-boost": false,
"cpu-throttle": true
}
}
}
}Set experiments.system_variant to one of function-gen1, function-gen2, or container. From the CLI and regression workflows, the same selection is exposed through the generic --system-variant option.
Gen1 and Gen2 package deployments use separate SeBS cache identities and separate cloud function names with short -gen1 and -gen2 suffixes. This avoids control-plane races when switching between package modes. In practice, regression and benchmark runs should still select one GCP package mode at a time for a given run.
Container deployments are the currently implemented Cloud Run-based path in SeBS. They are selected with container deployment and use a provider-specific function image built from Dockerfile.function.
At the deployment level, SeBS configures Cloud Run service properties:
"deployment": {
"name": "gcp",
"gcp": {
"region": "europe-west1",
"project_name": "your-gcp-project-id",
"credentials": "/path/to/project-credentials.json",
"configuration": {
"container": {
"environment": "gen2",
"vcpus": 1,
"gcp-concurrency": 80,
"worker-concurrency": 80,
"worker-threads": 8,
"min-instances": 0,
"max-instances": 20,
"cpu-boost": false,
"cpu-throttle": true
}
}
}
}For Python, SeBS uses functions-framework behind gunicorn rather than the framework's default development server, as recommended by Cloud Run performance guidance.
For Node.js, SeBS uses @google-cloud/functions-framework started directly with node rather than via npm start, as recommended by Cloud Run performance guidance.
For Java, we use java-function-invoker, and we disable tiered compilation to speed up startup time, as recommended by Cloud Run performance guidance.
Cloud Run containers can execute in two environments: gVisor-based gen1, and VM-based gen2.
The current GCP backend has the following practical limits:
- Gen1 is the primary managed-functions deployment path today.
- Gen2 supports the ZIP package deployment path and HTTP triggers.
- Library-trigger direct invocation remains Gen1-only.
- Cloud Run containers are implemented today and provide the most tuning control.
- GCP deployments currently reject
arm64, as arm64 instances are not available for GCR. - C++ packaging is not supported on GCP (but possible to be implemented on containers).
SeBS expects users to deploy and configure an OpenWhisk instance.
Below, you will find an example of instructions for deploying an OpenWhisk instance.
The configuration parameters of OpenWhisk for SeBS can be found
in configs/example.json under the key ['deployment']['openwhisk'].
In the subsections below, we discuss the meaning and use of each parameter.
To correctly deploy SeBS functions to OpenWhisk, following the
subsections on Toolchain and Docker configuration is particularly important.
For storage configuration in OpenWhisk, refer to the storage documentation, which covers both object storage and NoSQL requirements specific to OpenWhisk deployments.
Warning
Some benchmarks might require larger memory allocations, e.g., 2048 MB. Not all OpenWhisk deployments support this out-of-the-box. The deployment section below shows an example of changing the default function memory limit from 512 MB to a higher value.
In tools/openwhisk_preparation.py, we include scripts that help install
kind (Kubernetes in Docker) and deploy
OpenWhisk on a kind cluster. Alternatively, you can deploy to an existing
cluster by using offical deployment instructions:
./deploy/kind/start-kind.sh
helm install owdev ./helm/openwhisk -n openwhisk --create-namespace -f deploy/kind/mycluster.yaml
kubectl get pods -n openwhisk --watchTo change the maximum memory allocation per function, edit the max value under memory in file helm/openwhisk/values.yaml.
To run all benchmarks, we recommend at least "2048m".
We use OpenWhisk's CLI tool wsk
to manage the deployment of functions to OpenWhisk.
Please install wskand configure it to point to your OpenWhisk installation.
By default, SeBS assumes that wsk is available in the PATH.
To override this, set the configuration option wskExec to the location
of your wsk executable.
If you are using a local deployment of OpenWhisk with a self-signed
certificate, you can skip certificate validation with the wsk flag --insecure.
To enable this option, set wskBypassSecurity to true.
At the moment, all functions are deployed as web actions
that do not require credentials to invoke functions.
Furthermore, SeBS can be configured to remove the kind
cluster after finishing experiments automatically.
The boolean option removeCluster helps to automate the experiments
that should be conducted on fresh instances of the system.
SeBS enforces cold starts by updating the action with a new value of the
annotation sebs_cold_start. The invoker reuses a warm container only
when the action is identical to the one it was created for, and every update
changes the action version. Deleting and recreating the action does not work,
as the new action is identical to the old one and old containers are reused.
Old containers are not removed by OpenWhisk until they are evicted or
reach the idle timeout (10 minutes by default). When OpenWhisk is deployed on
Kubernetes, SeBS deletes their pods with kubectl after each cold start
enforcement. Set removeStaleContainers to false to disable it. Use kubectlExec and
kubernetesNamespace to configure the kubectl executable and the namespace
of the OpenWhisk deployment (openwhisk by default). SeBS verifies the namespace
by looking for the invoker pod deployed by the Helm chart, and skips the removal
with a warning when it is not found. Never delete pods of actions without updating them:
OpenWhisk does not detect removed pods and an invocation routed to a removed
container fails after the function timeout.
In FaaS platforms, the function's code can usually be deployed as a code package or a Docker image with all dependencies preinstalled. However, OpenWhisk has a very low code package size limit of only 48 megabytes. So, to circumvent this limit, we deploy functions using pre-built Docker images.
Note
On Python and Node.js, we create a full Docker image and upload the main handler file only to OpenWhisk, as this is required for actions. This is not possible on Java, as we need to compile the code into a JAR. Thus, we build the function image, extract the function JAR, and upload it with the action. In the future, if we want to create heavy JARs with complex dependencies, we might need to switch to full image deployment on Java as well.
Important: OpenWhisk requires that all Docker images are available in the registry, even if they have been cached on a system serving OpenWhisk functions. Function invocations will fail when the image is not available after a timeout with an error message that does not directly indicate image availability issues. Therefore, all SeBS benchmark functions are available on Docker Hub.
When adding new functions and extending existing functions with new languages
and new language versions, Docker images must be placed in the registry.
However, pushing the image to the default spcleth/serverless-benchmarks
repository on Docker Hub requires permissions.
To use a different Docker Hub repository, change the key
['general']['docker_repository'] in configs/systems.json.
Alternatively, OpenWhisk users can configure the FaaS platform to use a custom and
private Docker registry and push new images there.
A local Docker registry can speed up development when debugging a new function.
SeBS can use an alternative Docker registry - see dockerRegistry settings
in the example to configure the registry endpoint and credentials.
When the registry URL is not provided, SeBS will use Docker Hub.
When username and password are provided, SeBS will log in to the repository
and push new images before invoking functions.
See the documentation on the
Docker registry
and OpenWhisk configuration
for details.
Warning
This feature is experimental and has not been tested extensively. At the moment, it cannot be used on a kind cluster due to issues with Docker authorization on invoker nodes. See the OpenWhisk issue for details.
SeBS builds and deploys a new code package when constructing the local cache, when the function's contents have changed, and when the user requests a forced rebuild. In OpenWhisk, this setup is changed - SeBS will first attempt to verify if the image exists already in the registry and skip building the Docker image when possible. Then, SeBS can deploy seamlessly to OpenWhisk using default images available on Docker Hub. Furthermore, checking for image existence in the registry helps avoid failing invocations in OpenWhisk. For performance reasons, this check is performed only once when initializing the local cache for the first time.
When the function code is updated,
SeBS will build the image and push it to the registry.
Currently, the only available option of checking image existence in
the registry is pulling the image.
However, Docker's experimental manifest feature
allows checking image status without downloading its contents, saving bandwidth and time.
To use that feature in SeBS, set the experimentalManifest flag to true.
Start the storage with sebs storage start and pass the generated configuration to benchmark commands with --storage-configuration; see the storage documentation for details, including how to set the address advertised to functions with --external-address.
OpenWhisk has a shutdownStorage switch that controls the behavior of SeBS.
When set to true, SeBS will stop the object storage (RustFS or Minio) and ScyllaDB instances after finishing all work.