CommonWL

A Rust-framework for the Common Workflow Language (CWL) that supports parsing and execution.

Overview

commonwl is a Rust-based library crate for parsing and executing CWL documents (CommandLineTool, Workflow, ExpressionTool, Operation). It is based on the headless task execution framework crankshaft. It is developed for use in the Scientific Workflow Infrastructure (SciWIn).

Besides the library, this repo also ships:

• conformance — CLI used to evaluate CWL conformance against the official test suite.
• cwl-lsp — a language server providing diagnostics, symbols and formatting for CWL documents in editors. (🏗️ Under construction)

Getting Started

Installation

To use commonwl, you need to install and setup a Rust environment. Once Rust is installed, you can add the latest version of commonwl using the following command

cargo add commonwl

Parsing-only usage requires no extra features. To use the execution engine, enable the engine feature (add tes as well for the GA4GH Task Execution Service (TES) backend):

# enable execution engine feature
commonwl = { version = "0.9", features = ["engine"] } 
# enable the TES executor
commonwl = { version = "0.9", features = ["engine", "tes"] }

Usage

use commonwl::load_cwl_file;

let doc = load_cwl_file("workflow.cwl", true)?;
# Ok::<(), commonwl::Error>(())

CWL Engine

The CWL Engine features high conformance to the specification, passing all Tests for Workflow and ExpressionTool and nearly all tests for CommandLineTool. The conformance is dependent on the used TaskBackend. Currently there are the following existing and planned backends:

Backend	Status	Overall Conformance
Local	✔️
Docker*	✔️
TES	✔️
Slurm	🧾	-

✔️: Fully operational - 🏗️: Under Construction - 🧾: Planned

*=Uses Docker even if DockerRequirement is not specified.

Two tests fail due to f64 overflow outputting 1e42 instead of one with 42 zeros. serde_json stores all numbers as f64.

Storage Backends

File/Directory staging is storage-agnostic and works the same across all task backends:

Storage	Description
Local	Plain filesystem access.
S3	Any S3-compatible object store (used by the TES backend for remote input/output staging).
HTTP(S)	Read-only fetch of http:///https:// input locations.

Developing the TES Backend

The TES backend talks to a GA4GH Task Execution Service. To iterate on it locally you need a TES server plus S3-compatible storage. .dev/tes_env.sh spins both up (rustfs + Funnel):

.dev/tes_env.sh start    # start rustfs + funnel, wait until both are healthy
eval "$(.dev/tes_env.sh env)"   # export BACKEND=tes and the S3 credentials/endpoint
cargo build --release -p conformance
BACKEND=tes cwltest --test testdata/cwl/conformance_tests.yaml --tool target/release/conformance
.dev/tes_env.sh stop     # tear both down

Funnel has an upstream crash under concurrent load; .dev/tes_env.sh watchdog (run alongside a test run) restarts it automatically if that happens.

License

This work is dual-licensed under MIT and Apache 2.0. You can choose either one to use this work.

SPDX-License-Identifier: MIT OR Apache-2.0

See LICENSE-MIT and LICENSE-APACHE for the full license texts.

Funded by

DFG project number 501899475