From f47e7167ceb5aa15d48e974a9428de621963b1c0 Mon Sep 17 00:00:00 2001 From: Jeff Leach Date: Mon, 6 Jul 2026 17:12:48 -0500 Subject: [PATCH] Add MkDocs documentation site with GitHub Pages CI Stand up a full documentation site (MkDocs + Material theme) with 73 pages covering Getting Started, Installation, User Guide, Modeling, Administration, Developer/Adaptation, Deployment, and Reference. Content is migrated from in-repo sources and the CWS Confluence wiki, verified against 2.9.0 code, and sanitized of all JPL-internal data. Includes: - mkdocs.yml, requirements-docs.txt, docs/ (full nav tree) - .github/workflows/docs.yml (build + sanitize-lint + Javadoc + mike deploy) - tools/docs/sanitize-lint.sh (CI guard against JPL data leakage) - tools/docs/wiki-migration.md (repeatable import procedure) - De-hardcoded API version in SwaggerConfig and OpenApiController - Added maven-javadoc-plugin in a 'docs' profile (pom.xml) - Updated documentation.ftl, summary.ftl, README to point at the new site - Refreshed stale Camunda 7.20 links to 7.24 Co-Authored-By: Claude --- .github/workflows/docs.yml | 146 +++++++++++ .gitignore | 4 + README.md | 2 +- .../jpl/cws/controller/OpenApiController.java | 5 +- .../jpl/cws/controller/SwaggerConfig.java | 8 +- docs/administration/index.md | 27 ++ docs/administration/logs-history.md | 135 ++++++++++ docs/administration/monitoring.md | 214 ++++++++++++++++ docs/administration/security.md | 146 +++++++++++ docs/administration/users.md | 103 ++++++++ docs/administration/workers.md | 79 ++++++ docs/assets/custom.css | 37 +++ docs/deployment/aws.md | 93 +++++++ docs/deployment/docker.md | 171 +++++++++++++ docs/deployment/index.md | 39 +++ docs/deployment/upgrade.md | 116 +++++++++ docs/developer/adaptation.md | 45 ++++ docs/developer/architecture.md | 94 +++++++ docs/developer/contributing.md | 39 +++ docs/developer/custom-initiators.md | 95 +++++++ docs/developer/custom-tasks.md | 54 ++++ docs/developer/external-database.md | 65 +++++ docs/developer/index.md | 45 ++++ docs/developer/security-plugin.md | 44 ++++ docs/developer/task-types.md | 127 ++++++++++ docs/developer/web-integration.md | 81 ++++++ docs/getting-started/concepts.md | 74 ++++++ docs/getting-started/index.md | 46 ++++ docs/getting-started/lifecycle.md | 61 +++++ docs/getting-started/quickstart.md | 84 +++++++ docs/index.md | 95 +++++++ docs/install/building.md | 108 ++++++++ docs/install/certificates.md | 63 +++++ docs/install/configuration.md | 42 ++++ docs/install/database.md | 57 +++++ docs/install/elasticsearch.md | 41 +++ docs/install/index.md | 40 +++ docs/install/ldap.md | 47 ++++ docs/install/modeler.md | 46 ++++ docs/install/prerequisites.md | 91 +++++++ docs/install/requirements.md | 51 ++++ docs/install/running.md | 41 +++ docs/install/security-considerations.md | 49 ++++ docs/macros.py | 55 ++++ docs/modeling/best-practices.md | 91 +++++++ docs/modeling/dmn.md | 147 +++++++++++ docs/modeling/examples.md | 148 +++++++++++ docs/modeling/index.md | 26 ++ docs/modeling/parallel-subprocess.md | 129 ++++++++++ docs/modeling/script-recipes.md | 236 ++++++++++++++++++ docs/modeling/tips.md | 119 +++++++++ docs/reference/changelog.md | 6 + docs/reference/configuration.md | 121 +++++++++ docs/reference/external.md | 27 ++ docs/reference/index.md | 27 ++ docs/reference/javadoc.md | 30 +++ docs/reference/rest-api.md | 56 +++++ docs/user-guide/console/configuration.md | 16 ++ docs/user-guide/console/deployments.md | 35 +++ docs/user-guide/console/history.md | 25 ++ docs/user-guide/console/home-summary.md | 13 + docs/user-guide/console/index.md | 26 ++ docs/user-guide/console/initiators.md | 36 +++ docs/user-guide/console/logs.md | 44 ++++ docs/user-guide/console/processes.md | 41 +++ docs/user-guide/console/tasklist.md | 37 +++ docs/user-guide/console/workers.md | 32 +++ docs/user-guide/deploying.md | 38 +++ docs/user-guide/index.md | 28 +++ docs/user-guide/initiators/cron.md | 32 +++ docs/user-guide/initiators/file.md | 32 +++ docs/user-guide/initiators/index.md | 26 ++ .../initiators/internal-external.md | 40 +++ docs/user-guide/initiators/message-arrival.md | 31 +++ docs/user-guide/initiators/repeating-delay.md | 31 +++ docs/user-guide/launching/from-process.md | 29 +++ docs/user-guide/launching/index.md | 12 + docs/user-guide/launching/manual.md | 26 ++ docs/user-guide/launching/rest.md | 115 +++++++++ docs/user-guide/snippets.md | 35 +++ install/cws-ui/documentation.ftl | 11 +- install/cws-ui/summary.ftl | 2 +- mkdocs.yml | 205 +++++++++++++++ pom.xml | 32 +++ requirements-docs.txt | 7 + tools/docs/sanitize-lint.sh | 66 +++++ tools/docs/wiki-migration.md | 85 +++++++ 87 files changed, 5447 insertions(+), 9 deletions(-) create mode 100644 .github/workflows/docs.yml create mode 100644 docs/administration/index.md create mode 100644 docs/administration/logs-history.md create mode 100644 docs/administration/monitoring.md create mode 100644 docs/administration/security.md create mode 100644 docs/administration/users.md create mode 100644 docs/administration/workers.md create mode 100644 docs/assets/custom.css create mode 100644 docs/deployment/aws.md create mode 100644 docs/deployment/docker.md create mode 100644 docs/deployment/index.md create mode 100644 docs/deployment/upgrade.md create mode 100644 docs/developer/adaptation.md create mode 100644 docs/developer/architecture.md create mode 100644 docs/developer/contributing.md create mode 100644 docs/developer/custom-initiators.md create mode 100644 docs/developer/custom-tasks.md create mode 100644 docs/developer/external-database.md create mode 100644 docs/developer/index.md create mode 100644 docs/developer/security-plugin.md create mode 100644 docs/developer/task-types.md create mode 100644 docs/developer/web-integration.md create mode 100644 docs/getting-started/concepts.md create mode 100644 docs/getting-started/index.md create mode 100644 docs/getting-started/lifecycle.md create mode 100644 docs/getting-started/quickstart.md create mode 100644 docs/index.md create mode 100644 docs/install/building.md create mode 100644 docs/install/certificates.md create mode 100644 docs/install/configuration.md create mode 100644 docs/install/database.md create mode 100644 docs/install/elasticsearch.md create mode 100644 docs/install/index.md create mode 100644 docs/install/ldap.md create mode 100644 docs/install/modeler.md create mode 100644 docs/install/prerequisites.md create mode 100644 docs/install/requirements.md create mode 100644 docs/install/running.md create mode 100644 docs/install/security-considerations.md create mode 100644 docs/macros.py create mode 100644 docs/modeling/best-practices.md create mode 100644 docs/modeling/dmn.md create mode 100644 docs/modeling/examples.md create mode 100644 docs/modeling/index.md create mode 100644 docs/modeling/parallel-subprocess.md create mode 100644 docs/modeling/script-recipes.md create mode 100644 docs/modeling/tips.md create mode 100644 docs/reference/changelog.md create mode 100644 docs/reference/configuration.md create mode 100644 docs/reference/external.md create mode 100644 docs/reference/index.md create mode 100644 docs/reference/javadoc.md create mode 100644 docs/reference/rest-api.md create mode 100644 docs/user-guide/console/configuration.md create mode 100644 docs/user-guide/console/deployments.md create mode 100644 docs/user-guide/console/history.md create mode 100644 docs/user-guide/console/home-summary.md create mode 100644 docs/user-guide/console/index.md create mode 100644 docs/user-guide/console/initiators.md create mode 100644 docs/user-guide/console/logs.md create mode 100644 docs/user-guide/console/processes.md create mode 100644 docs/user-guide/console/tasklist.md create mode 100644 docs/user-guide/console/workers.md create mode 100644 docs/user-guide/deploying.md create mode 100644 docs/user-guide/index.md create mode 100644 docs/user-guide/initiators/cron.md create mode 100644 docs/user-guide/initiators/file.md create mode 100644 docs/user-guide/initiators/index.md create mode 100644 docs/user-guide/initiators/internal-external.md create mode 100644 docs/user-guide/initiators/message-arrival.md create mode 100644 docs/user-guide/initiators/repeating-delay.md create mode 100644 docs/user-guide/launching/from-process.md create mode 100644 docs/user-guide/launching/index.md create mode 100644 docs/user-guide/launching/manual.md create mode 100644 docs/user-guide/launching/rest.md create mode 100644 docs/user-guide/snippets.md create mode 100644 mkdocs.yml create mode 100644 requirements-docs.txt create mode 100755 tools/docs/sanitize-lint.sh create mode 100644 tools/docs/wiki-migration.md diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 00000000..7dc3b10b --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,146 @@ +name: CWS Docs Site + +# Builds the MkDocs documentation site, guards it against JPL-internal content, +# generates the aggregated Javadoc, and publishes versioned docs to GitHub +# Pages (the gh-pages branch) via mike. +# +# One-time setup in repo Settings → Pages: set the source to the "gh-pages" +# branch, root folder. The first deploy (push to develop or a manual run) +# creates that branch. + +on: + push: + branches: [main, develop] + paths: + - 'docs/**' + - 'mkdocs.yml' + - 'requirements-docs.txt' + - 'tools/docs/**' + - '.github/workflows/docs.yml' + pull_request: + paths: + - 'docs/**' + - 'mkdocs.yml' + - 'requirements-docs.txt' + - 'tools/docs/**' + - '.github/workflows/docs.yml' + release: + types: [published] + workflow_dispatch: + +permissions: + contents: write + +# Never run two deploys against gh-pages at once. +concurrency: + group: docs-deploy + cancel-in-progress: false + +jobs: + build-and-deploy: + runs-on: ubuntu-latest + steps: + - name: Check out the repo + uses: actions/checkout@v6 + with: + fetch-depth: 0 # mike needs full history of the gh-pages branch + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Install docs dependencies + run: pip install -r requirements-docs.txt + + - name: Guard against JPL-internal content + run: bash tools/docs/sanitize-lint.sh docs + + - name: Set up JDK 17 + uses: actions/setup-java@v5 + with: + java-version: '17' + distribution: 'temurin' + + # --- Maven credentials for Camunda Enterprise Edition dependencies --- + # Mirrors the CWS CI workflow so the Javadoc aggregate build can resolve + # EE artifacts. On fork PRs these secrets are absent; the Javadoc step is + # allowed to fail and the docs still build/deploy without it. + - name: Configure Maven security + run: | + mkdir -p ~/.m2 + echo " + ${{ secrets.MAVEN_MASTER_PASSWORD }} + " > ~/.m2/settings-security.xml + + - name: Configure Maven settings + run: | + echo " + + + ${{ secrets.MAVEN_REPO_ID }} + ${{ secrets.MAVEN_USERNAME }} + ${{ secrets.MAVEN_ENCRYPTED_PASSWORD }} + + + " > ~/.m2/settings.xml + + - name: Configure Camunda Download Credentials + run: | + echo "machine downloads.camunda.cloud + login ${{ secrets.CAMUNDA_DOWNLOAD_LOGIN }} + password ${{ secrets.CAMUNDA_DOWNLOAD_PASSWORD }}" > ~/.netrc + chmod 400 ~/.netrc + + - name: Cache Maven packages + uses: actions/cache@v5 + with: + path: ~/.m2/repository + key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }} + restore-keys: | + ${{ runner.os }}-m2- + + - name: Generate aggregated Javadoc + id: javadoc + continue-on-error: true + run: mvn -q -P core,docs -DskipTests javadoc:aggregate + + - name: Note if Javadoc was skipped + if: steps.javadoc.outcome != 'success' + run: echo "::warning::Javadoc generation failed or was skipped; the site will publish without the /javadoc reference." + + # Validate on every event (incl. PRs) — fails on broken links, bad nav, + # or macro errors. The on_post_build hook folds Javadoc into the site. + - name: Validate site (strict build) + run: mkdocs build --strict + + - name: Configure git identity + if: github.event_name != 'pull_request' + run: | + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + + - name: Deploy versioned docs (mike) + if: github.event_name != 'pull_request' + run: | + set -euo pipefail + git fetch origin gh-pages --depth=1 || true + + if [ "${{ github.event_name }}" = "release" ]; then + RAW="${{ github.event.release.tag_name }}" + VERSION="${RAW#v}" + export CWS_VERSION="$VERSION" + echo "Deploying release version $VERSION (alias: latest)" + mike deploy --push --update-aliases "$VERSION" latest + mike set-default --push latest + else + # Push to develop/main or a manual run: publish the moving "dev" docs. + export CWS_VERSION="$(grep -E "CWS_VER=" utils.sh | head -1 | sed -E "s/.*CWS_VER=['\"]?([^'\"# ]+).*/\1/")" + echo "Deploying dev docs (display version ${CWS_VERSION})" + mike deploy --push dev + # Until the first release publishes a "latest" alias, point the site + # root at dev so it resolves. + if ! mike list 2>/dev/null | grep -q 'latest'; then + mike set-default --push dev + fi + fi diff --git a/.gitignore b/.gitignore index 44bb1084..671b8648 100644 --- a/.gitignore +++ b/.gitignore @@ -38,3 +38,7 @@ install/logging/logstash-*.zip /install/camunda-distro-zips/ *.cnf + +# Documentation site (MkDocs) +/site/ +.venv/ diff --git a/README.md b/README.md index d1dc05fc..b4147ad9 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ While this repository is mostly complete, the documentation will be a work-in-pr While documentation is still in the works, please feel free to [open an issue](https://github.com/NASA-AMMOS/commoan-workflow-service/issues/new/choose) with your inquiry. -See the [wiki](https://github.com/NASA-AMMOS/common-workflow-service/wiki) for more information. +See the [CWS documentation site](https://nasa-ammos.github.io/common-workflow-service/) for the full user guide, installation guide, and reference documentation. # Installation diff --git a/cws-service/src/main/java/jpl/cws/controller/OpenApiController.java b/cws-service/src/main/java/jpl/cws/controller/OpenApiController.java index 4e53ef2f..f9ea5c6b 100644 --- a/cws-service/src/main/java/jpl/cws/controller/OpenApiController.java +++ b/cws-service/src/main/java/jpl/cws/controller/OpenApiController.java @@ -44,6 +44,9 @@ public class OpenApiController { @Autowired private ApplicationContext applicationContext; + @org.springframework.beans.factory.annotation.Value("${cws.version}") + private String cwsVersion; + @GetMapping(value = "/api-docs", produces = MediaType.APPLICATION_JSON_VALUE) @ResponseBody public String getApiDocs() { @@ -71,7 +74,7 @@ private OpenAPI createOpenApiSpec() { .info(new Info() .title("CWS API") .description("Documentation of the endpoints used by CWS. Once authenticated, requests can be made to these endpoints.\nTo authenticate, right click on this page --> Inspect --> Click the 'Application' tab --> Select the URL under the Cookies tab on the left --> Copy the value of the cwsToken cookie.") - .version("2.9.0") // Update this each CWS release + .version(cwsVersion) .license(new License() .name("Apache 2.0") .url("https://github.com/NASA-AMMOS/common-workflow-service?tab=Apache-2.0-1-ov-file"))) diff --git a/cws-service/src/main/java/jpl/cws/controller/SwaggerConfig.java b/cws-service/src/main/java/jpl/cws/controller/SwaggerConfig.java index c3e252b3..496d448e 100644 --- a/cws-service/src/main/java/jpl/cws/controller/SwaggerConfig.java +++ b/cws-service/src/main/java/jpl/cws/controller/SwaggerConfig.java @@ -1,5 +1,6 @@ package jpl.cws.controller; +import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import io.swagger.v3.oas.models.OpenAPI; @@ -11,14 +12,17 @@ @Configuration public class SwaggerConfig { - + + @Value("${cws.version}") + private String cwsVersion; + @Bean public OpenAPI api() { return new OpenAPI() .info(new Info() .title("CWS API") .description("Documentation of the endpoints used by CWS. Once authenticated, requests can be made to these endpoints.\nTo authenticate, right click on this page --> Inspect --> Click the 'Application' tab --> Select the URL under the Cookies tab on the left --> Copy the value of the cwsToken cookie.") - .version("2.9.0") // update this each CWS release + .version(cwsVersion) .license(new License() .name("Apache 2.0") .url("https://github.com/NASA-AMMOS/common-workflow-service?tab=Apache-2.0-1-ov-file"))) diff --git a/docs/administration/index.md b/docs/administration/index.md new file mode 100644 index 00000000..bb21b6fc --- /dev/null +++ b/docs/administration/index.md @@ -0,0 +1,27 @@ +# Administration + +CWS administration covers the day-to-day tasks needed to keep a deployment healthy: managing users and security, configuring workers, reviewing logs, and monitoring system resources. + +## Topics + +| Page | What it covers | +|------|---------------| +| [Security & Roles](security.md) | LDAP integration, authentication schemes, CWS token auth, security filters | +| [User Administration](users.md) | Adding users, groups, and Camunda permissions | +| [Worker Management](workers.md) | Viewing workers, executor threads, enabling process definitions per worker | +| [Log & History Management](logs-history.md) | Log page usage, filtering, history retention, Elasticsearch index management | +| [Resource Monitoring](monitoring.md) | Grafana and Prometheus setup for CWS infrastructure monitoring | + +## Quick reference + +**Accessing admin pages** + +The CWS web console provides direct links to the Camunda Cockpit, TaskList, and Admin pages in the top-right navigation bar. These are Camunda's own management UIs, seamlessly integrated into the CWS interface. + +**Roles overview** + +CWS uses Camunda's built-in authorization model. Users belong to groups; groups receive authorizations on applications, process definitions, and tasks. LDAP mode syncs users and groups from your directory service automatically. + +**Default admin credential** + +In Camunda auth mode, the default password is `changeme`. Change it immediately after installation. diff --git a/docs/administration/logs-history.md b/docs/administration/logs-history.md new file mode 100644 index 00000000..98cd830e --- /dev/null +++ b/docs/administration/logs-history.md @@ -0,0 +1,135 @@ +# Log & History Management + +CWS provides centralized logging through the **Logs** page in the web console, backed by Elasticsearch. History retention is controlled by two configuration properties that govern how long process instance data and log records are kept. + +## Logs Page + +Navigate to **Logs** in the CWS console sidebar. This page aggregates log messages from the console host and all connected workers in one view. + +### Filtering + +The Logs page offers several filters to narrow results: + +| Filter | Description | +|--------|-------------| +| **Process Definitions** | Show logs for one or more specific process definitions | +| **Log Sources** | Limit to Console logs, Worker logs, or both | +| **Process Instances** | Enter a specific process instance ID to see its full log trail. You can also navigate here from the Processes page with an instance pre-selected. | +| **Log Level** | Show only DEBUG, INFO, WARN, ERROR, or combinations | +| **Search by Keyword** | Free-text search within log message content | +| **Start Date / End Date** | Restrict results to a time range | + +### Additional columns + +Click the column selector to enable optional columns: + +| Column | Shows | +|--------|-------| +| **CWS Host** | IP or hostname of the worker that produced the message | +| **CWS Host ID** | Internal worker ID for the message source | +| **Thread Name** | Thread on which the logged activity ran | +| **Process Definition Key** | The key of the relevant process definition | +| **Instance ID** | The process instance ID for the logged event | + +### Useful filter combinations + +**Debugging a failed run:** Select the process definition, set Log Level to WARN + ERROR, enable the Log Level column. This surfaces only problematic messages for that definition. + +**Tracing a specific instance:** Enter the instance ID in the Process Instances field, select All log levels, enable the Instance ID and Thread Name columns. + +**Comparing worker output:** Enable the CWS Host and CWS Host ID columns to see which worker handled which tasks. + +## Camunda Cockpit + +For live process monitoring (as opposed to log text), use the **Cockpit** page accessible from the top-right navigation. Cockpit provides: + +- A list of all deployed definitions with running instance counts +- A graphical BPMN diagram view with instance heatmaps (blue circles showing where tokens are waiting) +- Version history — view instances that are still on older deployed versions +- Filtering by Business Key, start date, and process variables +- Per-instance drill-down: variables, incidents, audit trail, user task assignments + +## History Retention Configuration + +CWS stores process history in both the database and Elasticsearch. Two properties control how much is kept. + +### `history_level` + +Set in `cws-configuration.properties` (must be the same value on the console and all workers). + +| Level | What is stored | +|-------|---------------| +| `none` | Nothing | +| `activity` | Process instance start/end, activity instances | +| `audit` | Adds variable updates | +| `full` | Adds form properties and user operation log entries | + +```properties +history_level=full +``` + +The default and recommended value for most deployments is `full`. Lowering this reduces database growth but limits what Cockpit can display for completed instances. + +Reference: [Camunda 7.24 History](https://docs.camunda.org/manual/7.24/user-guide/process-engine/history/) + +### `history_days_to_live` + +Controls how many days of history data is retained before automatic cleanup. This applies to: + +- Process instance history in the database +- Log entries +- Elasticsearch indices + +```properties +history_days_to_live=7 +``` + +CWS runs a scheduled cleanup job that removes data older than this threshold. Increase this value if you need longer audit trails; decrease it to manage storage on high-volume deployments. + +!!! note + Individual process definitions can also set their own `camunda:historyTimeToLive` attribute in the BPMN XML, which overrides the global default for that definition. + +### Per-definition history TTL + +In the BPMN modeler, set the **History Time To Live** field on the process properties panel. In the XML this appears as: + +```xml + +``` + +This value is in days. Setting it to `0` disables history retention for that definition. + +## Elasticsearch Index Management + +CWS writes log messages to Elasticsearch using date-based indices. The index prefix is configured via: + +```properties +elasticsearch_index_prefix=cws +``` + +Indices take the form `-YYYY.MM.DD`. Old indices are pruned automatically when their age exceeds `history_days_to_live`. + +### Authentication + +If your Elasticsearch cluster requires authentication: + +```properties +elasticsearch_use_auth=y +elasticsearch_username=your_username +elasticsearch_password=your_password +``` + +Set `elasticsearch_protocol=https` if your cluster uses TLS. + +### Checking index health + +Use Kibana or the Elasticsearch REST API to inspect index size and document counts: + +```bash +curl http://:9200/_cat/indices/cws-*?v +``` + +## Further reading + +- [Camunda 7.24 History](https://docs.camunda.org/manual/7.24/user-guide/process-engine/history/) +- [Camunda 7.24 History Cleanup](https://docs.camunda.org/manual/7.24/user-guide/process-engine/history/history-cleanup/) diff --git a/docs/administration/monitoring.md b/docs/administration/monitoring.md new file mode 100644 index 00000000..6fc294bc --- /dev/null +++ b/docs/administration/monitoring.md @@ -0,0 +1,214 @@ +# Resource Monitoring + +CWS deployments can optionally be monitored with [Grafana](https://grafana.com/) and [Prometheus](https://prometheus.io/). This page walks through the setup and explains how to build dashboards for CWS infrastructure. + +## Architecture Overview + +``` + ┌─────────────────┐ + ┌──────────────┐ │ Console Host │ + │ Worker Hosts │────▶│ Prometheus │◀─── Grafana (port 3000) + │ node_exporter│ │ (port 9090) │ + │ (port 9100) │ └────────┬────────┘ + └──────────────┘ │ + ┌──────────────┐ │ scrapes + │ Database Host│◀─────────────┘ + │ node_exporter│ + │ (port 9100) │ + │mysqld_exporter│ + │ (port 9104) │ + └──────────────┘ +``` + +Prometheus scrapes `node_exporter` (system metrics) from every host, and `mysqld_exporter` (database metrics) from the database host. Grafana queries Prometheus and renders dashboards. + +## Requirements + +- SSH access to each worker machine, the database host, and the console host +- Network connectivity: the Prometheus host must reach each worker on port 9100 and the database host on ports 9100 and 9104 +- Grafana must reach Prometheus on port 9090 +- A read-only MariaDB/MySQL user for `mysqld_exporter` + +## Setting up Prometheus Exporters + +### Database host + +Two exporters are needed: `node_exporter` for system metrics and `mysqld_exporter` for database metrics. + +First create a read-only database user: + +```sql +CREATE USER 'mysqld_exporter'@'localhost' + IDENTIFIED BY 'StrongPassword' + WITH MAX_USER_CONNECTIONS 3; +GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysqld_exporter'@'localhost'; +``` + +Then run this setup script on the database host (adjust paths and versions as needed): + +```bash +#!/bin/bash +mkdir -p ~/prometheus/exporters +cd ~/prometheus/exporters + +# mysqld_exporter +wget -q https://github.com/prometheus/mysqld_exporter/releases/download/v0.12.0/mysqld_exporter-0.12.0.linux-amd64.tar.gz +tar zxf mysqld_exporter-0.12.0.linux-amd64.tar.gz +cd mysqld_exporter-0.12.0.linux-amd64 + +cat > .my.cnf < mysqld_exporter.log 2>&1 & + +cd ~/prometheus/exporters + +# node_exporter +wget -q https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz +tar zxf node_exporter-0.18.1.linux-amd64.tar.gz +cd node_exporter-0.18.1.linux-amd64 +nohup ./node_exporter > node_exporter.log 2>&1 & +``` + +Replace `YOUR_PASSWORD_HERE` with the password you set above. + +### Worker hosts + +Run this script on each worker you want to monitor: + +```bash +#!/bin/bash +mkdir -p ~/prometheus/exporters +cd ~/prometheus/exporters + +wget -q https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz +tar zxf node_exporter-0.18.1.linux-amd64.tar.gz +cd node_exporter-0.18.1.linux-amd64 +nohup ./node_exporter > node_exporter.log 2>&1 & +``` + +## Setting up Prometheus + +Install Prometheus on the console host (or any host that can reach all exporters). + +Create `prometheus.yml`: + +```yaml +global: + scrape_interval: 5s + evaluation_interval: 5s + +scrape_configs: + - job_name: node + static_configs: + - labels: + alias: cws + targets: + - "db-host:9100" + - "worker1-host:9100" + - "worker2-host:9100" + + - job_name: mysql + static_configs: + - labels: + alias: cws + targets: + - "db-host:9104" +``` + +Replace the target hostnames with your actual hosts. + +Then start Prometheus: + +```bash +#!/bin/bash +mkdir -p ~/prometheus +cd ~/prometheus + +wget -q https://github.com/prometheus/prometheus/releases/download/v2.11.1/prometheus-2.11.1.linux-amd64.tar.gz +tar zxf prometheus-2.11.1.linux-amd64.tar.gz +cd prometheus-2.11.1.linux-amd64 + +nohup ./prometheus --config.file=/path/to/prometheus.yml > prometheus.log 2>&1 & +``` + +Verify targets are up at `http://:9090/targets`. + +### Troubleshooting exporters + +If a target shows as DOWN in Prometheus: + +```bash +# Check if node_exporter is running +ps -ax | grep node_exporter + +# Check if mysqld_exporter is running +ps -ax | grep mysqld_exporter +``` + +A missing process means the exporter failed to start. Check the log file in its directory for errors. + +## Setting up Grafana + +Install Grafana on the same host as Prometheus (simplest setup): + +```bash +#!/bin/bash +mkdir -p ~/grafana +cd ~/grafana + +wget -q https://dl.grafana.com/oss/release/grafana-6.2.5.linux-amd64.tar.gz +tar -zxf grafana-6.2.5.linux-amd64.tar.gz +cd grafana-6.2.5.linux-amd64 + +nohup ./bin/grafana-server web > grafana.log 2>&1 & +``` + +Grafana is now available at `http://localhost:3000/`. The default credentials are `admin` / `admin` — you will be prompted to change the password on first login. + +For newer Grafana releases see the [official download page](https://grafana.com/grafana/download). + +## Configuring Dashboards + +### Add data sources + +1. Open Grafana and go to **Configuration > Data Sources** +2. Add a **Prometheus** data source pointing to `http://localhost:9090` +3. Optionally add a **MySQL** data source pointed at your CWS database — this lets you query CWS tables directly in dashboards + +### Recommended dashboards + +**System metrics (node_exporter):** +Import dashboard ID `1860` (Node Exporter Full) from the [Grafana dashboard library](https://grafana.com/grafana/dashboards/1860). + +**MySQL metrics:** +Install the Percona plugin and enable it: + +```bash +# Run from Grafana install directory +./bin/grafana-cli plugins install percona-percona-app +``` + +Once enabled, Percona provides the **InnoDB Overview** and **MySQL Overview** dashboards. These require the MySQL data source to be named `CWS MySQL Database`. + +### Reverse proxy (optional) + +To serve Grafana under a path prefix (e.g. `/grafana`) rather than its own port, configure the `root_url` in `grafana.ini`: + +```ini +[server] +root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/ +serve_from_sub_path = true +``` + +Then configure your web server or load balancer to proxy `/grafana` to port 3000. + +## Further reading + +- [Prometheus documentation](https://prometheus.io/docs/) +- [Grafana documentation](https://grafana.com/docs/) +- [node_exporter releases](https://github.com/prometheus/node_exporter/releases) +- [mysqld_exporter releases](https://github.com/prometheus/mysqld_exporter/releases) diff --git a/docs/administration/security.md b/docs/administration/security.md new file mode 100644 index 00000000..0b62b9a6 --- /dev/null +++ b/docs/administration/security.md @@ -0,0 +1,146 @@ +# Security & Roles + +CWS supports multiple authentication schemes to accommodate different deployment environments. The scheme is selected at install time and affects how users authenticate to both the CWS UI and its REST API. + +## Authentication Schemes + +| Scheme | Description | Default password | +|--------|-------------|-----------------| +| **LDAP** | Users authenticate with their LDAP credentials. CWS issues its own session token (`cwsToken`). | Your LDAP password | +| **Camunda** | Users and passwords managed in the Camunda database. | `changeme` (must be changed immediately) | +| **Custom** | Project-defined authentication plugin. | Depends on implementation | + +!!! warning + The Camunda scheme ships with the default password `changeme`. Change it immediately after installation through the Camunda Admin page. + +## CWS Token Authentication (LDAP scheme) + +When running in LDAP mode, CWS issues a session token after successful authentication. This token is stored as the `cwsToken` cookie and can be used to authenticate REST API calls from scripts or other automation. + +### Generating a token + +Use the `refresh_cws_token.sh` script bundled with CWS: + +```bash +cd cws +./refresh_cws_token.sh +``` + +You will be prompted for your username and password. On success the script outputs: + +- `cws_token.txt` — the raw token value, suitable for use in BPMN processes making REST calls +- `cookies.txt` — a curl-compatible cookie file for subsequent requests + +Example output: + +``` +CWS authorization scheme is : LDAP +CWS token is : 'CCE85F56A8AF0637B58B201E0BB1CE6A' +cookie file is : cookies.txt +token file is : cws_token.txt +``` + +### Using the token in curl + +**Inline token:** + +```bash +curl https://:38443/cws-ui/rest/process/my_proc/schedule \ + -b "cwsToken=CCE85F56A8AF0637B58B201E0BB1CE6A" \ + --data "param1=value1" +``` + +**Cookie file:** + +```bash +curl https://:38443/cws-ui/rest/process/my_proc/schedule \ + -b "/path/to/cookies.txt" \ + --data "param1=value1" +``` + +## HTTP Basic Authentication + +All schemes support HTTP Basic Auth as a fallback. Pass credentials directly with curl: + +```bash +curl https://:38443/cws-ui/rest/process/my_proc/schedule \ + -u myusername:mypassword \ + --data "param1=value1" +``` + +For GET requests, omit the `--data` flag. + +## LDAP Identity Provider Plugin + +CWS can be configured to use the Camunda `LdapIdentityProviderPlugin`, which synchronizes users and groups from your LDAP directory into Camunda's identity model. + +### Configuration + +The plugin is configured in `$CWS_HOME/server/conf/bpm-platform.xml`. Uncomment the LDAP section and set the following properties: + +```xml + + org.camunda.bpm.identity.impl.ldap.plugin.LdapIdentityProviderPlugin + + ldaps://your-ldap-server:636 + false + dc=example,dc=com + + + ou=personnel + (objectclass=person) + uid + givenName + sn + mail + userpassword + + + ou=personnel + (|(cn=my-app-admin)(cn=my-app-users)) + cn + cn + uniqueMember + + +``` + +**Important:** The `groupSearchFilter` must restrict which groups are returned. Directories with hundreds of groups will cause performance problems and potential timeouts if the filter is too broad. + +### User search options + +**All users in the directory:** + +``` +userSearchFilter: (objectclass=person) +``` + +**Specific users only:** + +``` +userSearchFilter: (|(uid=alice)(uid=bob)(uid=carol)) +``` + +## Security Filters + +CWS applies security at multiple layers: + +1. **LDAP filter** — validates credentials against the directory when in LDAP mode +2. **Camunda security filter** — enforces Camunda's own authorization model on cockpit/tasklist/admin pages +3. **General web security filter** — covers the CWS REST API and UI endpoints + +These are configured through Spring Security and the Camunda process engine configuration. Refer to the adaptation guide if you need to customize filter behavior for your deployment. + +## SSL / TLS + +CWS requires SSL certificates to be installed before startup: + +- `install/.keystore` — server keystore +- `install/tomcat_lib/cws_truststore.jks` — trust store + +The keystore password is stored in `~/.cws/creds` with permissions set to `400`. See the installation guide for the certificate setup procedure. + +## Further reading + +- [Camunda 7.24 Identity / LDAP](https://docs.camunda.org/manual/7.24/user-guide/process-engine/identity-service/) +- [Camunda 7.24 Authorization](https://docs.camunda.org/manual/7.24/user-guide/process-engine/authorization-service/) diff --git a/docs/administration/users.md b/docs/administration/users.md new file mode 100644 index 00000000..88d50bb3 --- /dev/null +++ b/docs/administration/users.md @@ -0,0 +1,103 @@ +# User Administration + +User and permission management in CWS is handled through the **Camunda Admin** page, which is accessible from the top-right navigation bar in the CWS web console. + +## Accessing the Admin Page + +Click the **Admin** button in the top-right corner of the CWS console. This opens the Camunda Admin UI, which is CWS's primary control hub for user management. + +!!! note + In LDAP mode, users and groups are automatically synchronized from your directory service. You still manage *authorizations* (what each user/group can do) through the Admin page, but you do not create users manually — they exist in LDAP. + +## Users + +### Adding a user (Camunda auth mode) + +1. Navigate to **Admin > Users** +2. Click **Create new user** +3. Fill in the user ID, first name, last name, email, and initial password +4. Click **Create** + +The new user can log in immediately with the password you set. Prompt them to change it on first login. + +### Editing a user + +Select a user from the list to update their profile information or reset their password. + +### Removing a user + +Select the user and click **Delete**. This removes the user from Camunda's local identity store. In LDAP mode, users must be deactivated in LDAP itself. + +## Groups + +Groups allow batch permission assignment. A user inherits all authorizations granted to any group they belong to. + +### Creating a group + +1. Navigate to **Admin > Groups** +2. Click **Create new group** +3. Provide a group ID and name +4. Click **Create** + +### Adding members to a group + +1. Open the group +2. Click **Add member** +3. Search for and select the user + +In LDAP mode, group membership is controlled in LDAP. The Admin page will reflect the current LDAP state. + +## Authorizations + +Authorizations define what a user or group can do within CWS. Navigate to **Admin > Authorizations** to manage them. + +### Application authorizations + +Control which Camunda applications (Cockpit, Tasklist, Admin) a user or group can access. + +| Application | Purpose | +|-------------|---------| +| `cockpit` | View and manage running process instances | +| `tasklist` | Claim and complete user tasks | +| `admin` | Manage users, groups, and authorizations | + +To grant a group access to an application: + +1. Go to **Admin > Authorizations > Application** +2. Click **Create new authorization** +3. Set type to **Grant**, resource type to **Application**, and select the group +4. Enter the application name (e.g., `cockpit`) +5. Check the **Access** permission and save + +### Process definition authorizations + +Restrict which users or groups can read, create, or delete specific process definitions and their instances. + +Common permission patterns: + +| Permission | Allows | +|------------|--------| +| `READ` | View the process definition | +| `CREATE_INSTANCE` | Manually start an instance | +| `READ_HISTORY` | View past process instance data in Cockpit | +| `DELETE_HISTORY` | Remove historical instance records | + +To grant a group access to a specific process definition: + +1. Go to **Admin > Authorizations > Process Definition** +2. Create a new authorization for the group +3. Set the Resource ID to the process definition key (or `*` for all definitions) +4. Choose the required permissions + +### Task authorizations + +Control which users or groups can see and interact with user tasks within process instances. + +## Tenants + +CWS supports Camunda's multi-tenancy model. Tenants can isolate process definitions and instances between different teams or projects on a shared CWS instance. Configure tenants under **Admin > Tenants**. + +## Further reading + +- [Camunda 7.24 User Management](https://docs.camunda.org/manual/7.24/webapps/admin/user-management/) +- [Camunda 7.24 Authorization](https://docs.camunda.org/manual/7.24/user-guide/process-engine/authorization-service/) diff --git a/docs/administration/workers.md b/docs/administration/workers.md new file mode 100644 index 00000000..73e90799 --- /dev/null +++ b/docs/administration/workers.md @@ -0,0 +1,79 @@ +# Worker Management + +A CWS worker is a component capable of executing process instances using the BPMN 2.0 engine. You must have at least one worker running to execute processes. Large deployments can run many workers across multiple hosts. + +## Viewing Workers + +Navigate to the **Workers** tab in the CWS web console. The page lists all registered workers and their current status. + +For each worker you can see: + +- Worker ID and host +- Current status (active / inactive) +- **Configuration** — expand to see all settings this worker was started with +- **Process Definitions** — expand to see which process definitions this worker handles + +## Worker Configuration + +Expanding the **Configuration** column for a worker reveals its startup settings. The most important tunable from the UI is the number of **executor threads**. + +### Executor threads + +Executor threads control how many process instances a worker can execute concurrently. Increasing this value allows more parallel work but consumes more memory and CPU on the worker host. + +To change the executor thread count for a running worker, expand its configuration and update the value. The change takes effect without restarting the worker. + +!!! tip + Start conservatively (e.g. 4–8 threads) and increase based on observed CPU and memory utilization. Overcommitting threads on a memory-constrained host will degrade overall throughput. + +## Process Definitions per Worker + +Expanding the **Process Definitions** section for a worker shows every process definition registered to that worker. From this view you can: + +### Enable / Disable a process definition + +Toggle a process definition on or off for a given worker. Disabling a definition on a worker prevents that worker from picking up new instances of that definition. Instances already in progress will complete normally. + +This is useful for: +- Draining a worker before maintenance +- Routing specific process definitions to specific worker hosts +- Temporarily pausing execution of a definition without undeploying it + +### Set a concurrency limit + +The **Limit** field controls the maximum number of instances of a specific process definition that can run concurrently on this worker. This is separate from the overall executor thread count. + +For example, if a process definition is resource-intensive (heavy disk I/O or network), set its limit to a small value to prevent it from monopolizing the worker's threads. + +## Worker Types + +CWS workers can be started in different modes depending on what work they should perform: + +| Mode | Description | +|------|-------------| +| `run_all` | Handles all process types (default) | +| `run_models_only` | Executes only BPMN service tasks running in-process | +| `run_external_tasks_only` | Polls for and executes external tasks only | + +The mode is set at worker startup via the `worker_type` configuration property. + +## Adding a New Worker + +Workers are registered automatically when started. To add a worker: + +1. Provision a host with Java 17 and the CWS distribution +2. Create a configuration file pointing to the shared database and Elasticsearch cluster +3. Start the worker with `./dev.sh` or your deployment script + +The worker will appear in the Workers tab once it connects to the CWS console and registers itself. + +## Removing a Worker + +Stopping a worker process removes it from active rotation. The worker record remains visible in the UI until its registration expires. There is no explicit delete action required. + +Before stopping a worker, disable all its process definitions (see above) and wait for any in-progress instances to complete to avoid abrupt termination. + +## Further reading + +- [Camunda 7.24 External Tasks](https://docs.camunda.org/manual/7.24/user-guide/process-engine/external-tasks/) +- [CWS Configuration Reference](../reference/configuration.md) diff --git a/docs/assets/custom.css b/docs/assets/custom.css new file mode 100644 index 00000000..81fce7c2 --- /dev/null +++ b/docs/assets/custom.css @@ -0,0 +1,37 @@ +/* CWS documentation site — light custom styling on top of Material. */ + +:root { + --md-primary-fg-color: #2e3a87; + --md-primary-fg-color--light: #4a57b5; + --md-primary-fg-color--dark: #1d2765; + --md-accent-fg-color: #3d5afe; +} + +/* Give the hero landing cards a little breathing room. */ +.cws-cards { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(15rem, 1fr)); + gap: 1rem; + margin: 1.5rem 0; +} + +.cws-card { + border: 1px solid var(--md-default-fg-color--lightest); + border-radius: 0.5rem; + padding: 1rem 1.25rem; + transition: border-color 120ms, box-shadow 120ms; +} + +.cws-card:hover { + border-color: var(--md-accent-fg-color); + box-shadow: 0 2px 8px rgba(0, 0, 0, 0.08); +} + +.cws-card h3 { + margin-top: 0; +} + +/* Slightly wider content column for reference tables. */ +.md-grid { + max-width: 62rem; +} diff --git a/docs/deployment/aws.md b/docs/deployment/aws.md new file mode 100644 index 00000000..206a9453 --- /dev/null +++ b/docs/deployment/aws.md @@ -0,0 +1,93 @@ +# Deploying on AWS + +This page describes a general architecture for running CWS on AWS. It covers the required infrastructure components, auto-scaling workers, and CloudWatch integration. All account-specific values (VPC IDs, AMI IDs, subnet IDs, etc.) are your own to supply. + +## Required AWS components + +| Component | AWS service | Notes | +|-----------|-------------|-------| +| Networking | VPC + security groups | Console, database, and ES must be able to reach each other | +| Console instance | EC2 | Runs CWS with `install_type=2` | +| Worker instances | EC2 | Run CWS with `install_type=3`; scale independently | +| Database | RDS (MariaDB/MySQL) or EC2 | One instance shared by all CWS nodes | +| Elasticsearch | Self-managed EC2 or Amazon OpenSearch | One cluster shared by all CWS nodes | +| IAM | Role + instance profile | Required for CloudWatch metrics and optional S3 access | + +## Security group rules + +At minimum: + +- Console → database: port `3306` +- Console → Elasticsearch: port `9200` +- Workers → console: ports `38443` (HTTPS), `31616` (ActiveMQ) +- Workers → database: port `3306` +- Workers → Elasticsearch: port `9200` +- Inbound to console from users: ports `38080` (HTTP redirect) and `38443` (HTTPS) + +## IAM permissions + +The EC2 instance profile for CWS nodes needs: + +- `cloudwatch:PutMetricData` — required for the `queueMaxPendingDuration` metric used by auto-scaling +- `s3:GetObject` / `s3:PutObject` on your CWS S3 bucket — if using S3 storage for process artifacts + +## Infrastructure as code + +CWS includes Terraform scripts under the repository for automating the EC2 setup. Refer to the scripts in `install/` for the canonical resource definitions. Adapt them to your VPC layout, AMI IDs, and naming conventions. + +## Auto-scaling workers + +CWS publishes a CloudWatch custom metric — `queueMaxPendingDuration` — that represents the age of the oldest pending item in the worker queue. You can drive an EC2 Auto Scaling Group off this metric to scale workers up when work accumulates and down when the queue is idle. + +### Enable auto-scaling on the console + +Add to the console's `config.properties`: + +```properties +cws_enable_cloud_autoscaling=y +``` + +### Enable auto-registration on workers + +Workers that join the cluster through auto-scaling must register their process definitions automatically: + +```properties +startup_autoregister_process_defs=true +``` + +### CloudWatch alarms + +Create two alarms on the `queueMaxPendingDuration` metric: + +**Scale-up alarm** — triggers when pending duration exceeds your threshold (e.g., > 300 seconds for 2 evaluation periods). Associate it with a scale-out policy on the Auto Scaling Group. + +**Scale-down alarm** — triggers when pending duration drops below a lower threshold (e.g., ≤ 30 seconds for 5 evaluation periods). Associate it with a scale-in policy. + +### Launch template + +Create an EC2 Launch Template for worker nodes. In the **User data** field, pass the CWS worker configuration as URL-encoded key=value pairs, for example: + +``` +INSTALL_TYPE=worker&DB_HOST=&DB_USER=&DB_PASS=&CWS_CONSOLE_HOST= +``` + +Attach the Launch Template to an IAM instance profile that has the CloudWatch and S3 permissions described above. + +### Auto Scaling Group + +Create an EC2 Auto Scaling Group using the Launch Template above. Attach two **automatic scaling policies** — one for each CloudWatch alarm. + +!!! note "Manual process definition enablement" + When a new auto-scaled worker joins the cluster, you may need to manually enable process definitions on it through the CWS console. The `startup_autoregister_process_defs=true` setting reduces this, but verify that new workers pick up definitions as expected in your environment. + +## Using S3 for storage + +CWS can read and write files to S3. Enable it by adding the S3 bucket name and region to `config.properties` and ensuring the IAM role attached to CWS EC2 instances has the appropriate S3 permissions. See `install/example-cws-configuration.properties` for the full list of S3-related properties. + +## CloudWatch log aggregation + +For centralized log collection from multiple CWS nodes, configure the CloudWatch Logs agent (or the AWS-provided unified agent) on each instance to ship the Tomcat and CWS log directories. The default log path inside a CWS installation is: + +``` +/server/apache-tomcat-11.0.20/logs/ +``` diff --git a/docs/deployment/docker.md b/docs/deployment/docker.md new file mode 100644 index 00000000..d2007586 --- /dev/null +++ b/docs/deployment/docker.md @@ -0,0 +1,171 @@ +# Deploying with Docker + +CWS ships with several Docker Compose configurations under `install/docker/`. This guide covers building the image, running the all-in-one stack, and common variations. + +## Prerequisites + +- Docker and Docker Compose installed +- A valid Camunda EE license file at `~/.camunda/license.txt` +- Recommended Docker resources: 5 CPUs, 14 GB RAM, 1 GB swap, 64 GB disk + +## Building the CWS image + +The CWS Docker image is based on Oracle Linux 9 and bundles Java 17 and the CWS server package. + +```bash +cd install/docker/cws-image +./build.sh +``` + +`build.sh` will: + +1. Run `./build.sh` from the repository root to produce `dist/cws_server.tar.gz` +2. Copy the package into the image build context +3. Build and tag the image as `nasa-ammos/common-workflow-service:2.9.0` + +If you have already built the package separately, `build.sh` skips the Maven build and uses the existing artifact. + +The resulting image tag follows the pattern `nasa-ammos/common-workflow-service:`. + +## All-in-one stack (console + worker + database + Elasticsearch) + +The `console-db-es-ls-kibana` compose file starts a full CWS environment on a single host: + +```bash +cd install/docker/console-db-es-ls-kibana +docker-compose up +``` + +This brings up: + +| Container | Image | Exposed ports | +|-----------|-------|---------------| +| `cws-db` | `mariadb:10.11` | `3306` | +| `cws-es` | `elasticsearch:8.12.0` | `9200`, `9300` | +| `cws-console` | `nasa-ammos/common-workflow-service:2.9.0` | `38080`, `38443`, `31616` | +| `cws-worker1` | `nasa-ammos/common-workflow-service:2.9.0` | — | +| `ldapsearch` | OpenLDAP (used for authentication) | `389` | + +Access the CWS UI at `https://localhost:38443/cws-ui/` (HTTP on `38080` redirects to HTTPS). + +To stop the stack: + +```bash +docker-compose down +``` + +### Configuration + +The console reads `config.properties` mounted as a read-only volume. Edit `console-db-es-ls-kibana/config.properties` before starting. Key settings: + +```properties +hostname=cws-console +install_type=2 # 1=console+worker, 2=console only, 3=worker only +database_host=db +elasticsearch_host=es +elasticsearch_port=9200 +cws_console_host=cws-console +amq_host=cws-console +``` + +The worker reads `worker-config.properties` from the same directory. + +## Elasticsearch-only container + +Use the `es-only` setup when you need a standalone Elasticsearch instance — for example, to support a non-Docker CWS installation during development. + +```bash +cd install/docker/es-only +docker-compose up +``` + +This starts Elasticsearch 8.12.0 on port `9200` with security disabled (`xpack.security.enabled=false`). Verify it is running: + +```bash +curl http://localhost:9200/_cluster/health +``` + +Then configure CWS to connect: + +```properties +elasticsearch_protocol=http +elasticsearch_host=localhost +elasticsearch_port=9200 +elasticsearch_use_auth=n +``` + +## Database-only container + +To run only MariaDB in Docker: + +```bash +docker run -d \ + -p 3306:3306 \ + -e MYSQL_DATABASE= \ + -e MYSQL_ROOT_PASSWORD= \ + -e TZ=America/Los_Angeles \ + --name cws-db \ + mariadb:10.11 +``` + +Test the connection: + +```bash +mysql -h 127.0.0.1 -u root -p +``` + +Then configure CWS: + +```properties +database_type=mariadb +database_host=127.0.0.1 +database_name= +database_username=root +database_password= +``` + +## Adding additional workers + +The `worker-ls` compose file adds a standalone worker that joins an existing CWS cluster. + +```bash +cd install/docker/worker-ls +# Edit config.properties to point at the console host +docker-compose up +``` + +The worker compose file uses `cws-network` as an external network, so it must be on the same Docker network as the console stack. The worker's `config.properties` must set: + +```properties +install_type=3 +cws_console_host= +amq_host= +database_host= +elasticsearch_host= +``` + +To scale further, repeat this pattern with unique `hostname` and `container_name` values per worker. + +## Volume mounts for certificates + +By default the compose files use a self-signed certificate baked into the image. To supply your own certificates, uncomment and update the volume entries in `docker-compose.yml`: + +```yaml +volumes: + - ../../.keystore:/home/cws_user/cws/server/apache-tomcat-11.0.20/conf/.keystore:ro + - ../../tomcat_lib/cws_truststore.jks:/home/cws_user/cws/server/apache-tomcat-11.0.20/lib/cws_truststore.jks:ro + - ~/.cws/creds:/root/.cws/creds:ro +``` + +- `.keystore` must be a PKCS12 or JKS keystore with your server certificate +- `cws_truststore.jks` is the outbound trust store +- `~/.cws/creds` is a single-line file containing the keystore password, with permissions `400` + +## Deploying without external database or Elasticsearch + +If you already have dedicated database and Elasticsearch services: + +1. In `docker-compose.yml`, remove the `db` and/or `es` service blocks. +2. Remove the corresponding `depends_on` entries from the `cws` and `cws-worker` services. +3. Set the `DB_HOST` / `ES_HOST` environment variables to your external service hostnames. +4. Update `config.properties` to match. diff --git a/docs/deployment/index.md b/docs/deployment/index.md new file mode 100644 index 00000000..30189e0a --- /dev/null +++ b/docs/deployment/index.md @@ -0,0 +1,39 @@ +# Deployment + +CWS can be deployed in several configurations depending on your infrastructure and scale requirements. All deployments require three core services to be accessible: + +- **Database** — MariaDB or MySQL (one instance per cluster) +- **Elasticsearch** — version 8.12.0+ (one instance per cluster) +- **CWS** — one or more instances (console, workers, or combined) + +## Deployment options + +| Option | Description | +|--------|-------------| +| [Docker](docker.md) | All-in-one stack or split services using Docker Compose — recommended for development and smaller deployments | +| [AWS](aws.md) | EC2-based deployment with optional auto-scaling workers | + +## Console vs. worker roles + +A CWS installation runs in one of three modes, set by `install_type` in `config.properties`: + +| `install_type` | Role | +|----------------|------| +| `1` | Console **and** Worker (default) | +| `2` | Console only | +| `3` | Worker only | + +For production workloads, run the console as `install_type=2` and scale workers separately as `install_type=3`. Workers connect back to the console via the ActiveMQ broker (`amq_host`). + +## TLS certificates + +CWS requires two certificate files at startup: + +- `install/.keystore` — Tomcat SSL keystore +- `install/tomcat_lib/cws_truststore.jks` — trust store for outbound TLS + +The keystore password must be stored in `~/.cws/creds` with permissions `400`. See the [Docker deployment guide](docker.md#volume-mounts-for-certificates) for how to mount these files into containers. + +## Upgrading + +See the [Upgrade & Migration guide](upgrade.md) for procedures to move between CWS versions, including database schema migrations and Elasticsearch data migration. diff --git a/docs/deployment/upgrade.md b/docs/deployment/upgrade.md new file mode 100644 index 00000000..1cff0882 --- /dev/null +++ b/docs/deployment/upgrade.md @@ -0,0 +1,116 @@ +# Upgrade & Migration + +This guide covers the general procedure for upgrading CWS and documents version-specific steps for known breaking changes. + +## General upgrade procedure + +!!! warning "Back up before upgrading" + Always back up your database and any custom configuration files before starting an upgrade. + +1. **Stop all CWS instances** — console and all workers: + + ```bash + cd + ./stop_cws.sh --shutdown_all + ``` + +2. **Move the existing installation aside** — you will need to copy data from it: + + ```bash + mv -backup + ``` + +3. **Install the new version** — unpack the new CWS server package and run the configuration script with your existing `config.properties`: + + ```bash + tar zxvf cws_server-.tar.gz + cd cws_server- + ./configure.sh /path/to/your/config.properties + ``` + +4. **Run any required database migration scripts** — check the version-specific notes below for your upgrade path. Scripts are located under `install/upgrade/` in the new package. + +5. **Start the new CWS console** and immediately stop it (console only) — this initialises the new schema: + + ```bash + ./start_cws.sh + ./stop_cws.sh + ``` + +6. **Migrate Elasticsearch data** if required — see [Elasticsearch data migration](#elasticsearch-data-migration) below. + +7. **Restart CWS** on the console and all workers: + + ```bash + ./start_cws.sh + ``` + +## Elasticsearch data migration + +If you are upgrading to an Elasticsearch version that is not backward-compatible with your existing data, you must migrate the data directory. + +1. Stop CWS on the console. +2. Copy the old Elasticsearch `data` directory to the new installation: + + ```bash + mv /server/elasticsearch-/data \ + /server/elasticsearch-/data_orig + cp -R /server/elasticsearch-/data \ + /server/elasticsearch-/data + ``` + +3. Start the new CWS. Elasticsearch will reindex the migrated data on first start. + +If using an external Elasticsearch cluster (the recommended configuration), back up and restore your indices using the standard Elasticsearch snapshot/restore API rather than copying data directories. + +## Version-specific notes + +### v2.3 → v2.4 + +**Database schema change:** A new column `max_num_running_procs` was added to the `cws_worker` table. Run the provided upgrade script before restarting CWS: + +```bash +cd +./install/upgrade/upgrade_2.3_to_2.4.sh +``` + +The script also removes stale worker rows from previous deployments. + +The `max_num_running_procs` value controls the maximum number of concurrently running process instances per worker. It is configurable per worker row in the database and takes effect immediately without a restart. + +### v2.0 → v2.1 + +No database changes required. Follow the [general upgrade procedure](#general-upgrade-procedure). + +### v1.8 → v2.0 + +**Camunda engine upgrade:** CWS v1.8 used Camunda 7.10; CWS v2.0 uses Camunda 7.13. If you want to retain your existing workflow history, you must migrate the Camunda database schema in three steps. Scripts are in `/sql/upgrade/`: + +```bash +# Execute in order — substitute mariadb_ prefix with mysql_ if using MySQL +mariadb_engine_7.10_to_7.11.sql +mariadb_engine_7.11_to_7.12.sql +mariadb_engine_7.12_to_7.13.sql +``` + +Procedure: + +1. Stop all CWS instances. +2. Back up the database. +3. Execute the three SQL scripts against your database in the order listed. +4. Upgrade all CWS installations to v2.0. +5. Restart CWS. + +## Docker upgrades + +When upgrading a Docker-based deployment: + +1. Pull or rebuild the new image (`nasa-ammos/common-workflow-service:`). +2. Update the image tag in `docker-compose.yml`. +3. Apply any database migration scripts by running them against the database container: + + ```bash + docker exec -i cws-db mysql -u root -p cws < upgrade_script.sql + ``` + +4. Bring the stack back up with `docker-compose up`. diff --git a/docs/developer/adaptation.md b/docs/developer/adaptation.md new file mode 100644 index 00000000..4245863f --- /dev/null +++ b/docs/developer/adaptation.md @@ -0,0 +1,45 @@ +# Adapting CWS for a Mission + +CWS can be adapted and tailored for a specific mission or project **without +forking the core codebase**. The `cws-adaptation` module is the designated +extension point. + +## What you can add + +| Extension | Location | +| --- | --- | +| Custom Java code | `cws-adaptation/src/main/java/` | +| Custom REST API endpoints | Same package, using Spring `@Controller`/`@RestController` | +| Custom process initiators | Package `jpl.cws.process.initiation.custom` | +| An external database | SQL templates in `install/sql/` + Java extending `DbService` | +| Custom UI elements | Adaptation-specific FTL templates and JavaScript | + +## Getting started + +1. Ensure you have a local build environment set up (see + [Building from Source](../install/building.md)). +2. Work within the `cws-adaptation` module — it depends on `cws-service` and + `cws-core` and is packaged into the console WAR. +3. Build your adaptation: `cd cws-adaptation && mvn clean package`. +4. Deploy the resulting `cws-adaptation.jar` into the console's + `WEB-INF/lib/`. + +## Adaptation workers modal + +Customize the Deployments page workers view via the JavaScript function +`addAdaptationWorkersInfo` in +`cws-ui/src/main/webapp/js/adaptation-workers-modal.js`: + +```javascript +function addAdaptationWorkersInfo(dataProcKey, listWorkers) { + // Your custom logic here + return; +} +``` + +## Further reading + +- [External Database](external-database.md) — connect to a separate schema. +- [Custom Tasks](custom-tasks.md) — add new task types. +- [Custom Initiators](custom-initiators.md) — create triggers. +- [Security Plugins](security-plugin.md) — custom authentication schemes. diff --git a/docs/developer/architecture.md b/docs/developer/architecture.md new file mode 100644 index 00000000..f2be751f --- /dev/null +++ b/docs/developer/architecture.md @@ -0,0 +1,94 @@ +# Architecture + +CWS is a layered, service-oriented, multi-module Maven project built on the +Camunda BPMN engine. + +## Key technologies + +| Technology | Version | Role | +| --- | --- | --- | +| Java | 17 | Runtime (enforced by Maven) | +| Spring Framework | 7.0.x | Dependency injection, web, transactions | +| Camunda BPM | 7.24.x (Enterprise) | Workflow engine and BPMN execution | +| Apache Artemis | — | Message queue for external-task communication | +| MyBatis | — | Database ORM and query mapping | +| Apache Tomcat | 11.x | Application server | +| Elasticsearch | 8.12+ | Log and history aggregation | +| MariaDB / MySQL | — | Relational datastore | + +See [Requirements & Compatibility](../install/requirements.md) for exact +versions. + +## Modules + +CWS is composed of these Maven modules: + +| Module | Responsibility | +| --- | --- | +| **cws-core** | Foundation: configuration, database services, logging, security. | +| **cws-tasks** | BPMN task implementations (email, command line, REST, file operations, sleep). | +| **cws-service** | Business logic: process initiators, scheduling, REST APIs. | +| **cws-engine-service** | Engine-specific services and external-task handling. | +| **cws-engine** *(WAR)* | Camunda engine integration. | +| **cws-adaptation-engine** | Adaptation hooks for the engine layer. | +| **cws-adaptation** | Project-specific customizations. | +| **cws-ui** *(WAR)* | Web-based user interface (the console). | +| **cws-installer** | Installation and configuration utilities. | +| **cws-test** | Integration and system testing framework. | + +## Module dependencies + +```mermaid +graph TD + core[cws-core] + tasks[cws-tasks] + service[cws-service] + installer[cws-installer] + engsvc[cws-engine-service] + adapteng[cws-adaptation-engine] + engine[cws-engine WAR] + adaptation[cws-adaptation] + ui[cws-ui WAR] + + core --> tasks + core --> service + core --> installer + core --> engsvc + tasks --> engsvc + engsvc --> adapteng + adapteng --> engine + engine --> adaptation + service --> adaptation + core --> adaptation + adaptation --> ui +``` + +## Layered design + +- **Foundation (`cws-core`)** provides cross-cutting services used everywhere: + configuration, the centralized `DbService` for database access, logging, and + security filters (LDAP, Camunda, and general web security). +- **Task and service layers (`cws-tasks`, `cws-service`, `cws-engine-service`)** + implement workflow behavior, the external-task engine, scheduling, and the + REST API. +- **Engine integration (`cws-engine`, `cws-adaptation-engine`)** packages the + Camunda engine as a deployable WAR with adaptation hooks. +- **Presentation and customization (`cws-ui`, `cws-adaptation`)** deliver the + web console and the project-specific adaptation layer. + +## External task processing + +Work is distributed via **Apache Artemis (ActiveMQ)** message queuing. Workers +can run as separate JVM processes, and the external task service in +`cws-engine-service` distributes tasks to them. Workers support multiple run +modes (for example: run everything, run models only, or run external tasks +only). + +## Extending CWS + +Rather than fork the codebase, extend CWS through the adaptation layer: + +- [Adapting CWS for a Mission](adaptation.md) +- [Writing Custom Tasks](custom-tasks.md) (extend the `CwsTask` base class) +- [Developing Custom Initiators](custom-initiators.md) +- [Custom Security Scheme Plugins](security-plugin.md) diff --git a/docs/developer/contributing.md b/docs/developer/contributing.md new file mode 100644 index 00000000..df82acde --- /dev/null +++ b/docs/developer/contributing.md @@ -0,0 +1,39 @@ +# Building & Contributing + +CWS is open source under the [Apache License 2.0](https://github.com/NASA-AMMOS/common-workflow-service/blob/main/LICENSE). +Contributions are welcome. + +## Building + +See the [Installation → Building from Source](../install/building.md) guide for +the full build procedure, including the personal dev script, running tests, and +dependency checks. + +## Contributing workflow + +The full contribution guidelines are in +[CONTRIBUTING.md](https://github.com/NASA-AMMOS/common-workflow-service/blob/main/CONTRIBUTING.md). +The highlights: + +1. **Fork** the repository and create a feature branch from `develop`. +2. **Make your changes** — follow existing coding patterns and conventions. +3. **Test** — run `./test.sh` to execute unit and integration tests. +4. **Open a pull request** against `develop` with a clear description of the + change. + +## Branching + +| Branch | Purpose | +| --- | --- | +| `main` | Release branch — tagged releases are cut from here. | +| `develop` | Default development branch — PRs target this. | + +## Code of conduct + +All contributors are expected to follow the project's +[Code of Conduct](https://github.com/NASA-AMMOS/common-workflow-service/blob/main/CODE_OF_CONDUCT.md). + +## Reporting issues + +[Open an issue](https://github.com/NASA-AMMOS/common-workflow-service/issues/new/choose) +on GitHub for bugs, feature requests, or documentation gaps. diff --git a/docs/developer/custom-initiators.md b/docs/developer/custom-initiators.md new file mode 100644 index 00000000..eb7f2dd4 --- /dev/null +++ b/docs/developer/custom-initiators.md @@ -0,0 +1,95 @@ +# Developing Custom Initiators + +CWS supports custom process initiators — both **internal** (runs inside the +CWS JVM) and **external** (runs outside CWS, calls the REST API). + +## Internal initiator development + +An internal initiator extends `CwsProcessInitiator` and runs within the CWS +console process. + +### Steps + +1. Navigate to `cws-adaptation/src/main/java/` and the package + `jpl.cws.process.initiation.custom`. + +2. Create a class extending `CwsProcessInitiator`: + + ```java + public class MyInitiator extends CwsProcessInitiator { + + private int delayMs; + + public MyInitiator(String procDefKey, int delayMs) { + super(procDefKey); + this.delayMs = delayMs; + } + + @Override + public void run() { + while (isActive()) { + // Check your trigger condition + if (shouldStart()) { + scheduleProcess( + procVariables, // Map + null, // procBusinessKey (null = auto) + "my-trigger-key" // initiationKey + ); + } + Thread.sleep(delayMs); + } + } + + @Override + public ConstructorArgumentValues getConstructorArgumentValues() { + ConstructorArgumentValues args = new ConstructorArgumentValues(); + args.addGenericArgumentValue(getProcDefKey()); + args.addGenericArgumentValue(delayMs); + return args; + } + + @Override + public boolean isValid() { + return delayMs > 0; + } + } + ``` + +3. Build and deploy: + + ```bash + cd cws-adaptation + mvn clean package + ``` + + Copy the resulting `cws-adaptation.jar` to the console's `WEB-INF/lib/`. + +4. Register your initiator in `cws-process-initiators.xml` (under + `cws-ui/src/main/resources/`) so it appears in the console's Initiators page. + +### Key points + +- Always check `isActive()` in your loop — this allows CWS to stop the + initiator cleanly. +- Use `Thread.sleep()` in loops to avoid busy-waiting. +- Call `scheduleProcess()` when your trigger condition is met. + +## External initiator development + +An external initiator is any program that calls the CWS +[REST API](../user-guide/launching/rest.md) to schedule processes. + +```bash +curl -k -X POST \ + "https://:38443/cws-ui/rest/process//schedule" \ + -H "cwsToken: " \ + --data "variable1=value1" +``` + +External initiators can be written in any language and run anywhere with +network access to the CWS console. + +## Choosing internal vs. external + +See [Internal & External Initiators](../user-guide/initiators/internal-external.md) +for a comparison. diff --git a/docs/developer/custom-tasks.md b/docs/developer/custom-tasks.md new file mode 100644 index 00000000..eb397518 --- /dev/null +++ b/docs/developer/custom-tasks.md @@ -0,0 +1,54 @@ +# Writing Custom Tasks + +CWS lets you create your own task types by extending the `CwsTask` base class +in the `cws-tasks` module. + +## Template method pattern + +`CwsTask` uses the template method pattern. To create a custom task: + +1. Create a new Java class extending `CwsTask`. +2. Override the required lifecycle methods. +3. Register your task with the engine so it appears in the modeler. +4. Build and deploy. + +## Minimal example + +```java +package jpl.cws.task; + +public class MyCustomTask extends CwsTask { + + @Override + public void initParams() { + // Read input parameters from the process context + } + + @Override + public void executeTask() { + // Your task logic here + // Use log.info(...) for output + // Set output variables on the execution context + } +} +``` + +## Input and output variables + +- **Inputs** are read from the process execution context in `initParams()`. +- **Outputs** are set back on the context so downstream tasks can use them. +- All variables are scoped by the task ID (see + [Built-in Task Types](task-types.md) for the scoping convention). + +## Building and deploying + +1. Add your class to the `cws-tasks` module (or `cws-adaptation` for + project-specific tasks). +2. Build: `mvn clean package` +3. Deploy the resulting JAR into the CWS console `WEB-INF/lib/` (or rebuild + the full distribution). + +## See also + +- [Built-in Task Types](task-types.md) — reference for the shipped tasks. +- [Architecture](architecture.md) — how the task layer fits into CWS. diff --git a/docs/developer/external-database.md b/docs/developer/external-database.md new file mode 100644 index 00000000..92d55299 --- /dev/null +++ b/docs/developer/external-database.md @@ -0,0 +1,65 @@ +# External Database + +The **Adaptation External Database** feature lets you connect to a database +separate from the core CWS schema, so your project-specific data lives +independently. + +## Setup + +### 1. Add SQL templates + +Place the following files in `install/sql/`: + +| File | Targets | +| --- | --- | +| `adaptation.sql.template` | Core database (`cws_dev`) | +| `adaptation_core.sql.template` | Core database (`cws_dev`) | +| `adaptation_external.sql.template` | External database (`cws_external_db` or your name) | + +### 2. Define your schema + +Add your `CREATE TABLE` statements in `adaptation_external.sql.template`: + +```sql +CREATE TABLE IF NOT EXISTS `my_project_table` ( + id BIGINT AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(255) NOT NULL, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; +``` + +### 3. Write Java code + +Extend `DbService` and use the appropriate JDBC template: + +| Template | Database | +| --- | --- | +| `jdbcTemplate` | Core CWS database | +| `jdbcAdaptationTemplate` | Your external adaptation database | + +```java +@Service +public class MyProjectDbService extends DbService { + + public List getRecords() { + return jdbcAdaptationTemplate.query( + "SELECT * FROM my_project_table", + new MyRecordRowMapper() + ); + } +} +``` + +### 4. Configure the connection + +The external database connection details are specified in your +[configuration properties](../install/configuration.md) and applied during +installation. + +## Notes + +- The external database is created and managed alongside the core CWS database + during installation. +- Both databases can be on the same host or on different hosts. +- Use the adaptation SQL templates to version-control your schema alongside + CWS. diff --git a/docs/developer/index.md b/docs/developer/index.md new file mode 100644 index 00000000..2a7d8b62 --- /dev/null +++ b/docs/developer/index.md @@ -0,0 +1,45 @@ +# Developer & Adaptation Guide + +This section covers extending CWS — adding custom tasks, initiators, and +security plugins, connecting external databases, and adapting CWS for a +mission-specific project. + +
+ +- :material-sitemap-outline: __[Architecture](architecture.md)__ + + Module structure, layers, and the external-task engine. + +- :material-puzzle-outline: __[Adapting CWS](adaptation.md)__ + + Tailor CWS for a mission without forking the codebase. + +- :material-database-cog-outline: __[External Database](external-database.md)__ + + Connect to a project-specific database alongside the core CWS schema. + +- :material-code-braces: __[Custom Tasks](custom-tasks.md)__ + + Write your own task types by extending `CwsTask`. + +- :material-format-list-checks: __[Built-in Task Types](task-types.md)__ + + Reference for the tasks that ship with CWS. + +- :material-timer-cog-outline: __[Custom Initiators](custom-initiators.md)__ + + Develop internal or external initiators. + +- :material-shield-lock-outline: __[Security Plugins](security-plugin.md)__ + + Implement a custom authentication/security scheme. + +- :material-web: __[Web Integration & CORS](web-integration.md)__ + + Enable CORS and use CWS Web Components. + +- :material-source-branch: __[Building & Contributing](contributing.md)__ + + Build CWS from source and contribute back. + +
diff --git a/docs/developer/security-plugin.md b/docs/developer/security-plugin.md new file mode 100644 index 00000000..b9efbd7c --- /dev/null +++ b/docs/developer/security-plugin.md @@ -0,0 +1,44 @@ +# Custom Security Scheme Plugins + +CWS supports pluggable security schemes. You can implement a custom +authentication/authorization mechanism to replace or augment the built-in LDAP +and Camunda security. + +## How it works + +CWS uses security filters to intercept requests and authenticate users. The +`identity_plugin_type` configuration setting selects which identity provider +is active. + +To implement a custom security scheme: + +1. Create a class in `cws-adaptation` that implements the security filter + interface. +2. Register your filter in the web application configuration. +3. Set `identity_plugin_type` and related settings in your + [configuration file](../install/configuration.md). + +## Extension points + +| Class / Interface | Purpose | +| --- | --- | +| `CwsLdapSecurityFilter` | The built-in LDAP security filter (reference implementation). | +| `CwsCamundaSecurityFilter` | The built-in Camunda security filter. | + +Study these implementations in `cws-core` to understand the contract your +custom filter must satisfy. + +## Configuration + +Set the filter class names in your configuration properties: + +```properties +ldap_security_filter_class=your.package.CustomSecurityFilter +camunda_security_filter_class=your.package.CustomCamundaFilter +``` + +## See also + +- [Security & Roles (admin)](../administration/security.md) — configuring the + built-in security modes. +- [LDAP Security](../install/ldap.md) — LDAP-specific installation. diff --git a/docs/developer/task-types.md b/docs/developer/task-types.md new file mode 100644 index 00000000..ec0fccdc --- /dev/null +++ b/docs/developer/task-types.md @@ -0,0 +1,127 @@ +# Built-in Task Types + +CWS ships several built-in task types you can drop into a process without +writing any code. To use one, drag a task box onto the modeler canvas, make it +a **Service Task**, then select the CWS task type from the element type list. + +!!! note "Task-scoped variables" + Variables produced by a task are **scoped by the ID of the task that + generates them**. For example, an `exitValue` produced by a Command Line + task with the ID `Task_1ydalhn` is stored as `Task_1ydalhn_exitValue`. This + lets multiple tasks of the same type appear in one process without their + output variables colliding. + +## The built-in tasks + +| Task type | Description | +| --- | --- | +| **Command Line Execution** | Invoke an executable program, as if launched from a command-line prompt, on an enabled worker. | +| **Log Message** | Write a message to the log file. | +| **Send Email** | Send an email. | +| **Sleep** | Pause for a specified amount of time. | +| **REST GET** | Make a REST `GET` call to a URL with the specified parameters. | +| **REST POST** | Make a REST `POST` call to a URL with the specified parameters. | +| **Set Variables** | Set process variables from a properties file. | +| **Move File** | Move a file from one location to another. | +| **Schedule Process** | Schedule another process for execution. | + +You can also write your own — see [Writing Custom Tasks](custom-tasks.md). + +## Command Line Execution + +Runs an executable program on a worker that is enabled for the process +definition. + +!!! warning "Executables only — not shell commands" + The Command Line task invokes **executables**, not shell scripting + constructs. It uses + [Apache Commons Exec](https://commons.apache.org/proper/commons-exec/) + behind the scenes, so only platform-independent invocations are allowed — + **not** environment variables, `if`/`then`/`else`, redirection, piping, or + other OS/shell-specific constructs. + + Allowed: + + ```bash + echo "hello world" + ``` + + Not allowed (uses shell redirection): + + ```bash + echo "hello world" >> my_output_file.txt + ``` + + To reference an environment variable, use the CWS expression form rather + than a shell variable: + + ```bash + echo "my home directory is ${cws.getEnv("HOME")}" # allowed + echo "my home directory is $HOME" # NOT allowed + ``` + +Executables must be on the `PATH`; otherwise specify the full absolute path to +the executable. + +### Options + +| Option | Purpose | +| --- | --- | +| **Command** | The command to execute. | +| **Working Directory** | Current working directory for the command (e.g. `/home/user`). | +| **Success Value(s)** | Comma-separated exit values considered a success (e.g. `0,4,10`). | +| **throwOnFailures** | If `true`, a non-success exit throws a catchable `BpmnError` (which a [boundary error event](https://docs.camunda.org/manual/7.24/reference/bpmn20/events/error-events/) can catch). | +| **Exit Event Map** | Key/value pairs mapping exit codes to named events (e.g. `0=success,1=fail`). | +| **throwOnTruncateVariable** | If `true`, throw a `BpmnError` when the output variable is truncated for length. Default `false`. | +| **timeout** | Seconds to allow the command to run before timing out. | +| **retries** / **retryDelay** | Number of retries on timeout, and the delay (ms) before retrying. | +| **Pre-condition** | An expression that must evaluate to `true` for the task to proceed. | +| **onPreConditionFail** | Behavior when the pre-condition fails. | + +### Input and output variables + +The task reads its inputs from an `in` JSON object (populated from the modeler +options: `command`, `workingDir`, `successfulValues`, `exitCodeEvents`, +`throwOnFailures`, `throwOnTruncatedVariable`, `timeout`, `retries`, +`retryDelay`) and writes results to an `out` JSON object: + +| `out` field | Meaning | +| --- | --- | +| `exitCode` | Exit code returned by the program. | +| `success` | Whether `exitCode` is one of the success values. | +| `event` | The event mapped from `exitCode` via the exit event map. | +| `stdout` | Standard output. | +| `stderr` | Standard error. | +| `lockedTime` | Start time of the command. | + +Access an output field in a model expression with the task ID and a JSON path, +for example: + +``` +Task_id_out.jsonPath("$.exitCode").numberValue() != 5 +Task_id_out.jsonPath("$.success").boolValue() == true +Task_id_out.jsonPath("$.stdout").stringValue() == "my text output" +``` + +### Command Line Execution (short-lived / blocking) + +A blocking variant of the Command Line task, for short-lived work. It differs +from the standard task in that it: + +- **Blocks** the process thread instead of releasing it for other work. +- Is guaranteed to run on the **same worker** as the previous activity (unless + you set an async flag). +- Does **not** have the `timeout` / `retry` settings of the standard task. + +!!! warning + Because it blocks, it is bound by the **process engine** timeout and retry + settings — by default a 5-minute timeout with up to 3 immediate retries. + Only use it for executions that reliably finish within that window; + retries on tasks with side effects can cause errors. + +## See also + +- [Script Task Recipes](../modeling/script-recipes.md) — parsing JSON/REST + responses, setting variables, and logging from script tasks. +- [Writing Custom Tasks](custom-tasks.md) — implement your own task by + extending `CwsTask`. \ No newline at end of file diff --git a/docs/developer/web-integration.md b/docs/developer/web-integration.md new file mode 100644 index 00000000..c81ebcab --- /dev/null +++ b/docs/developer/web-integration.md @@ -0,0 +1,81 @@ +# Web Integration & CORS + +CWS supports integration with external web applications through CORS +configuration and experimental Web Components. + +## Enabling CORS + +By default, the CWS REST API only accepts requests from the console's own +origin. To call the API from other hosts (external dashboards, custom UIs, +single-page apps), enable CORS by adding a filter to the Tomcat `web.xml`. + +### For the CWS REST API + +Edit: + +``` +/server/apache-tomcat-/webapps/cws-ui/WEB-INF/web.xml +``` + +### For the Camunda REST API + +Edit: + +``` +/server/apache-tomcat-/webapps/engine-rest/WEB-INF/web.xml +``` + +### Filter configuration + +Add after the last `` in the XML: + +```xml + + CorsFilter + org.apache.catalina.filters.CorsFilter + + cors.allowed.origins + https://your-app.example.com + + + cors.allowed.methods + GET,POST,HEAD,OPTIONS,PUT + + + cors.allowed.headers + Content-Type,X-Requested-With,accept,Authorization,Origin,Access-Control-Request-Method,Access-Control-Request-Headers,Last-Modified,X-Auth-Token + + + cors.exposed.headers + Access-Control-Allow-Origin,Access-Control-Allow-Credentials + + + cors.support.credentials + true + + + + CorsFilter + /* + +``` + +Replace `https://your-app.example.com` with your allowed origins +(comma-separated for multiple). + +See the [Tomcat CORS Filter documentation](https://tomcat.apache.org/tomcat-11.0-doc/config/filter.html#CORS_Filter) +for all available options. + +## CWS Web Components (Beta) + +CWS provides experimental Web Components that you can embed in external web +pages to display CWS data (process status, worker health, etc.) without +building a custom integration from scratch. + +!!! warning "Beta" + Web Components are experimental and subject to change. + +See the `project_webapp_root` +[configuration setting](../reference/configuration.md#host-installation) to set +up a project-specific webapp inside the CWS server that is accessible without +CWS security. diff --git a/docs/getting-started/concepts.md b/docs/getting-started/concepts.md new file mode 100644 index 00000000..8232cda4 --- /dev/null +++ b/docs/getting-started/concepts.md @@ -0,0 +1,74 @@ +# Core Concepts + +A short tour of the building blocks you'll work with in CWS. + +## Processes (BPMN) + +A **process** is a workflow defined in [BPMN 2.0](https://docs.camunda.org/manual/7.24/reference/bpmn20/) +— a standard graphical notation for modeling business processes. You design a +process as a diagram of tasks, gateways, and events, then deploy it to CWS. A +running copy of a deployed process is a **process instance**. + +CWS runs BPMN on top of the Camunda 7 engine, so any BPMN construct Camunda +supports is available, plus CWS's own task types and extensions. + +## Tasks + +A **task** is a single step in a process. CWS ships several built-in task +types — for example command-line execution, REST calls, email, file +operations, and sleep — and lets you add your own. See +[Built-in Task Types](../developer/task-types.md) and +[Writing Custom Tasks](../developer/custom-tasks.md). + +Tasks that CWS executes outside the engine are handled by the **external task** +mechanism (below). + +## Workers & external tasks + +CWS distributes work using an **external task engine**. Instead of the engine +executing every task in-process, eligible tasks are placed on a queue and +picked up by **workers**. + +- A **worker** is a CWS process that fetches and executes external tasks. +- Workers can run as separate JVM processes and on separate machines, letting + you scale execution horizontally. +- Worker behavior is bounded by configuration such as the maximum number of + concurrently running process instances per worker. + +This separation means the console stays responsive while heavy or long-running +work happens on the workers. + +## Initiators + +An **initiator** starts process instances automatically in response to some +trigger, so you don't have to launch them by hand. CWS includes initiators such +as: + +- **[Cron](../user-guide/initiators/cron.md)** — start on a schedule. +- **[File](../user-guide/initiators/file.md)** — start when a file appears. +- **[Message Arrival](../user-guide/initiators/message-arrival.md)** — start + when a message is received. +- **[Repeating Delay](../user-guide/initiators/repeating-delay.md)** — start + repeatedly after a delay. + +You can also develop [custom initiators](../developer/custom-initiators.md). + +## Adaptations + +An **adaptation** tailors CWS for a specific mission or project — custom Java, +custom REST endpoints, custom initiators, and project-specific configuration — +without modifying the core CWS codebase. See +[Adapting CWS for a Mission](../developer/adaptation.md). + +## Snippets + +**Snippets** are small pieces of reusable code you can invoke from your +processes, making it easy to share logic across workflows. See +[Snippets](../user-guide/snippets.md). + +## Logging & history + +CWS records auditable logs and process history, backed by **Elasticsearch** for +search and aggregation. How much history is retained, and for how long, is +controlled by the `history_level` and `history_days_to_live` +[configuration settings](../reference/configuration.md#history-retention). diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md new file mode 100644 index 00000000..a6c063d6 --- /dev/null +++ b/docs/getting-started/index.md @@ -0,0 +1,46 @@ +# Getting Started + +New to CWS? This section gets you from zero to a running instance and explains +the concepts you'll use every day. + +
+ +- :material-rocket-launch-outline: __[Quickstart with Docker](quickstart.md)__ + + Stand up a full CWS stack — console, worker, database, and Elasticsearch — + on a single machine with Docker. + +- :material-lightbulb-on-outline: __[Core Concepts](concepts.md)__ + + Understand processes, workers, external tasks, initiators, and adaptations + and how they fit together. + +- :material-timeline-clock-outline: __[Process Instance Lifecycle](lifecycle.md)__ + + Follow a process instance from launch through completion. + +
+ +## What is CWS? + +CWS wraps the [Camunda 7 BPMN engine](https://camunda.com/products/camunda-platform/bpmn-engine/) +with the pieces you need to run workflows in production: + +- A **web console** for deploying, launching, monitoring, and troubleshooting + processes. +- An **external task engine** that distributes work to one or more **workers**, + which can run as separate JVM processes or on separate machines. +- **Initiators** that start processes automatically — on a schedule, when a + file arrives, when a message is received, and more. +- **Adaptation layers** and **code snippets** for tailoring CWS to a specific + mission or project without forking the codebase. +- **Auditable logging** with Elasticsearch-backed history. + +## Which path is right for me? + +| I want to… | Start here | +| --- | --- | +| Try CWS quickly on one machine | [Quickstart with Docker](quickstart.md) | +| Install CWS for real (dev or production) | [Installation](../install/index.md) | +| Learn the moving parts first | [Core Concepts](concepts.md) | +| Understand the codebase / extend CWS | [Architecture](../developer/architecture.md) | diff --git a/docs/getting-started/lifecycle.md b/docs/getting-started/lifecycle.md new file mode 100644 index 00000000..30f6a8e4 --- /dev/null +++ b/docs/getting-started/lifecycle.md @@ -0,0 +1,61 @@ +# Process Instance Lifecycle + +When a process is started in CWS — whether manually, via an initiator, or +through the REST API — it goes through a defined lifecycle. + +## Status transitions + +```mermaid +stateDiagram-v2 + [*] --> pending + pending --> inSchedulerQueue + inSchedulerQueue --> claimedByWorker + claimedByWorker --> running + running --> success : completed normally + running --> fail : error / exception + pending --> disabled : user action + disabled --> pending : user action + running --> incident : engine incident raised + incident --> running : retry + fail --> resolved : user marks resolved +``` + +## Status descriptions + +| Status | Meaning | +| --- | --- | +| **pending** | Queued and waiting for a worker to pick it up. | +| **inSchedulerQueue** | Accepted by the scheduler, waiting for assignment. | +| **claimedByWorker** | A worker has claimed the instance. | +| **running** | Actively executing on a worker. | +| **success** | Completed normally (reached an end event). | +| **fail** | An error prevented successful completion. | +| **incident** | The engine raised one or more incidents (retryable). | +| **disabled** | User manually disabled the pending instance. | +| **resolved** | A failed instance marked as acknowledged by the user. | + +## Where to see status + +- **[Deployments page](../user-guide/console/deployments.md)** — color-coded + bars showing the aggregate status distribution per process. +- **[Processes page](../user-guide/console/processes.md)** — individual + instances with their current status and available actions. +- **[REST API](../user-guide/launching/rest.md#monitoring-instance-status)** — + poll the `/process-instance//status` endpoint. + +## What happens on failure + +When a process instance ends in `fail`: + +- The failure is visible on the Processes page. +- The [Logs page](../user-guide/console/logs.md) shows the error context. +- You can **mark it as resolved** (acknowledged, counted as completed in + statistics) or investigate and retry if the process supports it. + +## Incidents + +An **incident** is a retryable failure raised by the Camunda engine (e.g. a +transient error, a timed-out external task). CWS displays incidents in pink on +the Deployments page. You can **retry** incidents from the Processes page, which +restarts execution from the last successful +[commit point](https://docs.camunda.org/manual/7.24/user-guide/process-engine/transactions-in-processes/#asynchronous-continuations). diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md new file mode 100644 index 00000000..d008b493 --- /dev/null +++ b/docs/getting-started/quickstart.md @@ -0,0 +1,84 @@ +# Quickstart with Docker + +The fastest way to try CWS is the single-machine Docker stack, which runs the +database, Elasticsearch, the CWS console, and a worker together on one host. + +!!! warning "For evaluation and development" + This all-in-one stack is meant for trying CWS and local development. For a + real deployment, see the [Installation guide](../install/index.md). + +## Prerequisites + +- [Docker](https://docs.docker.com/get-docker/) with at least **4 CPUs and + 10 GB memory** allocated (Docker → Settings → Resources). +- A local clone of the + [CWS repository](https://github.com/NASA-AMMOS/common-workflow-service). + +## Steps + +1. **Build the CWS Docker image.** From the repository, run the `build.sh` + script in the CWS image directory: + + ```bash + cd install/docker/cws-image + ./build.sh + ``` + + Update the version in `build.sh` if needed. + +2. **Provide a keystore password file.** CWS uses SSL certificates that need a + password at startup. Create the credentials file and restrict its + permissions: + + ```bash + mkdir -p ~/.cws + echo "changeit" > ~/.cws/creds # the default password for the bundled self-signed certs + chmod 700 ~/.cws + chmod 400 ~/.cws/creds + ``` + + See [Certificates & Keystore](../install/certificates.md) to use your own + certificates. + +3. **Create the shared Docker network** so additional workers can join: + + ```bash + docker network create cws-network + ``` + +4. **Review configuration.** Adjust `config.properties` and + `docker-compose.yml` in `install/docker/console-db-es-ls-kibana/` if you + need non-default settings. + +5. **Start the stack:** + + ```bash + cd install/docker/console-db-es-ls-kibana + docker-compose up + ``` + + This brings up the database, Elasticsearch, the CWS console, and one + worker. + +6. **Open the console.** Once startup completes, browse to the CWS console over + HTTPS on the configured console port and log in. + +## Adding more workers + +Each additional worker needs about **4 GB more memory**. To add a second worker +to this deployment: + +```bash +cd ../worker-ls +docker-compose up +``` + +For further workers, copy the `worker-ls` directory, adjust its +`config.properties` and `docker-compose.yml`, and run `docker-compose up` in +each new directory. + +## Next steps + +- [Deploy a process definition](../user-guide/deploying.md) +- [Launch and schedule processes](../user-guide/launching/index.md) +- [Tour the web console](../user-guide/console/index.md) diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 00000000..9ca2c5c9 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,95 @@ +# Common Workflow Service + +The **Common Workflow Service (CWS)** is an open-source, enterprise workflow +management platform from [NASA-AMMOS](https://ammos.nasa.gov/). It is built on +top of the [Camunda BPMN workflow engine](https://camunda.com/products/camunda-platform/bpmn-engine/) +and extends it with an intuitive web console, auditable logging, a powerful +external-task engine, custom process initiators, code snippets, and a +mission-adaptation layer. + +This site is the home for the CWS **user guide**, **installation guide**, +**administration** and **adaptation** documentation, and pointers to the +generated **API and code reference**. + +!!! tip "New to CWS?" + Start with the [Quickstart](getting-started/quickstart.md) to stand up CWS + with Docker, then read the [Core Concepts](getting-started/concepts.md) to + learn how processes, workers, initiators, and adaptations fit together. + +## Explore the documentation + +
+ +- :material-rocket-launch-outline: __Getting Started__ + + --- + + Stand up CWS quickly and learn the core concepts behind BPMN processes, + workers, and initiators. + + [:octicons-arrow-right-24: Get started](getting-started/index.md) + +- :material-download-outline: __Installation__ + + --- + + Requirements, prerequisites, database and Elasticsearch setup, building + from source, and configuration. + + [:octicons-arrow-right-24: Install CWS](install/index.md) + +- :material-monitor-dashboard: __User Guide__ + + --- + + Tour the web console, deploy process definitions, and launch, schedule, + and monitor your workflows. + + [:octicons-arrow-right-24: Use CWS](user-guide/index.md) + +- :material-sitemap-outline: __Modeling__ + + --- + + Design BPMN processes with best practices, examples, and reusable script + task recipes. + + [:octicons-arrow-right-24: Model workflows](modeling/index.md) + +- :material-shield-account-outline: __Administration__ + + --- + + Manage users and security, workers, logging, and history retention. + + [:octicons-arrow-right-24: Administer CWS](administration/index.md) + +- :material-code-braces: __Developer & Adaptation__ + + --- + + Architecture, adapting CWS for a mission, custom tasks and initiators, and + contributing. + + [:octicons-arrow-right-24: Extend CWS](developer/index.md) + +
+ +## API & code reference + +- **[REST API](reference/rest-api.md)** — the CWS console serves interactive + Swagger UI for its REST endpoints. +- **[Javadoc](reference/javadoc.md)** — generated API documentation for the + CWS Java modules. +- **[Configuration properties](reference/configuration.md)** — every + configurable CWS setting. + +## About this documentation + +This documentation is versioned alongside CWS releases. The version you are +reading is shown in the selector at the top of the page. The current release is +**CWS <<< cws_version >>>**. + +CWS is released under the [Apache License 2.0](https://github.com/NASA-AMMOS/common-workflow-service/blob/main/LICENSE). +Found a problem or a gap? Please +[open an issue](https://github.com/NASA-AMMOS/common-workflow-service/issues/new/choose). diff --git a/docs/install/building.md b/docs/install/building.md new file mode 100644 index 00000000..293698cd --- /dev/null +++ b/docs/install/building.md @@ -0,0 +1,108 @@ +# Building from Source + +CWS is built with Maven. For development, the project convention is a small +**personal build script** that sets your environment values and calls +`dev.sh`, which builds CWS and starts the console and workers. + +## A personal build script + +Create a script such as `dev-yourname.sh` in the repository root. The template +below works for local development. Set `ES_PROTOCOL` (`HTTP` or `HTTPS`) and +`ES_HOST` to match your Elasticsearch: + +```bash +#!/bin/bash +# File: dev-yourname.sh + +HOSTNAME=localhost + +# Used in cws-test +echo "$HOSTNAME" > cws-test/src/test/resources/hostname.txt + +SECURITY="camunda" + +# Stop CWS if it is currently running +./stop_dev.sh + +# DB config +DB_TYPE=mariadb +DB_HOST=127.0.0.1 +DB_NAME=cws_dev # must match the database you created beforehand +DB_USER=root # must match the user you created beforehand +DB_PASS= # can also be supplied via environment variables +DB_PORT=3306 # mariadb default + +USER= # your username +CLOUD= # enable cloudwatch monitoring (leave blank to disable) + +EMAIL_LIST="{email}" + +ADMIN_FIRST="{first}" +ADMIN_LAST="{last}" +ADMIN_EMAIL="{email}" + +# Elasticsearch config +ES_PROTOCOL="HTTP" # 'HTTP' or 'HTTPS' +ES_HOST="localhost" +ES_PORT=9200 +ES_USE_AUTH=n +ES_USERNAME="na" +ES_PASSWORD="na" + +# Number of workers to start (1 is the minimum) +NUM_WORKERS=1 + +# Max concurrently running process instances per worker (default 16, min 1) +WORKER_MAX_NUM_RUNNING_PROCS=16 + +# Days until abandoned workers are cleaned from the cws_workers table +WORKER_ABANDONED_DAYS=1 + +# Run the dev script +./dev.sh `pwd` ${USER} ${DB_TYPE} ${DB_HOST} ${DB_PORT} ${DB_NAME} ${DB_USER} ${DB_PASS} ${ES_PROTOCOL} ${ES_HOST} ${ES_PORT} ${ES_USE_AUTH} ${ES_USERNAME} ${ES_PASSWORD} ${CLOUD} ${SECURITY} ${HOSTNAME} ${EMAIL_LIST} ${ADMIN_FIRST} ${ADMIN_LAST} ${ADMIN_EMAIL} ${NUM_WORKERS} ${WORKER_MAX_NUM_RUNNING_PROCS} ${WORKER_ABANDONED_DAYS} +``` + +Run it from the repository root: + +```bash +./dev-yourname.sh +``` + +The script builds CWS, verifies your configuration, and starts the console and +workers. When everything is up, it prints a link to the console dashboard. + +## Full (non-dev) build + +To produce a server distribution without the dev workflow: + +```bash +./build.sh +``` + +This cleans the module library directories, runs +`mvn -DskipTests -Dskip.integration.tests clean install -P core`, and creates +the server distribution via `create_server_dist.sh`. + +## Checking dependencies for security vulnerabilities + +```bash +mvn dependency-check:check +# or +mvn dependency-check:aggregate +``` + +## Running tests + +```bash +./test.sh +``` + +This runs the unit and integration tests and produces JaCoCo code-coverage +reports. To run a single test, use Maven directly: + +```bash +mvn test -Dtest=ClassName#methodName +mvn integration-test -Dit.test=IntegrationTestClass +``` + +Next: [Configuration Reference](configuration.md). \ No newline at end of file diff --git a/docs/install/certificates.md b/docs/install/certificates.md new file mode 100644 index 00000000..6102eb0d --- /dev/null +++ b/docs/install/certificates.md @@ -0,0 +1,63 @@ +# Certificates & Keystore + +The CWS web console runs over TLS and needs a Tomcat **keystore** and +**truststore**, plus the keystore password. + +## Required files + +| File | Path | Purpose | +| --- | --- | --- | +| Keystore | `install/.keystore` | Tomcat server certificate. | +| Truststore | `install/tomcat_lib/cws_truststore.jks` | Trusted certificates. | +| Password | `~/.cws/creds` | Plaintext keystore password. | + +Lock down the credentials file so only you can read it: + +```bash +chmod 700 ~/.cws/ +chmod 400 ~/.cws/creds +``` + +See the [Apache Tomcat SSL How-To](https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html) +for background on Tomcat TLS. + +## Generate self-signed certificates + +For development or open-source use, generate a self-signed keystore and +truststore with the project's script: + +```bash +cd cws-certs +./generate-certs.sh +``` + +!!! warning + Running `generate-certs.sh` **replaces** the existing keystore and + truststore in `install/` with new certificates. + +## Certificates and the Docker image + +The published CWS Docker image ships with self-signed certificates that use the +default password `changeit`, so it runs without extra configuration. + +To use your own certificates with the image: + +1. Generate them with `generate-certs.sh`. +2. Make them available to the container — either copy them in before startup or + (more easily) use volume mounts. The `docker-compose.yml` in + `install/docker/` has commented-out volume lines for this. + +Inside the image, CWS looks for the keystore files at: + +``` +/home/cws_user/cws/server/apache-tomcat-11.0.20/conf/.keystore +/home/cws_user/cws/server/apache-tomcat-11.0.20/lib/cws_truststore.jks +``` + +Provide the keystore password as a plaintext file mounted at `/root/.cws/creds`. + +!!! note + This certificate password is **not** the password you use to log into the + CWS interface — it only unlocks the certificates themselves. + +Next: [Building from Source](building.md). diff --git a/docs/install/configuration.md b/docs/install/configuration.md new file mode 100644 index 00000000..4a99cb3b --- /dev/null +++ b/docs/install/configuration.md @@ -0,0 +1,42 @@ +# Configuration Reference + +CWS is configured with a properties file that you pass to the configurator. + +## How configuration is applied + +1. Copy the example configuration and fill in your values: + + ```bash + cp install/example-cws-configuration.properties my-cws-configuration.properties + # edit my-cws-configuration.properties, filling in the [YourXXX] placeholders + ``` + +2. Run the configurator with your file: + + ```bash + ./configure.sh my-cws-configuration.properties + ``` + +Preset defaults are drawn from `install/installerPresets.properties`, +`install/example-cws-configuration.properties`, and `utils.sh`. + +## Where to find every setting + +The complete, grouped list of configuration properties — host and installation, +database, security and LDAP, ports, messaging, email, Elasticsearch, history +retention, workers, and optional AWS settings — is documented in the reference: + +

Configuration Properties reference

+ +## Key choices at install time + +| Setting | Why it matters | +| --- | --- | +| `install_type` | Whether this host is a Console, a Worker, or both. | +| `database_*` | Connection to your MariaDB/MySQL schema. | +| `elasticsearch_*` | Connection to your Elasticsearch cluster. | +| `identity_plugin_type` / `cws_ldap_url` | Authentication backend (e.g. LDAP). | +| `history_level` | Must be identical across the Console and all Workers. | +| `hostname` / `amq_host` / `cws_console_host` | How components find each other. | + +Next: [Running & Stopping CWS](running.md). diff --git a/docs/install/database.md b/docs/install/database.md new file mode 100644 index 00000000..ed1502ee --- /dev/null +++ b/docs/install/database.md @@ -0,0 +1,57 @@ +# Database Setup + +CWS stores its state in a **MariaDB** or **MySQL** database. All database access +is centralized through `cws-core`'s `DbService`. This page shows a Dockerized +MariaDB suitable for development. + +## Run MariaDB in Docker + +Create a MariaDB container and a database instance for CWS: + +```bash +docker run -d -p 3306:3306 \ + -e MYSQL_DATABASE= \ + -e MYSQL_ROOT_PASSWORD= \ + -e TZ=Etc/UTC \ + --name mdb1011 mariadb:10.11 +``` + +- Replace `` with your desired database name (for example + `cws_dev`). +- Replace `` with a password of your choice. +- `TZ` sets the container timezone — set it to whatever your environment + requires (e.g. `Etc/UTC` or a region such as `America/New_York`). + +!!! important + The database name and password you use here must match the values in your + [build/configuration](building.md). + +## Connect to MariaDB + +```bash +mysql -h 127.0.0.1 -u root -p +``` + +Enter the password you set above when prompted. + +!!! note + Directly accessing MariaDB via the MySQL monitor assumes CWS has been built + (the build script carries the information required to access the database). + See [Building from Source](building.md). + +Make sure your CWS database (e.g. `cws_dev`) exists in the running MariaDB +instance before building CWS. + +## Presets & defaults + +Preset configuration variables (such as default SMTP and LDAP settings) live in: + +- `install/installerPresets.properties` +- `install/example-cws-configuration.properties` +- `utils.sh` + +See the [Configuration Reference](../reference/configuration.md) for the full +list of database settings (`database_type`, `database_host`, `database_port`, +`database_name`, `database_username`, `database_password`). + +Next: [Elasticsearch Setup](elasticsearch.md). diff --git a/docs/install/elasticsearch.md b/docs/install/elasticsearch.md new file mode 100644 index 00000000..c5bbdf8d --- /dev/null +++ b/docs/install/elasticsearch.md @@ -0,0 +1,41 @@ +# Elasticsearch Setup + +CWS uses **Elasticsearch 8.12.0+** for log and history aggregation. You can +point CWS at any externally-configured cluster, or run the provided Dockerized +Elasticsearch for development. + +## Run Elasticsearch in Docker + +In a terminal dedicated to Elasticsearch, start the provided compose stack: + +```bash +cd install/docker/es-only +docker-compose up -d +``` + +This is a self-contained way to run Elasticsearch and serves as an alternative +to installing it directly. + +## Connecting CWS to Elasticsearch + +CWS supports secure (HTTPS, with or without authentication) and insecure (HTTP) +clusters. The relevant [configuration settings](../reference/configuration.md#elasticsearch) +are: + +| Setting | Purpose | +| --- | --- | +| `elasticsearch_protocol` | `HTTP` or `HTTPS`. | +| `elasticsearch_host` | Cluster hostname. | +| `elasticsearch_port` | Port (default `9200`). | +| `elasticsearch_index_prefix` | Prefix for CWS indices. | +| `elasticsearch_use_auth` | `y`/`n` — whether the cluster requires auth. | +| `elasticsearch_username` / `elasticsearch_password` | Credentials when auth is enabled. | + +## History retention + +The amount of history CWS keeps in Elasticsearch (and the database) is governed +by `history_level` and `history_days_to_live`. All Console and Worker hosts +**must** use the same `history_level`. See +[Log & History Management](../administration/logs-history.md). + +Next: [Certificates & Keystore](certificates.md). diff --git a/docs/install/index.md b/docs/install/index.md new file mode 100644 index 00000000..d6bdee2b --- /dev/null +++ b/docs/install/index.md @@ -0,0 +1,40 @@ +# Installation + +This guide covers installing CWS for development or production from source. To +simply try CWS on one machine, use the +[Docker quickstart](../getting-started/quickstart.md) instead. + +## Overview + +A CWS installation involves: + +1. Meeting the [requirements](requirements.md) and installing the + [prerequisites](prerequisites.md). +2. Setting up a [database](database.md) (MariaDB or MySQL). +3. Setting up an [Elasticsearch](elasticsearch.md) cluster. +4. Providing [SSL certificates](certificates.md) (keystore and truststore). +5. [Building CWS](building.md) from source. +6. Supplying a [configuration](configuration.md) and running the configurator. +7. [Running and stopping](running.md) the console and workers. + +Optional / environment-specific topics: + +- [LDAP security](ldap.md) +- [Installing the Modeler](modeler.md) +- [Installation security considerations](security-considerations.md) + +## Installation types + +CWS can be installed as a **Console**, a **Worker**, or **both** on the same +host. You select this with the `install_type` configuration setting: + +| `install_type` | Meaning | +| --- | --- | +| `1` | Console and Worker | +| `2` | Console only | +| `3` | Worker only | + +A typical deployment has one Console host and one or more Worker hosts, all +pointing at a shared database, message broker, and Elasticsearch cluster. + +Start with [Requirements & Compatibility](requirements.md). diff --git a/docs/install/ldap.md b/docs/install/ldap.md new file mode 100644 index 00000000..a7246d2f --- /dev/null +++ b/docs/install/ldap.md @@ -0,0 +1,47 @@ +# LDAP Security + +CWS can authenticate users against an **LDAP** (or LDAPS) directory. This is +the recommended security mode for production deployments. + +## Configuration + +Set the following in your [configuration properties](../reference/configuration.md#security-authentication): + +| Property | Value | +| --- | --- | +| `identity_plugin_type` | `LDAP` | +| `cws_ldap_url` | Your LDAP(S) URL, e.g. `ldaps://:636` | +| `ldap_identity_plugin_class` | `org.camunda.bpm.identity.impl.ldap.plugin.LdapIdentityProviderPlugin` | +| `ldap_security_filter_class` | `jpl.cws.core.web.CwsLdapSecurityFilter` | +| `admin_user` | The LDAP username of the initial CWS administrator. | + +## How it works + +- On login, CWS authenticates the user against the configured LDAP server. +- The `admin_user` receives full administrative permissions at first login. +- Additional users and permissions are managed from the Camunda Admin page (see + [User Administration](../administration/users.md)). + +## OpenLDAP for development + +For development and testing without an enterprise LDAP server, CWS provides a +Dockerized OpenLDAP setup: + +```bash +cd cws-opensource-ldap +docker-compose up -d +``` + +See `cws-opensource-ldap/README.md` for LDIF customization and user +provisioning. + +## Camunda security mode (alternative) + +If LDAP is not available, CWS can use Camunda's built-in identity service +(`SECURITY="camunda"` in the dev script). In this mode, users are managed +entirely within Camunda's database rather than an external directory. + +## See also + +- [Security & Roles (admin)](../administration/security.md) +- [Custom Security Scheme Plugins](../developer/security-plugin.md) diff --git a/docs/install/modeler.md b/docs/install/modeler.md new file mode 100644 index 00000000..8227c6c0 --- /dev/null +++ b/docs/install/modeler.md @@ -0,0 +1,46 @@ +# Installing the Modeler + +The **CWS Modeler** is a desktop application for designing BPMN process +definitions that are then deployed to CWS. + +## Installation + +CWS provides install scripts for the modeler: + +=== "macOS" + + ```bash + cd install/modeler + ./install_mac_modeler.sh + ``` + +=== "Linux" + + ```bash + cd install/modeler + ./install_linux_modeler.sh + ``` + +The script downloads and configures the Camunda Modeler with CWS-specific +element templates (`elements.json`) so the built-in CWS task types appear in +the properties panel. + +## Using the modeler + +1. Open the modeler application. +2. Create or open a `.bpmn` file. +3. Drag elements onto the canvas — service tasks, gateways, events, etc. +4. For CWS-specific tasks, select the element and choose the CWS task type + from the properties panel (Command Line Execution, Email, REST GET, etc.). +5. Save the `.bpmn` file and [deploy it](../user-guide/deploying.md) to CWS. + +## CWS element templates + +The `elements.json` file (in `install/modeler/`) defines the CWS task-type +templates. If you add [custom tasks](../developer/custom-tasks.md), update this +file to make them available in the modeler's UI. + +## See also + +- [Modeling](../modeling/index.md) — best practices, examples, and tips. +- [Built-in Task Types](../developer/task-types.md) diff --git a/docs/install/prerequisites.md b/docs/install/prerequisites.md new file mode 100644 index 00000000..bfe4375f --- /dev/null +++ b/docs/install/prerequisites.md @@ -0,0 +1,91 @@ +# Prerequisites + +Install and prepare the following before building CWS. See +[Requirements & Compatibility](requirements.md) for version details. + +## Java 17 JDK + +CWS runs only on a **JDK 17** (not a JRE). + +On macOS with Homebrew, Amazon Corretto 17 works well: + +```bash +brew install --cask corretto@17 +``` + +Then point `JAVA_HOME` at it in your shell startup (for example `.zprofile`): + +```bash +export JAVA_HOME=$(/usr/libexec/java_home -v17) +``` + +## Maven + +Maven downloads project dependencies and drives the build. + +```bash +brew install maven +mvn -v # verify +``` + +## Docker + +[Docker](https://docs.docker.com/get-docker/) runs an external Elasticsearch +and the MariaDB database container. Recommended Docker resource allocation: + +- CPUs: 5 +- Memory: 14 GB +- Swap: 1 GB +- Disk image size: 64 GB + +## Database (MariaDB or MySQL) + +Set up MariaDB or MySQL on your local machine or a remote host, and create: + +- A database (schema) for CWS — `cws_dev` is a good default for development. +- A database user with full (CRUD) access to that schema. + +See [Database Setup](database.md) for a Dockerized MariaDB. + +## Elasticsearch 8.12.0+ + +CWS requires an externally-configured Elasticsearch cluster. You may use a +secure (HTTPS) cluster with or without authentication, or an insecure (HTTP) +one. The [Elasticsearch Setup](elasticsearch.md) page provides a Dockerized +option. + +## Logstash 8.12.0+ (temporary build input) + +Download Logstash for your platform, then (only if it's a `.tar.gz`) +decompress it and re-zip it as `logstash-8.12.0.zip`, and place it in +`install/logging/`. This is a temporary step in the current installation +process. Download from the +[Elastic downloads page](https://www.elastic.co/downloads/logstash). + +## SSL keystore, truststore, and password + +The CWS web console requires TLS. You need: + +- A Tomcat **keystore** at `install/.keystore`. +- A **truststore** at `install/tomcat_lib/cws_truststore.jks`. +- A **credentials file** at `~/.cws/creds` containing the keystore password. + +You can generate open-source self-signed certificates with the project's +`generate-certs.sh` script — see [Certificates & Keystore](certificates.md). +Lock down the credentials file: + +```bash +chmod 700 ~/.cws/ +chmod 400 ~/.cws/creds +``` + +See the [Apache Tomcat SSL How-To](https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html) +for background. + +## A note on terminal tooling + +The development build scripts open additional terminal windows. On macOS they +currently do this via iTerm2, so running them from +[iTerm2](https://iterm2.com/) is the smoothest experience during development. + +Next: [Database Setup](database.md). diff --git a/docs/install/requirements.md b/docs/install/requirements.md new file mode 100644 index 00000000..8ef04212 --- /dev/null +++ b/docs/install/requirements.md @@ -0,0 +1,51 @@ +# Requirements & Compatibility + +## Runtime & build + +| Requirement | Minimum | Notes | +| --- | --- | --- | +| **Java** | 17 (JDK) | JDK required — **not** a JRE. Compatible with 17 and 21. Required for Spring 7.x. | +| **Maven** | 3.9.6 | Required for modern plugin support. | +| **Docker** | current | Used for MariaDB and Elasticsearch containers; recommended 5 CPUs / 14 GB memory / 1 GB swap / 64 GB disk. | +| **Database** | MariaDB or MySQL | A schema plus a user with full (CRUD) access. | +| **Elasticsearch** | 8.12.0+ | An externally-configured cluster (secure with/without auth, or insecure HTTP). | + +## Core framework versions + +| Library | Current version | Compatible | Notes | +| --- | --- | --- | --- | +| Spring Framework | 7.0.6 | 7.0.x | Stable | +| Camunda BPM | 7.24.6-ee | 7.24.x | **Enterprise Edition — a Camunda license is required** | +| Java | 17 | 17, 21 | LTS | + +!!! warning "Camunda Enterprise license" + CWS builds against Camunda **Enterprise Edition** (`7.24.6-ee`). Resolving + these artifacts and running the engine requires a valid Camunda license and + access to Camunda's enterprise repository. + +## Testing dependencies + +| Library | Current version | Notes | +| --- | --- | --- | +| JUnit | 4.13.2 | Migration to JUnit 5 is a future consideration. | + +## Checking compatibility & security + +CWS uses several tools to keep dependencies healthy: + +- **Maven Enforcer Plugin** — enforces the Java 17 and Maven 3.9.6+ requirements + automatically at build time. +- **Versions Maven Plugin** — check for updates manually: + + ```bash + mvn versions:display-property-updates + ``` + +- **OWASP Dependency-Check** — scan for known CVEs manually: + + ```bash + mvn clean dependency-check:aggregate # aggregate report + mvn clean dependency-check:check # per-module reports + ``` + +Next: [Prerequisites](prerequisites.md). diff --git a/docs/install/running.md b/docs/install/running.md new file mode 100644 index 00000000..6801b7ce --- /dev/null +++ b/docs/install/running.md @@ -0,0 +1,41 @@ +# Running & Stopping CWS + +## Starting CWS + +During development, your [personal build script](building.md) builds CWS, +verifies configuration, and starts the console and workers for you. When +startup finishes, it prints a link to the console dashboard. + +Under the hood, the installed distribution provides scripts for starting and +stopping CWS: + +- `install/start_cws.sh` — start the console/worker(s). +- `install/stop_cws.sh` — stop them. + +## Stopping CWS + +To stop the development instance — bringing down the console and all local +workers: + +```bash +./stop_dev.sh +``` + +## Verifying the instance + +Once CWS is running: + +1. Open the console in a browser over HTTPS on the configured console port. +2. Log in with your configured administrator credentials. +3. From the console you can [deploy process definitions](../user-guide/deploying.md), + [launch and schedule processes](../user-guide/launching/index.md), and watch + [workers](../user-guide/console/workers.md) pick up external tasks. + +## Troubleshooting startup + +- Confirm the database exists and the `database_*` settings are correct. +- Confirm Elasticsearch is reachable with the configured protocol/host/port. +- Confirm the keystore, truststore, and `~/.cws/creds` password file are in + place (see [Certificates & Keystore](certificates.md)). +- Check the console and worker logs under the Tomcat `logs/` directory + (`cws.log` and `catalina.out`). diff --git a/docs/install/security-considerations.md b/docs/install/security-considerations.md new file mode 100644 index 00000000..1321803b --- /dev/null +++ b/docs/install/security-considerations.md @@ -0,0 +1,49 @@ +# Installation Security Considerations + +This page covers security-related decisions and best practices to keep in mind +during CWS installation. + +## TLS / SSL + +- CWS **requires** HTTPS for the console. Ensure your keystore and truststore + are in place before starting (see [Certificates & Keystore](certificates.md)). +- Use certificates signed by a trusted CA for production. Self-signed + certificates are acceptable only for development. +- Restrict the `~/.cws/creds` keystore password file to owner-only permissions + (`chmod 400`). + +## Network exposure + +- The CWS console ports (`cws_web_port`, `cws_ssl_port`) should only be + exposed to trusted networks. +- The message broker port (`amq_port`) and JMX ports should **not** be exposed + to the public internet. +- Use firewall rules or security groups to limit access to known hosts. + +## Authentication + +- Use **LDAP** in production (see [LDAP Security](ldap.md)). Camunda security + mode is for development only. +- Rotate the `admin_user` password after initial setup. +- Set `cws_token_expiration_hours` to a reasonable value (default 24 hours); + shorter is more secure. + +## Database + +- Use a dedicated database user for CWS with only the permissions it needs + (CRUD on its schema, not global admin). +- Secure the database connection — use network-level controls or SSL if the + database is on a remote host. + +## Elasticsearch + +- If using HTTPS Elasticsearch with authentication + (`elasticsearch_use_auth=y`), keep the credentials out of version control. +- Elasticsearch should not be directly accessible from untrusted networks. + +## Docker deployments + +- The default Docker image ships with the `changeit` keystore password — change + it for any non-development use. +- Mount your own certificates and credentials via Docker volumes rather than + embedding them in images. diff --git a/docs/macros.py b/docs/macros.py new file mode 100644 index 00000000..06628e6b --- /dev/null +++ b/docs/macros.py @@ -0,0 +1,55 @@ +"""MkDocs-macros hook for the CWS documentation site. + +Single-sources the CWS version so pages can reference ``{{ cws_version }}`` +instead of hard-coding a release number. Resolution order: + +1. ``CWS_VERSION`` environment variable (set by CI), then +2. the ``CWS_VER`` export in ``utils.sh`` at the repo root, then +3. ``"dev"`` as a last resort. + +mkdocs runs with its working directory set to the folder containing +``mkdocs.yml`` (the repo root), so ``utils.sh`` is readable directly. +""" + +import os +import re +import shutil + +_UTILS_SH = "utils.sh" +_VER_RE = re.compile(r"""CWS_VER=['"]?([^'"\s#]+)""") + +# Where the aggregated Javadoc is generated (maven-javadoc-plugin +# javadoc:aggregate default), overridable for CI. When present, it is copied +# into the built site under /javadoc so it ships with every mike version. +_JAVADOC_SRC = os.environ.get("CWS_JAVADOC_DIR", "target/site/apidocs") + + +def _version_from_utils(): + try: + with open(_UTILS_SH, "r", encoding="utf-8") as fh: + for line in fh: + m = _VER_RE.search(line) + if m: + return m.group(1) + except OSError: + pass + return None + + +def define_env(env): + version = os.environ.get("CWS_VERSION") or _version_from_utils() or "dev" + env.variables["cws_version"] = version + + +def on_post_build(env): + """Copy generated Javadoc into the built site under /javadoc. + + No-op when the Javadoc has not been generated (e.g. local doc-only builds + without a JDK), so the site still builds cleanly. CI generates the Javadoc + before building, so it is included in every published version. + """ + if not os.path.isdir(_JAVADOC_SRC): + return + site_dir = env.conf["site_dir"] + dest = os.path.join(site_dir, "javadoc") + shutil.copytree(_JAVADOC_SRC, dest, dirs_exist_ok=True) diff --git a/docs/modeling/best-practices.md b/docs/modeling/best-practices.md new file mode 100644 index 00000000..1892f6b6 --- /dev/null +++ b/docs/modeling/best-practices.md @@ -0,0 +1,91 @@ +# Modeling Best Practices + +These guidelines help you build process models that are maintainable, debuggable, and resilient in production CWS deployments. + +## Naming + +**Use descriptive, verb-phrase names for tasks.** A task name should describe what it does, not what it is. Prefer `Validate Input File` over `Task1` or `Validation`. + +**Use noun phrases for events.** Start events name the trigger (`File Arrived`, `Request Received`) and end events name the outcome (`Process Complete`, `Error Reported`). + +**Use consistent prefixes for similar tasks.** If you have multiple command-line tasks, name them consistently: `EXEC: Run Preprocessor`, `EXEC: Run Analysis`, `EXEC: Package Results`. This makes the Logs page easier to scan. + +**Keep process definition IDs lowercase with underscores.** The ID is used as a key throughout CWS and in log messages. `my_data_pipeline` is easier to filter on than `MyDataPipeline`. + +## Async Continuations + +**Set `camunda:asyncBefore="true"` on the Start Event.** This causes Camunda to commit the process instance to the database before any task executes. Without it, a crash during the first task can leave no trace in the database. + +**Use async continuations on tasks that run after gateways.** If a gateway splits into branches that run long operations, add `camunda:asyncBefore="true"` to the first task of each branch. This ensures the branch choice is persisted before execution begins. + +**Use `camunda:exclusive="false"` on multi-instance sub-processes.** This allows multiple instances to run in parallel on the same job executor thread pool, rather than being serialized. + +```xml + + 10 + +``` + +## Error Handling + +**Use Error Boundary Events for recoverable task failures.** Attach a boundary event to a service task to catch errors and route to a recovery or notification path, rather than leaving the instance stuck in an incident state. + +**Distinguish incidents from expected errors.** Use Error End Events (with named error codes) for business-level failures that have a defined handling path. Let unexpected exceptions become incidents so the Cockpit alerts you. + +**Use a Terminate End Event for fatal conditions.** When an unrecoverable error is detected and the whole process instance should stop, route to a Terminate End Event rather than a plain End Event. Terminate stops all active tokens; plain End only terminates the current path. + +**Log at the error boundary.** Add a Log Task (or Script Task with a logging call) on the error boundary path before routing to the error handling flow. This gives you a log message tied to the specific instance when the error occurred. + +## Subprocess Design + +**Use call activities to decompose large processes.** A process that would have more than about 15 tasks is usually clearer as a parent process with several call activities invoking child process definitions. This also makes testing individual stages easier. + +**Use embedded sub-processes to group related tasks.** If several tasks share a common error boundary or transaction scope, wrap them in an embedded sub-process. Boundary events on the sub-process apply to all tasks inside it. + +**Keep child processes self-contained.** Child processes called via call activity should work from variables passed in at launch — they should not depend on implicit knowledge of the parent process's variable names. + +**Use input/output variable mappings on call activities.** Explicitly map what goes in and comes out: + +```xml + + + + + + +``` + +This makes data flow explicit and prevents variable namespace collisions. + +## Gateway Usage + +**Prefer Exclusive Gateways (XOR) for single-path decisions.** Use Exclusive Gateways when exactly one outgoing path should be taken. Always set a default sequence flow to catch unexpected conditions. + +**Use Parallel Gateways (AND) for fan-out/join, not XOR.** When multiple paths should all execute, use a Parallel Gateway. The joining Parallel Gateway waits for all branches to complete before proceeding. + +**Name gateway conditions.** Label each outgoing sequence flow from an Exclusive Gateway (e.g. `success`, `failure`, `skip`). Anonymous flows make diagrams hard to read. + +## Variable Management + +**Set variables explicitly in Script Tasks.** Avoid relying on side effects to create variables. Use `execution.setVariable("name", value)` explicitly. + +**Prefer process-scope variables over local variables for shared state.** Local variables are visible only within the current scope (sub-process, call activity). If downstream tasks need a value, set it on the execution scope that encompasses those tasks. + +**Initialize variables at the start of a process.** If a later task depends on a variable that might not exist (e.g. an optional initiator variable), initialize it to a safe default in the first Script Task. + +## History and Cleanup + +**Always set `camunda:historyTimeToLive`.** Camunda requires this on every process definition. Set it to a value that matches your operational retention needs (e.g. `30` days for 30 days of history). + +```xml + +``` + +**Remove completed instances from Elasticsearch.** CWS's `history_days_to_live` property governs automatic cleanup of both database history and Elasticsearch log indices. + +## Further reading + +- [Camunda 7.24 BPMN Reference](https://docs.camunda.org/manual/7.24/reference/bpmn20/) +- [Camunda 7.24 Best Practices](https://docs.camunda.org/manual/7.24/user-guide/process-engine/transactions-in-processes/) diff --git a/docs/modeling/dmn.md b/docs/modeling/dmn.md new file mode 100644 index 00000000..0a1c969b --- /dev/null +++ b/docs/modeling/dmn.md @@ -0,0 +1,147 @@ +# DMN Examples + +CWS supports DMN (Decision Model and Notation) decision tables, executed by the Camunda DMN engine. Decision tables externalize conditional logic from BPMN, making rules easier to read and maintain without modifying the process diagram. + +## What is a Decision Table? + +A DMN decision table maps input conditions to output values. Each row is a rule: if all input conditions in the row match, the corresponding outputs are returned. + +This is useful for: +- Routing decisions based on multiple conditions (e.g., priority level + data type → processing queue) +- Deriving configuration values from input parameters +- Replacing nested exclusive gateways with a more readable rules table + +## Creating a Decision Table + +Use the Camunda Modeler to create `.dmn` files. Deploy them to CWS the same way as BPMN files. + +1. Open Camunda Modeler +2. File > New File > DMN Table +3. Define input and output columns +4. Add rules (rows) +5. Save as `my_decision.dmn` + +## Hit Policies + +The hit policy determines what happens when multiple rules match: + +| Hit Policy | Behavior | +|------------|----------| +| `UNIQUE` | Exactly one rule may match; error if multiple match | +| `FIRST` | Returns the first matching rule (top to bottom) | +| `ANY` | Multiple rules may match, all must produce the same output | +| `COLLECT` | Returns all matching outputs as a list | + +For most routing decisions, `FIRST` is the most intuitive choice. + +## Calling a Decision Table from BPMN + +Use a **Business Rule Task** in your BPMN diagram to invoke a DMN decision: + +```xml + + from_start + to_gateway + +``` + +| Attribute | Purpose | +|-----------|---------| +| `camunda:decisionRef` | The ID of the deployed DMN decision definition | +| `camunda:resultVariable` | Process variable name to store the result | +| `camunda:mapDecisionResult` | How the result is mapped: `singleEntry`, `singleResult`, `collectEntries`, `resultList` | + +Use `singleEntry` when the table returns a single output column and one row. Use `resultList` when multiple rows may match (`COLLECT` hit policy). + +## Example: Priority-Based Routing + +This decision table maps a `priority` input to a `queue` output: + +| Input: `priority` | Output: `queue` | +|-------------------|----------------| +| `"high"` | `"urgent"` | +| `"medium"` | `"standard"` | +| `"low"` | `"batch"` | +| _(any other)_ | `"standard"` | + +In the BPMN, after the Business Rule Task: + +``` +${routingResult == "urgent"} → high-priority handler +${routingResult == "standard"} → normal handler +${routingResult == "batch"} → batch handler +``` + +## Example: Multi-Input Decision + +A decision table with two inputs — `fileSize` (in MB) and `processingMode` — determines `workerThreads`: + +| Input: `fileSize` | Input: `processingMode` | Output: `workerThreads` | +|-------------------|------------------------|------------------------| +| `< 100` | `"fast"` | `4` | +| `< 100` | `"safe"` | `2` | +| `>= 100` | `"fast"` | `8` | +| `>= 100` | `"safe"` | `4` | + +The DMN engine evaluates both conditions for each row. Use FEEL (Friendly Enough Expression Language) syntax for input expressions: + +| FEEL expression | Meaning | +|-----------------|---------| +| `< 100` | Less than 100 | +| `>= 100` | Greater than or equal to 100 | +| `"fast"` | String equality | +| `"fast", "express"` | String in set | +| _(blank)_ | Always matches (wildcard) | + +## DMN in the BPMN XML + +A full Business Rule Task with DMN call: + +```xml + + from_init + to_process + +``` + +`camunda:decisionRefBinding="latest"` always uses the most recently deployed version of the decision (recommended for most cases). + +## Accessing Results + +After the Business Rule Task, the result is available as a process variable: + +``` +// singleEntry — result is a scalar value +${workerThreads} + +// collectEntries — result is a list +${routeList[0]} +``` + +In a Script Task following the Business Rule Task: + +```groovy +// Access a singleEntry result +def threads = execution.getVariable("workerThreads") +execution.setVariable("maxThreads", threads) +``` + +## Testing Decision Tables + +The Camunda Modeler has a built-in test runner for DMN files. Open the `.dmn` file, click **Decision Table**, and use the test cases panel to provide sample inputs and verify expected outputs before deploying. + +## Further reading + +- [Camunda 7.24 DMN Reference](https://docs.camunda.org/manual/7.24/reference/dmn/) +- [Camunda 7.24 Business Rule Task](https://docs.camunda.org/manual/7.24/reference/bpmn20/tasks/business-rule-task/) +- [FEEL Language Reference](https://docs.camunda.org/manual/7.24/reference/dmn/feel/) \ No newline at end of file diff --git a/docs/modeling/examples.md b/docs/modeling/examples.md new file mode 100644 index 00000000..96769a4d --- /dev/null +++ b/docs/modeling/examples.md @@ -0,0 +1,148 @@ +# BPMN Examples + +CWS ships with a set of example process definitions in `install/dev/bpmn/`. These can be deployed to a running CWS instance for testing and as starting points for new processes. + +## Available Examples + +| File | Description | +|------|-------------| +| `log_core_vars.bpmn` | Logs CWS console host and port using built-in CWS variables | +| `cmd_sleep_n.bpmn` | Runs a command-line sleep task with a configurable duration | +| `simple_sleep_30.bpmn` | Minimal process: start → 30-second sleep → end | +| `external_pwd.bpmn` | External task that runs `pwd` as a command-line task | +| `test_parallel_gateway.bpmn` | Demonstrates fan-out/join with a Parallel Gateway | +| `test_parallel_gateway_sync_tasks.bpmn` | Parallel gateway with synchronized task completion | +| `test_parallel_with_subprocess.bpmn` | Multi-instance sub-process calling a child process definition | +| `test_error_handling.bpmn` | Comprehensive error handling patterns (boundary events, throw/catch) | +| `test_error_end_event.bpmn` | Error End Event that propagates to a parent process | +| `message_passing_example_parent.bpmn` | Parent process using message events to communicate with a child | +| `message_passing_example_child.bpmn` | Child process that receives and sends messages | +| `test_model_outputs.bpmn` | Demonstrates output variable mapping from service tasks | +| `test_out_var.bpmn` | Simple output variable capture from an external task | +| `test_schedule_task.bpmn` | Uses a timer intermediate event for scheduled execution | + +## Annotated Examples + +### Logging CWS context variables (`log_core_vars.bpmn`) + +This is the simplest useful process — it logs the console hostname and port, demonstrating how to use CWS built-in variables: + +```xml + + + + to_log + + + + + + + + + none + + + ABORT_PROCESS + + + to_log + to_end + + + + to_end + + +``` + +Key points: +- `camunda:historyTimeToLive="30"` is set on the process — required by Camunda +- `camunda:asyncBefore="true"` on the Start Event ensures the instance is persisted before execution +- `preCondition` is set to `none` to skip pre-condition evaluation + +### Parallel sub-process with call activity (`test_parallel_with_subprocess.bpmn`) + +This pattern runs a child process definition N times in parallel: + +```xml + + + 100 + + + + to_child + + + + to_child + to_sub_end + + + + to_sub_end + + +``` + +The `camunda:exclusive="false"` attribute allows multiple instances to run concurrently on the job executor thread pool. + +### Command-line task (`external_pwd.bpmn`) + +The `CmdLineExecTask` is a CWS external task that executes a shell command: + +```xml + + + + pwd + + + . + + + 0 + + + true + + + +``` + +- `camunda:topic="__CWS_CMD_TOPIC__"` routes the task to the CWS command executor +- `successExitValues` lists exit codes that are considered successful (comma-separated) +- `throwOnFailures=true` causes a non-success exit code to throw an error (and create an incident) + +### Error handling (`test_error_handling.bpmn`) + +This example demonstrates multiple error-handling scenarios: + +- A task that succeeds normally +- A task with `throwOnFailures=false` that continues even on failure +- A task with an **Error Boundary Event** that catches the error and routes to a recovery path +- An **Error End Event** that propagates the error to the parent process + +Deploy this process to observe how CWS handles each scenario in the Cockpit and Logs pages. + +## Deploying the Examples + +To deploy any example to your running CWS instance: + +1. Open the CWS web console +2. Navigate to **Processes** +3. Click **Deploy Process** +4. Select the `.bpmn` file from `install/dev/bpmn/` + +Some examples (like `test_parallel_with_subprocess.bpmn`) depend on a child process definition (`test.bpmn`) also being deployed. Deploy the child first. \ No newline at end of file diff --git a/docs/modeling/index.md b/docs/modeling/index.md new file mode 100644 index 00000000..862b7221 --- /dev/null +++ b/docs/modeling/index.md @@ -0,0 +1,26 @@ +# Modeling + +CWS uses BPMN 2.0 as its process definition language, executed by the Camunda 7.24 engine. This section covers how to design, organize, and deploy process models that run reliably in CWS. + +## Topics + +| Page | What it covers | +|------|---------------| +| [Best Practices](best-practices.md) | Naming conventions, error handling, async continuations, subprocess design | +| [Tips & Tricks](tips.md) | Practical modeler tips for common CWS patterns | +| [Parallel Sub-Processes](parallel-subprocess.md) | Running tasks in parallel using multi-instance sub-processes | +| [BPMN Examples](examples.md) | Annotated examples from the CWS example library | +| [DMN Examples](dmn.md) | Decision table examples for routing and rule evaluation | +| [Script Task Recipes](script-recipes.md) | Groovy and JavaScript snippets for common scripting needs | + +## Tooling + +Models are created with the **Camunda Modeler** desktop application, which produces `.bpmn` and `.dmn` files. Deploying a model to CWS is done through the CWS web console (Processes page > Deploy) or via the REST API. + +Download the Camunda Modeler from [camunda.com/download/modeler/](https://camunda.com/download/modeler/). + +## CWS-specific conventions + +- Every process definition must have `camunda:historyTimeToLive` set (in days) to comply with Camunda's history cleanup requirements. Use the process properties panel or set it in the XML: `camunda:historyTimeToLive="30"`. +- The Start Event should have `camunda:asyncBefore="true"` to ensure the process instance is committed to the database before execution begins. +- External tasks use the topic `__CWS_CMD_TOPIC__` for command-line execution tasks. diff --git a/docs/modeling/parallel-subprocess.md b/docs/modeling/parallel-subprocess.md new file mode 100644 index 00000000..3121291a --- /dev/null +++ b/docs/modeling/parallel-subprocess.md @@ -0,0 +1,129 @@ +# Parallel Tasks & Sub-Processes + +Running tasks in parallel is one of the most common modeling needs in CWS. This page covers the two main approaches: Parallel Gateways for a fixed set of branches, and Multi-Instance Sub-Processes for a dynamic number of parallel executions. + +## Parallel Gateway (fixed branches) + +Use a Parallel Gateway when you have a known, fixed number of branches that should all execute simultaneously. + +``` +Start → [task A] → ╔══╗ → [branch 1] ─┐ + ║AND║ ├→ ╔══╗ → [join task] → End + ╚══╝ → [branch 2] ─┘ ╚══╝ +``` + +The opening Parallel Gateway fans out to all branches. The closing Parallel Gateway waits for all branches to complete before the flow continues. + +**Key points:** + +- All outgoing paths from a Parallel Gateway are taken unconditionally — no conditions on sequence flows +- The joining Parallel Gateway blocks until every incoming branch arrives +- If any branch fails, the instance goes to an incident state; fix and retry + +**BPMN example (`test_parallel_gateway.bpmn`):** + +The `test_parallel_gateway.bpmn` example in `install/dev/bpmn/` demonstrates this pattern. It fans out from a Parallel Gateway into several concurrent paths and rejoins at a second Parallel Gateway before the End Event. + +## Multi-Instance Sub-Process (dynamic parallelism) + +Use a Multi-Instance Sub-Process when you need to run the same sub-process body N times in parallel, where N may be determined at runtime. + +This is the pattern used by `test_parallel_with_subprocess.bpmn` in `install/dev/bpmn/`. + +### Fixed cardinality + +To run exactly N parallel iterations, set the **Loop Cardinality** in the sub-process multi-instance properties: + +```xml + + + 100 + + + +``` + +Setting `camunda:exclusive="false"` is critical — it allows multiple instances to be acquired by the job executor pool simultaneously rather than being serialized. + +### Collection-based iteration + +To iterate over a list variable (e.g. a list of file names), use the collection and element variable properties: + +```xml + + +``` + +Inside the sub-process, `${currentFile}` holds the value for the current iteration. + +### Calling a child process from each iteration + +A common pattern is to have the sub-process body contain a Call Activity that invokes a separate child process definition for each item. This keeps the child process definition simple and testable independently: + +```xml + + + ${itemCount} + + + + to_child + + + + to_child + to_sub_end + + + + to_sub_end + + +``` + +Setting `camunda:asyncAfter="true"` on the sub-process End Event gives the job executor a checkpoint after each iteration completes. + +## Worker Concurrency Considerations + +Parallel execution depends on having sufficient executor threads. If you launch 50 parallel sub-processes but the worker has only 4 executor threads, at most 4 will run simultaneously. + +Match your parallelism to your worker configuration: + +- Set the worker's executor thread count high enough to allow meaningful parallelism +- Use the **Limit** field on the Workers page to cap how many instances of the parent process run at once +- Each parallel branch uses one thread for the duration of the task; short tasks can share threads more efficiently than long-running ones + +## Collecting Results from Parallel Branches + +To aggregate output from parallel iterations, use a collection output variable: + +```xml + + +``` + +After all iterations complete, `${results}` contains the list of collected output values. + +## Error Handling in Parallel Branches + +By default, an error in one parallel branch creates an incident for that instance but does not affect other branches. The joining gateway will not complete until all branches either finish or are resolved. + +To cancel all remaining branches when one fails, use an **Error Boundary Event** on the sub-process itself with a Terminate End Event downstream. This cancels the entire sub-process scope, including all active parallel instances. + +## Further reading + +- [Camunda 7.24 Multi-Instance](https://docs.camunda.org/manual/7.24/reference/bpmn20/tasks/task-markers/#multi-instance) +- [Camunda 7.24 Parallel Gateway](https://docs.camunda.org/manual/7.24/reference/bpmn20/gateways/parallel-gateway/) +- [Camunda 7.24 Sub-Processes](https://docs.camunda.org/manual/7.24/reference/bpmn20/subprocesses/) \ No newline at end of file diff --git a/docs/modeling/script-recipes.md b/docs/modeling/script-recipes.md new file mode 100644 index 00000000..ec02c8e5 --- /dev/null +++ b/docs/modeling/script-recipes.md @@ -0,0 +1,236 @@ +# Script Task Recipes + +Script Tasks in CWS use either **Groovy** or **JavaScript** (Nashorn engine). These recipes cover the most common scripting needs. All examples use Groovy unless noted. + +## Setting a Variable + +```groovy +execution.setVariable("myVar", "someValue") +``` + +Set a numeric or boolean: + +```groovy +execution.setVariable("count", 42) +execution.setVariable("isReady", true) +``` + +Set a variable from an expression result: + +```groovy +def path = execution.getVariable("inputDir") + "/" + execution.getVariable("filename") +execution.setVariable("fullPath", path) +``` + +## Reading a Variable + +```groovy +def inputFile = execution.getVariable("inputFile") +``` + +Read with a default value if the variable might not exist: + +```groovy +def retries = execution.getVariable("retryCount") ?: 0 +execution.setVariable("retryCount", retries + 1) +``` + +## Posting a Log Message + +CWS log messages from Script Tasks are written through the SLF4J logger that Camunda injects: + +```groovy +execution.setVariable("logMessage", "Processing file: " + execution.getVariable("inputFile")) +``` + +Alternatively, use a Log Task (service task with `camunda:class="jpl.cws.task.LogTask"`) for structured log output — this is the preferred approach when logging is the sole purpose of the step. + +For debug output within a Script Task, write to standard output (visible in worker logs): + +```groovy +println "DEBUG: inputFile = " + execution.getVariable("inputFile") +``` + +## Parsing JSON from a File + +```groovy +import groovy.json.JsonSlurper + +def filePath = execution.getVariable("jsonFilePath") +def jsonText = new File(filePath).text +def data = new JsonSlurper().parseText(jsonText) + +execution.setVariable("recordCount", data.records.size()) +execution.setVariable("firstRecord", data.records[0].id) +``` + +For a JSON file with this structure: + +```json +{ + "records": [ + {"id": "rec001", "status": "ready"}, + {"id": "rec002", "status": "pending"} + ] +} +``` + +## Parsing JSON from a REST Response + +When following a REST Task, the response body is available as a process variable (configured in the REST Task's output variable field, typically `responseBody`): + +```groovy +import groovy.json.JsonSlurper + +def responseBody = execution.getVariable("responseBody") +def data = new JsonSlurper().parseText(responseBody) + +execution.setVariable("statusCode", data.status) +execution.setVariable("resultId", data.result.id) +``` + +Check the HTTP status before parsing: + +```groovy +import groovy.json.JsonSlurper + +def statusCode = execution.getVariable("httpStatusCode") as Integer +if (statusCode == 200) { + def body = new JsonSlurper().parseText(execution.getVariable("responseBody")) + execution.setVariable("result", body.data) +} else { + execution.setVariable("result", null) + throw new RuntimeException("REST call failed with status: " + statusCode) +} +``` + +## Building a JSON Payload + +```groovy +import groovy.json.JsonOutput + +def payload = [ + jobId: execution.getVariable("jobId"), + inputFile: execution.getVariable("inputFile"), + timestamp: new Date().toInstant().toString() +] + +execution.setVariable("requestBody", JsonOutput.toJson(payload)) +``` + +## Constructing a File Path + +```groovy +def baseDir = execution.getVariable("outputDir") +def filename = execution.getVariable("productName") + "_" + + new Date().format("yyyyMMdd") + ".dat" + +execution.setVariable("outputPath", baseDir + "/" + filename) +``` + +## Working with Lists + +Collect values into a list across multiple tasks: + +```groovy +// Initialize (typically in first script task) +def results = [] +execution.setVariable("results", results) +``` + +Append to the list in a later task: + +```groovy +def results = execution.getVariable("results") +results.add(execution.getVariable("latestResult")) +execution.setVariable("results", results) +``` + +Iterate over a list variable: + +```groovy +def fileList = execution.getVariable("fileList") +fileList.each { file -> + println "File: " + file +} +``` + +## Checking if a Variable Exists + +```groovy +def val = execution.getVariableLocal("optionalVar") +if (val == null) { + execution.setVariable("optionalVar", "default") +} +``` + +## JavaScript (Nashorn) Snippets + +For teams that prefer JavaScript, Camunda's Nashorn engine supports ECMAScript 5.1: + +```javascript +// Set a variable +execution.setVariable("myVar", "value"); + +// Read a variable +var inputFile = execution.getVariable("inputFile"); + +// Increment a counter +var count = execution.getVariable("count"); +execution.setVariable("count", count + 1); + +// Check condition and set result +var status = execution.getVariable("status"); +execution.setVariable("isSuccess", status === "OK"); +``` + +!!! note + The Nashorn JavaScript engine (included with Java) is deprecated as of Java 15 and removed in Java 17. Use Groovy for new script tasks in CWS deployments running on Java 17. + +## CWS Code Snippets (CustomMethods) + +For logic that needs to be reusable across multiple process definitions, use the CWS Snippets mechanism instead of Script Tasks. Snippets are Java methods in the `CustomMethods` class, editable from the CWS web console under **Snippets**. + +Call a snippet in any BPMN expression: + +``` +${cws.methodName(arg1, arg2)} +``` + +Example — echo a value: + +```java +package jpl.cws.core.code; + +public class CustomMethods { + public String echo(String arg1) { + return arg1; + } +} +``` + +Example — generate a UUID: + +```java +package jpl.cws.core.code; +import java.util.UUID; + +public class CustomMethods { + public String getRandUuid() { + return UUID.randomUUID().toString(); + } +} +``` + +Use snippets in a sequence flow condition: + +``` +${cws.isDataReady() == "true"} +``` + +Snippets update dynamically — no process redeployment needed when you modify the snippet. + +## Further reading + +- [Camunda 7.24 Script Tasks](https://docs.camunda.org/manual/7.24/user-guide/process-engine/scripting/) +- [Camunda 7.24 Expression Language](https://docs.camunda.org/manual/7.24/user-guide/process-engine/expression-language/) \ No newline at end of file diff --git a/docs/modeling/tips.md b/docs/modeling/tips.md new file mode 100644 index 00000000..2e3cc978 --- /dev/null +++ b/docs/modeling/tips.md @@ -0,0 +1,119 @@ +# Modeling Tips & Tricks + +Practical tips for working with the Camunda Modeler and deploying processes in CWS. + +## Modeler shortcuts + +| Action | Shortcut | +|--------|----------| +| Open properties panel | Click element, then `F4` or the wrench icon | +| Quick-create connected task | Click element, hover edge, click the task icon | +| Align selected elements | Select multiple, right-click > Align | +| Toggle XML editor | Click the code button in the toolbar | +| Validate diagram | Click the checkmark button; errors show in the bottom panel | + +## Setting the asyncBefore flag + +Select the Start Event or any task, open the properties panel, go to the +**General** tab, and check **Asynchronous Before** under the job configuration +section. + +This should be set on the Start Event of every process deployed to CWS so that +the process engine creates a save point before execution begins. + +## Using CWS built-in variables in expressions + +CWS exposes several variables accessible in BPMN expression language using the +dollar-brace syntax: + +- `cws.hostname` — Hostname of the CWS console +- `cws.port` — Port of the CWS console + +These are useful in Log Task messages, for example: +`Console is at` followed by `cws.hostname` and `cws.port` references. + +## Calling code snippets from BPMN + +CWS snippets are Java methods in the `CustomMethods` class, callable via +dollar-brace expressions (e.g. `cws.methodName(arg1, arg2)`) in any expression +field — sequence flow conditions, task parameters, log messages, etc. + +Changes to snippets take effect immediately — no redeployment needed. + +## Using pre-conditions on tasks + +CWS service tasks support a `preCondition` field that is evaluated before the +task runs. If the condition evaluates to false, the `onPreConditionFail` action +is taken instead of executing the task. + +| `onPreConditionFail` value | Behavior | +|---------------------------|----------| +| `SKIP_TASK` | Skip this task and continue the flow | +| `ABORT_PROCESS` | Terminate the process instance | + +Set `preCondition` to `none` to disable this check (default behavior). + +## Deploying a process + +From the CWS console: + +1. Go to the **Deployments** page. +2. Click **Deploy Process**. +3. Select your `.bpmn` file. +4. Confirm the deployment. + +On success, the new version appears in the process list. Existing running +instances continue on their current version. + +## Inspecting a running instance in Cockpit + +1. Click **Cockpit** in the console navigation. +2. Select the process definition. +3. Find the instance in the list (filter by business key if needed). +4. Click the instance ID to open the detail view. + +The detail view shows: + +- Current token positions on the BPMN diagram (blue circles with count). +- All process variables with their current values. +- Any active incidents with the error message and stack trace. +- The full audit trail of completed activities. + +## Testing with Camunda Tasklist + +You can manually start a process instance from the **Tasklist** page. This is +useful for testing a process end-to-end with specific variables. + +!!! note + Instances started via Tasklist are not recorded in the CWS Processes tab. + Use this only for testing; production triggering should go through CWS + initiators or the REST API. + +## Viewing process variables mid-run + +In Cockpit, select a running instance, then look at the **Variables** panel. +You can see all current variable values and their scopes. This is the fastest +way to check why a gateway took an unexpected path. + +## Incident resolution + +When a task fails with an unhandled exception, Camunda creates an **Incident**. +Incidents appear in Cockpit as red markers on the diagram. To resolve: + +1. Fix the root cause (repair a file, fix a script, update a variable). +2. In Cockpit, click the incident and choose **Retry**. +3. The task will execute again from the beginning. + +If you need to update a variable before retrying, use the Variables panel to +edit it, then retry the incident. + +## Cleaning up stuck instances + +If a process instance is stuck and cannot be recovered, cancel it from Cockpit: + +1. Select the instance. +2. Click **Cancel Process Instance**. + +This terminates the instance and marks it as cancelled in the history. Any +resources held by external tasks will be released when the worker's lock timeout +expires. diff --git a/docs/reference/changelog.md b/docs/reference/changelog.md new file mode 100644 index 00000000..28022a9e --- /dev/null +++ b/docs/reference/changelog.md @@ -0,0 +1,6 @@ +# Changelog + +The full CWS changelog is maintained in the repository root and reproduced +below. + +--8<-- "CHANGELOG.md" diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md new file mode 100644 index 00000000..b69860a4 --- /dev/null +++ b/docs/reference/configuration.md @@ -0,0 +1,121 @@ +# Configuration Properties + +CWS is configured with a properties file passed to the configurator +(`./configure.sh .properties`). A complete, annotated +starting point ships in the repository at +[`install/example-cws-configuration.properties`](https://github.com/NASA-AMMOS/common-workflow-service/blob/main/install/example-cws-configuration.properties). + +Copy that file, fill in the values marked `[YourXXX]`, and run the configurator. +The tables below group the settings by area. Values shown are the example +defaults; adjust them for your environment. + +## Host & installation + +| Property | Description | +| --- | --- | +| `hostname` | Hostname of the machine you are installing CWS on (Console, Worker, or both). Must be reachable by all other components. | +| `install_type` | `1` = Console and Worker, `2` = Console only, `3` = Worker only. | +| `cws_console_host` | Host running the Console. Only needed when installing a non-console (Worker-only) host. | +| `project_webapp_root` | Optional. Name of a webapp created inside the CWS web server, accessible without CWS security; logging in redirects to its `index.html`. | +| `brand_header` | Text shown in the console header. | + +## Database + +| Property | Description | +| --- | --- | +| `database_type` | `mariadb` or `mysql`. | +| `database_host` | Database hostname; must be reachable by all Workers and the Console. | +| `database_port` | Database port (default `3306`). | +| `database_name` | Database schema name. | +| `database_username` | User with CRUD access to the schema. | +| `database_password` | Password for the database user. | + +## Security & authentication + +| Property | Description | +| --- | --- | +| `identity_plugin_type` | Identity backend, e.g. `LDAP`. | +| `cws_ldap_url` / `cws_ldap_url_default` | LDAP(S) URL, e.g. `ldaps://:636`. | +| `ldap_identity_plugin_class` | Camunda LDAP identity provider plugin class. | +| `ldap_security_filter_class` | CWS LDAP security filter class. | +| `camunda_security_filter_class` | CWS Camunda security filter class. | +| `admin_user` | LDAP username of the initial CWS administrator. | +| `admin_firstname`, `admin_lastname`, `admin_email` | Administrator identity (required with Camunda security). | +| `cws_token_expiration_hours` | Hours a CWS security token stays valid before re-authentication (default `24`). | + +## Network ports + +| Property | Default | Description | +| --- | --- | --- | +| `cws_web_port` | `38080` | HTTP port. | +| `cws_ssl_port` | `38443` | HTTPS port. | +| `cws_ajp_port` | `38009` | AJP port. | +| `cws_shutdown_port` | `38005` | Tomcat shutdown port. | +| `cws_console_ssl_port` | `38443` | Console HTTPS port. | +| `amq_port` | `31616` | Message broker port. | +| `cws_amq_jmx_port` | `37099` | Broker JMX port. | +| `cws_jmx_port` | `31099` | CWS JMX port. | + +## Messaging (broker) + +| Property | Description | +| --- | --- | +| `amq_host` | Host of the CWS message broker. For a Console, use the same value as `hostname`; for a Worker, use the Console's hostname. | + +## Email & notifications + +| Property | Description | +| --- | --- | +| `notify_users_email` | `y`/`n` — email users when assigned a task. | +| `email_subject` | Task-assignment email subject template (supports `CWS_*` tokens). | +| `email_body` | Task-assignment email body template. | +| `smtp_hostname` | SMTP server hostname. | +| `smtp_port` | SMTP port (default `25`). | +| `cws_notification_emails` | Comma-separated addresses that receive alerts for major system errors (DB/JMS/auth failures). | + +## Elasticsearch + +| Property | Description | +| --- | --- | +| `elasticsearch_protocol` | `HTTP` or `HTTPS`. | +| `elasticsearch_host` | Elasticsearch hostname. | +| `elasticsearch_port` | Port (default `9200`). | +| `elasticsearch_index_prefix` | Prefix for CWS indices. | +| `elasticsearch_use_auth` | `y`/`n` — whether the cluster requires authentication. | +| `elasticsearch_username`, `elasticsearch_password` | Credentials when `elasticsearch_use_auth=y`. | +| `user_provided_logstash` | `y`/`n` — if `y`, CWS will not install or start its own Logstash. | + +## History & retention + +| Property | Description | +| --- | --- | +| `history_level` | Amount of history stored: `none`, `activity`, `audit`, or `full`. Console and all Workers **must** use the same value. See the [Camunda history docs](https://docs.camunda.org/manual/7.24/user-guide/process-engine/history/). | +| `history_days_to_live` | Days to keep history (process instance history, log files, Elasticsearch indices) before automatic purge. High values can consume large disk over time. | + +## Workers + +| Property | Description | +| --- | --- | +| `worker_max_num_running_procs` | Max actively running process instances per worker (integer ≥ 1; default `16`). Configurable per worker in the `cws_worker` DB table after install. | +| `worker_abandoned_days` | Days before an unseen worker's row is cleaned from the `cws_workers` table (integer ≥ 1). | +| `startup_autoregister_process_defs` | Whether to auto-register process definitions at startup. | + +## AWS (optional autoscaling & S3/SQS initiators) + +These settings are only relevant when running on AWS with autoscaling or the +S3/SQS initiators. They are optional and disabled by default. + +| Property | Description | +| --- | --- | +| `cws_enable_cloud_autoscaling` | `y`/`n` — publish metrics to CloudWatch for autoscaling. Requires a valid CloudWatch endpoint. | +| `aws_cloudwatch_endpoint` | CloudWatch endpoint used when autoscaling is enabled. | +| `metrics_publishing_interval` | Seconds between metric publications (default `10`). | +| `aws_default_region` | Default AWS region (defaults to `us-west-2`). | +| `aws_sqs_dispatcher_sqsUrl` | SQS queue URL for the S3 initiator. | +| `aws_sqs_dispatcher_msgFetchLimit` | SQS fetch limit, `1`–`10` (default `1`). | + +!!! tip + Presets and default values also live in + [`install/installerPresets.properties`](https://github.com/NASA-AMMOS/common-workflow-service/blob/main/install/installerPresets.properties) + and `utils.sh`. See [Configuration Reference](../install/configuration.md) in + the installation guide for how these are applied. diff --git a/docs/reference/external.md b/docs/reference/external.md new file mode 100644 index 00000000..c15c878c --- /dev/null +++ b/docs/reference/external.md @@ -0,0 +1,27 @@ +# External Resources + +CWS is built on the Camunda 7 BPMN engine. These upstream resources are useful +alongside the CWS documentation. The Camunda links are pinned to **7.24**, the +version CWS currently builds against. + +## Camunda & BPMN + +| Resource | Description | +| --- | --- | +| [Camunda BPMN 2.0 Implementation Reference](https://docs.camunda.org/manual/7.24/reference/bpmn20/) | Reference for the BPMN elements CWS processes are built from. | +| [Camunda 7.24 User Guide](https://docs.camunda.org/manual/7.24/user-guide/) | Engine concepts: history, external tasks, scripting, and more. | +| [Camunda REST API (7.24)](https://docs.camunda.org/rest/camunda-bpm-platform/7.24/) | The underlying engine REST API. | +| [Camunda process engine history](https://docs.camunda.org/manual/7.24/user-guide/process-engine/history/) | Background for the `history_level` setting. | +| [BPMN Quick Guide](https://www.bpmnquickguide.com/quickguide/index.html) | A concise, vendor-neutral introduction to BPMN notation. | + +## CWS project + +| Resource | Description | +| --- | --- | +| [CWS on GitHub](https://github.com/NASA-AMMOS/common-workflow-service) | Source, issues, and releases. | +| [NASA-AMMOS](https://ammos.nasa.gov/) | The Advanced Multi-Mission Operations System program. | +| [CWS REST API](rest-api.md) | Interactive Swagger UI served by a running CWS instance. | + +!!! note "Camunda version" + If you upgrade the Camunda engine used by CWS, update these links to the + matching Camunda documentation version. diff --git a/docs/reference/index.md b/docs/reference/index.md new file mode 100644 index 00000000..15a1dd7e --- /dev/null +++ b/docs/reference/index.md @@ -0,0 +1,27 @@ +# Reference + +Generated and reference material for CWS. + +
+ +- :material-api: __[REST API](rest-api.md)__ + + The CWS console serves interactive Swagger UI for its REST endpoints. + +- :material-language-java: __[Javadoc](javadoc.md)__ + + Aggregated Java API documentation for the CWS modules. + +- :material-cog-outline: __[Configuration Properties](configuration.md)__ + + Every configurable CWS setting, grouped by area. + +- :material-history: __[Changelog](changelog.md)__ + + Release history and notable changes. + +- :material-link-variant: __[External Resources](external.md)__ + + Camunda, BPMN, and other upstream documentation. + +
diff --git a/docs/reference/javadoc.md b/docs/reference/javadoc.md new file mode 100644 index 00000000..1db3d902 --- /dev/null +++ b/docs/reference/javadoc.md @@ -0,0 +1,30 @@ +# Javadoc + +Aggregated Java API documentation for the CWS modules (`cws-core`, +`cws-tasks`, `cws-service`, and others) is generated from source and published +alongside this site. + +

Open the CWS Javadoc

+ +The Javadoc is most useful when [writing custom tasks](../developer/custom-tasks.md), +[custom initiators](../developer/custom-initiators.md), or otherwise +[adapting CWS for a mission](../developer/adaptation.md), where you extend CWS +base classes and call into its APIs directly. + +!!! note "Availability" + The Javadoc is generated during the documentation build. If the link above + does not resolve for a particular version, that version was published + without the API docs (for example, a build where the Java toolchain was + unavailable). + +## Generating Javadoc locally + +From the repository root, with a JDK 17 toolchain and Maven configured to +resolve the project's dependencies: + +```bash +mvn -P core,docs -DskipTests javadoc:aggregate +``` + +The aggregated output is written to `target/site/apidocs/`. Open +`target/site/apidocs/index.html` in a browser. diff --git a/docs/reference/rest-api.md b/docs/reference/rest-api.md new file mode 100644 index 00000000..ed0dd62a --- /dev/null +++ b/docs/reference/rest-api.md @@ -0,0 +1,56 @@ +# REST API + +CWS exposes a REST API for launching and managing processes, querying history, +managing workers, and more. The running CWS console serves **interactive +Swagger UI** documentation for these endpoints, generated directly from the +live server so it always matches the deployed version. + +## Accessing the API documentation + +On a running CWS instance, open the Swagger UI at: + +``` +https://:/cws-ui/api-docs +``` + +You can also reach it from the console's **Documentation** page. The raw +OpenAPI specification is available at: + +``` +https://:/cws-ui/v3/api-docs +``` + +!!! info "Why isn't the API rendered here?" + The OpenAPI specification is built dynamically by the running server from + the live controller definitions, so it reflects exactly the endpoints and + version of your deployment. Rather than ship a snapshot that could drift + out of date, this site points you at the interactive docs served by your + own instance. + +## Authenticating requests + +The API is protected by a `cwsToken` API key sent as a header. To obtain your +token from a browser session that is already logged in to CWS: + +1. Open the CWS console and log in. +2. Open your browser's developer tools → **Application** tab. +3. Under **Cookies**, select the CWS console URL. +4. Copy the value of the **`cwsToken`** cookie. + +In Swagger UI, use the authorization control to supply this value as the +`cwsToken` header. For scripted requests, send it as a header: + +```bash +curl -H "cwsToken: " \ + "https://:/cws-ui/rest/..." +``` + +Tokens expire after a configurable period (see +[`cws_token_expiration_hours`](configuration.md#security-authentication)), +after which you must re-authenticate to obtain a new one. + +## Enabling remote (CORS) access + +By default the API is intended to be called from the CWS console origin. To +call it from other hosts, CORS must be enabled for the console. See +[Web Integration & CORS](../developer/web-integration.md). diff --git a/docs/user-guide/console/configuration.md b/docs/user-guide/console/configuration.md new file mode 100644 index 00000000..3d850789 --- /dev/null +++ b/docs/user-guide/console/configuration.md @@ -0,0 +1,16 @@ +# Configuration Page + +The **Configuration** page in the console displays the current CWS system +configuration. It shows key information such as: + +- **CWS version** — the running release version. +- **Java version** — the JDK powering this instance. +- **Database** — type, host, and schema in use. +- **Elasticsearch** — connection parameters. +- **Security mode** — the configured authentication/identity plugin. +- **Worker settings** — max processes, abandoned-worker cleanup, etc. + +This page is read-only in the console. To change settings, edit the +[configuration file](../../install/configuration.md) and re-run the +configurator, or update specific values in the database as noted in the +[Configuration Properties reference](../../reference/configuration.md). diff --git a/docs/user-guide/console/deployments.md b/docs/user-guide/console/deployments.md new file mode 100644 index 00000000..ac0e1ef3 --- /dev/null +++ b/docs/user-guide/console/deployments.md @@ -0,0 +1,35 @@ +# Deployments Page + +The **Deployments** page is the starting point for managing your process +definitions. From here you can: + +- View all deployed process definitions. +- Deploy new BPMN process definitions (created in the modeler). +- Enable or disable workers for specific processes. +- View instance statistics for each deployment. + +## Instance statistics + +The **Instance Statistics** column shows a color-coded bar representing the +status distribution of all instances for each process: + +| Color | Meaning | +| --- | --- | +| :material-circle:{ style="color: green" } Green | Completed successfully (or marked *resolved*). | +| :material-circle:{ style="color: red" } Red | Failed to complete. | +| :material-circle:{ style="color: gold" } Yellow | Pending — waiting for an available worker or executor threads. | +| :material-circle:{ style="color: orange" } Orange | Failed to start. | +| :material-circle:{ style="color: blue" } Blue | Currently running. | +| :material-circle:{ style="color: hotpink" } Pink | Raised one or more incidents. | + +Click on a colored section to jump to the [Processes page](processes.md) filtered +to that status for the selected deployment. + +!!! note "Incidents and subprocesses" + Incidents are reported for the **parent** instance. A single pink entry may + represent multiple subprocess-level failures. Click through to the Camunda + cockpit to inspect individual incidents. + +## Deploying a new process + +See [Deploying Processes](../deploying.md) for the full workflow. diff --git a/docs/user-guide/console/history.md b/docs/user-guide/console/history.md new file mode 100644 index 00000000..e5862d2e --- /dev/null +++ b/docs/user-guide/console/history.md @@ -0,0 +1,25 @@ +# History Page + +The **History** page provides a searchable view of completed and ended process +instances. Use it to review past workflow executions, check outcomes, and audit +process history. + +The history page draws from the same Elasticsearch-backed data store as the +[Logs page](logs.md), but is oriented around process instances rather than +individual log lines. + +## What you'll see + +- Process definition name and version. +- Instance ID and business key. +- Start and end time. +- Final status (completed, failed, etc.). + +## Retention + +How long history is retained is controlled by the `history_days_to_live` +[configuration property](../../reference/configuration.md#history-retention). +Older data is automatically purged to manage disk usage. + +See [Log & History Management](../../administration/logs-history.md) for +details on retention policies. diff --git a/docs/user-guide/console/home-summary.md b/docs/user-guide/console/home-summary.md new file mode 100644 index 00000000..c2029b4d --- /dev/null +++ b/docs/user-guide/console/home-summary.md @@ -0,0 +1,13 @@ +# Home & Summary + +The **Home / Summary** page is the landing page you see after logging in. It +provides an at-a-glance view of your CWS instance health and activity. + +The summary displays: + +- The current CWS version and Java version. +- A link to the [User's Guide](../../getting-started/index.md) for first-time + users. +- High-level status of workers, processes, and the system. + +From here, navigate to the other console pages using the tabs or sidebar. diff --git a/docs/user-guide/console/index.md b/docs/user-guide/console/index.md new file mode 100644 index 00000000..ad3ee79a --- /dev/null +++ b/docs/user-guide/console/index.md @@ -0,0 +1,26 @@ +# The CWS Web Console + +The CWS web console is your primary interface for managing workflows. Access it +over HTTPS at the hostname and port configured during installation. + +If this is the first time logging in, use the administrator credentials you +specified during [configuration](../../install/configuration.md). + +## Console pages + +Once logged in, the navigation gives you access to: + +| Page | Purpose | +| --- | --- | +| **[Home & Summary](home-summary.md)** | At-a-glance status of your CWS instance. | +| **[Deployments](deployments.md)** | Deploy process definitions, enable workers, and view instance statistics. | +| **[Processes](processes.md)** | View and manage individual process instances and their statuses. | +| **[History](history.md)** | Completed process instance history. | +| **[Workers](workers.md)** | View worker status, configuration, and enabled process definitions. | +| **[Initiators](initiators.md)** | Configure automatic triggers that start processes. | +| **[Configuration](configuration.md)** | View system configuration and version info. | +| **[Logs](logs.md)** | Filter and search log messages from the console and workers. | +| **[Camunda Tasklist](tasklist.md)** | Complete manual user tasks assigned within a process. | +| **Snippets** | Manage reusable [code snippets](../snippets.md). | +| **Cockpit** | Camunda's built-in process inspection and monitoring tool. | +| **Admin** | Camunda's user/group/permission management. | diff --git a/docs/user-guide/console/initiators.md b/docs/user-guide/console/initiators.md new file mode 100644 index 00000000..faebb188 --- /dev/null +++ b/docs/user-guide/console/initiators.md @@ -0,0 +1,36 @@ +# Initiators Page + +The **Initiators** page in the console lets you create, configure, and manage +automatic triggers for your process definitions. + +An **initiator** starts process instances automatically when a condition is met, +without manual intervention. From this page you can: + +- View all configured initiators and their status (active/inactive). +- Create new initiators for any deployed process definition. +- Edit initiator parameters (schedule, file path, etc.). +- Enable or disable initiators. + +## Initiator types + +CWS ships several built-in initiator types: + +| Type | Trigger | +| --- | --- | +| [Cron](../initiators/cron.md) | A cron-like schedule. | +| [File](../initiators/file.md) | A file appears at a watched location. | +| [Message Arrival](../initiators/message-arrival.md) | A message is received. | +| [Repeating Delay](../initiators/repeating-delay.md) | Repeatedly, after a configurable delay. | + +Custom initiators (internal or external) can also be developed — see +[Custom Initiators](../../developer/custom-initiators.md). + +## Creating an initiator + +1. Click **Create Initiator**. +2. Select the process definition to trigger. +3. Choose an initiator type and fill in the type-specific parameters. +4. Save and enable. + +The initiator runs in the background; once active, it starts instances +automatically according to its configuration. diff --git a/docs/user-guide/console/logs.md b/docs/user-guide/console/logs.md new file mode 100644 index 00000000..5b5bc6b7 --- /dev/null +++ b/docs/user-guide/console/logs.md @@ -0,0 +1,44 @@ +# Logs Page + +The **Logs** page lets you view and filter log messages from the console and all +connected workers in one place. + +## Filtering options + +| Filter | Purpose | +| --- | --- | +| **Process Definitions** | Show logs for a specific process definition. | +| **Log Sources** | Show only Console or only Worker logs. | +| **Process Instances** | Enter a process instance ID to see its specific messages. (Clicking an instance on the [Processes page](processes.md) sends you here with the ID pre-filled.) | +| **Log Level** | Filter by severity — useful for showing only warnings and errors during debugging. | +| **Search by Keyword** | Free-text search within log messages. | +| **Start Date / End Date** | Limit results to a time range. | + +## Additional columns + +Beyond the default columns, you can enable: + +| Column | Shows | +| --- | --- | +| **CWS Host** | The host IP for each worker log message. | +| **CWS Host ID** | The worker ID for the message. | +| **Thread Name** | The thread on which the logged item ran. | +| **Process Definition Key** | The PD key, when the message relates to a specific definition. | +| **Instance ID** | The process instance ID. | + +## Useful configurations + +A few examples of productive filter setups: + +- Select a target process definition + log levels *Warning* and *Error* + + enable the *Log Level* column → quickly spot failures within a specific + workflow. +- Filter by a specific process instance ID → trace the full execution path of + one run. +- Set a date range + keyword → find a known error that occurred during a + specific time window. + +## Related + +- [Log & History Management (admin)](../../administration/logs-history.md) — + controlling history retention and Elasticsearch index lifecycle. diff --git a/docs/user-guide/console/processes.md b/docs/user-guide/console/processes.md new file mode 100644 index 00000000..6c1965fe --- /dev/null +++ b/docs/user-guide/console/processes.md @@ -0,0 +1,41 @@ +# Processes Page + +The **Processes** page shows individual process instances and their statuses. +Use it to monitor, troubleshoot, and take action on running or completed +instances. + +## What you'll see + +For each process instance the page shows its status, the worker it ran on, and +its timestamp. Selecting an instance takes you to the +[Logs page](logs.md) with a pre-set filter for that instance, letting you see +all relevant log output without sifting through the entire log. + +## Filtering + +Use the filter controls to narrow the view: + +- **Status** — select a status radio (Pending, Running, Completed, Failed, + Incident, etc.) and click **Filter** to show only matching instances. +- **Process Definition** — limit to a specific deployment. + +This is especially useful for debugging: filtering by *Failed* shows you +exactly which instances failed, on which worker, and when — then clicking +through to the logs gives you the full context. + +## Process actions + +Depending on the instance status, you can perform bulk actions via the +**Actions** dropdown: + +| Instance status | Action | Description | +| --- | --- | --- | +| Pending | Disable selected rows | Changes status to *disabled*; prevents workers from processing them. | +| Disabled | Enable selected rows | Changes status back to *pending*; allows workers to process them. | +| Incident | Retry all selected | Retries execution from the last successful [commit point](https://docs.camunda.org/manual/7.24/user-guide/process-engine/transactions-in-processes/#asynchronous-continuations). | +| Failed to Start | Retry all selected | Changes status to *pending*, enabling workers to pick them up. | +| Failed | Mark as resolved | Acknowledges the failure; the instance is counted as *completed* in statistics and displays green on the deployments page. | + +!!! warning "Same-type selection required" + All selected rows must have the **same status** for an action to be + available. If you select rows of mixed status, all actions are disabled. diff --git a/docs/user-guide/console/tasklist.md b/docs/user-guide/console/tasklist.md new file mode 100644 index 00000000..ec854bba --- /dev/null +++ b/docs/user-guide/console/tasklist.md @@ -0,0 +1,37 @@ +# Camunda Tasklist + +The **Tasklist** page is Camunda's built-in interface for **user tasks** — manual +steps in a process that require human interaction before the workflow can +continue. + +## How user tasks work + +When a process reaches a user task, execution pauses until a person claims and +completes it. The Tasklist shows all tasks assigned to you (or unassigned tasks +you can claim). + +For each task you'll see: + +- Task name. +- The process instance it belongs to. +- Assignee (or unassigned). +- Due date and creation date. +- Priority. + +## Completing a task + +1. Open the Tasklist and find your assigned task (or claim an unassigned one). +2. Review any form or instructions attached to the task. +3. Fill in required information or confirm completion. +4. Submit — the process continues from where it paused. + +## When to use user tasks + +User tasks are useful for: + +- **Approval gates** — a human must approve before the process continues. +- **Data entry** — collect information that can't be automated. +- **Review steps** — confirm outputs before downstream processing. + +For more on modeling user tasks, see [Modeling](../../modeling/index.md) and the +[Camunda User Task reference](https://docs.camunda.org/manual/7.24/reference/bpmn20/tasks/user-task/). diff --git a/docs/user-guide/console/workers.md b/docs/user-guide/console/workers.md new file mode 100644 index 00000000..7a37c875 --- /dev/null +++ b/docs/user-guide/console/workers.md @@ -0,0 +1,32 @@ +# Workers Page + +The **Workers** page displays information about all workers connected to your +console. + +## Worker details + +A **worker** is a CWS component that executes process instances using the BPMN +engine. You need at least one worker to execute processes, but can scale +horizontally with as many as needed. + +For each worker you'll see: + +- **Status** — whether the worker is active and connected. +- **Configuration** — expand to view the worker's settings, including the + number of **executor threads** (how many concurrent process instances the + worker can run simultaneously). You can adjust this value here. + +## Process definitions per worker + +Under **Process Definitions**, expand to see all processes the worker is +configured to handle. From here you can: + +- **Enable / disable** specific process definitions on a per-worker basis. +- Set a **Limit** — the maximum number of threads that should run concurrently + for a particular process on this worker. + +This lets you control how work is distributed across your worker fleet: you can +dedicate specific workers to specific processes, throttle expensive processes, +or ensure critical workflows get priority. + +See also: [Worker Management (admin)](../../administration/workers.md). diff --git a/docs/user-guide/deploying.md b/docs/user-guide/deploying.md new file mode 100644 index 00000000..6637de5d --- /dev/null +++ b/docs/user-guide/deploying.md @@ -0,0 +1,38 @@ +# Deploying Processes + +Before you can launch a process, you must **deploy** the BPMN process +definition to CWS. + +## How to deploy + +1. Open the [Deployments page](console/deployments.md) in the CWS console. +2. Click the **Deploy** button. +3. Upload your `.bpmn` file (created in the CWS modeler or any compatible BPMN + editor). +4. CWS validates and registers the process definition. + +Once deployed, the process appears in the Deployments table and is available +for [launching](launching/index.md) manually, via the REST API, or through an +[initiator](initiators/index.md). + +## Enabling workers + +After deployment, you need to enable at least one worker for the process before +instances can execute: + +1. On the Deployments page, expand the process entry. +2. Use the worker toggle to enable one or more workers. + +You can control per-worker limits from the [Workers page](console/workers.md). + +## Redeploying / updating a process + +Deploying a new version of an existing process definition creates a new +version while preserving the previous one. Running instances continue on the +version they were started with; new instances use the latest version. + +## Auto-registering processes at startup + +If `startup_autoregister_process_defs` is set to `true` in your +[configuration](../install/configuration.md), processes placed in the +configured BPMN directory are deployed automatically when CWS starts. diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md new file mode 100644 index 00000000..87915842 --- /dev/null +++ b/docs/user-guide/index.md @@ -0,0 +1,28 @@ +# User Guide + +This section covers day-to-day usage of CWS: the web console, deploying and +running processes, initiators, and snippets. + +
+ +- :material-monitor-dashboard: __[Web Console](console/index.md)__ + + Tour the pages of the CWS console and learn what each provides. + +- :material-upload: __[Deploying Processes](deploying.md)__ + + Deploy BPMN process definitions to CWS. + +- :material-play-circle-outline: __[Launching & Scheduling](launching/index.md)__ + + Start processes manually, from another process, or via the REST API. + +- :material-timer-cog-outline: __[Initiators](initiators/index.md)__ + + Auto-start processes on a schedule, file arrival, message, or custom trigger. + +- :material-code-braces-box: __[Snippets](snippets.md)__ + + Inject reusable code into your process definitions. + +
diff --git a/docs/user-guide/initiators/cron.md b/docs/user-guide/initiators/cron.md new file mode 100644 index 00000000..188ed500 --- /dev/null +++ b/docs/user-guide/initiators/cron.md @@ -0,0 +1,32 @@ +# Cron Initiator + +The **Cron Initiator** starts a process instance on a cron-like schedule. + +## Configuration + +| Field | Description | +| --- | --- | +| **Process Definition** | The deployed process to start. | +| **Cron Expression** | A standard cron expression defining the schedule (e.g. `0 0/5 * * * ?` = every 5 minutes). | +| **Enabled** | Toggle the initiator on/off without deleting it. | + +## How it works + +CWS evaluates the cron expression at startup and schedules a timer. Each time +the timer fires, a new process instance is created and queued for a worker. + +## Example + +To run a cleanup process every night at midnight: + +``` +0 0 0 * * ? +``` + +## Notes + +- The cron expression uses the [Quartz cron syntax](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) + (6–7 fields: seconds minutes hours day-of-month month day-of-week [year]). +- If the process takes longer than the interval, instances can overlap. Use + executor thread limits on the [Workers page](../console/workers.md) to + control concurrency. diff --git a/docs/user-guide/initiators/file.md b/docs/user-guide/initiators/file.md new file mode 100644 index 00000000..1c27da3b --- /dev/null +++ b/docs/user-guide/initiators/file.md @@ -0,0 +1,32 @@ +# File Initiator + +The **File Initiator** starts a process instance when a file appears at a +watched directory location. + +## Configuration + +| Field | Description | +| --- | --- | +| **Process Definition** | The deployed process to start. | +| **Watch Directory** | The filesystem path to monitor for new files. | +| **File Pattern** | Optional glob or regex to match specific filenames. | +| **Enabled** | Toggle the initiator on/off. | + +## How it works + +CWS polls the watch directory. When a new file matching the pattern appears, a +process instance is created. The filename (and optionally its full path) is +passed into the process as a variable so the workflow can act on it. + +## Use cases + +- Processing incoming data files from an external system. +- Triggering a pipeline when a partner drops a file in a shared location. +- Watch a staging directory for new artifacts to validate. + +## Notes + +- The polling interval depends on CWS internal settings. +- Ensure the watch directory is accessible by the CWS console host. +- For S3-based file triggers, see the S3 Initiator / SQS integration + (configured via the `aws_sqs_dispatcher_*` settings). diff --git a/docs/user-guide/initiators/index.md b/docs/user-guide/initiators/index.md new file mode 100644 index 00000000..80844bb7 --- /dev/null +++ b/docs/user-guide/initiators/index.md @@ -0,0 +1,26 @@ +# Initiators + +An **initiator** automatically starts process instances in response to a +trigger — no manual intervention required. Configure initiators from the +[Initiators page](../console/initiators.md) in the console. + +## Built-in initiator types + +| Type | Starts a process when… | +| --- | --- | +| **[Cron](cron.md)** | A cron schedule fires. | +| **[File](file.md)** | A file appears at a watched location. | +| **[Message Arrival](message-arrival.md)** | A message is received on the broker. | +| **[Repeating Delay](repeating-delay.md)** | A configurable delay elapses, then repeats. | + +## Custom initiators + +CWS supports developing your own initiators: + +- **[Internal initiators](internal-external.md)** — run inside the CWS + console JVM. +- **[External initiators](internal-external.md)** — run outside CWS and + schedule processes via the REST API. + +See [Developing Custom Initiators](../../developer/custom-initiators.md) for +the development guide. diff --git a/docs/user-guide/initiators/internal-external.md b/docs/user-guide/initiators/internal-external.md new file mode 100644 index 00000000..3e55f3c5 --- /dev/null +++ b/docs/user-guide/initiators/internal-external.md @@ -0,0 +1,40 @@ +# Internal & External Initiators + +Beyond the built-in initiator types (cron, file, message, repeating delay), CWS +supports **custom initiators** that you develop yourself. + +## Internal initiators + +An **internal initiator** runs inside the CWS console JVM. It has direct access +to CWS services and can schedule processes without making REST calls. + +- **Use when:** your trigger logic needs tight integration with CWS internals + or must run with low latency. +- **Develop by:** implementing the internal initiator interface and registering + it with CWS. See + [Developing Custom Initiators](../../developer/custom-initiators.md). + +## External initiators + +An **external initiator** runs outside CWS — in a separate process, on a +different host, or as part of another system. It schedules processes by calling +the CWS [REST API](../launching/rest.md). + +- **Use when:** the trigger source is an external system (a monitoring tool, a + CI/CD pipeline, a partner's service) and you don't want to couple it to the + CWS JVM. +- **Develop by:** writing a script or service that detects the trigger condition + and `POST`s to the CWS schedule endpoint. + +## Choosing between them + +| Consideration | Internal | External | +| --- | --- | --- | +| Deployment | Inside CWS (must redeploy CWS to update) | Independent lifecycle | +| Access to CWS internals | Full | REST API only | +| Language | Java | Any (Python, Bash, Go…) | +| Failure isolation | Shares CWS JVM | Isolated process | + +For most cases, an external initiator is simpler and more maintainable. Use +an internal initiator only when you need direct access to CWS services that +aren't exposed via REST. diff --git a/docs/user-guide/initiators/message-arrival.md b/docs/user-guide/initiators/message-arrival.md new file mode 100644 index 00000000..557b4a3d --- /dev/null +++ b/docs/user-guide/initiators/message-arrival.md @@ -0,0 +1,31 @@ +# Message Arrival Initiator + +The **Message Arrival Initiator** starts a process instance when a message is +received on the CWS message broker (Apache Artemis). + +## Configuration + +| Field | Description | +| --- | --- | +| **Process Definition** | The deployed process to start. | +| **Queue / Topic** | The broker destination to listen on. | +| **Enabled** | Toggle on/off. | + +## How it works + +CWS listens on the configured broker queue. When a message arrives, it creates +a process instance and passes the message content (or selected fields) as +process variables. + +## Use cases + +- Event-driven workflows triggered by an upstream system posting to the broker. +- Decoupled architectures where producers don't call the CWS REST API directly. +- Fanout patterns where one message triggers one or more workflows. + +## Notes + +- The message broker must be reachable at the `amq_host`:`amq_port` configured + during installation. +- Messages that fail to start a process are dead-lettered per broker + configuration. diff --git a/docs/user-guide/initiators/repeating-delay.md b/docs/user-guide/initiators/repeating-delay.md new file mode 100644 index 00000000..4e8611d1 --- /dev/null +++ b/docs/user-guide/initiators/repeating-delay.md @@ -0,0 +1,31 @@ +# Repeating Delay Initiator + +The **Repeating Delay Initiator** starts a process instance repeatedly, waiting +a fixed delay between launches. + +## Configuration + +| Field | Description | +| --- | --- | +| **Process Definition** | The deployed process to start. | +| **Delay** | Time to wait between launches (e.g. seconds or minutes). | +| **Enabled** | Toggle on/off. | + +## How it works + +After the initiator is enabled, CWS waits the configured delay, then launches +an instance. Once that instance is scheduled (not necessarily completed), CWS +waits the delay again and repeats. + +## Difference from Cron + +- **Cron** fires at absolute wall-clock times regardless of the previous + instance's state. +- **Repeating Delay** measures the interval *from the end of the last launch*, + so it naturally spaces executions even if the process duration varies. + +## Use cases + +- Polling an external system at a regular interval. +- Producing a heartbeat or watchdog process. +- Steady-state load generation for testing. diff --git a/docs/user-guide/launching/from-process.md b/docs/user-guide/launching/from-process.md new file mode 100644 index 00000000..0c563d31 --- /dev/null +++ b/docs/user-guide/launching/from-process.md @@ -0,0 +1,29 @@ +# Scheduling from Another Process + +A running process can schedule another process as part of its own flow using +the built-in **Schedule Process** task type. + +## How it works + +1. In your BPMN model, add a Service Task of type **Schedule Process** (see + [Built-in Task Types](../../developer/task-types.md)). +2. Configure it with the **process definition key** of the process you want to + start. +3. Optionally pass variables from the parent process to the child. +4. At runtime, when the parent reaches this task, CWS schedules the target + process for execution on an enabled worker. + +## Use cases + +- **Orchestration** — a parent workflow coordinates multiple child workflows. +- **Chained processing** — output from one process triggers the next step in a + pipeline. +- **Fan-out** — dynamically start multiple instances based on loop data. + +## Relationship to the parent instance + +The child process runs as a separate, independent instance. It does **not** +share a transaction with the parent. If you need tighter coupling (shared scope, +error propagation), consider using a +[Call Activity](https://docs.camunda.org/manual/7.24/reference/bpmn20/subprocesses/call-activity/) +instead. diff --git a/docs/user-guide/launching/index.md b/docs/user-guide/launching/index.md new file mode 100644 index 00000000..4fe7214a --- /dev/null +++ b/docs/user-guide/launching/index.md @@ -0,0 +1,12 @@ +# Launching & Scheduling Processes + +CWS provides three ways to start a deployed process: + +| Method | Use case | +| --- | --- | +| **[Manually](manual.md)** | Ad-hoc launches from the console UI. | +| **[From another process](from-process.md)** | One process schedules another as part of its flow. | +| **[Via the REST API](rest.md)** | External systems or scripts start processes programmatically. | + +For *automatic* launching based on triggers (schedule, file arrival, message), +see [Initiators](../initiators/index.md). diff --git a/docs/user-guide/launching/manual.md b/docs/user-guide/launching/manual.md new file mode 100644 index 00000000..beb131bf --- /dev/null +++ b/docs/user-guide/launching/manual.md @@ -0,0 +1,26 @@ +# Launching Processes Manually + +The simplest way to run a deployed process is to launch it manually from the +CWS console. + +## Steps + +1. Open the [Deployments page](../console/deployments.md). +2. Find your process definition in the list. +3. Click the **Launch** (play) button for the process. +4. Optionally set process variables or a business key in the launch dialog. +5. Confirm — an instance is created with status *pending* and picked up by an + enabled worker. + +Track the instance on the [Processes page](../console/processes.md) or via the +[Logs page](../console/logs.md). + +## When to use manual launch + +- Testing a process during development. +- Running a one-off workflow that doesn't need automation. +- Ad-hoc execution where no initiator is configured. + +For repeated or event-driven execution, use an +[Initiator](../initiators/index.md) or the +[REST API](rest.md) instead. diff --git a/docs/user-guide/launching/rest.md b/docs/user-guide/launching/rest.md new file mode 100644 index 00000000..47a1fde0 --- /dev/null +++ b/docs/user-guide/launching/rest.md @@ -0,0 +1,115 @@ +# Scheduling via the REST API + +External systems and scripts can start CWS processes programmatically through +the REST API. This is useful for integrating CWS into pipelines, CI/CD, or +other automation. + +## Endpoint + +``` +POST https://:/cws-ui/rest/process//schedule +``` + +### Authentication + +All CWS REST endpoints require authentication. Include your `cwsToken` header +(see [REST API reference](../../reference/rest-api.md) for how to obtain it), +or use HTTP basic authentication (`-u username:password`) for scripted calls. + +### Parameters + +Pass process variables as form-encoded key/value pairs in the request body. + +## Example (cURL) + +Schedule a process named `my_process`, setting a variable `input_file` to +`/data/input.csv`: + +```bash +curl -k -X POST \ + "https://:38443/cws-ui/rest/process/my_process/schedule" \ + -H "cwsToken: " \ + --data "input_file=/data/input.csv" +``` + +### Response + +```json +{ + "uuid": "254c73fe-c31d-482b-a193-86dd1e5bc9cd", + "createdTime": "2024-06-15 13:46:50", + "procDefKey": "my_process", + "procPriority": 10, + "procVariables": { + "procDefKey": "my_process", + "priority": 10, + "input_file": "/data/input.csv", + "uuid": "254c73fe-c31d-482b-a193-86dd1e5bc9cd", + "procBusinessKey": "254c73fe-c31d-482b-a193-86dd1e5bc9cd" + }, + "procBusinessKey": "254c73fe-c31d-482b-a193-86dd1e5bc9cd", + "status": "pending" +} +``` + +## Monitoring instance status + +Use the returned `uuid` to poll the status endpoint: + +``` +GET https://:/cws-ui/rest/process-instance//status +``` + +The `status` field transitions through: + +``` +pending → inSchedulerQueue → claimedByWorker → running → success +``` + +If the process fails, `status` will be `fail` and `errorMessage` may contain +details. + +### Status response examples + +**Pending (not yet started):** + +```json +{ + "status": "pending", + "procDefKey": "my_process", + "uuid": "07d3311f-ef5e-49cb-af60-426f19f3b12b" +} +``` + +**Completed successfully:** + +```json +{ + "status": "complete", + "procDefKey": "my_process", + "procInstId": "5eb4a8fd-f78f-11e5-9041-685b357b8867", + "uuid": "07d3311f-ef5e-49cb-af60-426f19f3b12b", + "startTime": "2024-06-15 15:24:44", + "endTime": "2024-06-15 15:24:44", + "duration": 808, + "endActivityId": "EndEvent_1" +} +``` + +**Failed:** + +```json +{ + "status": "fail", + "procDefKey": "my_process", + "procInstId": "b46129f8-6eb5-11e4-9b38-10ddb1f141ab", + "startTime": "2024-06-15 15:59:02", + "endTime": "2024-06-15 15:59:02", + "duration": 335, + "endActivityId": "ServiceTask_1" +} +``` + +!!! tip + For additional querying on a completed instance, use the `procInstId` with + the [Camunda REST API](https://docs.camunda.org/rest/camunda-bpm-platform/7.24/). diff --git a/docs/user-guide/snippets.md b/docs/user-guide/snippets.md new file mode 100644 index 00000000..6d977f37 --- /dev/null +++ b/docs/user-guide/snippets.md @@ -0,0 +1,35 @@ +# Snippets + +**Snippets** are small, reusable pieces of code that you can inject into your +process definitions from the CWS console. + +## What snippets are for + +- Share common logic across multiple processes without duplicating it in each + BPMN model. +- Centrally manage utility functions (string processing, date formatting, + variable transformations) that many workflows use. +- Change shared behavior in one place — all processes using the snippet pick up + the update on their next execution. + +## Managing snippets + +From the **Snippets** tab in the CWS console: + +- **View** — see all available snippets and their code. +- **Create** — write a new snippet with a name and body. +- **Edit** — update an existing snippet's code. +- **Delete** — remove a snippet that is no longer needed. + +## Using a snippet in a process + +Reference a snippet by name in a script task or expression within your BPMN +model. The CWS engine resolves the snippet at execution time and runs its code +in the script context. + +## Best practices + +- Keep snippets focused — one snippet, one responsibility. +- Name snippets clearly so their purpose is obvious in the modeler. +- Test snippets with a simple process before relying on them in production + workflows. diff --git a/install/cws-ui/documentation.ftl b/install/cws-ui/documentation.ftl index 489b2071..5936f4cd 100644 --- a/install/cws-ui/documentation.ftl +++ b/install/cws-ui/documentation.ftl @@ -72,13 +72,16 @@ Resource - CWS Wiki + + CWS Documentation Site + User guide, install guide, and reference + - Camunda BPMN 2.0 Implementation Reference + Camunda BPMN 2.0 Implementation Reference - BP Incubator BPMN Quick Guide + BP Incubator BPMN Quick Guide @@ -87,7 +90,7 @@ - Camunda API Documentation + Camunda API Documentation diff --git a/install/cws-ui/summary.ftl b/install/cws-ui/summary.ftl index 520e957a..ee6abaa7 100644 --- a/install/cws-ui/summary.ftl +++ b/install/cws-ui/summary.ftl @@ -98,7 +98,7 @@


- New to CWS? Click here for the User's Guide + New to CWS? Click here for the User's Guide diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 00000000..ca523e63 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,205 @@ +site_name: Common Workflow Service +site_description: >- + User guide, installation guide, and reference documentation for the + NASA-AMMOS Common Workflow Service (CWS). +site_url: https://nasa-ammos.github.io/common-workflow-service/ +site_author: NASA-AMMOS + +repo_name: NASA-AMMOS/common-workflow-service +repo_url: https://github.com/NASA-AMMOS/common-workflow-service +edit_uri: edit/develop/docs/ + +copyright: >- + Made available under the Apache License 2.0. + +theme: + name: material + language: en + icon: + repo: fontawesome/brands/github + features: + - navigation.tabs + - navigation.sections + - navigation.top + - navigation.instant + - navigation.instant.progress + - navigation.tracking + - navigation.indexes + - navigation.footer + - toc.follow + - search.suggest + - search.highlight + - search.share + - content.code.copy + - content.code.annotate + - content.tabs.link + - content.action.edit + palette: + - media: "(prefers-color-scheme)" + toggle: + icon: material/brightness-auto + name: Switch to light mode + - media: "(prefers-color-scheme: light)" + scheme: default + primary: indigo + accent: indigo + toggle: + icon: material/brightness-7 + name: Switch to dark mode + - media: "(prefers-color-scheme: dark)" + scheme: slate + primary: indigo + accent: indigo + toggle: + icon: material/brightness-4 + name: Switch to system preference + +extra: + version: + provider: mike + default: latest + social: + - icon: fontawesome/brands/github + link: https://github.com/NASA-AMMOS/common-workflow-service + - icon: fontawesome/solid/globe + link: https://ammos.nasa.gov/ + +extra_css: + - assets/custom.css + +plugins: + - search + - macros: + module_name: docs/macros + include_yaml: [] + j2_variable_start_string: '<<<' + j2_variable_end_string: '>>>' + j2_block_start_string: '<<%' + j2_block_end_string: '%>>' + - redirects: + redirect_maps: {} + +markdown_extensions: + - abbr + - admonition + - attr_list + - def_list + - footnotes + - md_in_html + - tables + - toc: + permalink: true + toc_depth: 3 + - pymdownx.betterem + - pymdownx.caret + - pymdownx.details + - pymdownx.emoji: + emoji_index: !!python/name:material.extensions.emoji.twemoji + emoji_generator: !!python/name:material.extensions.emoji.to_svg + - pymdownx.highlight: + anchor_linenums: true + line_spans: __span + pygments_lang_class: true + - pymdownx.inlinehilite + - pymdownx.keys + - pymdownx.mark + - pymdownx.snippets: + base_path: + - . + - docs + check_paths: true + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format + - pymdownx.tabbed: + alternate_style: true + - pymdownx.tasklist: + custom_checkbox: true + +nav: + - Home: index.md + - Getting Started: + - getting-started/index.md + - Quickstart (Docker): getting-started/quickstart.md + - Core Concepts: getting-started/concepts.md + - Process Instance Lifecycle: getting-started/lifecycle.md + - Installation: + - install/index.md + - Requirements: install/requirements.md + - Prerequisites: install/prerequisites.md + - Database Setup: install/database.md + - Elasticsearch: install/elasticsearch.md + - Certificates & Keystore: install/certificates.md + - Building from Source: install/building.md + - Configuration Reference: install/configuration.md + - Running & Stopping: install/running.md + - LDAP Security: install/ldap.md + - Modeler: install/modeler.md + - Security Considerations: install/security-considerations.md + - User Guide: + - user-guide/index.md + - Web Console: + - user-guide/console/index.md + - Home & Summary: user-guide/console/home-summary.md + - Processes: user-guide/console/processes.md + - Deployments: user-guide/console/deployments.md + - History: user-guide/console/history.md + - Workers: user-guide/console/workers.md + - Initiators: user-guide/console/initiators.md + - Configuration: user-guide/console/configuration.md + - Logs: user-guide/console/logs.md + - Camunda Tasklist: user-guide/console/tasklist.md + - Deploying Processes: user-guide/deploying.md + - Launching & Scheduling: + - user-guide/launching/index.md + - Manually: user-guide/launching/manual.md + - From Another Process: user-guide/launching/from-process.md + - Via REST API: user-guide/launching/rest.md + - Initiators: + - user-guide/initiators/index.md + - File Initiator: user-guide/initiators/file.md + - Cron Initiator: user-guide/initiators/cron.md + - Message Arrival: user-guide/initiators/message-arrival.md + - Repeating Delay: user-guide/initiators/repeating-delay.md + - Internal & External: user-guide/initiators/internal-external.md + - Snippets: user-guide/snippets.md + - Modeling: + - modeling/index.md + - Best Practices: modeling/best-practices.md + - Tips & Tricks: modeling/tips.md + - Parallel Tasks & Sub-Processes: modeling/parallel-subprocess.md + - BPMN Examples: modeling/examples.md + - DMN Examples: modeling/dmn.md + - Script Task Recipes: modeling/script-recipes.md + - Administration: + - administration/index.md + - Security & Roles: administration/security.md + - User Administration: administration/users.md + - Worker Management: administration/workers.md + - Log & History Management: administration/logs-history.md + - Resource Monitoring: administration/monitoring.md + - Developer & Adaptation: + - developer/index.md + - Architecture: developer/architecture.md + - Adapting CWS for a Mission: developer/adaptation.md + - External Database: developer/external-database.md + - Custom Tasks: developer/custom-tasks.md + - Built-in Task Types: developer/task-types.md + - Custom Initiators: developer/custom-initiators.md + - Security Scheme Plugins: developer/security-plugin.md + - Web Integration & CORS: developer/web-integration.md + - Building & Contributing: developer/contributing.md + - Deployment: + - deployment/index.md + - Docker: deployment/docker.md + - AWS: deployment/aws.md + - Upgrade & Migration: deployment/upgrade.md + - Reference: + - reference/index.md + - REST API: reference/rest-api.md + - Javadoc: reference/javadoc.md + - Configuration Properties: reference/configuration.md + - Changelog: reference/changelog.md + - External Resources: reference/external.md diff --git a/pom.xml b/pom.xml index fd81857b..80916c15 100644 --- a/pom.xml +++ b/pom.xml @@ -73,6 +73,7 @@ 3.5.0 3.1.4 3.21.0 + 3.11.2 3.4.2 1.4.0 2.16 @@ -118,6 +119,37 @@ cws-test + + + + docs + + + + org.apache.maven.plugins + maven-javadoc-plugin + ${maven-javadoc-plugin.version} + + none + false + false + true + false + -Xdoclint:none + CWS ${project.version} API + CWS ${project.version} API + + + + + diff --git a/requirements-docs.txt b/requirements-docs.txt new file mode 100644 index 00000000..f82e6515 --- /dev/null +++ b/requirements-docs.txt @@ -0,0 +1,7 @@ +# Documentation site build dependencies (MkDocs + Material). +# Install into a virtualenv: python3 -m venv .venv && . .venv/bin/activate && pip install -r requirements-docs.txt +mkdocs-material>=9.5,<10 +mike>=2.1,<3 +mkdocs-macros-plugin>=1.0,<2 +mkdocs-redirects>=1.2,<2 +pymdown-extensions>=10.7 diff --git a/tools/docs/sanitize-lint.sh b/tools/docs/sanitize-lint.sh new file mode 100755 index 00000000..7fccd98f --- /dev/null +++ b/tools/docs/sanitize-lint.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# +# sanitize-lint.sh — guard the public documentation against JPL-internal data. +# +# Scans the docs/ content tree for a denylist of JPL-specific patterns +# (internal hostnames, the CAM system, MOZART/PGE examples, internal ticket +# IDs, AWS account specifics, etc.). Exits non-zero if any match is found so +# CI fails before such content is ever published to GitHub Pages. +# +# Usage: +# tools/docs/sanitize-lint.sh [DOCS_DIR] +# +# DOCS_DIR defaults to "docs". Only the content tree is scanned — this script +# and tools/docs/ (which necessarily name the denylisted terms) are excluded. +# +# To intentionally allow a specific line (rare, e.g. a sanitized illustrative +# example), append the marker: + +set -euo pipefail + +DOCS_DIR="${1:-docs}" + +if [[ ! -d "$DOCS_DIR" ]]; then + echo "sanitize-lint: docs directory '$DOCS_DIR' not found" >&2 + exit 2 +fi + +# Each entry: "