Skip to content

Latest commit

 

History

109 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

karate-connect

1. What is karate-connect ?

It is a fat JAR (available here) with Karate Core and extensions to connect to other systems :

  • karate-connect-<version>-standalone.jar

The extensions require some tools. Three Docker images are also available with everything installed :

  • lectratech/karate-connect:<version>-minimal (only the standalone JAR in a JRE image)

  • lectratech/karate-connect:<version> (Python packages & standard kubectl installation)

  • lectratech/karate-connect:<version>-aks (Python packages & Azure Kubernetes Service kubectl installation)

2. What is an extension ?

A set of Kotlin classes, Karate features, Javascript functions. This set is callable in your Karate project.

How to call a function from an extension ?
* def result1 = <extension>.<value>
* def result2 = <extension>.<feature>.<function>(args)
Examples
* json cliConfig     = snowflake.cliConfigFromEnv
* def rabbitmqClient = rabbitmq.topology.createClient({ host: "localhost", port: 5672 })

3. How to build karate-connect ?

Requirements
  • JDK 21+

  • Kotlin

  • Gradle

  • Docker and Docker Compose

  • Python 3.x & pip

Snowflake requirements
  • Define a src/test/resources/snowflake/snowflake.properties with your Snowflake information

    • Example: src/test/resources/snowflake/snowflake.template.properties

    • Note: privateKeyBase64 is a base64 one-line encoded private key : cat my-private-key.pem | base64 -w0

  • Your Snowflake role has to be allowed to create/drop schemas/stages/tables, create/execute/drop tasks.

Commands
  • source .envrc to install Python packages in a virtual environment (or direnv allow if you prefer the great direnv tool)

  • ./gradlew build to build the fat JAR with all tests

  • ./gradlew build -DtestExtensions=…​ to build the fat JAR with only Karate tests on the given extensions (useful if you do not have a Snowflake account)

  • docker compose build to build the 3 Docker images locally :

    • karate-connect:minimal

    • karate-connect

    • karate-connect:aks

3.1. Optional: devenv

devenv provides the whole build/test toolchain (JDK 21, Python CLIs, kubectl) without installing anything system-wide. It is opt-in and does not interfere with .envrc/py_venv.

devenv shell   # enter the dev environment
devenv test    # fat JAR + tests on $TEST_EXTENSIONS, same check as CI

Once inside the shell, the following scripts are available :

  • kc-build : fat JAR + tests on $TEST_EXTENSIONS (defaults to rabbitmq,kafka,kubernetes)

  • kc-build-all : fat JAR + tests on every extension (Snowflake credentials required)

  • kc-test : tests only

  • kc-docker-build : build the Docker images (uses the host Docker daemon)

  • kc-headers : re-apply the license headers

Note
Entering the shell clears any KAFKA_*/RABBITMQ_*/SNOWFLAKE_* environment variables inherited from your own shell, since the configFromEnv test scenarios require them to be unset. Snowflake and dbt tests are skipped unless src/test/resources/snowflake/snowflake.properties exists (see the Snowflake requirements above).

4. How to run karate-connect on your features

4.1. Docker usage

To run all features in <feature_path> having maybe some extensions & reports generated in <report_path>
docker run --rm \
    -v <features_path>:/features \
    -v <reports_path>:/target/karate-reports \
    -e KARATE_EXTENSIONS=<ext1>,<ext2>... \
    lectratech/karate-connect:<version> <karate_args>
Note
KARATE_EXTENSIONS, reports_path, karate_args are optional
With a specific karate-config.js
docker run --rm \
    ... \
    -v <my-specific-karate-config.js>:/karate-config.js \
    lectratech/karate-connect:<version> <karate_args>
Example of karate-config.js
function fn() {
    const myFunction = (input) => input.toUpperCase();
    return {
        myValue: "foo",
        myFunction: myFunction
    };
}

4.2. Java usage

java -Dextensions=<ext1>,<ext2>... -jar karate-connect-<version>-standalone.jar <karate_args>

5. Using karate-connect from Nix (flake.nix)

flake.nix (and the nix/.nix files it imports) is a reusable Nix module for *other Nix projects that want to run karate-connect’s Karate CLI against their own features, with every setting customizable (extensions, features path, `karate-config.js location, tags, threads, report format, environment variables, JVM args, …​). This is for consumers of karate-connect, not for building karate-connect itself (that is still done with Gradle, see above).

Note
The JAR used at runtime is fetched from a pinned GitHub Release, not rebuilt from source.

Three adapters share the same option set (defined below), so their behavior never diverges : a flake-parts module (flakeModules.default), a plain function for consumers without flake-parts (lib.mkKarateRun), and a devenv module (devenvModules.default).

5.1. Options

Every "run" (a karate-connect.runs.<name> entry, or the attribute set passed to lib.mkKarateRun) supports the following options :

Option Default Description

jar

pinned release JAR

The karate-connect standalone JAR to run. Defaults to a pinned release fetched from GitHub Releases. Override with a custom build (e.g. a fork, or a locally built -standalone.jar) if needed.

jdk

pkgs.temurin-bin-21

JDK used to run the JAR. Defaults to JDK 21, matching karate-connect’s own Gradle toolchain and Docker builder image.

extensions

[ "base" ]

List of extensions to load, passed as -Dextensions=<ext1>,<ext2>,…​. One of base, rabbitmq, kafka, snowflake, dbt, kubernetes. base is always loaded by karate-connect itself regardless of this list.

featuresPath

"features"

Path to the feature file(s) or directory to run, resolved at runtime against the invoking shell’s working directory (kept as a plain string, not a Nix path, so it is never copied into the store).

featuresMountPath

null

Absolute path at which featuresPath should appear to the running JVM, bind-mounted at run time — the same remapping Docker usage does with -v <features_path>:/features (see the Docker image’s VOLUME /features). Implemented with bubblewrap (bwrap) on Linux, or bindfs on Darwin (see the note below the table); evaluation fails with a clear error on any other platform. Useful when feature files were written assuming that Docker layout (e.g. reading fixtures from a hardcoded absolute path). Example: with featuresPath = "it/features" and featuresMountPath = "/features", the host directory it/features (resolved against the invoking shell’s working directory) is bind-mounted onto /features for the JVM process only — no Nix store copy, no change to the real filesystem outside the wrapped process. Leave null (the default) to run directly against featuresPath, with no remapping. Must be a real subpath (e.g. /features), not / itself.

karateConfigDir

null

Directory containing a custom karate-config.js, passed as -Dkarate.config.dir=<dir>. Resolved at runtime against the invoking shell’s working directory. Leave null to use Karate’s default resolution (classpath / current directory).

karateConfigMountPath

null

Absolute path at which karateConfigDir should appear to the running JVM, bind-mounted at run time — the same remapping Docker usage does with -v <my-specific-karate-config.js>:/karate-config.js. Implemented the same way as featuresMountPath (bubblewrap on Linux, bindfs on Darwin). karate-config.js (or code it calls into) sometimes reads auxiliary files via a hardcoded absolute path too, the same problem featuresMountPath solves for featuresPath. Example: with karateConfigDir = "it/config" and karateConfigMountPath = "/karate-config", the host directory it/config is bind-mounted onto /karate-config for the JVM process only, and -Dkarate.config.dir is set to the mounted path. Requires karateConfigDir to also be set. Leave null (the default) to run directly against karateConfigDir, with no remapping. Must be a real subpath (e.g. /karate-config), not / itself — featuresMountPath and karateConfigMountPath share the same sandbox on Linux, built as one fresh writable root with /nix, /dev, /proc, /etc, /run, /tmp and the caller’s working directory re-bound onto it, so mounting directly onto / would hide those.

outputDir

"target/karate-reports"

Report output directory, passed as -o.

tags

null

Tag expression filter, passed as -t. Example: "@smoke".

threads

1

Parallel thread count, passed as -T.

format

[ "junit:xml" "cucumber:json" ]

Report formats, comma-joined and passed as -f.

env

null

Karate environment name, passed as -e (karate.env). Example: "dev".

name

null

Scenario name filter, passed as -n.

environmentVariables

{ }

OS environment variables exported before running, e.g. the RABBITMQ_*/KAFKA_*/SNOWFLAKE_* variables read by karate-connect’s extensions, or any variable a consumer’s own karate-config.js relies on. Example: { RABBITMQ_HOST = "localhost"; KAFKA_BOOTSTRAP_SERVERS = "localhost:9092"; }.

jvmArgs

[ ]

Extra JVM arguments (e.g. heap size, custom system properties). Example: [ "-Xmx2g" ].

extraArgs

[ ]

Escape hatch: extra arguments appended verbatim to the Karate CLI invocation.

extraClasspath

[ ]

Extra directories added to the JVM classpath alongside the JAR and featuresPath, resolved at runtime against the invoking shell’s working directory. Needed when a feature reads a sibling resource via classpath:…​ (e.g. a mock JSON fixture) from outside featuresPath. Example: [ "src/test/fixtures" ].

Note
featuresMountPath and karateConfigMountPath require either Linux (bubblewrap, zero-config) or macOS (bindfs). macOS has no equivalent of Linux user/mount namespaces, so on Darwin these options fall back to bindfs, a FUSE filesystem, with two consequences the Linux path doesn’t have: (1) it requires macFUSE to be installed and approved by the user once — a system extension outside of Nix’s control; (2) unlike the disposable Linux sandbox, bindfs mounts onto the real filesystem, so the mount target (e.g. /features) must already exist as a directory the invoking user owns (create it once with sudo mkdir -p /features && sudo chown "$(whoami)" /features) — the run fails with a clear message otherwise. The Darwin path is less exercised than the Linux one; treat it as best-effort. If neither is workable, running the Docker image directly (Docker Desktop already provides the equivalent Linux environment on macOS) remains the simplest option.

5.2. Usage with flake-parts (flakeModules.default)

Exposes perSystem.karate-connect.runs.<name>, producing a packages/apps pair named karate-test-<name>.

Full example flake.nix
{
  description = "My project's Karate integration tests";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
    karate-connect.url = "github:lectra-tech/karate-connect";
  };

  outputs =
    inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      systems = [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];

      imports = [ inputs.karate-connect.flakeModules.default ];

      perSystem = { pkgs, ... }: {
        karate-connect.runs.default = {
          extensions = [ "rabbitmq" "kafka" ];
          featuresPath = "src/test/features";
          karateConfigDir = "src/test";
          tags = "@smoke";
          threads = 4;
          environmentVariables = {
            RABBITMQ_HOST = "localhost";
            KAFKA_BOOTSTRAP_SERVERS = "localhost:9092";
          };
        };

        # A second, independent run: e.g. a Kubernetes-only smoke test.
        karate-connect.runs.kubernetes-smoke = {
          extensions = [ "kubernetes" ];
          featuresPath = "src/test/features/kubernetes";
          tags = "@kubernetes";
        };

        # A third run: features and config written for the Docker layout,
        # reading fixtures/config from hardcoded absolute paths. `it/features`
        # and `it/config` (on disk, next to this flake.nix) are bind-mounted
        # onto `/features` and `/karate-config` for the JVM only, mirroring
        # `docker run -v it/features:/features -v it/config/karate-config.js:/karate-config.js ...`.
        karate-connect.runs.docker-style = {
          extensions = [ "rabbitmq" ];
          featuresPath = "it/features";
          featuresMountPath = "/features";
          karateConfigDir = "it/config";
          karateConfigMountPath = "/karate-config";
        };
      };
    };
}
nix run .#karate-test-default             # runs the "default" run
nix run .#karate-test-kubernetes-smoke     # runs the "kubernetes-smoke" run
nix run .#karate-test-docker-style         # runs the "docker-style" run, /features bind-mounted
nix flake check                            # if wired into `checks.*`, see this repo's own flake.nix

# Any argument passed after `--` replaces `featuresPath` for that invocation only,
# exactly like `docker run <image> <karate_args>` overrides the image's default `CMD`:
nix run .#karate-test-default -- src/test/features/foo.feature

5.3. Usage without flake-parts (lib.mkKarateRun)

A plain function for consumers who do not use flake-parts. It takes the same options plus pkgs/lib/name, and returns { package, app }.

{
  description = "My project's Karate integration tests";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    karate-connect.url = "github:lectra-tech/karate-connect";
  };

  outputs = { self, nixpkgs, karate-connect, ... }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };
      run = karate-connect.lib.mkKarateRun {
        inherit pkgs;
        name = "default";
        extensions = [ "rabbitmq" "kafka" ];
        featuresPath = "src/test/features";
        tags = "@smoke";
      };
    in
    {
      packages.${system}.karate-test = run.package;
      apps.${system}.karate-test = run.app;
    };
}
nix run .#karate-test

5.4. Usage with devenv (devenvModules.default)

Exposes karate-connect.runs.<name>, adding a karate-<name> script to the devenv shell.

Full example devenv.yaml
inputs:
  nixpkgs:
    url: github:cachix/devenv-nixpkgs/rolling
  karate-connect:
    url: github:lectra-tech/karate-connect
Full example devenv.nix
{ inputs, ... }:
{
  imports = [ inputs.karate-connect.devenvModules.default ];

  karate-connect.runs.default = {
    extensions = [ "rabbitmq" "kafka" ];
    featuresPath = "src/test/features";
    karateConfigDir = "src/test";
    tags = "@smoke";
    threads = 4;
    environmentVariables = {
      RABBITMQ_HOST = "localhost";
      KAFKA_BOOTSTRAP_SERVERS = "localhost:9092";
    };
  };

  # Features and config written for the Docker layout, reading fixtures/config
  # from hardcoded absolute paths. `it/features` and `it/config` (on disk,
  # next to this devenv.nix) are bind-mounted onto `/features` and
  # `/karate-config` for the JVM only, mirroring
  # `docker run -v it/features:/features -v it/config/karate-config.js:/karate-config.js ...`.
  karate-connect.runs.docker-style = {
    extensions = [ "rabbitmq" ];
    featuresPath = "it/features";
    featuresMountPath = "/features";
    karateConfigDir = "it/config";
    karateConfigMountPath = "/karate-config";
  };
}
devenv shell                        # enter the shell, exposes karate-default & karate-docker-style
devenv shell -- karate-default      # run the "default" run non-interactively
devenv shell -- karate-docker-style # run the "docker-style" run, /features bind-mounted

# Any argument passed to the script replaces `featuresPath` for that invocation only,
# exactly like `docker run <image> <karate_args>` overrides the image's default `CMD`:
devenv shell -- karate-default src/test/features/foo.feature

5.5. Dogfooding

This repo dogfoods the module against its own, fully-mocked kubernetes feature (no kubectl/network access needed) as checks.kubernetes-smoke :

nix run .#karate-test-kubernetes-smoke   # this repo's own dogfood run
nix flake check                          # includes the same run as checks.kubernetes-smoke

6. Extensions

6.1. base

Some common functions added to the Karate DSL

6.1.1. Functions

* string res = base.random.uuid()                             # ex: res='8cd07583-cf24-4373-ad58-f1c9303501c5'
* def millis = base.time.currentTimeMillis()                  # ex: millis=1738851217499
* string now = base.time.offsetDateTimeNow()                  # ex: now='2025-01-01T15:10:00.629772630+01:00'
* string str = base.json.toString({foo:"bar"})                # str='{"foo":"bar"}'
* string str = base.json.readLines("file.json")               # str='{"id":"1c4b..."}\n{"id":"2a02..."}' with file.json = '{"id":"#(base.random.uuid())"}\n{"id":"#(base.random.uuid())"}'
* def bool   = base.assert.withEpsilon(0.1234, 0.12, epsilon) # bool=true if epsilon=1E-2, false if epsilon=1E-3
* string res = base.hash.md5("hello world")                   # res='5eb63bbbe01eeed093cb22bb8f5acdc3'
* string res = base.hash.sha1("hello world")                  # res='2aae6c35c94fcfb415dbe95f408b9ce91ee846ed'
* string res = base.hash.sha224("hello world")                # res='2f05477fc24bb4faefd86517156dafdecec45b8ad3cf2522a563582b'
* string res = base.hash.sha256("hello world")                # res='b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9'
* string res = base.hash.sha384("hello world")                # res='5eb63bbbfdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bde01eeed093cb22bb8f5acdc3'
* string res = base.hash.sha512("hello world")                # res='309ecc489c12d6eb4cc40f50c902f2b4d0ed77ee511a7c7a9bcd3ca86d4cd86f989dd35bc5ff499670da34255b45b0cfd830e81f605dcf7dc5542e93ae9cd76f'
Note
This extension is loaded by default.

6.1.2. More info

6.2. rabbitmq

Rabbitmq topology creation & messages publication/consumption

6.2.1. Config

Rabbitmq client config
* json rabbitmqConfigFromEnv = rabbitmq.configFromEnv # environment variable RABBITMQ_HOST, RABBITMQ_PORT, RABBITMQ_VIRTUAL_HOST, RABBITMQ_USERNAME, RABBITMQ_PASSWORD, RABBITMQ_SSL
* json rabbitmqConfigFromValue = { host: "localhost", port:5672, virtualHost:"default", username:"guest", password:"guest", ssl:false }
* json rabbitmqConfigFromJsonFile = read("my-rabbitmq-config.json")
Rabbitmq client
* def rabbitmqClient = rabbitmq.topology.createClient(rabbitmqConfig)
Note
It will be closed automatically when no longer in use.
Tip
It should be created only once! Declare it in your karate-config.js
karate-config.js with your rabbitmqClient
function fn() {
  const rabbitmqConfig = ...;
  const rabbitmqClient = karate.callSingle("classpath:rabbitmq/topology.feature@createClient", rabbitmqConfig).result;
  return {
    "rabbitmqClient": rabbitmqClient
  };
}

6.2.2. Topology

These operations should not normally be performed by Karate. Nevertheless, it is possible if you need.

Exchange creation
* json exchangeConfig = ({ rabbitmqClient, name: "<myexchange>", type: "direct|topic|fanout|headers", durable: true(default)|false, autoDelete: true|false(default) })
* json result = rabbitmq.topology.exchange(exchangeConfig)
* match result.status == "OK"
Queue creation
* json queueConfig = ({ rabbitmqClient, name: "<myqueue>", type: "classic|quorum|stream", durable: true(default)|false, exclusive: true|false(default), autoDelete: true|false(default) })
* json result = rabbitmq.topology.queue(queueConfig)
* match result.status == "OK"
Binding creation (between an exchange and a queue)
* json bindingConfig = ({ rabbitmqClient, exchangeName: "<myexchange>", queueName: "<myqueue>", routingKey: "<my.routing.key>" })
* json result = rabbitmq.topology.bind(bindingConfig)
* match result.status == "OK"

6.2.3. Message

Message publication
* json publishConfig = ({ rabbitmqClient, exchangeName: "<myexchange>", routingKey: "<my.routing.key>" })
* json headers = { header1: "foo", header2: "bar" }
* json properties = ({ headers, contentType: "text/plain" })
* json message = ({ body: "hello world", properties })
* json result = rabbitmq.message.publish({...publishConfig, message})
* match result.status == "OK"
Table 1. Available properties
name type default value

contentType

string

"application/json"

contentEncoding

string

"UTF-8"

deliveryMode

number

null

priority

number

null

correlationId

string

"<uuid>"

replyTo

string

null

expiration

string

null

messageId

string

"<uuid>"

timestamp

number

nb milliseconds since January 1, 1970, 00:00:00 GMT, until now

type

string

null

userId

string

null

appId

string

null

clusterId

string

null

headers

map<string,string>

empty map

Message consumption
* json consumeConfig = ({ rabbitmqClient, queueName: "<myqueue>", timeoutSeconds: <nbSeconds>(default 60), minNbMessages: <nbNeededMessages>(default 1) })
* json result = rabbitmq.message.consume(consumeConfig)
* match result.status == "OK"
* match result.data[0].properties.contentType == "text/plain"
* match result.data[0].body == "hello world"
* json bodyAsJson = result.data[0].body # cast to JSON
Note
  • The consumption is waiting for minNbMessages messages during timeoutSeconds seconds.

  • If the number of messages is not reached during timeoutSeconds seconds, the consumption fails.

  • Set minNbMessages to 0 for no failure if no message is received during timeoutSeconds seconds.

Message publication & consumption (RPC: Remote Procedure Call)
* json publishAndConsumeConfig = ({ rabbitmqClient, exchangeName: "<myexchange>", routingKey: "<my.routing.key>", timeoutSeconds: <nbSeconds>(default 60) })
* json message = ({ body: "ping", properties: { contentType: "text/plain" } })
* json result = rabbitmq.message.publishAndConsume({...publishAndConsumeConfig, message})
* match result.status == "OK"
* match result.data.properties.contentType == "text/plain"
* match result.data.body == "pong"
Note
  • If message.properties.replyTo is set, this queue name must exist and the client will wait for 1 message in this queue for the response, during timeoutSeconds seconds.

  • If message.properties.replyTo is not set, a temporary reply-to queue will created and used for the response.

6.2.4. More info

6.3. kafka

Kafka topics creation & subjects registration & messages production/consumption

6.3.1. Config

Kafka client config
# built from environment variables KAFKA_BOOTSTRAP_SERVERS, KAFKA_SCHEMA_REGISTRY_URL,
# KAFKA_SCHEMA_REGISTRY_BASIC_AUTH_CREDENTIALS_SOURCE, KAFKA_SCHEMA_REGISTRY_KEY, KAFKA_SCHEMA_REGISTRY_SECRET
* json kafkaConfigNoSecurityFromEnv = kafka.configFromEnv

# built from previous environment variables +
# KAFKA_SECURITY_PROTOCOL, KAFKA_SASL_MECHANISM, KAFKA_USERNAME, KAFKA_PASSWORD
* json kafkaConfigSaslFromEnv = kafka.kafkaConfigSaslFromEnv

# define your own Kafka client config
* json kafkaConfigFromValue = { "bootstrap.servers": "localhost:9092", "schema.registry.url": "http://localhost:8081" }

# read a JSON file with your Kafka client config
* json kafkaConfigFromJsonFile = read("my-kafka-config.json")
All available Kafka parameters, with example values
{
    "bootstrap.servers": "localhost:9092",
    "schema.registry.url": "http://localhost:8081",
    "basic.auth.credentials.source": "USER_INFO",
    "basic.auth.user.info": "myregistrykey:myregistrysecret",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username='myuser' password='mypassword';",
    "karate.connect.consumer.group.id.prefix": "FOO-"
}
Note
  • Created consumer group ID will be a UUID, prefixed by karate.connect.consumer.group.id.prefix if set

  • If basic.auth.credentials.source is set, only USER_INFO (basic authentication) is supported (required for Confluent Cloud Schema Registry)

Kafka client
* def kafkaClient = kafka.topology.createClient(kafkaConfig)
Note
It will be closed automatically (and all created consumer groups) when no longer in use.
Tip
It should be created only once! Declare it in your karate-config.js
karate-config.js with your kafkaClient
function fn() {
  const kafkaConfig = ...;
  const kafkaClient = karate.callSingle("classpath:kafka/topology.feature@createClient", kafkaConfig).result;
  return {
    "kafkaClient": kafkaClient
  };
}

6.3.2. Topology

These operations should not normally be performed by Karate. Nevertheless, it is possible if you need.

Topic creation
* json topicConfig = ({ kafkaClient, topic: "mytopic", partitions: 2(default 1), replicationFactor: 1(default 1)})
* json result = kafka.topology.createTopic(topicConfig)
* match result.status == "OK"
Subject registration (AVRO or PROTOBUF or JSON)
* string subjectName = "mysubject"
* string subjectType = "AVRO|PROTOBUF|JSON"
* string schemaString = "..."
* json registerResult = kafka.topology.registerSubject({ kafkaClient, subjectName, subjectType, schemaString })
* match registerResult.status == "OK"
* match registerResult.schemaId == "#number? _ > 0"

6.3.3. Message

Message publication with headers
* json record = ({ key: "myKey", value: "my message", headers: { header1: "foo", header2: "bar" } })
* json result = kafka.message.produce({ kafkaClient, topic: "mytopic", record })
* match result.status == "OK"
* match result.recordMetadata.serializedKeySize == "#number? _ > 0"
* match result.recordMetadata.serializedValueSize == "#number? _ > 0"
* match result.recordMetadata.topic == "mytopic"
* match result.recordMetadata.partition == 0
* match result.recordMetadata.timestamp == "#number? _ > 0"
* match result.recordMetadata.offset == 0
Note
Only Map<String, String> headers are supported.
Message publication with subjects
* json record = ({ key: '{ "foo": "foo1" }' , value: '{ "bar": 42 }' })
* json result = kafka.message.produce({ kafkaClient, topic: "mytopic", keySubject: "my-key-avro-subject", valueSubject: "my-value-json-subject", record })
* match result.status == "OK"
Consumer creation
* json result = kafka.message.subscribe({ kafkaClient, topic: "mytopic" })
* match result.status == "OK"
* json consumer = result.consumer
* match consumer.topic == "mytopic"
# if `karate.connect.consumer.group.id.prefix` is set, groupId = "<prefix><uuid>", else "<uuid>"
* match consumer.groupId == "#uuid"
Note
A created consumer will read a topic from the latest record by partition, when its subscription has been initialized.
Message consumption
* json consumer = ...
* json result = kafka.message.consume({ consumer, pollDurationSeconds: 1(default 1), timeoutSeconds: 60(default 60), maxMessages: 1(default 100) })
* match result.status == "OK"
* match (result.data.length) == 1
* match result.data[0] == { "key":"mykey", "value":"my message", "headers":{ ... } }
Note
  • A loop is created : the consumption is polling during pollDurationSeconds seconds.

  • The loop is stopped if at least maxMessages messages are found or timeoutSeconds seconds is reached.

  • If more records are found, only the first maxMessages records are returned.

Message consumption by key (same behavior as consume, but with a key for record filtering)
* json consumer = ...
* json result = kafka.message.consumeByKey({ consumer, key: "mykey" pollDurationSeconds: 1(default 1), timeoutSeconds: 60(default 60), maxMessages: 1(default 100) })
* match result.status == "OK"
* match (result.data.length) == 1
* match result.data[0] == { "key":"mykey", "value":"my message", "headers":{ ... } }
Message consumption with subjects
* json consumer = ...
* json result = kafka.message.consume({ consumer, pollDurationSeconds: 1, timeoutSeconds: 60, maxMessages: 1, keySubject: "my-key-avro-subject", valueSubject: "my-value-json-subject" })
* match result.status == "OK"
* match (result.data.length) == 1
* match result.data[0].key == { "foo": "foo1" }
* match result.data[0].value == { "bar": 42 }

6.3.4. More info

6.4. snowflake

Snowflake CLI / Snowflake REST API calls

Note

For the fat JAR usage, you have to install snowflake-cli Python package

6.4.1. Config

CLI config
* json cliConfigFromEnv = snowflake.cliConfigFromEnv # environment variable SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_PRIVATE_KEY_PATH, PRIVATE_KEY_PASSPHRASE
* json cliConfigFromValue = { account: "xxx.yyy.azure", user: "<MY_USER>", privateKeyPath: "<path/file.pem>", privateKeyPassphrase: "****" }
* json cliConfigFromJsonFile = read("my-cli-config.json")
Snowflake config
* json snowflakeConfigConfigFromEnv = snowflake.snowflakeConfigFromEnv # environment variable SNOWFLAKE_ROLE, SNOWFLAKE_WAREHOUSE, SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA
* json snowflakeConfigConfigFromValue = { role: "<MY_ROLE>", warehouse: "<MY_WH>", database: "<MY_DB>", schema: "<MY_SCHEMA>" }
* json snowflakeConfigFromJsonFile = read("my-snowflake-config.json")

6.4.2. CLI

JWT generation
* string jwt = snowflake.cli.generateJwt(cliConfig)
* match jwt === '#regex .+\\..+\\..+'
Tip
The JWT should be created once. Declare it in your karate-config.js
karate-config.js with jwt generation
function fn() {
  const cliConfig = ...;
  const jwt = karate.callSingle("classpath:snowflake/cli.feature@generateJwt", cliConfig).result;
  return {
    "jwt": jwt,
    "cliConfig": cliConfig,
    ...
  };
}
SQL statement execution (directly with the CLI)
* string statement = "SELECT FOO, BAR FROM MY_TABLE"
* json result = snowflake.cli.runSql({ statement, cliConfig, snowflakeConfig })
* match result.status == "OK"
* match result.output == [ { "FOO": 1, "BAR": "bar1" }, { "FOO": 2, "BAR": "bar2" } ]
Note
Limitations for SQL statement through CLI is not yet analyzed.
CSV file import into table
# <file>.csv :
# FOO,BAR
# 1,bar1
# 2,bar2

* string fileAbsolutePath = karate.toAbsolutePath("<relativePath>/<file>.csv")
* string tableName = "<MY_TABLE>"
* json result = snowflake.cli.putCsvIntoTable({ fileAbsolutePath, tableName, cliConfig, snowflakeConfig })
* match result.status == "OK"
JSON-line file import into table
# <file>.json :
# {"FOO":1,"BAR":"bar1"}
# {"FOO":2,"BAR":"bar2"}

* string fileAbsolutePath = karate.toAbsolutePath("<relativePath>/<file>.json")
* string tableName = "<MY_TABLE>"
* json result = snowflake.cli.putJsonIntoTable({ fileAbsolutePath, tableName, cliConfig, snowflakeConfig })
* match result.status == "OK"

6.4.3. REST API

SQL statement execution
* json restConfig = ({ jwt, cliConfig, snowflakeConfig })
* string statement = "SELECT FOO, BAR FROM MY_TABLE"
* json result = snowflake.rest.runSql({ ...restConfig, statement})
* match result.status == "OK"
* match (result.data.length) == 1
* match result.data[0].FOO == 1
* match result.data[0].BAR == "bar1"
Note
  • Limitations for SQL statement is not yet fully analyzed.

  • Default HTTP retry strategy: karate.configure("retry", {count: 10, interval: 5000})

  • Default readTimeout: karate.configure("readTimeout", 240000);

  • If HTTP 202 is returned (long SQL statement), a GET request loop (with a statementHandle) will wait for a HTTP 200, according to the HTTP retry strategy.

  • Pagination: TODO

Schema cloning
* json restConfig = ({ jwt, cliConfig, snowflakeConfig })
* json result = snowflake.rest.cloneSchema({...restConfig, schemaToClone: "<MY_SOURCE_SCHEMA>", schemaToCreate: "<MY_TARGET_SCHEMA>"})
* match result.status == "OK"
Schema dropping
* json restConfig = ({ jwt, cliConfig, snowflakeConfig })
* json result = snowflake.rest.dropSchema({...restConfig, schemaToDrop: "<MY_SCHEMA>"})
* match result.status == "OK"
Staging table (RECORD_METADATA JSON_VARIANT, RECORD_VALUE JSON_VARIANT) insertion - Useful for a Kafka Connect usage
* string table = "<MY_TABLE>"
# Single row
* json result = snowflake.rest.insertRowIntoStagingTable({...restConfigLocal, table, recordMetadata: {...}, recordValue: {...}})
* match result.status == "OK"
# Single row from files
* json result = snowflake.rest.insertRowIntoStagingTable({...restConfigLocal, table, recordMetadataFile: "<file-metadata-path>", recordValueFile: "<file-value-path>"})
* match result.status == "OK"
# Many rows
* json result = snowflake.rest.insertRowsIntoStagingTable({...restConfigLocal, table, records: [ {recordMetadata: {...}, recordValue: {...}}, ... ]})
* match result.status == "OK"
Task status checking
* string taskName = "<MY_TASK>"
* json restConfig = ({ jwt, cliConfig, snowflakeConfig })
* json result = snowflake.rest.runSql({...restConfig, statement: "EXECUTE TASK "+taskName})
* match result.status == "OK"
* json result = snowflake.rest.checkTaskStatus({...restConfig, taskName})
* match result.status == "OK"
Note
checkTaskStatus will use the retry strategy to wait for the task completion.
Task cloning and execution - Useful to ignore the parent task and test only the task code
* string taskName = "<MY_TASK>"
* json restConfig = ({ jwt, cliConfig, snowflakeConfig })
* json result = snowflake.rest.cloneAndExecuteTask({...restConfig, taskName})
* match result.status == "OK"
Note
cloneAndExecuteTask will execute a temporary copy of the task taskName (without the parent task) and will wait for its completion.

6.4.4. More info

6.5. kubernetes

Kubectl calls

Note
  • For the fat JAR usage

  • For the Docker image usage

    • you have to mount your .kube directory in /root/.kube to use your Kubernetes configuration.

6.5.1. CronJob

Job creation from a CronJob
# Mandatory parameters
* string namespace = "my-namespace"
* string cronJobName = "my-cronjob-name"
* string jobName = "my-created-job-name"
# Optional parameters with default values
* def timeoutSeconds = 60 # (default)
# Run the job
* json result = kubernetes.cronJob.runJob({namespace, cronJobName, jobName, timeoutSeconds})
* match result.status == "OK"
* match result.jobDescription.metadata.name == jobName
* match result.jobDescription.spec.template.spec.containers[0].name == cronJobName
* match result.executeJobMessage == "job.batch/my-created-job-name created"
* match result.waitForJobCompletionMessage == "job.batch/my-created-job-name condition met"
* match result.deleteJobMessage == "job.batch/my-created-job-name deleted"
Job creation from a CronJob with more optional parameters
# Mandatory parameters
* string namespace = "my-namespace"
* string cronJobName = "my-cronjob-name"
* string jobName = "my-created-job-name"
# Optional parameters with default values
* def timeoutSeconds = 60 # (default)
# More optional parameters (no default values)
* json env = { "MY_ENV1": "MY_VALUE1", "MY_ENV2": "MY_VALUE2" }
* json command = [ "my-command" ]
* json args = [ "arg1", "arg2" ]
# Run the job
* json result = kubernetes.cronJob.runJob({namespace, cronJobName, jobName, timeoutSeconds, env, command, args})
* match result.status == "OK"
* match result.jobDescription.metadata.name == jobName
* match result.jobDescription.spec.template.spec.containers[0].name == cronJobName
* match result.jobDescription.spec.template.spec.containers[0].env == [ { "name":"MY_ENV1", "value":"MY_VALUE1" } , { "name":"MY_ENV2", "value":"MY_VALUE2" } ]
* match result.jobDescription.spec.template.spec.containers[0].command == [ "my-command" ]
* match result.jobDescription.spec.template.spec.containers[0].args == [ "arg1", "arg2" ]
* match result.executeJobMessage == "job.batch/my-created-job-name created"
* match result.waitForJobCompletionMessage == "job.batch/my-created-job-name condition met"
* match result.deleteJobMessage == "job.batch/my-created-job-name deleted"
Note
  • env: if the cronjob has already some environment variables, they will be merged with the new ones.

  • command and args: if the cronjob has already some command or args, they will be replaced by the new ones.

6.5.2. More info

6.6. dbt

Dbt calls

Note

For the fat JAR usage, you have to install dbt-snowflake Python package

6.6.1. CLI

DBT execution
# nominal case
* json result = dbt.cli.run({})
* match result.status == "OK"
* karate.log(result.output)
# with optional parameters
* json env = { "X": "valueX", "Y": "valueY" }
* string select = "my_model"
* string profilesDir = "/path/to/.dbt"
* string projectDir = "/path/to/dbtProject"
* string extra = "..."
* json result = dbt.cli.run({env, select, profilesDir, projectDir, extra})
* match result.status == "OK"
* karate.log(result.output)

6.6.2. More info

8. Contributing

9. Coding guidelines

TODO

10. Code of Conduct

TODO

11. Licensing

The code is licensed under Apache License, Version 2.0.

The documentation and logo are licensed under Creative Commons Attribution-ShareAlike 4.0 International Public License.

About

Karate enriched with extensions to connect to other systems

Resources

Contributing

Stars

4 stars

Watchers

4 watching

Forks

Releases

Packages

Used by

Contributors

Languages