About the project

Arcane is a Kubernetes-native data streaming platform powered by extendable plugin architecture.

Arcane is designed to be:

• Kubernetes-native: It runs data streaming plugins as a Kubernetes jobs and uses Kubernetes API to manage the lifecycle of data streams.

• Extendable: It allows you to extend the platform with your own data streaming plugins, which can be written in any programming language.

• Scalable: It can scale horizontally by adding more Kubernetes nodes to the cluster or vertically by adding more resources to the streaming jobs. It does not require any stateful components like Zookeeper or Kafka brokers which makes it easy to scale and manage.

• Lightweight: It has a small footprint and can run on any Kubernetes cluster, including local clusters like Minikube or Kind.

• Cloud-agnostic: It can run on any cloud provider that supports Kubernetes, including AWS, Azure, GCP, and on-premises clusters.

This repository contains the Arcane Operator, which is responsible for managing the lifecycle of Arcane data streams.

Table of Contents

• Getting started
  • Verify the installation
• Streaming plugins
  • Available ZIO-based streaming plugins
  • Available Akka-based streaming plugins
• Monitoring and observability
• Contributing
  • Extending the platform with your own plugins

Getting started

Install the Arcane CRDs before installing the Arcane Operator: https://github.com/SneaksAndData/arcane-crd. The operator chart expects the shared Arcane custom resource definitions to already exist in the cluster.

# Create a namespace for the operator installation
$ kubectl create namespace arcane

# Install the shared Arcane CRDs
$ helm install arcane-crd oci://ghcr.io/sneaksanddata/helm/arcane-crd \
  --version vX.Y.Z \
  --namespace arcane

# Install the operator in the created namespace
$ helm install arcane oci://ghcr.io/sneaksanddata/helm/arcane-operator \
  --version vX.Y.Z \
  --namespace arcane

The arcane-crd chart installs the shared Arcane CRDs:

• StreamClass
• StreamingJobTemplate
• BackfillRequest

Verify the installation

To verify the operator installation run the following command:

$ kubectl get pods -l app.kubernetes.io/name=arcane-operator --namespace arcane

It should produce the similar output:

NAME                               READY   STATUS    RESTARTS   AGE
arcane-operator-55988bbfcb-ql7qr   1/1     Running   0          25m

Once operator is installed, you can install the streaming plugins.

Platform roadmap

Please refer the roadmap for the Arcane Streaming Platform on the Arcane project page.

The most significant milestones are listed below:

• Support for ZIO-based streaming plugins
• Add contribution guidelines
• Rewrite the operator in Go
• Complete transition from Akka.NET to ZIO for the remaining streaming plugins

Streaming plugins

Available ZIO-based streaming plugins

• Sql Server streaming plugin: arcane-stream-sqlserver-change-tracking
• Microsoft Synapse streaming plugin: arcane-stream-microsoft-synapse-link
• Parquet streaming plugin: arcane-stream-parquet
• JSON streaming plugin: arcane-stream-json

Available Akka-based streaming plugins

• REST-api streaming plugin: arcane-stream-rest-api

User guide

Detailed Arcane user guide is available in the docs/user_guide.md file. User scenarios cheat sheet is available in the docs/user_scenarios.md file.

Monitoring and observability

-- TBD --

Contributing

-- TBD --

Extending the platform with plugins

-- TBD --