Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,6 @@ CLAUDE.md

temp/
docs/
tests/evals/results/*.jsonl
tests/evals/results/*.summary.json
tests/evals/eval.duckdb
77 changes: 77 additions & 0 deletions .golangci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# yaml-language-server: $schema=https://golangci-lint.run/jsonschema/golangci.jsonschema.json
version: "2"

run:
timeout: 5m
tests: true

linters:
default: standard
enable:
- staticcheck
- revive
- errcheck
- unused
- ineffassign
- govet
- funlen
- gocyclo
- unconvert
- misspell

settings:
funlen:
lines: 80
statements: 60
gocyclo:
min-complexity: 25
revive:
rules:
- name: exported
disabled: true
- name: package-comments
disabled: true
- name: var-naming
- name: error-return
- name: context-as-argument
- name: unexported-return

exclusions:
generated: lax
presets:
- comments
- common-false-positives
- legacy
- std-error-handling
rules:
- path: _test\.go
linters:
- funlen
- gocyclo
paths:
- temp

formatters:
enable:
- gofmt
- goimports
settings:
goimports:
local-prefixes:
- github.com/slingdata-io/sling-cli
exclusions:
paths:
- temp

issues:
new-from-merge-base: main
max-issues-per-linter: 0
max-same-issues: 0
uniq-by-line: true

output:
formats:
text:
path: stdout
print-linter-name: true
print-issued-lines: true
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Example [Replication](https://docs.slingdata.io/sling-cli/run/configuration/repl
---

Available Connectors:
- **Databases**: [`adbc`](https://docs.slingdata.io/connections/database-connections/adbc) [`azuredwh`](https://docs.slingdata.io/connections/database-connections/azuredwh) [`azuresql`](https://docs.slingdata.io/connections/database-connections/azuresql) [`azuretable`](https://docs.slingdata.io/connections/database-connections/azuretable) [`bigquery`](https://docs.slingdata.io/connections/database-connections/bigquery) [`bigtable`](https://docs.slingdata.io/connections/database-connections/bigtable) [`clickhouse`](https://docs.slingdata.io/connections/database-connections/clickhouse) [`d1`](https://docs.slingdata.io/connections/database-connections/d1) [`databricks`](https://docs.slingdata.io/connections/database-connections/databricks) [`duckdb`](https://docs.slingdata.io/connections/database-connections/duckdb) [`elasticsearch`](https://docs.slingdata.io/connections/database-connections/elasticsearch) [`exasol`](https://docs.slingdata.io/connections/database-connections/exasol) [`fabric`](https://docs.slingdata.io/connections/database-connections/fabric) [`mariadb`](https://docs.slingdata.io/connections/database-connections/mariadb) [`mongodb`](https://docs.slingdata.io/connections/database-connections/mongodb) [`motherduck`](https://docs.slingdata.io/connections/database-connections/motherduck) [`mysql`](https://docs.slingdata.io/connections/database-connections/mysql) [`odbc`](https://docs.slingdata.io/connections/database-connections/odbc) [`oracle`](https://docs.slingdata.io/connections/database-connections/oracle) [`postgres`](https://docs.slingdata.io/connections/database-connections/postgres) [`prometheus`](https://docs.slingdata.io/connections/database-connections/prometheus) [`proton`](https://docs.slingdata.io/connections/database-connections/proton) [`redshift`](https://docs.slingdata.io/connections/database-connections/redshift) [`snowflake`](https://docs.slingdata.io/connections/database-connections/snowflake) [`sqlite`](https://docs.slingdata.io/connections/database-connections/sqlite) [`sqlserver`](https://docs.slingdata.io/connections/database-connections/sqlserver) [`starrocks`](https://docs.slingdata.io/connections/database-connections/starrocks) [`trino`](https://docs.slingdata.io/connections/database-connections/trino)
- **Databases**: [`adbc`](https://docs.slingdata.io/connections/database-connections/adbc) [`azuredwh`](https://docs.slingdata.io/connections/database-connections/azuredwh) [`azuresql`](https://docs.slingdata.io/connections/database-connections/azuresql) [`azuretable`](https://docs.slingdata.io/connections/database-connections/azuretable) [`bigquery`](https://docs.slingdata.io/connections/database-connections/bigquery) [`bigtable`](https://docs.slingdata.io/connections/database-connections/bigtable) [`clickhouse`](https://docs.slingdata.io/connections/database-connections/clickhouse) [`d1`](https://docs.slingdata.io/connections/database-connections/d1) [`databricks`](https://docs.slingdata.io/connections/database-connections/databricks) [`duckdb`](https://docs.slingdata.io/connections/database-connections/duckdb) [`elasticsearch`](https://docs.slingdata.io/connections/database-connections/elasticsearch) [`exasol`](https://docs.slingdata.io/connections/database-connections/exasol) [`fabric`](https://docs.slingdata.io/connections/database-connections/fabric) [`mariadb`](https://docs.slingdata.io/connections/database-connections/mariadb) [`mongodb`](https://docs.slingdata.io/connections/database-connections/mongodb) [`motherduck`](https://docs.slingdata.io/connections/database-connections/motherduck) [`mysql`](https://docs.slingdata.io/connections/database-connections/mysql) [`odbc`](https://docs.slingdata.io/connections/database-connections/odbc) [`oracle`](https://docs.slingdata.io/connections/database-connections/oracle) [`postgres`](https://docs.slingdata.io/connections/database-connections/postgres) [`prometheus`](https://docs.slingdata.io/connections/database-connections/prometheus) [`proton`](https://docs.slingdata.io/connections/database-connections/proton) [`redshift`](https://docs.slingdata.io/connections/database-connections/redshift) [`scylladb`](https://docs.slingdata.io/connections/database-connections/scylladb) [`snowflake`](https://docs.slingdata.io/connections/database-connections/snowflake) [`sqlite`](https://docs.slingdata.io/connections/database-connections/sqlite) [`sqlserver`](https://docs.slingdata.io/connections/database-connections/sqlserver) [`starrocks`](https://docs.slingdata.io/connections/database-connections/starrocks) [`trino`](https://docs.slingdata.io/connections/database-connections/trino)

- **Data Lakes**:[`athena`](https://docs.slingdata.io/connections/datalake-connections/athena) [`Ducklake`](https://docs.slingdata.io/connections/datalake-connections/ducklake) [`iceberg`](https://docs.slingdata.io/connections/datalake-connections/iceberg) [`S3 Tables`](https://docs.slingdata.io/connections/datalake-connections/iceberg)

Expand Down
2 changes: 0 additions & 2 deletions api/specs/shopify_legacy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,6 @@ endpoints:
url: "{state.base_url}/product_variants.json" # Initial URL or next URL from pagination
updated_at_min: '{coalesce(env.UPDATED_AT_MIN, date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%S%z"))}'

sync: [ last_id ]

request:
url: '{state.url}'
parameters:
Expand Down
10 changes: 5 additions & 5 deletions cmd/sling/resource/llm_API_SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ endpoints:
{ coalesce(
env.LAST_UPDATED,
sync.last_updated,
date_format(date_add(now(), -30, 'day'), '%Y-%m-%dT%H:%M:%SZ')
date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%SZ")
)
}
page: 1 # Example for page-based pagination
Expand Down Expand Up @@ -976,7 +976,7 @@ endpoints:
start_timestamp: >
{ coalesce(
sync.last_sync_ts,
date_format(date_add(now(), -7, 'day'), '%Y-%m-%dT%H:%M:%SZ')
date_format(date_add(now(), -7, "day"), "%Y-%m-%dT%H:%M:%SZ")
)
}

Expand Down Expand Up @@ -1162,7 +1162,7 @@ endpoints:

You can use the following functions within `{...}` expressions in your API spec. Functions provide capabilities for data manipulation, type casting, date operations, control flow, and more.

**IMPORTANT:** Always use double quotes (`"`) for string literals in expressions, never single quotes (`'`). This is required by the [goval](https://github.com/maja42/goval) expression library that Sling uses.
**String literals:** double quotes (`"`) are preferred. Single-quoted SQL-style literals (`'%Y-%m-%d'`) are also accepted.

### String Functions

Expand Down Expand Up @@ -1207,7 +1207,7 @@ Uses Go's `time` package and `strftime` conventions via [timefmt-go](https://git
| `date_extract(date, part)` | Extracts part from date | `date`, `part` ("year", "month", "day", "hour", etc.) | Number | `date_extract(now(), "year")` → 2023 |
| `date_last(date[, period])` | Gets last day of period | `date`, `period` ("month", "year", default "month") | Time object | `date_last(now())` → Last day of current month |
| `date_first(date[, period])` | Gets first day of period | `date`, `period` ("month", "year", default "month") | Time object | `date_first(now())` → First day of current month |
| `range(start, end[, step])` | Creates array of time objects | `start`, `end` (time obj), `step` (string duration) | Array of Time objects | `range(date_add(now(),-2,'day'), now(), '1d')` → `[t-2d, t-1d, t]` |
| `range(start, end[, step])` | Creates array of time objects | `start`, `end` (time obj), `step` (string duration) | Array of Time objects | `range(date_add(now(),-2,"day"), now(), "1d")` → `[t-2d, t-1d, t]` |

*Date function `unit`/`part`/`period` parameters often accept: "year", "month", "week", "day", "hour", "minute", "second".*
*`range` function with dates requires time objects as start/end.*
Expand All @@ -1230,7 +1230,7 @@ Uses Go's `time` package and `strftime` conventions via [timefmt-go](https://git
# Format for API parameter (ISO 8601 with timezone)
request:
parameters:
updated_since: "{date_format(date_add(now(), -1, 'hour'), '%Y-%m-%dT%H:%M:%SZ')}"
updated_since: "{date_format(date_add(now(), -1, \"hour\"), \"%Y-%m-%dT%H:%M:%SZ\")}"
```

### Value Handling Functions
Expand Down
22 changes: 11 additions & 11 deletions cmd/sling/resource/llm_CONNECTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,9 +255,9 @@ API specifications define:
"type": "api",
"spec": "file:///path/to/my_api_spec.yaml",
"secrets": {
"api_key": "your-api-key",
"client_id": "your-client-id",
"client_secret": "your-client-secret"
"api_key": "${MY_API_API_KEY}",
"client_id": "${MY_API_CLIENT_ID}",
"client_secret": "${MY_API_CLIENT_SECRET}"
}
}
}
Expand Down Expand Up @@ -295,7 +295,7 @@ Create or update a connection in the Sling environment file:
"host": "localhost",
"user": "myuser",
"database": "mydb",
"password": "mypass",
"password": "${MY_POSTGRES_PASSWORD}",
"port": 5432
}
}
Expand All @@ -304,8 +304,8 @@ Create or update a connection in the Sling environment file:

**Important Notes:**
- Check existing connections with `list` before overwriting
- Sensitive credentials should be manually set in `~/.sling/env.yaml`
- The tool will provide the env.yaml file path after setting
- Never pass secret values. Use a `${VAR}` ref, or omit the secret field to scaffold a ref
- The tool returns `{path, line, missing}` so the user can replace refs in `env.yaml`

### Testing Connections

Expand Down Expand Up @@ -430,7 +430,7 @@ my_project:
type: postgres
host: localhost
user: myuser
password: mypass
password: ${PG_PASSWORD}
port: 5432
dbname: mydb
schema: public
Expand Down Expand Up @@ -605,10 +605,10 @@ streams:
"type": "api",
"spec": "file:///configs/salesforce_spec.yaml",
"secrets": {
"client_id": "your-client-id",
"client_secret": "your-client-secret",
"username": "api-user@company.com",
"password": "password-with-token"
"client_id": "${SALESFORCE_API_CLIENT_ID}",
"client_secret": "${SALESFORCE_API_CLIENT_SECRET}",
"username": "${SALESFORCE_API_USERNAME}",
"password": "${SALESFORCE_API_PASSWORD}"
}
}
}
Expand Down
3 changes: 1 addition & 2 deletions cmd/sling/resource/llm_CONNECTION_DATABASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ The `database` tool provides database-specific operations through various action
}
```

**Important**: Database operations require a Pro token and are rate-limited.
**Important**: Database operations are read-oriented and run locally through your configured connections.

---

Expand All @@ -85,7 +85,6 @@ Before using database operations:
1. **Connection Setup**: Database connection must be configured using the `connection` tool
2. **Connection Testing**: Verify connection works with `connection` tool `test` action
3. **Permissions**: Database user must have SELECT permissions on target objects
4. **Token**: Operations require a valid Sling CLI Pro token

### Operation Categories

Expand Down
3 changes: 1 addition & 2 deletions cmd/sling/resource/llm_CONNECTION_FILE.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ The `file_system` tool provides file system-specific operations through various
}
```

**Important**: File system operations require a Pro token and are rate-limited.
**Important**: File system operations are read-oriented and run locally through your configured connections.

---

Expand All @@ -85,7 +85,6 @@ Before using file system operations:
1. **Connection Setup**: File system connection must be configured using the `connection` tool
2. **Connection Testing**: Verify connection works with `connection` tool `test` action
3. **Permissions**: Connection must have appropriate read/write permissions
4. **Token**: Operations require a valid Sling CLI Pro token

### Operation Categories

Expand Down
46 changes: 20 additions & 26 deletions cmd/sling/resource/llm_PIPELINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,16 @@ Sling pipelines are powerful YAML-based workflow definitions that allow you to o

The Sling MCP tool provides these pipeline commands:
- `pipeline/docs` - Get documentation.
- `pipeline/parse` - Parse and validate the pipeline configuration.
- `pipeline/run` - Execute the pipeline.
- `pipeline/validate` - Parse and validate the pipeline configuration.

There is no MCP `run` action. Execute pipelines with the CLI: `sling run -p /path/to/pipeline.yaml`.
There is no MCP `build` action. Execute SQL models with the CLI: `sling build`.

---

## 2. Quick Start Guide

### Essential MCP Commands
### Essential Commands

```json
// Get pipeline documentation
Expand All @@ -66,21 +68,18 @@ The Sling MCP tool provides these pipeline commands:

// Parse a pipeline file
{
"action": "parse",
"action": "validate",
"input": {
"file_path": "/path/to/pipeline.yaml",
"working_dir": "/optional/work/dir"
}
}
```

// Run a pipeline
{
"action": "run",
"input": {
"file_path": "/path/to/pipeline.yaml",
"working_dir": "/optional/work/dir"
}
}
```bash
# Run a pipeline (CLI only — no MCP run action)
sling run -p /path/to/pipeline.yaml
sling run -p /path/to/pipeline.yaml --debug
```

### Basic Pipeline Structure
Expand Down Expand Up @@ -603,7 +602,7 @@ steps:
#### Parse Configuration
```json
{
"action": "parse",
"action": "validate",
"input": {
"file_path": "/path/to/pipeline.yaml",
"working_dir": "/optional/work/dir"
Expand All @@ -612,25 +611,20 @@ steps:
```

#### Execute Pipeline
```json
{
"action": "run",
"input": {
"file_path": "/path/to/pipeline.yaml",
"working_dir": "/optional/work/dir",
"env": {
"CUSTOM_VAR": "value"
}
}
}

There is no MCP `run` action. Use the CLI:

```bash
sling run -p /path/to/pipeline.yaml
sling run -p /path/to/pipeline.yaml --debug
```

### Development Workflow

1. **Write** your pipeline YAML file.
2. **Parse** the configuration to validate syntax: `{"action": "parse", "input": {"file_path": "my_pipeline.yaml"}}`.
2. **Validate** the configuration with MCP: `{"action": "validate", "input": {"file_path": "my_pipeline.yaml"}}`.
3. **Test** individual steps if possible (e.g., run `query` or `command` steps manually).
4. **Run** the full pipeline: `{"action": "run", "input": {"file_path": "my_pipeline.yaml"}}`.
4. **Run** the full pipeline with the CLI: `sling run -p my_pipeline.yaml`.
5. **Debug** by checking the logs and the output of each step.

---
Expand Down
8 changes: 4 additions & 4 deletions cmd/sling/resource/llm_PLATFORM.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

1. [Introduction and Overview](#1-introduction-and-overview)
2. [Authentication](#2-authentication)
3. [`sling project init`](#3-sling-project-init)
3. [`sling init`](#3-sling-init)
4. [`sling project status`](#4-sling-project-status)
5. [`sling project sync`](#5-sling-project-sync)
6. [`sling project jobs list`](#6-sling-project-jobs-list)
Expand Down Expand Up @@ -72,12 +72,12 @@ By default the CLI talks to `https://api.slingdata.io`. Override with:

---

## 3. `sling project init`
## 3. `sling init`

Create a `.sling.json` marker in the current directory so the CLI knows it is inside a project. Subsequent commands will be scoped to this directory tree.

```bash
sling project init
sling init
```

Effects:
Expand Down Expand Up @@ -550,7 +550,7 @@ The full `Job` JSON shape accepted by `save` and returned by `get`:
```bash
export SLING_PROJECT_TOKEN=...
cd ~/my-project
sling project init
sling init
# ... write replications/nightly.yaml ...
sling project sync -f
sling project jobs save --payload '{"name":"nightly","type":"replication","file_name":"replications/nightly.yaml","active":true,"schedules":["0 2 * * *"],"timezone":"UTC","config":{"mode":"incremental","threads":4}}'
Expand Down
Loading