diff --git a/.gitignore b/.gitignore index 6de8425a6..36bc38862 100644 --- a/.gitignore +++ b/.gitignore @@ -52,3 +52,6 @@ CLAUDE.md temp/ docs/ +tests/evals/results/*.jsonl +tests/evals/results/*.summary.json +tests/evals/eval.duckdb diff --git a/.golangci.yaml b/.golangci.yaml new file mode 100644 index 000000000..c0fd30fb2 --- /dev/null +++ b/.golangci.yaml @@ -0,0 +1,77 @@ +# yaml-language-server: $schema=https://golangci-lint.run/jsonschema/golangci.jsonschema.json +version: "2" + +run: + timeout: 5m + tests: true + +linters: + default: standard + enable: + - staticcheck + - revive + - errcheck + - unused + - ineffassign + - govet + - funlen + - gocyclo + - unconvert + - misspell + + settings: + funlen: + lines: 80 + statements: 60 + gocyclo: + min-complexity: 25 + revive: + rules: + - name: exported + disabled: true + - name: package-comments + disabled: true + - name: var-naming + - name: error-return + - name: context-as-argument + - name: unexported-return + + exclusions: + generated: lax + presets: + - comments + - common-false-positives + - legacy + - std-error-handling + rules: + - path: _test\.go + linters: + - funlen + - gocyclo + paths: + - temp + +formatters: + enable: + - gofmt + - goimports + settings: + goimports: + local-prefixes: + - github.com/slingdata-io/sling-cli + exclusions: + paths: + - temp + +issues: + new-from-merge-base: main + max-issues-per-linter: 0 + max-same-issues: 0 + uniq-by-line: true + +output: + formats: + text: + path: stdout + print-linter-name: true + print-issued-lines: true \ No newline at end of file diff --git a/README.md b/README.md index 54e4800de..78d913ec4 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ Example [Replication](https://docs.slingdata.io/sling-cli/run/configuration/repl --- Available Connectors: -- **Databases**: [`adbc`](https://docs.slingdata.io/connections/database-connections/adbc) [`azuredwh`](https://docs.slingdata.io/connections/database-connections/azuredwh) [`azuresql`](https://docs.slingdata.io/connections/database-connections/azuresql) [`azuretable`](https://docs.slingdata.io/connections/database-connections/azuretable) [`bigquery`](https://docs.slingdata.io/connections/database-connections/bigquery) [`bigtable`](https://docs.slingdata.io/connections/database-connections/bigtable) [`clickhouse`](https://docs.slingdata.io/connections/database-connections/clickhouse) [`d1`](https://docs.slingdata.io/connections/database-connections/d1) [`databricks`](https://docs.slingdata.io/connections/database-connections/databricks) [`duckdb`](https://docs.slingdata.io/connections/database-connections/duckdb) [`elasticsearch`](https://docs.slingdata.io/connections/database-connections/elasticsearch) [`exasol`](https://docs.slingdata.io/connections/database-connections/exasol) [`fabric`](https://docs.slingdata.io/connections/database-connections/fabric) [`mariadb`](https://docs.slingdata.io/connections/database-connections/mariadb) [`mongodb`](https://docs.slingdata.io/connections/database-connections/mongodb) [`motherduck`](https://docs.slingdata.io/connections/database-connections/motherduck) [`mysql`](https://docs.slingdata.io/connections/database-connections/mysql) [`odbc`](https://docs.slingdata.io/connections/database-connections/odbc) [`oracle`](https://docs.slingdata.io/connections/database-connections/oracle) [`postgres`](https://docs.slingdata.io/connections/database-connections/postgres) [`prometheus`](https://docs.slingdata.io/connections/database-connections/prometheus) [`proton`](https://docs.slingdata.io/connections/database-connections/proton) [`redshift`](https://docs.slingdata.io/connections/database-connections/redshift) [`snowflake`](https://docs.slingdata.io/connections/database-connections/snowflake) [`sqlite`](https://docs.slingdata.io/connections/database-connections/sqlite) [`sqlserver`](https://docs.slingdata.io/connections/database-connections/sqlserver) [`starrocks`](https://docs.slingdata.io/connections/database-connections/starrocks) [`trino`](https://docs.slingdata.io/connections/database-connections/trino) +- **Databases**: [`adbc`](https://docs.slingdata.io/connections/database-connections/adbc) [`azuredwh`](https://docs.slingdata.io/connections/database-connections/azuredwh) [`azuresql`](https://docs.slingdata.io/connections/database-connections/azuresql) [`azuretable`](https://docs.slingdata.io/connections/database-connections/azuretable) [`bigquery`](https://docs.slingdata.io/connections/database-connections/bigquery) [`bigtable`](https://docs.slingdata.io/connections/database-connections/bigtable) [`clickhouse`](https://docs.slingdata.io/connections/database-connections/clickhouse) [`d1`](https://docs.slingdata.io/connections/database-connections/d1) [`databricks`](https://docs.slingdata.io/connections/database-connections/databricks) [`duckdb`](https://docs.slingdata.io/connections/database-connections/duckdb) [`elasticsearch`](https://docs.slingdata.io/connections/database-connections/elasticsearch) [`exasol`](https://docs.slingdata.io/connections/database-connections/exasol) [`fabric`](https://docs.slingdata.io/connections/database-connections/fabric) [`mariadb`](https://docs.slingdata.io/connections/database-connections/mariadb) [`mongodb`](https://docs.slingdata.io/connections/database-connections/mongodb) [`motherduck`](https://docs.slingdata.io/connections/database-connections/motherduck) [`mysql`](https://docs.slingdata.io/connections/database-connections/mysql) [`odbc`](https://docs.slingdata.io/connections/database-connections/odbc) [`oracle`](https://docs.slingdata.io/connections/database-connections/oracle) [`postgres`](https://docs.slingdata.io/connections/database-connections/postgres) [`prometheus`](https://docs.slingdata.io/connections/database-connections/prometheus) [`proton`](https://docs.slingdata.io/connections/database-connections/proton) [`redshift`](https://docs.slingdata.io/connections/database-connections/redshift) [`scylladb`](https://docs.slingdata.io/connections/database-connections/scylladb) [`snowflake`](https://docs.slingdata.io/connections/database-connections/snowflake) [`sqlite`](https://docs.slingdata.io/connections/database-connections/sqlite) [`sqlserver`](https://docs.slingdata.io/connections/database-connections/sqlserver) [`starrocks`](https://docs.slingdata.io/connections/database-connections/starrocks) [`trino`](https://docs.slingdata.io/connections/database-connections/trino) - **Data Lakes**:[`athena`](https://docs.slingdata.io/connections/datalake-connections/athena) [`Ducklake`](https://docs.slingdata.io/connections/datalake-connections/ducklake) [`iceberg`](https://docs.slingdata.io/connections/datalake-connections/iceberg) [`S3 Tables`](https://docs.slingdata.io/connections/datalake-connections/iceberg) diff --git a/api/specs/shopify_legacy.yaml b/api/specs/shopify_legacy.yaml index 0416dd6f8..112d09708 100644 --- a/api/specs/shopify_legacy.yaml +++ b/api/specs/shopify_legacy.yaml @@ -100,8 +100,6 @@ endpoints: url: "{state.base_url}/product_variants.json" # Initial URL or next URL from pagination updated_at_min: '{coalesce(env.UPDATED_AT_MIN, date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%S%z"))}' - sync: [ last_id ] - request: url: '{state.url}' parameters: diff --git a/cmd/sling/resource/llm_API_SPEC.md b/cmd/sling/resource/llm_API_SPEC.md index 800680122..56378561c 100644 --- a/cmd/sling/resource/llm_API_SPEC.md +++ b/cmd/sling/resource/llm_API_SPEC.md @@ -159,7 +159,7 @@ endpoints: { coalesce( env.LAST_UPDATED, sync.last_updated, - date_format(date_add(now(), -30, 'day'), '%Y-%m-%dT%H:%M:%SZ') + date_format(date_add(now(), -30, "day"), "%Y-%m-%dT%H:%M:%SZ") ) } page: 1 # Example for page-based pagination @@ -976,7 +976,7 @@ endpoints: start_timestamp: > { coalesce( sync.last_sync_ts, - date_format(date_add(now(), -7, 'day'), '%Y-%m-%dT%H:%M:%SZ') + date_format(date_add(now(), -7, "day"), "%Y-%m-%dT%H:%M:%SZ") ) } @@ -1162,7 +1162,7 @@ endpoints: You can use the following functions within `{...}` expressions in your API spec. Functions provide capabilities for data manipulation, type casting, date operations, control flow, and more. -**IMPORTANT:** Always use double quotes (`"`) for string literals in expressions, never single quotes (`'`). This is required by the [goval](https://github.com/maja42/goval) expression library that Sling uses. +**String literals:** double quotes (`"`) are preferred. Single-quoted SQL-style literals (`'%Y-%m-%d'`) are also accepted. ### String Functions @@ -1207,7 +1207,7 @@ Uses Go's `time` package and `strftime` conventions via [timefmt-go](https://git | `date_extract(date, part)` | Extracts part from date | `date`, `part` ("year", "month", "day", "hour", etc.) | Number | `date_extract(now(), "year")` → 2023 | | `date_last(date[, period])` | Gets last day of period | `date`, `period` ("month", "year", default "month") | Time object | `date_last(now())` → Last day of current month | | `date_first(date[, period])` | Gets first day of period | `date`, `period` ("month", "year", default "month") | Time object | `date_first(now())` → First day of current month | -| `range(start, end[, step])` | Creates array of time objects | `start`, `end` (time obj), `step` (string duration) | Array of Time objects | `range(date_add(now(),-2,'day'), now(), '1d')` → `[t-2d, t-1d, t]` | +| `range(start, end[, step])` | Creates array of time objects | `start`, `end` (time obj), `step` (string duration) | Array of Time objects | `range(date_add(now(),-2,"day"), now(), "1d")` → `[t-2d, t-1d, t]` | *Date function `unit`/`part`/`period` parameters often accept: "year", "month", "week", "day", "hour", "minute", "second".* *`range` function with dates requires time objects as start/end.* @@ -1230,7 +1230,7 @@ Uses Go's `time` package and `strftime` conventions via [timefmt-go](https://git # Format for API parameter (ISO 8601 with timezone) request: parameters: - updated_since: "{date_format(date_add(now(), -1, 'hour'), '%Y-%m-%dT%H:%M:%SZ')}" + updated_since: "{date_format(date_add(now(), -1, \"hour\"), \"%Y-%m-%dT%H:%M:%SZ\")}" ``` ### Value Handling Functions diff --git a/cmd/sling/resource/llm_CONNECTION.md b/cmd/sling/resource/llm_CONNECTION.md index 820595632..89ac04abc 100644 --- a/cmd/sling/resource/llm_CONNECTION.md +++ b/cmd/sling/resource/llm_CONNECTION.md @@ -255,9 +255,9 @@ API specifications define: "type": "api", "spec": "file:///path/to/my_api_spec.yaml", "secrets": { - "api_key": "your-api-key", - "client_id": "your-client-id", - "client_secret": "your-client-secret" + "api_key": "${MY_API_API_KEY}", + "client_id": "${MY_API_CLIENT_ID}", + "client_secret": "${MY_API_CLIENT_SECRET}" } } } @@ -295,7 +295,7 @@ Create or update a connection in the Sling environment file: "host": "localhost", "user": "myuser", "database": "mydb", - "password": "mypass", + "password": "${MY_POSTGRES_PASSWORD}", "port": 5432 } } @@ -304,8 +304,8 @@ Create or update a connection in the Sling environment file: **Important Notes:** - Check existing connections with `list` before overwriting -- Sensitive credentials should be manually set in `~/.sling/env.yaml` -- The tool will provide the env.yaml file path after setting +- Never pass secret values. Use a `${VAR}` ref, or omit the secret field to scaffold a ref +- The tool returns `{path, line, missing}` so the user can replace refs in `env.yaml` ### Testing Connections @@ -430,7 +430,7 @@ my_project: type: postgres host: localhost user: myuser - password: mypass + password: ${PG_PASSWORD} port: 5432 dbname: mydb schema: public @@ -605,10 +605,10 @@ streams: "type": "api", "spec": "file:///configs/salesforce_spec.yaml", "secrets": { - "client_id": "your-client-id", - "client_secret": "your-client-secret", - "username": "api-user@company.com", - "password": "password-with-token" + "client_id": "${SALESFORCE_API_CLIENT_ID}", + "client_secret": "${SALESFORCE_API_CLIENT_SECRET}", + "username": "${SALESFORCE_API_USERNAME}", + "password": "${SALESFORCE_API_PASSWORD}" } } } diff --git a/cmd/sling/resource/llm_CONNECTION_DATABASE.md b/cmd/sling/resource/llm_CONNECTION_DATABASE.md index 7893e198a..e363c038a 100644 --- a/cmd/sling/resource/llm_CONNECTION_DATABASE.md +++ b/cmd/sling/resource/llm_CONNECTION_DATABASE.md @@ -49,6 +49,7 @@ The `database` tool provides database-specific operations through various action - `get_schemas` - Get list of schema names - `get_columns` - Get column metadata for a table - `query` - Execute read-only SQL queries +- `query_cancel` - Cancel a running query by `query_id` ### Basic Tool Usage @@ -72,7 +73,7 @@ The `database` tool provides database-specific operations through various action } ``` -**Important**: Database operations require a Pro token and are rate-limited. +**Important**: Database operations are read-oriented and run locally through your configured connections. --- @@ -85,7 +86,6 @@ Before using database operations: 1. **Connection Setup**: Database connection must be configured using the `connection` tool 2. **Connection Testing**: Verify connection works with `connection` tool `test` action 3. **Permissions**: Database user must have SELECT permissions on target objects -4. **Token**: Operations require a valid Sling CLI Pro token ### Operation Categories @@ -96,6 +96,7 @@ Before using database operations: #### Data Access Operations - `query` - Read-only SQL execution +- `query_cancel` - Cancel a running query --- @@ -235,6 +236,7 @@ Run read-only SQL queries on database connections: - `description` (optional but strongly recommended) - A brief description of the intent and expected result of this query. Explain *why* you are running this query and what the result should tell you. This is logged for observability so that query activity can be understood in context. **Always provide this when executing a query.** - `limit` (optional) - Maximum rows to return (default: 100) - `transient` (optional) - Use transient connection (default: false) +- `query_id` (optional) - Client-supplied id (`[A-Za-z0-9_-]`, max 64) so a concurrent `query_cancel` can target this query. Generated if omitted. Long queries are supported; interrupting the client (Ctrl+C / Escape) cancels the running query. #### Referencing a Query File with `file://` @@ -253,6 +255,24 @@ If the SQL query already exists as a file on disk, pass its path as a `file://` The path must be absolute. Sling reads the file and executes its contents as the SQL query. The same read-only restrictions below apply to file-loaded queries. +### Cancelling a Running Query + +Pass a `query_id` on `query`, then call `query_cancel` from a concurrent tool call (clients that pipeline JSON-RPC). Sequential clients should interrupt the first call (Ctrl+C / Escape); that cancels the query automatically. + +```json +{ + "action": "query_cancel", + "input": { + "query_id": "my-q1" + } +} +``` + +**Parameters:** +- `query_id` (required) - The id of the running query to cancel + +Idempotent. Returns `{cancelled: true, query_id}` if that query was running, or `{cancelled: false, query_id}` if it was not. + ### Query Safety and Restrictions **WARNING: Only use this tool for SELECT queries and other read-only operations. Never execute destructive queries such as DELETE, DROP, TRUNCATE, ALTER, UPDATE, INSERT, or any other data modification operations.** diff --git a/cmd/sling/resource/llm_CONNECTION_FILE.md b/cmd/sling/resource/llm_CONNECTION_FILE.md index 6dc7e47d1..5e43e3216 100644 --- a/cmd/sling/resource/llm_CONNECTION_FILE.md +++ b/cmd/sling/resource/llm_CONNECTION_FILE.md @@ -72,7 +72,7 @@ The `file_system` tool provides file system-specific operations through various } ``` -**Important**: File system operations require a Pro token and are rate-limited. +**Important**: File system operations are read-oriented and run locally through your configured connections. --- @@ -85,7 +85,6 @@ Before using file system operations: 1. **Connection Setup**: File system connection must be configured using the `connection` tool 2. **Connection Testing**: Verify connection works with `connection` tool `test` action 3. **Permissions**: Connection must have appropriate read/write permissions -4. **Token**: Operations require a valid Sling CLI Pro token ### Operation Categories diff --git a/cmd/sling/resource/llm_PIPELINE.md b/cmd/sling/resource/llm_PIPELINE.md index 746316f00..c0b02d589 100644 --- a/cmd/sling/resource/llm_PIPELINE.md +++ b/cmd/sling/resource/llm_PIPELINE.md @@ -48,14 +48,16 @@ Sling pipelines are powerful YAML-based workflow definitions that allow you to o The Sling MCP tool provides these pipeline commands: - `pipeline/docs` - Get documentation. -- `pipeline/parse` - Parse and validate the pipeline configuration. -- `pipeline/run` - Execute the pipeline. +- `pipeline/validate` - Parse and validate the pipeline configuration. + +There is no MCP `run` action. Execute pipelines with the CLI: `sling run -p /path/to/pipeline.yaml`. +There is no MCP `build` action. Execute SQL models with the CLI: `sling build`. --- ## 2. Quick Start Guide -### Essential MCP Commands +### Essential Commands ```json // Get pipeline documentation @@ -66,21 +68,18 @@ The Sling MCP tool provides these pipeline commands: // Parse a pipeline file { - "action": "parse", + "action": "validate", "input": { "file_path": "/path/to/pipeline.yaml", "working_dir": "/optional/work/dir" } } +``` -// Run a pipeline -{ - "action": "run", - "input": { - "file_path": "/path/to/pipeline.yaml", - "working_dir": "/optional/work/dir" - } -} +```bash +# Run a pipeline (CLI only — no MCP run action) +sling run -p /path/to/pipeline.yaml +sling run -p /path/to/pipeline.yaml --debug ``` ### Basic Pipeline Structure @@ -603,7 +602,7 @@ steps: #### Parse Configuration ```json { - "action": "parse", + "action": "validate", "input": { "file_path": "/path/to/pipeline.yaml", "working_dir": "/optional/work/dir" @@ -612,25 +611,20 @@ steps: ``` #### Execute Pipeline -```json -{ - "action": "run", - "input": { - "file_path": "/path/to/pipeline.yaml", - "working_dir": "/optional/work/dir", - "env": { - "CUSTOM_VAR": "value" - } - } -} + +There is no MCP `run` action. Use the CLI: + +```bash +sling run -p /path/to/pipeline.yaml +sling run -p /path/to/pipeline.yaml --debug ``` ### Development Workflow 1. **Write** your pipeline YAML file. -2. **Parse** the configuration to validate syntax: `{"action": "parse", "input": {"file_path": "my_pipeline.yaml"}}`. +2. **Validate** the configuration with MCP: `{"action": "validate", "input": {"file_path": "my_pipeline.yaml"}}`. 3. **Test** individual steps if possible (e.g., run `query` or `command` steps manually). -4. **Run** the full pipeline: `{"action": "run", "input": {"file_path": "my_pipeline.yaml"}}`. +4. **Run** the full pipeline with the CLI: `sling run -p my_pipeline.yaml`. 5. **Debug** by checking the logs and the output of each step. --- diff --git a/cmd/sling/resource/llm_PLATFORM.md b/cmd/sling/resource/llm_PLATFORM.md index b11bff0c9..56ab99f15 100644 --- a/cmd/sling/resource/llm_PLATFORM.md +++ b/cmd/sling/resource/llm_PLATFORM.md @@ -4,18 +4,18 @@ 1. [Introduction and Overview](#1-introduction-and-overview) 2. [Authentication](#2-authentication) -3. [`sling project init`](#3-sling-project-init) -4. [`sling project status`](#4-sling-project-status) -5. [`sling project sync`](#5-sling-project-sync) -6. [`sling project jobs list`](#6-sling-project-jobs-list) -7. [`sling project jobs status`](#7-sling-project-jobs-status) -8. [`sling project jobs trigger`](#8-sling-project-jobs-trigger) -9. [`sling project jobs get`](#9-sling-project-jobs-get) -10. [`sling project jobs save`](#10-sling-project-jobs-save) -11. [`sling project jobs delete`](#11-sling-project-jobs-delete) -12. [`sling project execs`](#12-sling-project-execs) -13. [`sling project files`](#13-sling-project-files) -14. [`sling project connections`](#14-sling-project-connections) +3. [`sling init`](#3-sling-init) +4. [`sling platform status`](#4-sling-platform-status) +5. [`sling platform sync`](#5-sling-platform-sync) +6. [`sling platform jobs list`](#6-sling-platform-jobs-list) +7. [`sling platform jobs status`](#7-sling-platform-jobs-status) +8. [`sling platform jobs trigger`](#8-sling-platform-jobs-trigger) +9. [`sling platform jobs get`](#9-sling-platform-jobs-get) +10. [`sling platform jobs save`](#10-sling-platform-jobs-save) +11. [`sling platform jobs delete`](#11-sling-platform-jobs-delete) +12. [`sling platform execs`](#12-sling-platform-execs) +13. [`sling platform files`](#13-sling-platform-files) +14. [`sling platform connections`](#14-sling-platform-connections) 15. [Job Payload Reference](#15-job-payload-reference) 16. [Common Workflows](#16-common-workflows) 17. [Troubleshooting](#17-troubleshooting) @@ -26,9 +26,9 @@ ### What is the Sling Platform? -The Sling Platform (hosted at `api.slingdata.io`, UI at `platform.slingdata.io`) is the commercial control plane for Sling. It stores projects, project files, scheduled jobs, execution history, monitors, and agents. The `sling project ...` subcommands in the CLI let you interact with a platform project from the terminal and from scripts — without touching the UI. +The Sling Platform (hosted at `api.slingdata.io`, UI at `platform.slingdata.io`) is the commercial control plane for Sling. It stores projects, project files, scheduled jobs, execution history, monitors, and agents. The `sling platform ...` subcommands in the CLI let you interact with a platform project from the terminal and from scripts — without touching the UI. -### When to use the `sling project` commands +### When to use the `sling platform` commands - You have a Sling Platform project and want to push local YAML file changes up (`sync`). - You want to script job lifecycle management (create, read, update, trigger) from CI or local automation. @@ -39,19 +39,19 @@ The Sling Platform (hosted at `api.slingdata.io`, UI at `platform.slingdata.io`) - **Project**: A workspace on the platform containing project files (YAML replications/pipelines/monitors/queries) and jobs. - **Project token**: A scoped API token (36-character UUID) that authenticates requests for one project. Created in `Settings > API Tokens` on the platform UI. -- **Project file**: A YAML/SQL file stored in the project (e.g. `replications/hackernews.yaml`). Synced from local disk via `sling project sync`. +- **Project file**: A YAML/SQL file stored in the project (e.g. `replications/hackernews.yaml`). Synced from local disk via `sling platform sync`. - **Job**: A scheduled or triggerable definition pointing at a project file, with its own `schedules`, `streams`, `config`, `notification_settings`, etc. Job IDs have the prefix `job_`. - **Execution**: One run of a job. Execution IDs have the prefix `exec_`. ### Relationship to the rest of the CLI -`sling project` commands are **only** for interacting with the hosted platform. They have nothing to do with running data movement locally — for that use `sling run ...`. The platform commands are thin REST clients over `https://api.slingdata.io` (or a self-hosted URL if `SLING_PLATFORM_HOST` is set). +`sling platform` commands are **only** for interacting with the hosted platform. They have nothing to do with running data movement locally — for that use `sling run ...`. The platform commands are thin REST clients over `https://api.slingdata.io` (or a self-hosted URL if `SLING_PLATFORM_HOST` is set). --- ## 2. Authentication -All `sling project ...` commands require a project token via environment variable: +All `sling platform ...` commands require a project token via environment variable: ```bash export SLING_PROJECT_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx @@ -72,12 +72,12 @@ By default the CLI talks to `https://api.slingdata.io`. Override with: --- -## 3. `sling project init` +## 3. `sling init` Create a `.sling.json` marker in the current directory so the CLI knows it is inside a project. Subsequent commands will be scoped to this directory tree. ```bash -sling project init +sling init ``` Effects: @@ -89,26 +89,26 @@ The `id` field is populated when `SLING_PROJECT_TOKEN` is validated — you do n --- -## 4. `sling project status` +## 4. `sling platform status` Print a summary of the linked platform project: project ID, name, organization, owner, plus a key/value table of status details (counts of files, jobs, executions, etc.). ```bash -sling project status +sling platform status ``` Uses `GET /project/get?type=status_details`. Token required. --- -## 5. `sling project sync` +## 5. `sling platform sync` Diff local project files against the platform's stored copies and push any that are new or locally newer. ```bash -sling project sync # prompts before pushing -sling project sync --force # skip the confirmation prompt -sling project sync -f # short form of --force +sling platform sync # prompts before pushing +sling platform sync --force # skip the confirmation prompt +sling platform sync -f # short form of --force ``` Flags: @@ -122,16 +122,16 @@ Behavior: --- -## 6. `sling project jobs list` +## 6. `sling platform jobs list` Print all jobs in the project as a table (default) or JSON. ```bash -sling project jobs list # all jobs, table -sling project jobs list --type replication # only replication jobs -sling project jobs list --file-name replications/x.yaml # filter by file name -sling project jobs list --name "Nightly" # filter by job name -sling project jobs list -o json | jq '.[].id' # JSON for scripting +sling platform jobs list # all jobs, table +sling platform jobs list --type replication # only replication jobs +sling platform jobs list --file-name replications/x.yaml # filter by file name +sling platform jobs list --name "Nightly" # filter by job name +sling platform jobs list -o json | jq '.[].id' # JSON for scripting ``` Flags: @@ -144,15 +144,15 @@ Columns (table mode): `ID`, `Name`, `File Name`, `Type`, `Status`, `Active`, `Ex --- -## 7. `sling project jobs status` +## 7. `sling platform jobs status` Per-job overview matching the platform home page: one row per job with the latest execution's status. Jobs with no runs in the last 60 days still appear with `-` placeholders. ```bash -sling project jobs status -sling project jobs status --name hacker # substring match on job name -sling project jobs status --id job_0vnx # substring match on job id -sling project jobs status -o json +sling platform jobs status +sling platform jobs status --name hacker # substring match on job name +sling platform jobs status --id job_0vnx # substring match on job id +sling platform jobs status -o json ``` Flags: @@ -168,17 +168,17 @@ Backed by `GET /project/dashboard?name=home_dashboard_job_history`; merged with --- -## 8. `sling project jobs trigger` +## 8. `sling platform jobs trigger` Kick off a job run. Optionally override which streams run and the mode. ```bash -sling project jobs trigger job_0vnx9sjkjzmd95nh -sling project jobs trigger job_0vnx9sjkjzmd95nh --wait -sling project jobs trigger job_0vnx9sjkjzmd95nh -w -sling project jobs trigger job_0vnx9sjkjzmd95nh --streams users,orders -sling project jobs trigger job_0vnx9sjkjzmd95nh --full-refresh -sling project jobs trigger job_0vnx9sjkjzmd95nh --streams u --full-refresh --wait +sling platform jobs trigger job_0vnx9sjkjzmd95nh +sling platform jobs trigger job_0vnx9sjkjzmd95nh --wait +sling platform jobs trigger job_0vnx9sjkjzmd95nh -w +sling platform jobs trigger job_0vnx9sjkjzmd95nh --streams users,orders +sling platform jobs trigger job_0vnx9sjkjzmd95nh --full-refresh +sling platform jobs trigger job_0vnx9sjkjzmd95nh --streams u --full-refresh --wait ``` Flags: @@ -190,41 +190,41 @@ The positional argument must begin with `job_` or the CLI rejects it as invalid. --- -## 9. `sling project jobs get` +## 9. `sling platform jobs get` Fetch a single job's full definition as JSON to stdout. ```bash -sling project jobs get job_0vnx9sjkjzmd95nh -sling project jobs get job_0vnx9sjkjzmd95nh > /tmp/job.json -sling project jobs get job_0vnx9sjkjzmd95nh | jq '.config' +sling platform jobs get job_0vnx9sjkjzmd95nh +sling platform jobs get job_0vnx9sjkjzmd95nh > /tmp/job.json +sling platform jobs get job_0vnx9sjkjzmd95nh | jq '.config' ``` -Output is pretty-printed JSON of the `Job` model. The shape is a round-trip — you can edit the JSON and pass it directly to `sling project jobs save --file`. +Output is pretty-printed JSON of the `Job` model. The shape is a round-trip — you can edit the JSON and pass it directly to `sling platform jobs save --file`. Uses `GET /project/job/get?job_id=`. The positional argument must begin with `job_`. --- -## 10. `sling project jobs save` +## 10. `sling platform jobs save` Create a job (if `id` is empty/missing) or update an existing one (if `id` is present). A single command handles both. ```bash # Create: omit id -sling project jobs save --payload '{"name":"nightly","type":"replication","file_name":"replications/nightly.yaml","active":true,"schedules":["0 2 * * *"],"timezone":"UTC","config":{}}' +sling platform jobs save --payload '{"name":"nightly","type":"replication","file_name":"replications/nightly.yaml","active":true,"schedules":["0 2 * * *"],"timezone":"UTC","config":{}}' # Update from edited file -sling project jobs get job_abc123 > /tmp/job.json +sling platform jobs get job_abc123 > /tmp/job.json # ... edit /tmp/job.json ... -sling project jobs save --file /tmp/job.json +sling platform jobs save --file /tmp/job.json # Short flags -sling project jobs save -p '{"name":"test","type":"replication","file_name":"r.yaml","config":{}}' -sling project jobs save -f /tmp/job.json +sling platform jobs save -p '{"name":"test","type":"replication","file_name":"r.yaml","config":{}}' +sling platform jobs save -f /tmp/job.json # Pipe from get (stdin via `-`) -sling project jobs get job_abc123 | sling project jobs save --file - +sling platform jobs get job_abc123 | sling platform jobs save --file - ``` Flags (exactly one required): @@ -253,14 +253,14 @@ The platform enforces plan limits and will reject the save with an error: --- -## 11. `sling project jobs delete` +## 11. `sling platform jobs delete` Delete a job by ID. ```bash -sling project jobs delete job_abc123 # prompts Y/N -sling project jobs delete job_abc123 -f # no prompt -sling project jobs delete job_abc123 --force # no prompt +sling platform jobs delete job_abc123 # prompts Y/N +sling platform jobs delete job_abc123 -f # no prompt +sling platform jobs delete job_abc123 --force # no prompt ``` Flags: @@ -272,7 +272,7 @@ Uses `POST /project/job/delete` with body `{"job_id": ""}`. --- -## 12. `sling project execs` +## 12. `sling platform execs` Manage and inspect executions. @@ -281,12 +281,12 @@ Manage and inspect executions. List recent executions with a job-name column, derived status, and optional time range. Newest first. ```bash -sling project execs list # default: 10 rows, table -sling project execs list --status error --limit 5 # filter by status -sling project execs list --job-id job_abc123 -o json # scope to one job, JSON -sling project execs list --since 24h # last 24 hours -sling project execs list --since 7d --status success -sling project execs list --since 2026-04-10 --until 2026-04-17 +sling platform execs list # default: 10 rows, table +sling platform execs list --status error --limit 5 # filter by status +sling platform execs list --job-id job_abc123 -o json # scope to one job, JSON +sling platform execs list --since 24h # last 24 hours +sling platform execs list --since 7d --status success +sling platform execs list --since 2026-04-10 --until 2026-04-17 ``` Flags: @@ -304,8 +304,8 @@ Columns (table): `Name`, `Exec ID`, `Job ID`, `Start Time`, `Rows`, `Bytes`, `St Fetch the full server-side status record for one execution as pretty-printed JSON. ```bash -sling project execs status exec_abc123 -sling project execs status exec_abc123 | jq '.status_map' +sling platform execs status exec_abc123 +sling platform execs status exec_abc123 | jq '.status_map' ``` Uses `GET /execution/list?filters=%7B%22exec_id%22%3A%22...%22%7D`. @@ -315,7 +315,7 @@ Uses `GET /execution/list?filters=%7B%22exec_id%22%3A%22...%22%7D`. Cancel a running execution. ```bash -sling project execs cancel exec_abc123 +sling platform execs cancel exec_abc123 ``` Uses `POST /execution/cancel` with body `{"exec_id": ""}`. @@ -325,11 +325,11 @@ Uses `POST /execution/cancel` with body `{"exec_id": ""}`. Print the full log output captured during an execution. Each task/step's log is prefixed with a `=== (status=...) ===` banner. The log contains ANSI color codes by default; use `--no-color` to strip them for grep/diff/save-to-file. ```bash -sling project execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X # all streams, colored -sling project execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X --no-color # plain text -sling project execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X --task users # only the "users" stream -sling project execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X --status error # only errored tasks/steps -sling project execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X -o json > tasks.json # raw task records +sling platform execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X # all streams, colored +sling platform execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X --no-color # plain text +sling platform execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X --task users # only the "users" stream +sling platform execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X --status error # only errored tasks/steps +sling platform execs log 3CUJmxllsG8YC1kDYGzU6jYjS5X -o json > tasks.json # raw task records ``` Flags: @@ -343,15 +343,15 @@ Backed by `POST /execution/replication-tasks` for replication/query/monitor exec --- -## 13. `sling project files` +## 13. `sling platform files` -Manage project files stored on the platform. For bulk push from local disk, prefer `sling project sync`. +Manage project files stored on the platform. For bulk push from local disk, prefer `sling platform sync`. ### `files list` ```bash -sling project files list -sling project files list -o json | jq '.[].name' +sling platform files list +sling platform files list -o json | jq '.[].name' ``` Columns (table): `Name`, `Size`, `Updated`. `-o json` emits the full file records. @@ -361,8 +361,8 @@ Columns (table): `Name`, `Size`, `Updated`. `-o json` emits the full file record Print a project file's body to stdout. Output is the raw file body, **not** JSON-wrapped, so shell redirection works: ```bash -sling project files get replications/hackernews.yaml -sling project files get replications/hackernews.yaml > /tmp/hackernews.yaml +sling platform files get replications/hackernews.yaml +sling platform files get replications/hackernews.yaml > /tmp/hackernews.yaml ``` ### `files save ` @@ -371,23 +371,23 @@ Create or update a project file. For Sling job files (replication/pipeline/monit ```bash # inline body -sling project files save replications/foo.yaml --body 'source: PG +sling platform files save replications/foo.yaml --body 'source: PG target: SF streams: public.users: {} ' # from a local file -sling project files save replications/foo.yaml -f ./local_repl.yaml +sling platform files save replications/foo.yaml -f ./local_repl.yaml # from stdin -cat ./local.yaml | sling project files save replications/foo.yaml -f - +cat ./local.yaml | sling platform files save replications/foo.yaml -f - # create an empty directory marker -sling project files save replications/archive/ --dir +sling platform files save replications/archive/ --dir # JSON response (useful for default_job_id) -sling project files save replications/foo.yaml -f ./local.yaml -o json | jq .default_job_id +sling platform files save replications/foo.yaml -f ./local.yaml -o json | jq .default_job_id ``` Flags: @@ -403,8 +403,8 @@ Backed by `POST /project/file/save` with `{"file": {"name", "body", "is_dir"}}`. Hard-delete a file on the platform. ```bash -sling project files delete replications/old.yaml # prompts Y/N -sling project files delete replications/old.yaml -f # no prompt +sling platform files delete replications/old.yaml # prompts Y/N +sling platform files delete replications/old.yaml -f # no prompt ``` Flags: @@ -417,20 +417,20 @@ Note: the server performs a hard delete immediately; the CLI prompt is the only Rename a file on the platform. The server also cascades the rename to any jobs referencing the old path. ```bash -sling project files rename replications/old.yaml replications/new.yaml +sling platform files rename replications/old.yaml replications/new.yaml ``` --- -## 14. `sling project connections` +## 14. `sling platform connections` Inspect connections configured for the project on the platform (from `env.yaml` and/or `.env.sling`). ### `connections list` ```bash -sling project connections list -sling project connections list -o json +sling platform connections list +sling platform connections list -o json ``` Columns (table): `Name`, `Type`, `Kind`, `Source`. @@ -440,7 +440,7 @@ Columns (table): `Name`, `Type`, `Kind`, `Source`. Test that a project connection is valid. Requires a running agent for most connection types (the request is forwarded to the connection's agent). ```bash -sling project connections test MY_POSTGRES +sling platform connections test MY_POSTGRES ``` On success: prints `Connection MY_POSTGRES: valid`. On failure: exits non-zero with the server/agent's error message. @@ -550,41 +550,41 @@ The full `Job` JSON shape accepted by `save` and returned by `get`: ```bash export SLING_PROJECT_TOKEN=... cd ~/my-project -sling project init +sling init # ... write replications/nightly.yaml ... -sling project sync -f -sling project jobs save --payload '{"name":"nightly","type":"replication","file_name":"replications/nightly.yaml","active":true,"schedules":["0 2 * * *"],"timezone":"UTC","config":{"mode":"incremental","threads":4}}' +sling platform sync -f +sling platform jobs save --payload '{"name":"nightly","type":"replication","file_name":"replications/nightly.yaml","active":true,"schedules":["0 2 * * *"],"timezone":"UTC","config":{"mode":"incremental","threads":4}}' ``` ### Edit a job's schedule ```bash -sling project jobs get job_abc123 > /tmp/j.json +sling platform jobs get job_abc123 > /tmp/j.json # edit schedules[] in /tmp/j.json -sling project jobs save --file /tmp/j.json +sling platform jobs save --file /tmp/j.json ``` ### Toggle `active` via jq ```bash -sling project jobs get job_abc123 \ +sling platform jobs get job_abc123 \ | jq '.active = false' \ - | sling project jobs save --file - + | sling platform jobs save --file - ``` ### Promote config from one job to another ```bash -sling project jobs get job_src > /tmp/src.json -sling project jobs get job_dst \ +sling platform jobs get job_src > /tmp/src.json +sling platform jobs get job_dst \ | jq --slurpfile src /tmp/src.json '.config = $src[0].config | .notification_settings = $src[0].notification_settings' \ - | sling project jobs save --file - + | sling platform jobs save --file - ``` ### Trigger + wait in CI ```bash -sling project jobs trigger job_abc123 --wait +sling platform jobs trigger job_abc123 --wait # exits non-zero on non-Success, so CI fails the step ``` @@ -627,3 +627,20 @@ The file is empty or malformed. A common cause: you tried to redirect `get` outp ### Self-hosted platform Set `SLING_PLATFORM_HOST=https://your-host.example.com` before any command. The token is validated against that host's `project/token/get` endpoint instead of `licensing.slingdata.io`. + +--- + +## 18. Platform MCP HTTP server + +The master also speaks MCP Streamable HTTP at `POST|GET|DELETE /mcp`. Authenticate with the same project token: + +``` +Authorization: Sling-Project-Token +``` + +```bash +claude mcp add --transport http sling-platform https:///mcp \ + --header "Authorization: Sling-Project-Token " +``` + +Tools: `project`, `file`, `job`, `execution`, `connection`, `compile`, `monitor`, `infra`. Call `docs` first. `job.run` returns `{exec_id}` — poll `execution.list`. diff --git a/cmd/sling/resource/llm_REPLICATION.md b/cmd/sling/resource/llm_REPLICATION.md index d5114df73..456fc442f 100644 --- a/cmd/sling/resource/llm_REPLICATION.md +++ b/cmd/sling/resource/llm_REPLICATION.md @@ -57,15 +57,15 @@ Sling replications are YAML or JSON configuration files that define data movemen The Sling MCP tool provides these replication commands: - `replication/docs` - Get documentation -- `replication/parse` - Parse and validate configuration -- `replication/compile` - Compile configuration with validation -- `replication/run` - Execute the replication +- `replication/validate` - Parse and compile configuration (default compile is true). `compile: false` is parse-only and does not mean the file is ready to run. + +There is no MCP `run` action. Execute replications with the CLI: `sling run -r /path/to/replication.yaml`. --- ## 2. Quick Start Guide -### Essential MCP Commands +### Essential Commands ```json // Get replication documentation @@ -76,20 +76,17 @@ The Sling MCP tool provides these replication commands: // Parse a replication file { - "action": "parse", + "action": "validate", "input": { "file_path": "/path/to/replication.yaml" } } +``` -// Run a replication -{ - "action": "run", - "input": { - "file_path": "/path/to/replication.yaml", - "mode": "incremental" - } -} +```bash +# Run a replication (CLI only — no MCP run action) +sling run -r /path/to/replication.yaml -m incremental +sling run -r /path/to/replication.yaml --streams table1 --debug ``` ### Basic Replication Structure @@ -1443,7 +1440,7 @@ streams: #### Parse Configuration ```json { - "action": "parse", + "action": "validate", "input": { "file_path": "/path/to/replication.yaml", "working_dir": "/optional/work/dir" @@ -1452,9 +1449,10 @@ streams: ``` #### Validate Configuration +Parsing compiles by default, which checks connections and streams. Prefer this when connections exist. `compile: false` is parse-only and does not mean the file is ready to run. ```json { - "action": "compile", + "action": "validate", "input": { "file_path": "/path/to/replication.yaml", "select_streams": ["table1", "table2"], // Optional @@ -1464,21 +1462,14 @@ streams: ``` #### Execute Replication -```json -{ - "action": "run", - "input": { - "file_path": "/path/to/replication.yaml", - "select_streams": ["specific_table"], // Optional: run specific streams - "working_dir": "/project/directory", // Optional: change directory - "range": "2024-01-01,2024-01-31", // Optional: backfill range - "mode": "incremental", // Optional: override mode - "env": { // Optional: environment variables - "CUSTOM_VAR": "value", - "SLING_THREADS": "5" - } - } -} + +There is no MCP `run` action. Use the CLI: + +```bash +sling run -r /path/to/replication.yaml +sling run -r /path/to/replication.yaml --streams specific_table -m incremental +sling run -r /path/to/replication.yaml --range 2024-01-01,2024-01-31 +sling run -r /path/to/replication.yaml --env '{CUSTOM_VAR: value, SLING_THREADS: "5"}' ``` ### Combining with Connection Tools @@ -1502,23 +1493,18 @@ Then test connections: } ``` -Finally run replication: -```json -{ - "action": "run", - "input": { - "file_path": "/path/to/replication.yaml" - } -} +Finally run the replication with the CLI: + +```bash +sling run -r /path/to/replication.yaml ``` ### Development Workflow -1. **Parse** configuration for syntax validation -2. **Compile** to check connections and streams -3. **Test** on subset of streams first -4. **Run** full replication -5. **Monitor** with debug/trace options +1. **Validate** configuration with MCP `replication/validate` (compile is the default) +2. **Test** on a subset of streams first: `sling run -r file.yaml --streams small_table` +3. **Run** the full replication: `sling run -r file.yaml` +4. **Monitor** with `--debug` / `--trace` --- @@ -1648,14 +1634,8 @@ hooks: ### Testing Strategies #### Test with Subsets First -```json -{ - "action": "run", - "input": { - "file_path": "/path/to/replication.yaml", - "select_streams": ["small_test_table"] - } -} +```bash +sling run -r /path/to/replication.yaml --streams small_test_table ``` #### Use Development Targets @@ -1780,23 +1760,15 @@ target_options: ### Debug Options #### Enable Debug Logging -```json -{ - "action": "run", - "input": { - "file_path": "/path/to/replication.yaml", - "env": { - "DEBUG": "true" - } - } -} +```bash +sling run -r /path/to/replication.yaml --debug ``` #### Parse Configuration Always validate before running: ```json { - "action": "parse", + "action": "validate", "input": { "file_path": "/path/to/replication.yaml" } @@ -1804,10 +1776,10 @@ Always validate before running: ``` #### Compile Configuration -Check connections and streams: +Parsing compiles by default, which checks connections and streams. Prefer this when connections exist. ```json { - "action": "compile", + "action": "validate", "input": { "file_path": "/path/to/replication.yaml" } @@ -1889,14 +1861,8 @@ streams: Sling automatically tracks progress, but you can: 1. **Run specific streams:** -```json -{ - "action": "run", - "input": { - "file_path": "/path/to/replication.yaml", - "select_streams": ["failed_stream"] - } -} +```bash +sling run -r /path/to/replication.yaml --streams failed_stream ``` 2. **Use retry configuration:** @@ -1906,4 +1872,4 @@ env: SLING_RETRY_DELAY: 60s ``` -This comprehensive guide provides everything needed to effectively use Sling replications with the MCP tool, from basic concepts to advanced troubleshooting techniques. \ No newline at end of file +This comprehensive guide provides everything needed to effectively use Sling replications: validate with MCP, then execute with `sling run`. \ No newline at end of file diff --git a/cmd/sling/resource/mcp.yaml b/cmd/sling/resource/mcp.yaml index 46bde6044..f1213a5bd 100644 --- a/cmd/sling/resource/mcp.yaml +++ b/cmd/sling/resource/mcp.yaml @@ -85,6 +85,7 @@ tools: - `"get_schemata"` - Get database schemata (schemas, tables, columns) - `"get_schemas"` - Get list of schema names - `"query"` - Execute SQL queries (read-only) + - `"query_cancel"` - Cancel a running query by query_id - `"get_columns"` - Get column metadata for a table - `input` (object, required): The input parameters for the specific action. The structure varies based on the action: @@ -103,13 +104,19 @@ tools: - `description` (string, required): A brief description of the intent and expected result of this query. Explain *why* you are running this query and what the result should tell you. Always provide this when executing a query. - `limit` (number, optional): The limit of rows to return (defaults to 100) - `transient` (boolean, optional): Whether to use a transient connection (default: false) + - `query_id` (string, optional): Client-supplied id (`[A-Za-z0-9_-]`, max 64) so a concurrent `query_cancel` can target this query. Generated if omitted. - This action executes a SQL query on a database connection and return the results. **WARNING: Only use this tool for SELECT queries and other read-only operations. Never execute destructive queries such as DELETE, DROP, TRUNCATE, ALTER, UPDATE, INSERT, or any other data modification operations.** + This action executes a SQL query on a database connection and return the results. Long queries are supported. Interrupting the client (Ctrl+C / Escape) cancels the running query. Pass an optional `query_id` to cancel from another concurrent call via `query_cancel`. **WARNING: Only use this tool for SELECT queries and other read-only operations. Never execute destructive queries such as DELETE, DROP, TRUNCATE, ALTER, UPDATE, INSERT, or any other data modification operations.** * If a destructive operation (e.g., dropping an object, deleting significant data, altering table structures) is deemed necessary, **DO NOT execute it directly**. Instead, formulate the required SQL query/statement and **return it to the USER for manual review and execution**. * Always use the least number of columns, to minimize computation and the amount of data being retrieved. * Always use the LIMIT clause to minimize the number of rows returned. * If the SQL is already saved as a file, pass `file:///abs/path/to/query.sql` as `query` instead of inlining the SQL text. + + **For `action: "query_cancel"`**: + - `query_id` (string, required): The id of the running query to cancel. + + Cancels a running `query`. Idempotent; returns `{cancelled: bool, query_id: string}`. `{cancelled: false}` means that id was not running. Requires a client that can issue a second tool call while `query` is in flight; sequential clients rely on interrupting the first call. **For `action: "get_columns"`**: - `connection` (string, required): The name of the database connection @@ -132,6 +139,14 @@ tools: "description": "Preview first 10 user records to understand table structure and data format" } } + + # Cancel a running query + { + "action": "query_cancel", + "input": { + "query_id": "my-q1" + } + } # Get all schemas in a database { @@ -322,12 +337,12 @@ tools: **Parameters**: - `action` (string, required): The API spec action to perform. Valid values: - - `"parse"` - Load and parse the API specification file, will return various details + - `"validate"` - Load and validate the API specification file, will return various details - `"docs"` - Fetch the Sling API specification documentation - `"test"` - Test an existing API Spec connection - `input` (object, required): The input parameters for the specific action. The structure varies based on the action: - **For `action: "parse"`**: + **For `action: "validate"`**: - `file_path` (string, required): The file path of the API specification. - `working_dir` (string, optional): Working directory to change to before running @@ -367,7 +382,7 @@ tools: # Parse API specification content { - "action": "parse", + "action": "validate", "input": { "file_path": "path/to/github_api.yaml", "working_dir": "/path/to/dir" @@ -383,31 +398,28 @@ tools: **Parameters**: - `action` (string, required): The replication action to perform. Valid values: - `"docs"` - Fetch documentation on how to use replications - - `"parse"` - Parse the content of a replication configuration file (non-compile validation) - - `"compile"` - Compile/validate a replication configuration - - `"run"` - Execute a replication configuration + - `"validate"` - Validate and compile a replication configuration file - `input` (object, required): The input parameters for the specific action. The structure varies based on the action: **For `action: "docs"`**: - No parameters required (empty object) - **For `action: "parse"`**: + **For `action: "validate"`**: - `file_path` (string, required): Path to the replication configuration file to read - - `working_dir` (string, optional): Working directory to change to before reading - - **For `action: "compile"`**: - - `file_path` (string, required): Path to the replication configuration file to compile + - `compile` (boolean, optional): Compile the configuration (default: true). Compile resolves + connections, interpolates `${VAR}` and `{var}` references, and returns the task list. + Set to false only for a parse-only syntax check. Parse-only does not mean the file is ready to run. + Prefer compile: true when connections exist. - `select_streams` (array, optional): List of specific streams to compile (default: all streams) - - `working_dir` (string, optional): Working directory to change to before compiling + - `working_dir` (string, optional): Working directory to change to before reading - **For `action: "run"`**: - - `file_path` (string, required): Path to the replication configuration file to execute - - `select_streams` (array, optional): List of specific streams to run (default: all streams) - - `working_dir` (string, optional): Working directory to change to before running - - `range` (string, optional): Backfill range for source options (e.g., "2024-01-01:2024-01-31") - - `mode` (string, optional): Mode override for replication (e.g., "full-refresh", "incremental") - - `env` (object, optional): Environment variables to set for the replication run + **To execute a replication, use the CLI** (there is no MCP `run` action): + ``` + sling run -r path/to/my_replication.yaml + sling run -r path/to/my_replication.yaml --streams table1,table2 -m incremental + sling run -r path/to/my_replication.yaml --debug + ``` **Output**: The output format depends on the action and audience: - **For User**: Human-readable messages, CSV tables, or status updates @@ -421,29 +433,11 @@ tools: "input": {} } - # Parse a replication file - { - "action": "parse", - "input": { - "file_path": "path/to/my_replication.yaml", - } - } - - # Compile a replication file - { - "action": "compile", - "input": { - "file_path": "path/to/my_replication.yaml", - } - } - - # Run a replication with specific streams + # Parse and compile a replication file (default; prefer this when connections exist) { - "action": "run", + "action": "validate", "input": { - "file_path": "path/to/my_replication.yaml", - "select_streams": ["table1", "table2"], - "mode": "incremental" + "file_path": "path/to/my_replication.yaml" } } ``` @@ -456,21 +450,28 @@ tools: **Parameters**: - `action` (string, required): The pipeline action to perform. Valid values: - `"docs"` - Fetch documentation on how to use pipelines - - `"parse"` - Parse and validate the content of a pipeline configuration file - - `"run"` - Execute a pipeline configuration + - `"validate"` - Validate the content of a pipeline configuration file - `input` (object, required): The input parameters for the specific action. The structure varies based on the action: **For `action: "docs"`**: - No parameters required (empty object) - **For `action: "parse"`**: + **For `action: "validate"`**: - `file_path` (string, required): Path to the pipeline configuration file to read. - `working_dir` (string, optional): Working directory to change to before reading - **For `action: "run"`**: - - `file_path` (string, required): Path to the pipeline configuration file to execute. - - `working_dir` (string, optional): Working directory to change to before running - - `env` (object, optional): Environment variables to set for the pipeline run. + **To execute a pipeline, use the CLI** (there is no MCP `run` action): + ``` + sling run -p path/to/my_pipeline.yaml + sling run -p path/to/my_pipeline.yaml --debug + ``` + + **To execute SQL models, use the CLI** (there is no MCP `build` action): + ``` + sling build + sling build --compile + sling build -s stg_users,fct_orders + ``` **Output**: The output format depends on the action and audience: - **For User**: Human-readable messages, CSV tables, or status updates @@ -486,30 +487,11 @@ tools: # Parse a pipeline file { - "action": "parse", - "input": { - "file_path": "path/to/my_pipeline.yaml" - } - } - - # Run a pipeline - { - "action": "run", + "action": "validate", "input": { "file_path": "path/to/my_pipeline.yaml" } } - - # Run a pipeline with environment variables - { - "action": "run", - "input": { - "file_path": "path/to/my_pipeline.yaml", - "env": { - "MY_VAR": "my_value" - } - } - } ``` @@ -547,7 +529,7 @@ prompts: The assistant should first attempt to use a browser mcp tool to fetch the documentation, as that will yield better content. If unable to use the browser mcp tool, the assistant should next attempt to directly fetch from the URL links (without a browser). - If the assistant is unable to obtain or fetch meaningful documentation from the URL links provided, stop and alert the user for further instruction. Recommend the user to download and activate the `browsermcp` (https://browsermcp.io/) which will provide tooling to access the browser. If the user wishes the assistant to proceed without having read the documentation, that is an option as well, although results are likely to be poor. + If the assistant is unable to obtain or fetch meaningful documentation from the URL links provided, stop and alert the user. Recommend `sling assist setup` so the agent-browser MCP server is wired. If the user wishes the assistant to proceed without having read the documentation, that is an option as well, although results are likely to be poor. 3. **Create Specification**: Now that the API is greatly understood, formulate and create a fully functional Sling API Spec in file path {spec_file_path}. If the assistant sees obstacles or issues in developing the spec, or does not have sufficient information, this is the time to alert the user with any concerns or questions for clarification. @@ -600,9 +582,9 @@ prompts: 1. **Get Documentation**: Call the tool `api_spec/docs` to obtain the latest Sling API specification guide - 2. **Load Existing Spec**: Use `api_spec/parse` to load the current specification content. + 2. **Load Existing Spec**: Use `api_spec/validate` to load the current specification content. - 3. **Read Raw Spec**: Examine the existing spec structure and patterns. If there were any parsing errors from the previous step, attempt to fix the structure of the file, and continue using `api_spec/parse` until there are no errors or determining that no progress can be made. + 3. **Read Raw Spec**: Examine the existing spec structure and patterns. If there were any parsing errors from the previous step, attempt to fix the structure of the file, and continue using `api_spec/validate` until there are no errors or determining that no progress can be made. 4. **Analyze Endpoint Documentation**: Navigate to the endpoint docs URL (if provided) or the main API docs, using the browser to fully understand: - Endpoint URL structure and parameters @@ -614,7 +596,7 @@ prompts: The assistant should first attempt to use a browser mcp tool to fetch the documentation, as that will yield better content. If unable to use the browser mcp tool, the assistant should next attempt to directly fetch from the URL links (without a browser). - If the assistant is unable to obtain or fetch meaningful documentation from the URL links provided, stop and alert the user for further instruction. Recommend the user to download and activate the `browsermcp` (https://browsermcp.io/) which will provide tooling to access the browser. If the user wishes the assistant to proceed without having read the documentation, that is an option as well, although results are likely to be poor. + If the assistant is unable to obtain or fetch meaningful documentation from the URL links provided, stop and alert the user. Recommend `sling assist setup` so the agent-browser MCP server is wired. If the user wishes the assistant to proceed without having read the documentation, that is an option as well, although results are likely to be poor. 5. **Implement Endpoint**: Add the new endpoint configuration to the spec file following existing patterns. @@ -654,9 +636,9 @@ prompts: ## Workflow Steps: 1. **Get Documentation**: Use `api_spec/docs` to access the Sling API specification guide - 2. **Load Existing Spec**: Use `api_spec/parse` to load the current specification content. + 2. **Load Existing Spec**: Use `api_spec/validate` to load the current specification content. - 3. **Read Raw Spec**: Examine the existing spec structure and patterns. If there were any parsing errors from the previous step, attempt to fix the structure of the file, and continue using `api_spec/parse` until there are no errors or determining that no progress can be made. + 3. **Read Raw Spec**: Examine the existing spec structure and patterns. If there were any parsing errors from the previous step, attempt to fix the structure of the file, and continue using `api_spec/validate` until there are no errors or determining that no progress can be made. 4. **Analyze API Documentation**: Navigate to the API docs using the browser to fully understand and verify: - Correct endpoint URL and parameters @@ -667,7 +649,7 @@ prompts: The assistant should first attempt to use a browser mcp tool to fetch the documentation, as that will yield better content. If unable to use the browser mcp tool, the assistant should next attempt to directly fetch from the URL links (without a browser). - If the assistant is unable to obtain or fetch meaningful documentation from the URL links provided, stop and alert the user for further instruction. Recommend the user to download and activate the `browsermcp` (https://browsermcp.io/) which will provide tooling to access the browser. If the user wishes the assistant to proceed without having read the documentation, that is an option as well, although results are likely to be poor. + If the assistant is unable to obtain or fetch meaningful documentation from the URL links provided, stop and alert the user. Recommend `sling assist setup` so the agent-browser MCP server is wired. If the user wishes the assistant to proceed without having read the documentation, that is an option as well, although results are likely to be poor. 5. **Test and Diagnose**: Use `api_spec/test` with debug enabled, specifying the endpoint name - Examine error messages and response details diff --git a/cmd/sling/sling_assist.go b/cmd/sling/sling_assist.go new file mode 100644 index 000000000..dd5b61ce5 --- /dev/null +++ b/cmd/sling/sling_assist.go @@ -0,0 +1,536 @@ +package main + +import ( + "context" + "errors" + "fmt" + "os" + "strings" + + "github.com/flarco/g" + "github.com/integrii/flaggy" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/slingdata-io/sling-cli/core/sling/assist" + "github.com/spf13/cast" +) + +var cliAssistFlags = []g.Flag{ + {Name: "id", Type: "string", Description: "investigate a failure by its id (an id prefix works)"}, + {Name: "out", ShortName: "o", Type: "string", Description: "emit prompt instead of launching: FILE, or - for stdout"}, + {Name: "agent", Type: "string", Description: "override profile.agent for this run only"}, + {Name: "model", Type: "string", Description: "pass --model to the harness for this run only"}, + {Name: "name", Type: "string", Description: "history slug override"}, + {Name: "non-interactive", Type: "bool", Description: "skip interactive prompts (use defaults / launch headless)"}, + {Name: "resume", Type: "string", Description: "resume a past session (omit id to pick)"}, +} + +var cliAssistSetupFlags = []g.Flag{ + {Name: "doctor", Type: "bool", Description: "print the health/install report and exit"}, + {Name: "agent", Type: "string", Description: "preferred agent for this setup"}, + {Name: "non-interactive", Type: "bool", Description: "skip interactive prompts (use defaults)"}, + {Name: "install", Type: "string", Description: "comma-separated components (e.g. mcp,skills)"}, + {Name: "uninstall", Type: "bool", Description: "remove all Sling skills + MCP wiring"}, + {Name: "reconfigure", Type: "bool", Description: "re-prompt the profile form before installing"}, + {Name: "scope", Type: "string", Description: "user (default) or project"}, + {Name: "clients", Type: "string", Description: "with --uninstall: comma-separated client names"}, +} + +// cliAssist is the `sling assist` top-level command. +var cliAssist = &g.CliSC{ + Name: "assist", + Description: "Get AI help to build, run and debug replications, pipelines, API Specs, etc.", + AdditionalHelpPrepend: "\n" + + " sling assist # first run: setup; after: greet with choices\n" + + " sling assist \"backfill orders\" # positional ask\n" + + " sling assist --id # investigate a failure\n sling assist --resume # pick a past session\n" + + " sling assist --resume # resume that session\n" + + " sling assist setup # re-run setup / change harness\n" + + " sling assist setup --doctor # report only\n" + + " sling assist error # look up an error signature\n" + + " sling assist report --id # review a redacted failure report\n" + + " sling assist report --id --github # open a prefilled GitHub issue\n" + + " sling assist report --id --email # send to support (confirm first)\n" + + " sling assist --out - | --out F # emit prompt, no launch\n" + + " sling assist --agent claude # one-run override\n" + + " sling assist --model sonnet # pass --model to the harness\n" + + "See https://docs.slingdata.io/sling-cli/assist", + ExecuteWithoutFlags: true, + Flags: cliAssistFlags, + SubComs: []*g.CliSC{ + { + Name: "setup", + Description: "install skills + MCP, or reconfigure / uninstall", + ExecuteWithoutFlags: true, + Flags: cliAssistSetupFlags, + }, + { + Name: "error", + Description: "look up guidance for an error signature", + Flags: []g.Flag{ + {Name: "json", Type: "bool", Description: "emit machine-readable JSON"}, + }, + PosFlags: []g.Flag{ + {Name: "signature", Type: "string", Description: "8-char error signature from a failed run"}, + }, + }, + { + Name: "report", + Description: "compose a redacted issue report from a failed run", + Flags: []g.Flag{ + {Name: "id", Type: "string", Description: "failure id (a unique prefix works)"}, + {Name: "title", Type: "string", Description: "override the report title"}, + {Name: "description", Type: "string", Description: "custom context shown above the error in the report"}, + {Name: "github", Type: "bool", Description: "open a prefilled GitHub issue after confirm"}, + {Name: "email", Type: "bool", Description: "send to support after confirm"}, + {Name: "submit", Type: "bool", Description: "skip the confirm prompt (for agents)"}, + }, + }, + }, + ExecProcess: processAssist, +} + +func init() { + cliAssist.Make().Add() +} + +// processAssist is the dispatcher. Auto-refresh runs once at the top so every +// subcommand below sees a fresh canonical bundle. If skills are already +// installed, AutoRefresh updates and prunes them. If none are installed, +// the user must run `sling assist setup`. +func processAssist(c *g.CliSC) (ok bool, err error) { + ok = true + if notice, refreshErr := assist.AutoRefresh(context.Background()); refreshErr == nil && notice != "" { + fmt.Fprintln(os.Stderr, notice) + } + + switch c.UsedSC() { + case "setup": + return ok, runAssistSetup(c) + case "error": + return ok, runAssistError(c) + case "report": + return ok, runAssistReport(c) + } + return ok, runAssistFlags(c) +} + +// runAssistFlags is the flags-only path: --resume, else first-run +// setup or probe+launch / --out. +func runAssistFlags(c *g.CliSC) error { + vals := flatVals(c) + + resumeSet, resumeID := resumeFromArgs(os.Args) + if resumeSet { + return runAssistResume(c, resumeID) + } + + _, profileExists, err := assist.LoadProfile() + if err != nil { + return err + } + if !profileExists && strings.TrimSpace(cast.ToString(vals["out"])) == "" { + return runAssistSetup(c) + } + return runAssistSession(c) +} + +func runAssistSession(c *g.CliSC) error { + vals := flatVals(c) + // Trailing args are the ask (flaggy cannot share PosFlags with SubComs). + ask := "" + if len(flaggy.TrailingArguments) > 0 { + ask = strings.TrimSpace(strings.Join(flaggy.TrailingArguments, " ")) + } + opts := assist.SessionOptions{ + Ask: ask, + ExecID: strings.TrimSpace(cast.ToString(vals["id"])), + Name: cast.ToString(vals["name"]), + Agent: cast.ToString(vals["agent"]), + Model: cast.ToString(vals["model"]), + Headless: cast.ToBool(vals["non-interactive"]), + } + applyAssistOut(&opts, cast.ToString(vals["out"])) + _, err := assist.Session(opts) + if code, ok := assist.ExitCodeOf(err); ok { + os.Exit(code) + } + return err +} + +// applyAssistOut maps --out onto the session emit options. +// "-" means stdout; any other value is a file path. +func applyAssistOut(opts *assist.SessionOptions, out string) { + out = strings.TrimSpace(out) + switch out { + case "": + case "-": + opts.Print = true + default: + opts.OutputFile = out + } +} + +func runAssistResume(c *g.CliSC, id string) error { + vals := flatVals(c) + if id == "" { + e, err := assist.PickHistoryEntry() + if err != nil { + if errors.Is(err, assist.ErrUserAborted) { + return nil + } + return err + } + id = e.ID + } + opts := assist.SessionOptions{ + ResumeID: id, + ResumeSet: true, + Agent: cast.ToString(vals["agent"]), + Model: cast.ToString(vals["model"]), + Headless: cast.ToBool(vals["non-interactive"]), + } + applyAssistOut(&opts, cast.ToString(vals["out"])) + _, err := assist.Session(opts) + if code, ok := assist.ExitCodeOf(err); ok { + os.Exit(code) + } + return err +} + +// padAssistResumeFlag lets flaggy accept a bare `--resume` (picker) as `--resume=`. +func padAssistResumeFlag(args []string) []string { + assist := false + for _, a := range args[1:] { + if a == "assist" { + assist = true + break + } + if strings.HasPrefix(a, "-") { + continue + } + break + } + if !assist { + return args + } + out := make([]string, 0, len(args)+1) + for i := 0; i < len(args); i++ { + a := args[i] + if a == "--resume" { + if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") { + out = append(out, "--resume=") + continue + } + } + out = append(out, a) + } + return out +} + +func resumeFromArgs(args []string) (present bool, id string) { + for i := 1; i < len(args); i++ { + a := args[i] + if a == "--resume" { + if i+1 < len(args) && args[i+1] != "" && !strings.HasPrefix(args[i+1], "-") { + return true, args[i+1] + } + return true, "" + } + if strings.HasPrefix(a, "--resume=") { + return true, strings.TrimPrefix(a, "--resume=") + } + } + return false, "" +} + +func parseScope(v string) assist.Scope { + if strings.EqualFold(v, "project") { + return assist.ScopeProject + } + return assist.ScopeUser +} + +// runAssistSetup is the unified setup entry point. One command does everything +// install/doctor/uninstall used to do — doctor runs implicitly each call to +// detect current state, then we branch by flag intent or interactive choice. +func runAssistSetup(c *g.CliSC) error { + vals := flatVals(c) + + // --uninstall is the only path that skips doctor (we're tearing it all + // down anyway; running doctor first would just be noise). + if cast.ToBool(vals["uninstall"]) { + return runSetupUninstallAll(vals) + } + + // Doctor always runs first; we use its result both to render state for + // the user and to decide what the implicit "next action" should be. + // Scope must match install scope so project installs aren't reported broken. + report, err := assist.Doctor(context.Background(), assist.DoctorOptions{ + Scope: parseScope(cast.ToString(vals["scope"])), + }) + if err != nil { + return err + } + _, profileExists, _ := assist.LoadProfile() + + // --doctor: print and exit. + if cast.ToBool(vals["doctor"]) { + fmt.Fprint(os.Stdout, report.Render()) + if !report.OK { + return g.Error("doctor reported failures") + } + return nil + } + + // --install : skip the picker, install whatever was named. + if comps := cast.ToString(vals["install"]); comps != "" { + // Split "mcp,skills" into lower-case names. + var parts []string + for _, p := range strings.Split(comps, ",") { + p = strings.TrimSpace(strings.ToLower(p)) + if p != "" { + parts = append(parts, p) + } + } + return runSetupInstall(vals, parts, profileExists, report) + } + + // Non-interactive without an explicit verb → behave like first-run install + // of all components (back-compat with how the old `install --non-interactive` + // behaved). + if cast.ToBool(vals["non-interactive"]) { + return runSetupInstall(vals, allComponents(), profileExists, report) + } + + // Subsequent run (profile exists): show doctor output, then drop into the + // interactive "what do you want to do?" form. + if profileExists { + fmt.Fprint(os.Stdout, report.Render()) + fmt.Fprintln(os.Stdout, "") + action, err := assist.RunSetupActionForm(report) + if err != nil { + return err + } + switch action { + case assist.SetupActionRefresh: + return runSetupInstall(vals, allComponents(), profileExists, report) + case assist.SetupActionInstallMissing: + comps := report.MissingComponents() + if len(comps) == 0 { + comps = allComponents() + } + return runSetupInstall(vals, comps, profileExists, report) + case assist.SetupActionUninstall: + return runSetupUninstallAll(vals) + case assist.SetupActionReconfigure: + vals["reconfigure"] = true + return runSetupInstall(vals, allComponents(), profileExists, report) + case assist.SetupActionExit: + return nil + } + return nil + } + + // First run (no profile): harness confirm (agents + bundled opencode fallback). + if len(assist.DetectedClients()) == 0 && len(assist.RankedCLIAgents()) == 0 { + return g.Error("no AI agent on $PATH; install one of: claude, codex, gemini, cursor, opencode, pi, grok") + } + prefill := assist.DefaultProfile() + prefill.Agent = assist.RecommendedAgent() + result, err := assist.RunHarnessConfirmForm(prefill) + if err != nil { + if errors.Is(err, assist.ErrUserAborted) { + return nil + } + return err + } + prefill.Agent = result.Agent + prefill.HintInErrors = result.HintInErrors + prefill.DefaultInstallScope = result.Scope + if err := assist.SaveProfile(prefill); err != nil { + return err + } + vals["agent"] = result.Agent + vals["scope"] = result.Scope + return runSetupInstall(vals, result.Components, true, report) +} + +// runSetupInstall runs the profile form (if needed) then installs the +// requested components. components is the canonical set; we translate it +// to SkillsOnly/MCPOnly for the existing Install API. +func runSetupInstall(vals map[string]any, components []string, _ bool, _ *assist.DoctorReport) error { + opts := assist.InstallOptions{ + Reconfigure: cast.ToBool(vals["reconfigure"]), + Scope: parseScope(cast.ToString(vals["scope"])), + NonInteractive: cast.ToBool(vals["non-interactive"]), + DefaultAgent: cast.ToString(vals["agent"]), + } + hasSkills := containsString(components, "skills") + hasMCP := containsString(components, "mcp") + if !hasSkills && !hasMCP { + return g.Error("no components selected; pass --install mcp,skills or pick at least one in the form") + } + opts.SkillsOnly = hasSkills && !hasMCP + opts.MCPOnly = hasMCP && !hasSkills + + // Interactive first-run (or --reconfigure): show the profile form before + // falling through to Install(). + if !opts.NonInteractive { + prof, exists, _ := assist.LoadProfile() + needForm := !exists || opts.Reconfigure + if needForm { + if len(assist.DetectedClients()) == 0 && len(assist.RankedCLIAgents()) == 0 { + return g.Error("no AI agent on $PATH; install one of: claude, codex, gemini, cursor, opencode, pi, grok") + } + prefill := prof + if !exists { + prefill = assist.DefaultProfile() + prefill.Agent = assist.RecommendedAgent() + } + result, err := assist.RunInstallForm(prefill) + if err != nil { + if errors.Is(err, assist.ErrUserAborted) { + return nil + } + return err + } + prefill.Agent = result.Agent + prefill.HintInErrors = result.HintInErrors + prefill.DefaultInstallScope = result.Scope + if err := assist.SaveProfile(prefill); err != nil { + return err + } + opts.Reconfigure = false + opts.DefaultAgent = "" + opts.Scope = parseScope(result.Scope) + } + } + + res, err := assist.Install(context.Background(), opts) + if err != nil { + return err + } + // Install summary: profile, canonical skills, then each wired client. + if res.ProfileWritten { + fmt.Fprintf(os.Stdout, "%s Wrote AI profile to %s\n", + env.GreenString("✓"), env.CyanString(env.HomeDirEnvFile)) + } + fmt.Fprintf(os.Stdout, "%s Wrote canonical skills to %s\n", + env.GreenString("✓"), env.CyanString(res.CanonicalSkillsDir)) + fmt.Fprintln(os.Stdout, "") + fmt.Fprintln(os.Stdout, env.BlueString("Wired clients:")) + yesNo := func(v bool) string { + if v { + return env.GreenString("yes") + } + return env.DarkGrayString("no") + } + for _, row := range res.WiredClients { + mark := env.GreenString("✓") + if !row.WroteSkills && !row.WroteMCP { + mark = env.YellowString("⊘") + } + auth := row.Authed.YesNo() + authOut := env.DarkGrayString(auth) + if row.Authed == assist.AuthOK { + authOut = env.GreenString(auth) + } + fmt.Fprintf(os.Stdout, " %s %-8s skills=%s mcp=%s authed=%s %s\n", + mark, row.Name, yesNo(row.WroteSkills), yesNo(row.WroteMCP), authOut, + env.DarkGrayString(row.Notes)) + } + fmt.Fprintln(os.Stdout, "") + fmt.Fprintf(os.Stdout, "Run %s to verify.\n", env.CyanString("`sling assist setup --doctor`")) + return nil +} + +// runSetupUninstallAll wipes everything (skills + mcp from every detected +// client + the canonical bundle). No interactive form — `--uninstall` is the +// blunt instrument; per-client/per-component selection isn't worth a separate +// surface. +func runSetupUninstallAll(vals map[string]any) error { + opts := assist.UninstallOptions{ + All: true, + Scope: parseScope(cast.ToString(vals["scope"])), + } + if v := cast.ToString(vals["clients"]); v != "" { + opts.IncludeClients = strings.Split(v, ",") + } + if err := assist.Uninstall(context.Background(), opts); err != nil { + return err + } + fmt.Fprintln(os.Stdout, env.GreenString("✓ Removed Sling skills + MCP wiring from detected clients.")) + return nil +} + +func allComponents() []string { return []string{"skills", "mcp"} } + +func containsString(xs []string, x string) bool { + for _, v := range xs { + if v == x { + return true + } + } + return false +} + +func runAssistReport(c *g.CliSC) error { + vals := flatVals(c) + id := strings.TrimSpace(cast.ToString(vals["id"])) + if id == "" { + return g.Error("usage: sling assist report --id ") + } + return assist.RunReport(assist.ReportCmd{ + ExecID: id, + Title: strings.TrimSpace(cast.ToString(vals["title"])), + Description: strings.TrimSpace(cast.ToString(vals["description"])), + GitHub: cast.ToBool(vals["github"]), + Email: cast.ToBool(vals["email"]), + Submit: cast.ToBool(vals["submit"]), + }) +} + +func runAssistError(c *g.CliSC) error { + vals := flatVals(c) + sig := strings.TrimSpace(cast.ToString(vals["signature"])) + if sig == "" { + return g.Error("usage: sling assist error ") + } + result, err := assist.LookupError(sig) + if err != nil { + return err + } + if cast.ToBool(vals["json"]) { + fmt.Println(g.Marshal(result)) + return nil + } + fmt.Printf("error_signature: %s\n", result.Signature) + fmt.Printf("status: %s\n", result.Status) + if result.Title != "" { + fmt.Printf("title: %s\n", result.Title) + } + if result.Guidance != "" { + fmt.Println() + fmt.Println(result.Guidance) + } + if result.DocsURL != "" { + fmt.Printf("\nDocs: %s\n", result.DocsURL) + } + fmt.Println() + fmt.Println("Next steps:") + fmt.Printf(" sling assist # open assist (offers to investigate failures)\n") + if result.Status == "unknown" || result.Status == "pending" { + fmt.Printf(" sling assist report --id # share a redacted report\n") + } + return nil +} + +// flatVals returns the val map from the active subcommand. CliSC stores per- +// subcommand flag values in c.Vals; we just pass that through. +func flatVals(c *g.CliSC) map[string]any { + out := map[string]any{} + for k, v := range c.Vals { + out[k] = v + } + return out +} diff --git a/cmd/sling/sling_assist_test.go b/cmd/sling/sling_assist_test.go new file mode 100644 index 000000000..5542fa1f3 --- /dev/null +++ b/cmd/sling/sling_assist_test.go @@ -0,0 +1,243 @@ +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core/dbio" + "github.com/slingdata-io/sling-cli/core/dbio/connection" +) + +func TestAssistBrowseFlagRemoved(t *testing.T) { + for _, f := range cliAssistFlags { + if f.Name == "browse" || f.Name == "cdp" || f.Name == "setup" || f.Name == "doctor" || f.Name == "task" || f.Name == "path" { + t.Fatalf("removed flag still present: %s", f.Name) + } + } + foundResume, foundModel := false, false + for _, f := range cliAssistFlags { + if f.Name == "resume" { + foundResume = true + } + if f.Name == "model" { + foundModel = true + } + } + if !foundResume || !foundModel { + t.Fatal("missing --resume or --model on sling assist") + } + hasSetup, setupHasDoctor := false, false + for _, sc := range cliAssist.SubComs { + if sc.Name == "setup" { + hasSetup = true + for _, f := range sc.Flags { + if f.Name == "doctor" { + setupHasDoctor = true + } + } + } + if sc.Name == "investigate" || sc.Name == "history" { + t.Fatalf("removed subcommand still present: %s", sc.Name) + } + } + hasReport := false + for _, sc := range cliAssist.SubComs { + if sc.Name == "report" { + hasReport = true + } + } + if !hasSetup { + t.Fatal("missing setup subcommand") + } + if !hasReport { + t.Fatal("missing report subcommand") + } + if !setupHasDoctor { + t.Fatal("missing --doctor on sling assist setup") + } +} + +func TestPadAssistResumeFlag(t *testing.T) { + got := padAssistResumeFlag([]string{"sling", "assist", "--resume"}) + if len(got) < 3 || got[2] != "--resume=" { + t.Fatalf("bare resume: %v", got) + } + got = padAssistResumeFlag([]string{"sling", "assist", "--resume", "--out"}) + if got[2] != "--resume=" { + t.Fatalf("resume before flag: %v", got) + } + got = padAssistResumeFlag([]string{"sling", "assist", "--resume", "sess1"}) + if got[2] != "--resume" || got[3] != "sess1" { + t.Fatalf("resume with id: %v", got) + } +} + +func TestResumeFromArgs(t *testing.T) { + ok, id := resumeFromArgs([]string{"sling", "assist", "--resume="}) + if !ok || id != "" { + t.Fatalf("empty resume: %v %q", ok, id) + } + ok, id = resumeFromArgs([]string{"sling", "assist", "--resume", "abc"}) + if !ok || id != "abc" { + t.Fatalf("id: %v %q", ok, id) + } + ok, _ = resumeFromArgs([]string{"sling", "assist", "--out"}) + if ok { + t.Fatal("resume not set") + } +} + +// parseKVList is kept in this test file so TestCLI still compiles after +// the production helper was removed in the assist redesign. +func parseKVList(s string) map[string]string { + out := map[string]string{} + for _, pair := range splitKVPairs(s) { + kv := strings.SplitN(pair, "=", 2) + if len(kv) != 2 { + continue + } + k := strings.TrimSpace(kv[0]) + v := unquoteKV(strings.TrimSpace(kv[1])) + if strings.HasPrefix(v, "@") && !strings.HasPrefix(v, "@@") { + if b, err := os.ReadFile(v[1:]); err == nil { + v = string(b) + } + } + out[k] = v + } + return out +} + +func splitKVPairs(s string) []string { + var parts []string + var b strings.Builder + var quote byte + for i := 0; i < len(s); i++ { + c := s[i] + if quote != 0 { + if c == '\\' && i+1 < len(s) { + b.WriteByte(c) + i++ + b.WriteByte(s[i]) + continue + } + if c == quote { + quote = 0 + } + b.WriteByte(c) + continue + } + if c == '"' || c == '\'' { + quote = c + b.WriteByte(c) + continue + } + if c == ',' { + parts = append(parts, b.String()) + b.Reset() + continue + } + b.WriteByte(c) + } + if b.Len() > 0 { + parts = append(parts, b.String()) + } + return parts +} + +func unquoteKV(v string) string { + if len(v) < 2 { + return v + } + if (v[0] == '"' && v[len(v)-1] == '"') || (v[0] == '\'' && v[len(v)-1] == '\'') { + inner := v[1 : len(v)-1] + inner = strings.ReplaceAll(inner, `\"`, `"`) + inner = strings.ReplaceAll(inner, `\'`, `'`) + return inner + } + return v +} + +func TestParseKVListQuotedComma(t *testing.T) { + got := parseKVList(`Intention="Select id, email, name. Range 2024-01-01,2024-12-31",Path=./out.yaml`) + if got["Intention"] != "Select id, email, name. Range 2024-01-01,2024-12-31" { + t.Fatalf("Intention=%q", got["Intention"]) + } + if got["Path"] != "./out.yaml" { + t.Fatalf("Path=%q", got["Path"]) + } +} + +func TestParseKVListAtFile(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "intention.txt") + body := "Load fixtures/data/*.csv and range 2024-01-01,2024-12-31" + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + got := parseKVList("Intention=@" + p + ",Path=./out.yaml") + if got["Intention"] != body { + t.Fatalf("Intention=%q", got["Intention"]) + } + if got["Path"] != "./out.yaml" { + t.Fatalf("Path=%q", got["Path"]) + } +} + +func TestParseKVListUnquotedStillSplits(t *testing.T) { + got := parseKVList("A=1,B=2") + if got["A"] != "1" || got["B"] != "2" { + t.Fatalf("%v", got) + } +} + +func TestOverlaySpecConn(t *testing.T) { + dir := t.TempDir() + specPath := filepath.Join(dir, "draft.yaml") + if err := os.WriteFile(specPath, []byte("name: draft\n"), 0o644); err != nil { + t.Fatal(err) + } + + apiConn, err := connection.NewConnection("MY_API", dbio.TypeApi, g.M("type", "api", "spec", "baseline")) + if err != nil { + t.Fatal(err) + } + otherConn, err := connection.NewConnection("LOCAL", dbio.TypeFileLocal, g.M("type", "file")) + if err != nil { + t.Fatal(err) + } + entries := connection.ConnEntries{ + {Name: "MY_API", Connection: apiConn}, + {Name: "LOCAL", Connection: otherConn}, + } + + out, err := overlaySpecConn(entries, "MY_API", "draft.yaml", dir) + if err != nil { + t.Fatal(err) + } + + got := out.Get("MY_API").Connection.Data["spec"] + want := "file://" + specPath + if got != want { + t.Fatalf("overlay spec=%q want %q", got, want) + } + // original entries stay untouched + if entries.Get("MY_API").Connection.Data["spec"] != "baseline" { + t.Fatalf("original entry was mutated") + } + if out.Get("LOCAL").Connection.Data["spec"] != nil { + t.Fatalf("unrelated entry changed") + } + + // missing file errors + if _, err := overlaySpecConn(entries, "MY_API", "nope.yaml", dir); err == nil { + t.Fatal("expected error for missing spec file") + } + // unknown connection errors + if _, err := overlaySpecConn(entries, "NOPE", specPath, dir); err == nil { + t.Fatal("expected error for unknown connection") + } +} diff --git a/cmd/sling/sling_build.go b/cmd/sling/sling_build.go new file mode 100644 index 000000000..2e7c79f94 --- /dev/null +++ b/cmd/sling/sling_build.go @@ -0,0 +1,285 @@ +package main + +import ( + "bufio" + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/flarco/g" + "github.com/integrii/flaggy" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/slingdata-io/sling-cli/core/sling/build" + "github.com/spf13/cast" + "gopkg.in/yaml.v3" +) + +var cliBuildFlags = []g.Flag{ + { + Name: "target", + ShortName: "t", + Type: "string", + Description: "Target connection (required if no sling_build.yml).", + }, + { + Name: "select", + ShortName: "s", + Type: "string", + Description: "Model selector (glob, tag:xxx, +model for upstream).", + }, + { + Name: "exclude", + Type: "string", + Description: "Exclude models matching pattern.", + }, + { + Name: "full-refresh", + ShortName: "f", + Type: "bool", + Description: "Force full-refresh for all models.", + }, + { + Name: "schema", + Type: "string", + Description: "Override dev schema (forces dev mode, cannot combine with --prod).", + }, + { + Name: "prod", + Type: "bool", + Description: "Force prod mode (overrides yml mode: dev).", + }, + { + Name: "vars", + Type: "string", + Description: "Variables as YAML/JSON string.", + }, + { + Name: "compile", + ShortName: "c", + Type: "bool", + Description: "Compile only — show SQL + DAG, don't execute.", + }, + { + Name: "list", + ShortName: "l", + Type: "bool", + Description: "List selected models and exit.", + }, + { + Name: "fail-fast", + ShortName: "x", + Type: "bool", + Description: "Stop on first failure (in-flight models finish).", + }, + { + Name: "no-seeds", + Type: "bool", + Description: "Skip seed loading.", + }, + { + Name: "range", + Type: "string", + Description: "Backfill range for incremental models: 'start,end[,step]'. E.g. '2024-01-01,2024-12-31,1mo'. Does not advance SLING_STATE.", + }, + { + Name: "threads", + Type: "string", + Description: "Parallel model executions (default: 4).", + }, + { + Name: "recursive", + ShortName: "R", + Type: "bool", + Description: "Recursively discover sling_build.yml in immediate subdirectories.", + }, + { + Name: "test", + Type: "bool", + Description: "Run declarative data tests only (skip materialization).", + }, + { + Name: "json", + Type: "bool", + Description: "Emit machine-readable JSON for --compile / --list.", + }, + { + Name: "debug", + ShortName: "d", + Type: "bool", + Description: "Set logging level to DEBUG.", + }, + { + Name: "trace", + Type: "bool", + Description: "Set logging level to TRACE.", + }, +} + +var cliBuild = &g.CliSC{ + Name: "build", + Description: "Build and execute SQL models", + AdditionalHelpPrepend: "\nA lightweight SQL model builder with dependency resolution, Jinja templating, and incremental materializations.", + ExecuteWithoutFlags: true, + Flags: cliBuildFlags, + PosFlags: []g.Flag{ + { + Name: "path", + Type: "string", + Description: "The project directory path (default: current directory).\n", + Required: false, + }, + }, + ExecProcess: processBuild, +} + +func init() { + cliBuild.Make().Add() +} + +func processBuild(c *g.CliSC) (ok bool, err error) { + ok = true + + opts := build.BuildOptions{ + Threads: build.DefaultThreads, + } + + projectPath := "." + compileMode := false + + for k, v := range c.Vals { + switch k { + case "path": + if p := cast.ToString(v); p != "" { + projectPath = p + } + case "target": + opts.Target = cast.ToString(v) + case "select": + if s := cast.ToString(v); s != "" { + opts.Select = strings.Split(s, ",") + } + case "exclude": + if s := cast.ToString(v); s != "" { + opts.Exclude = strings.Split(s, ",") + } + case "full-refresh": + opts.FullRefresh = cast.ToBool(v) + case "schema": + opts.Schema = cast.ToString(v) + case "prod": + opts.Prod = cast.ToBool(v) + case "vars": + if varsStr := cast.ToString(v); varsStr != "" { + varsMap := make(map[string]any) + if err := yaml.Unmarshal([]byte(varsStr), &varsMap); err != nil { + return ok, g.Error(err, "could not parse --vars") + } + opts.Vars = varsMap + } + case "compile": + compileMode = cast.ToBool(v) + case "list": + opts.List = cast.ToBool(v) + case "fail-fast": + opts.FailFast = cast.ToBool(v) + case "no-seeds": + opts.NoSeeds = cast.ToBool(v) + case "range": + if s := cast.ToString(v); s != "" { + opts.Range = g.String(s) + } + case "threads": + if t := cast.ToInt(v); t > 0 { + opts.Threads = t + } + case "recursive": + opts.Recursive = cast.ToBool(v) + case "test": + opts.Test = cast.ToBool(v) + case "json": + opts.JSON = cast.ToBool(v) + case "debug": + if cast.ToBool(v) { + os.Setenv("DEBUG", "LOW") + env.InitLogger() + } + case "trace": + if cast.ToBool(v) { + os.Setenv("DEBUG", "TRACE") + env.InitLogger() + } + } + } + + opts.Compile = compileMode + + // Validate flag combinations + if opts.Prod && opts.Schema != "" { + return ok, g.Error("cannot combine --prod and --schema") + } + + // If there's no sling_build.yml at the path and the user gave us nothing to + // work with (no --target, no -r), show help instead of walking the tree. + // This avoids slurping every .sql file under cwd as "models". + if opts.Target == "" && !opts.Recursive { + if _, err := os.Stat(filepath.Join(projectPath, build.ConfigFileName)); os.IsNotExist(err) { + flaggy.ShowHelp("") + return ok, nil + } + } + + // Build and compile + b, err := build.NewBuild(projectPath, opts) + if err != nil { + return ok, g.Error(err, "could not load build project") + } + + if err := b.Compile(); err != nil { + return ok, g.Error(err, "could not compile build project") + } + + if opts.List { + if opts.JSON { + b.PrintListJSON() + } else { + b.PrintListOutput() + } + return ok, nil + } + + if compileMode { + if opts.JSON { + b.PrintCompileJSON() + } else { + b.PrintCompileOutput() + } + return ok, nil + } + + // Execute the build + if err := b.Execute(); err != nil { + return ok, g.Error(err, "build execution failed") + } + return ok, nil +} + +// askPrompt writes label and reads one answer. It returns an error on EOF so +// callers that loop on empty input cannot spin when stdin closes. +func askPrompt(reader *bufio.Reader, label string) (string, error) { + fmt.Print(label) + input, err := reader.ReadString('\n') + if err != nil && input == "" { + return "", g.Error(err, "could not read input") + } + return strings.TrimSpace(input), nil +} + +// isInteractive reports whether stdin is a TTY. +func isInteractive() bool { + fi, err := os.Stdin.Stat() + if err != nil { + return false + } + return (fi.Mode() & os.ModeCharDevice) != 0 +} diff --git a/cmd/sling/sling_cli.go b/cmd/sling/sling_cli.go index 720979ad8..1770be7ff 100755 --- a/cmd/sling/sling_cli.go +++ b/cmd/sling/sling_cli.go @@ -17,6 +17,7 @@ import ( "github.com/slingdata-io/sling-cli/core" "github.com/slingdata-io/sling-cli/core/env" "github.com/slingdata-io/sling-cli/core/sling" + "github.com/slingdata-io/sling-cli/core/sling/assist" "github.com/slingdata-io/sling-cli/core/store" "github.com/flarco/g" @@ -60,6 +61,12 @@ var cliRunFlags = []g.Flag{ Type: "string", Description: "The directory path file to use to run nested replications/pipelines.\n", }, + { + Name: "job", + ShortName: "j", + Type: "string", + Description: "The job key from the project manifest (sling_project.yml) to run locally.\n", + }, { Name: "src-conn", ShortName: "", @@ -216,12 +223,6 @@ var cliRun = &g.CliSC{ ExecProcess: processRun, } -var cliInteractive = &g.CliSC{ - Name: "it", - Description: "launch interactive mode", - ExecProcess: slingPrompt, -} - var cliUpdate = &g.CliSC{ Name: "update", Description: "Update Sling to the latest version", @@ -367,10 +368,26 @@ var cliConns = &g.CliSC{ Name: "key=value properties...", ShortName: "", Type: "string", - Description: "The key=value properties to set. See https://docs.slingdata.io/sling-cli/environment#set-connections", + Description: "The key=value properties to set. Secret fields omitted with --type are written as ${NAME_KEY} refs. See https://docs.slingdata.io/sling-cli/environment#set-connections", }, }, Flags: []g.Flag{ + { + Name: "type", + Type: "string", + Description: "Connection type (postgres, s3, api, ...). Use ${NAME_KEY} refs for secrets.", + }, + { + Name: "output", + ShortName: "o", + Type: "string", + Description: "Output format: text (default), json. Overrides SLING_OUTPUT.", + }, + { + Name: "stdin", + Type: "bool", + Description: "Read a YAML or JSON property map from stdin.", + }, { Name: "home-dir", Type: "string", @@ -559,6 +576,27 @@ func main() { os.Exit(exitCode) } +// startAssistLogCapture buffers the run log tail for the assist failure +// snapshot. Only for commands that write one (see writeFailure below). +// Reads os.Args: g.CliObj is not populated until g.CliProcess runs. +func startAssistLogCapture() { + cmd, sub := "", "" + for _, a := range os.Args[1:] { + if strings.HasPrefix(a, "-") { + continue + } + if cmd == "" { + cmd = a + continue + } + sub = a + break + } + if cmd == "run" || (cmd == "conns" && g.In(sub, "test", "discover")) { + env.StartLogCapture() + } +} + func cliInit(done chan struct{}) int { defer close(done) @@ -592,12 +630,20 @@ func cliInit(done chan struct{}) int { os.Args = []string{os.Args[0], "runner"} case len(os.Args) > 2 && os.Args[1] == "agent": os.Args = append([]string{os.Args[0], "runner"}, os.Args[2:]...) + // 'sling project' into 'sling platform' + case len(os.Args) == 2 && os.Args[1] == "project": + os.Args = []string{os.Args[0], "platform"} + case len(os.Args) > 2 && os.Args[1] == "project": + os.Args = append([]string{os.Args[0], "platform"}, os.Args[2:]...) } + os.Args = padAssistResumeFlag(os.Args) + flaggy.ShowHelpOnUnexpectedDisable() flaggy.Parse() setSentry() + startAssistLogCapture() ok, err := g.CliProcess() if err != nil || env.TelMap["error"] != nil { @@ -615,9 +661,39 @@ func cliInit(done chan struct{}) int { Track(eventName) } - g.PrintFatal(err) + // print main error + env.PrintFatal(err) + + // append failure hint + if g.CliObj != nil && (g.In(g.CliObj.Name, "run", "build") || g.In(g.CliObj.UsedSC(), "test", "discover")) { + errMsg := getErrString(err) + snap := assist.FailureSnapshot{ + ExecID: env.ExecID, + ErrMsg: errMsg, + RunLog: env.RecentLogs(), + SignMeta: assist.MakeSignMeta(), + } + if g.CliObj != nil && g.CliObj.Name == "conns" { + snap.ConnName = cast.ToString(g.CliObj.Vals["name"]) + } else if g.CliObj != nil { + for _, key := range []string{"replication", "pipeline", "path"} { + if v, ok := g.CliObj.Vals[key]; ok { + snap.ConfigPath = cast.ToString(v) + break + } + } + } + + assist.WriteFailureSnapshot(snap) + assist.PrintFailureFooter(assist.FailureFooterOpts{ + ExecID: env.ExecID, + ErrMsg: errMsg, + SignMeta: snap.SignMeta, + }) + } return 1 } else if !ok { + // Always print classic help. flaggy.ShowHelp("") } @@ -631,15 +707,28 @@ func cliInit(done chan struct{}) int { return 0 } -func getErrString(err error) (errString string) { - if err != nil { - errString = err.Error() - E, ok := err.(*g.ErrType) - if ok && E.Debug() != "" { - errString = E.Debug() +func getErrString(err error) string { + if err == nil { + return "" + } + if eg, ok := err.(*g.ErrorGroup); ok { + parts := make([]string, 0, len(eg.Errors)) + for i, child := range eg.Errors { + s := getErrString(child) + if s == "" { + continue + } + if i < len(eg.Names) && eg.Names[i] != "" { + s = g.F("--------------------------- %s ---------------------------\n%s", eg.Names[i], s) + } + parts = append(parts, s) } + return strings.Join(parts, "\n") + } + if d := g.ErrMsgDebug(err); d != "" { + return d } - return + return err.Error() } func setSentry() { diff --git a/cmd/sling/sling_conns.go b/cmd/sling/sling_conns.go index 6a1d46686..ce580eacb 100644 --- a/cmd/sling/sling_conns.go +++ b/cmd/sling/sling_conns.go @@ -4,6 +4,7 @@ import ( "encoding/csv" "encoding/hex" "fmt" + "io" "os" "strings" "time" @@ -69,24 +70,72 @@ func processConns(c *g.CliSC) (ok bool, err error) { } g.Info("connection `%s` has been removed from %s", name, ec.EnvFile.Path) case "set": - if len(c.Vals) == 0 { + name := strings.ToUpper(cast.ToString(c.Vals["name"])) + if name == "" { flaggy.ShowHelp("") return ok, nil } + setOutput, outErr := ResolveOutputFormat(c, "json") + if outErr != nil { + return ok, outErr + } + asJSON = setOutput == "json" + + kvMap := map[string]any{} + if cast.ToBool(c.Vals["stdin"]) { + stat, _ := os.Stdin.Stat() + if stat != nil && stat.Mode()&os.ModeCharDevice != 0 { + return ok, g.Error("stdin is a terminal; pipe a YAML or JSON property map") + } + raw, readErr := io.ReadAll(os.Stdin) + if readErr != nil { + return ok, g.Error(readErr, "could not read stdin") + } + stdinMap, parseErr := connection.ParsePropsInput(string(raw)) + if parseErr != nil { + return ok, parseErr + } + kvMap = stdinMap + } + kvArr := []string{cast.ToString(c.Vals["value properties..."])} - kvMap := map[string]interface{}{} for k, v := range g.KVArrToMap(append(kvArr, flaggy.TrailingArguments...)...) { k = strings.ToLower(k) + if k == "" { + continue + } kvMap[k] = v } - name := strings.ToUpper(cast.ToString(c.Vals["name"])) + if t := strings.TrimSpace(cast.ToString(c.Vals["type"])); t != "" { + kvMap["type"] = strings.ToLower(t) + } + + if err = connection.RejectLiteralSecrets(name, kvMap); err != nil { + return ok, err + } - err := ec.Set(name, kvMap) + err = ec.Set(name, kvMap) if err != nil { return ok, g.Error(err, "could not set %s (See https://docs.slingdata.io/sling-cli/environment)", name) } - g.Info("connection `%s` has been set in %s. Please test with `sling conns test %s`", name, ec.EnvFile.Path, name) + + loc, locErr := ec.EnvFile.LookupConnection(name) + if locErr != nil { + loc = env.ConnLocation{Path: ec.EnvFile.Path, Connection: name, Missing: []env.MissingRef{}} + } + + if asJSON { + fmt.Println(g.Marshal(loc)) + return ok, nil + } + + g.Info("connection `%s` has been set in %s:%d", name, loc.Path, loc.Line) + if len(loc.Missing) > 0 { + g.Info("set the env var(s), then: sling conns test %s", name) + } else { + g.Info("next: sling conns test %s", name) + } case "exec": env.SetTelVal("task", g.Marshal(g.M("type", sling.ConnExec))) @@ -305,9 +354,29 @@ func processConns(c *g.CliSC) (ok bool, err error) { env.SetTelVal("task", g.Marshal(g.M("type", sling.ConnTest))) name := cast.ToString(c.Vals["name"]) - if conn := entries.Get(name); conn.Name != "" { + conn := entries.Get(name) + if conn.Name != "" { env.SetTelVal("conn_type", conn.Connection.Type.String()) env.SetTelVal("conn_keys", lo.Keys(conn.Connection.Data)) + if g.IsDebugLow() { + g.Debug("connection %s properties: %s", name, g.Marshal(conn.Connection.Data)) + } + } + + refData := conn.Connection.Data + if len(refData) == 0 { + if cdata, ok := ef.Connections[strings.ToUpper(name)]; ok { + refData = cdata + } + } + if refs := connection.FindUnsetEnvRefs(refData); len(refs) > 0 { + loc, _ := ef.LookupConnection(strings.ToUpper(name)) + err = connection.FormatUnsetRefError(refs, loc) + if os.Getenv("SLING_OUTPUT") == "json" { + fmt.Println(g.Marshal(g.M("success", false, "error", g.ErrMsg(err)))) + return + } + return ok, err } // for testing specific endpoints diff --git a/cmd/sling/sling_init.go b/cmd/sling/sling_init.go new file mode 100644 index 000000000..552b08e8f --- /dev/null +++ b/cmd/sling/sling_init.go @@ -0,0 +1,155 @@ +package main + +import ( + "bufio" + "fmt" + "os" + "strings" + + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core/dbio/connection" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/slingdata-io/sling-cli/core/sling/project" + "github.com/spf13/cast" +) + +var initFlags = []g.Flag{ + {Name: "source", Type: "string", Description: "Source connection name"}, + {Name: "target", ShortName: "t", Type: "string", Description: "Target connection name"}, + {Name: "name", Type: "string", Description: "Project name (default: folder name)"}, + {Name: "yes", ShortName: "y", Type: "bool", Description: "Overwrite existing files without a prompt"}, + {Name: "force", Type: "bool", Description: "Allow init inside an existing project"}, + {Name: "test", Type: "bool", Description: "Test source and target connections"}, + {Name: "debug", ShortName: "d", Type: "bool", Description: "Set logging level to DEBUG."}, +} + +var cliInitProject = &g.CliSC{ + Name: "init", + Description: "Create a Sling project in the current folder", + AdditionalHelpPrepend: "\nSee more details at https://docs.slingdata.io/", + Flags: initFlags, + ExecProcess: processInit, +} + +func init() { + cliInitProject.Make().Add() +} + +func processInit(c *g.CliSC) (ok bool, err error) { + if cast.ToBool(c.Vals["debug"]) { + os.Setenv("DEBUG", "LOW") + env.InitLogger() + } + ok = true + return ok, runInit(initOpts(c), cast.ToBool(c.Vals["test"])) +} + +func initOpts(c *g.CliSC) project.Options { + return project.Options{ + Source: cast.ToString(c.Vals["source"]), + Target: cast.ToString(c.Vals["target"]), + Name: cast.ToString(c.Vals["name"]), + Yes: cast.ToBool(c.Vals["yes"]), + Force: cast.ToBool(c.Vals["force"]), + } +} + +func runInit(opts project.Options, testConns bool) error { + if err := resolveInitConns(&opts, testConns); err != nil { + return err + } + + res, err := project.Init(opts) + if err != nil { + return err + } + + for _, f := range res.Files { + g.Info("wrote `%s`", f) + } + fmt.Println() + fmt.Println(" Next: Add .sql models to schema folders and run 'sling build'") + fmt.Println(" Run a job locally with 'sling run -j '. Schedules fire on the platform once this folder is linked.") + fmt.Println() + + if os.Getenv("SLING_PROJECT_TOKEN") != "" { + fmt.Println("To see the platform jobs for this folder, run `sling platform jobs list`.") + } + return nil +} + +// resolveInitConns fills in missing source/target via interactive prompts and +// optionally tests both connections. +func resolveInitConns(opts *project.Options, testConns bool) error { + opts.Source = strings.TrimSpace(opts.Source) + opts.Target = strings.TrimSpace(opts.Target) + + if opts.Source == "" || opts.Target == "" { + if err := promptMissingConns(opts); err != nil { + return err + } + } + + if testConns { + entries := connection.GetLocalConns() + for _, name := range []string{opts.Source, opts.Target} { + if err := testNamedConn(entries, name); err != nil { + return err + } + } + } + return nil +} + +// promptMissingConns lists local connections and asks for any of source/target +// not already provided via flags. +func promptMissingConns(opts *project.Options) error { + entries := connection.GetLocalConns() + if len(entries) == 0 { + return g.Error("no connections found. Run `sling assist` or `sling conns set --type` to add a connection.") + } + if !isInteractive() { + return g.Error("source and target are required; pass --source and --target with --yes") + } + + fmt.Println("\n Available connections:") + for i, conn := range entries { + fmt.Printf(" %d. %s (%s)\n", i+1, conn.Name, conn.Connection.Type.String()) + } + fmt.Println() + + reader := bufio.NewReader(os.Stdin) + var err error + if opts.Source == "" { + if opts.Source, err = askPrompt(reader, " ? Source connection: "); err != nil { + return err + } + } + if opts.Target == "" { + if opts.Target, err = askPrompt(reader, " ? Target connection: "); err != nil { + return err + } + } + + if opts.Source == "" || opts.Target == "" { + return g.Error("source and target are required") + } + return nil +} + +// testNamedConn tests one connection looked up by name in the given entries. +func testNamedConn(entries connection.ConnEntries, name string) error { + conn := entries.Get(name) + if conn.Name == "" { + return g.Error("connection %s not found", name) + } + ok, err := conn.Connection.Test() + conn.Connection.Close() + if err != nil { + return g.Error(err, "connection %s failed", name) + } + if !ok { + return g.Error("connection %s failed", name) + } + return nil +} diff --git a/cmd/sling/sling_prompt.go b/cmd/sling/sling_prompt.go deleted file mode 100644 index 90d117a34..000000000 --- a/cmd/sling/sling_prompt.go +++ /dev/null @@ -1,122 +0,0 @@ -package main - -import ( - "fmt" - "os" - "strings" - - "github.com/c-bata/go-prompt" - "github.com/flarco/g" - "github.com/slingdata-io/sling-cli/core" - "github.com/slingdata-io/sling-cli/core/dbio/connection" - "github.com/spf13/cast" -) - -var suggestions = []prompt.Suggest{} - -func init() { - suggList := [][]string{ - {"exit", "Exit interactive mode"}, - {cliRun.Name, cliRun.Description}, - {cliConns.Name, cliConns.Description}, - } - for _, sl := range suggList { - suggestions = append(suggestions, prompt.Suggest{Text: sl[0], Description: sl[1]}) - } -} - -func completer(in prompt.Document) []prompt.Suggest { - localSuggestions := []prompt.Suggest{} - - w := in.GetWordBeforeCursor() - blocks := strings.Split(in.Text, " ") - // trimmedBlocks := strings.Split(strings.TrimSpace(in.Text), " ") - - lastWord := blocks[len(blocks)-1] - prevWord := lastWord - - if len(blocks) > 1 { - prevWord = blocks[len(blocks)-2] - } - - switch blocks[0] { - case cliRun.Name: - // collect strings flags - stringFlags := []string{} - for _, f := range cliRun.Flags { - if f.Type == "string" { - stringFlags = append(stringFlags, f.Name) - } - } - - // suggestions based on previous word - switch { - case g.In(prevWord, "src-conn", "tgt-conn"): - for _, conn := range connection.GetLocalConns() { - localSuggestions = append(localSuggestions, prompt.Suggest{Text: conn.Name, Description: conn.Description}) - } - return prompt.FilterHasPrefix(localSuggestions, w, true) - case g.In(prevWord, stringFlags...): - return []prompt.Suggest{} - } - - // suggest normal flags - for _, f := range cliRun.Flags { - localSuggestions = append(localSuggestions, prompt.Suggest{Text: f.Name, Description: f.Description}) - } - return prompt.FilterHasPrefix(localSuggestions, w, true) - - case cliConns.Name: - for _, f := range cliConns.Flags { - localSuggestions = append(localSuggestions, prompt.Suggest{Text: f.Name, Description: f.Description}) - } - return prompt.FilterHasPrefix(localSuggestions, w, true) - case "": - return []prompt.Suggest{} - } - return prompt.FilterHasPrefix(suggestions, w, true) -} - -func executor(in string) { - in = strings.TrimSpace(in) - - blocks := strings.Split(in, " ") - switch blocks[0] { - case "exit": - fmt.Println("exiting") - os.Exit(0) - case cliRun.Name: - cliRun.Vals = g.M(cast.ToSlice(blocks[1:])...) - _, err := cliRun.ExecProcess(cliRun) - g.LogError(err) - case cliConns.Name: - if len(blocks) == 1 { - return - } - for _, subCom := range cliConns.SubComs { - if subCom.Name == blocks[1] { - subCom.Vals = g.M(cast.ToSlice(blocks[2:])...) - _, err := subCom.ExecProcess(subCom) - g.LogError(err) - } - } - } - println(in) -} - -func slingPrompt(c *g.CliSC) (ok bool, err error) { - fmt.Println("sling - An Extract-Load tool") - fmt.Println("Slings data from a data source to a data target.\nVersion " + core.Version) - - p := prompt.New( - executor, - completer, - prompt.OptionPrefix("sling > "), - // prompt.OptionLivePrefix(livePrefix), - prompt.OptionTitle("sling"), - ) - - p.Run() - - return -} diff --git a/cmd/sling/sling_run.go b/cmd/sling/sling_run.go index bc48eb757..6ca697550 100755 --- a/cmd/sling/sling_run.go +++ b/cmd/sling/sling_run.go @@ -23,6 +23,8 @@ import ( "github.com/slingdata-io/sling-cli/core/dbio/iop" "github.com/slingdata-io/sling-cli/core/env" "github.com/slingdata-io/sling-cli/core/sling" + "github.com/slingdata-io/sling-cli/core/sling/project" + "github.com/slingdata-io/sling-cli/core/sling/validate" "github.com/flarco/g" "github.com/spf13/cast" @@ -50,6 +52,7 @@ func processRun(c *g.CliSC) (ok bool, err error) { } var replicationCfgPath, pipelineCfgPath, directoryPath string + var jobKey, barePath string showExamples := false selectStreams := []string{} @@ -88,10 +91,17 @@ func processRun(c *g.CliSC) (ok bool, err error) { case "directory": env.SetTelVal("run_mode", "directory") directoryPath = cast.ToString(v) + case "job": + jobKey = cast.ToString(v) case "path": filePath := cast.ToString(v) fileInfo, err := os.Stat(filePath) if err != nil { + // Missing path: try it as a job key after the flag loop. + if jobKey == "" && strings.TrimSpace(filePath) != "" { + barePath = filePath + continue + } return true, g.Error(err, "error accessing path: %s", filePath) } @@ -252,6 +262,23 @@ func processRun(c *g.CliSC) (ok bool, err error) { env.InitLogger() } + if key := lo.Ternary(jobKey != "", jobKey, barePath); key != "" { + path, isPipeline, jobErr := resolveJob(key, cfg, &selectStreams) + if jobErr != nil { + if jobKey == "" { + return true, g.Error(jobErr, "error accessing path: %s", barePath) + } + return true, jobErr + } + if isPipeline { + env.SetTelVal("run_mode", "pipeline") + pipelineCfgPath = path + } else { + env.SetTelVal("run_mode", "replication") + replicationCfgPath = path + } + } + if showExamples { println(examples) return ok, nil @@ -297,7 +324,7 @@ runReplication: return ok, g.Error(err, "failure running directory (see docs @ https://docs.slingdata.io)") } } else if pipelineCfgPath != "" { - err = runPipeline(pipelineCfgPath) + err = runPipeline(pipelineCfgPath, cfg.Env) if err != nil { return ok, g.Error(err, "failure running pipeline (see docs @ https://docs.slingdata.io)") } @@ -311,8 +338,17 @@ runReplication: // run task, add replication config for md5 rc := cfg.AsReplication() - // run as replication is stream is wildcard - if cfg.HasWildcard() { + // stdin only counts as the source if no source connection is given, + // since Options.StdIn is also true for any non-interactive shell + isStdInOut := (cfg.Options.StdIn && cfg.Source.Conn == "") || cfg.Options.StdOut + + if isStdInOut && (cfg.WithRetries() || cfg.WithChunking() || cfg.WithThreads()) { + g.Warn("chunking, threads and retries are not supported with stdin/stdout") + } + + // run as replication if stream is wildcard, or if a pro feature + // needs the replication path + if cfg.HasWildcard() || (cfg.RequiresPro() && !isStdInOut) { replicationCfgPath = path.Join(env.GetTempFolder(), g.NewTsID("replication.temp")+".json") err = os.WriteFile(replicationCfgPath, []byte(g.Marshal(rc)), 0775) if err != nil { @@ -337,6 +373,57 @@ runReplication: return ok, err } +// resolveJob turns a manifest key into a replication/pipeline path. +// Streams, mode, and variables from the spec fill cfg only when the CLI left them empty. +func resolveJob(key string, cfg *sling.Config, selectStreams *[]string) (path string, isPipeline bool, err error) { + wd, err := os.Getwd() + if err != nil { + return "", false, g.Error(err, "could not get working directory") + } + root, spec, err := project.ResolveJob(wd, key) + if err != nil { + return "", false, err + } + + file := strings.TrimSpace(spec.File) + if file == "" { + return "", false, g.Error("job %s has no file", key) + } + if !filepath.IsAbs(file) { + file = filepath.Join(root, file) + } + body, err := os.ReadFile(file) + if err != nil { + return "", false, g.Error(err, "could not read the file for job %s", key) + } + + kind := validate.DetectFileKind(body, file) + if kind != validate.KindReplication && kind != validate.KindPipeline { + return "", false, g.Error("job %s file %s is a %s; expected a replication or pipeline", key, spec.File, kind) + } + + if len(spec.Streams) > 0 && len(*selectStreams) == 0 { + *selectStreams = spec.Streams + } + if spec.Mode != "" && strings.TrimSpace(string(cfg.Mode)) == "" { + cfg.Mode = sling.Mode(spec.Mode) + } + if len(spec.Variables) > 0 { + if cfg.Env == nil { + cfg.Env = map[string]string{} + } + for k, v := range spec.Variables { + if _, set := cfg.Env[k]; !set { + cfg.Env[k] = v + } + } + } + if len(spec.Schedules) > 0 { + g.Debug("schedules fire on the platform after deploy; this run is manual") + } + return file, kind == validate.KindPipeline, nil +} + func runTask(cfg *sling.Config, replication *sling.ReplicationConfig) (err error) { var task *sling.TaskExecution @@ -689,10 +776,11 @@ func replicationRun(cfgPath string, cfgOverwrite *sling.Config, selectStreams .. return eG.Err() } -func runPipeline(pipelineCfgPath string) (err error) { +func runPipeline(pipelineCfgPath string, overlay map[string]string) (err error) { g.Debug("Sling version: %s (%s %s)", core.Version, runtime.GOOS, runtime.GOARCH) - pipeline, err := sling.LoadPipelineConfigFromFile(pipelineCfgPath) + // Job / --env values overlay the file env. + pipeline, err := sling.LoadPipelineConfigFromFile(pipelineCfgPath, overlay) if err != nil { return g.Error(err, "could not load pipeline: %s", pipelineCfgPath) } @@ -780,7 +868,7 @@ func runDirectory(directoryPath string) (err error) { switch runFile.Type { case sling.RunFilePipeline: - err = runPipeline(runFile.File.RelPath) + err = runPipeline(runFile.File.RelPath, nil) case sling.RunFileReplication: err = runReplication(runFile.File.RelPath, nil) } diff --git a/cmd/sling/sling_validate.go b/cmd/sling/sling_validate.go new file mode 100644 index 000000000..cf6e628f6 --- /dev/null +++ b/cmd/sling/sling_validate.go @@ -0,0 +1,131 @@ +package main + +import ( + "fmt" + "os" + "strings" + + "github.com/flarco/g" + "github.com/integrii/flaggy" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/slingdata-io/sling-cli/core/sling/project" + "github.com/slingdata-io/sling-cli/core/sling/validate" + "github.com/spf13/cast" +) + +var cliValidate = &g.CliSC{ + Name: "validate", + Description: "Validate Sling files", + AdditionalHelpPrepend: "\nDetect the file kind and check the structure. Compile is the default, which confirms the file is ready to run. Compile needs connections and replaces ${VAR}. Use --parse-only for a syntax check. With no paths, validate the project from the current directory.", + Flags: []g.Flag{ + {Name: "quiet", ShortName: "q", Type: "bool", Description: "Exit code only. No output."}, + {Name: "parse-only", Type: "bool", Description: "Skip compilation step, parse structure only. Do not replace ${VAR}."}, + {Name: "detailed", Type: "bool", Description: "Show one table per kind (streams, steps, endpoints, connections)."}, + {Name: "ndjson", Type: "bool", Description: "One JSON object per line."}, + {Name: "output", ShortName: "o", Type: "string", Description: "Output format: json. Human table is the TTY default."}, + {Name: "json", Type: "bool", Description: "Emit one JSON object keyed by path."}, + {Name: "debug", ShortName: "d", Type: "bool", Description: "Set logging level to DEBUG."}, + {Name: "trace", Type: "bool", Description: "Set logging level to TRACE."}, + }, + PosFlags: []g.Flag{ + { + Name: "paths...", + Type: "string", + Description: "Files or folders to validate.", + Required: false, + }, + }, + ExecProcess: processValidate, +} + +func init() { + cliValidate.Make().Add() +} + +func processValidate(c *g.CliSC) (ok bool, err error) { + ok = true + + if cast.ToBool(c.Vals["trace"]) { + os.Setenv("DEBUG", "TRACE") + env.InitLogger() + } else if cast.ToBool(c.Vals["debug"]) { + os.Setenv("DEBUG", "LOW") + env.InitLogger() + } + + paths := collectValidatePaths(c) + if len(paths) == 0 { + // Bare invocation inside a project validates the project root. + wd, wdErr := os.Getwd() + if wdErr == nil { + if root, findErr := project.FindRoot(wd); findErr == nil && root != "" { + g.Debug("validating project root %s", root) + paths = []string{root} + } + } + } + if len(paths) == 0 { + flaggy.ShowHelp("") + return ok, nil + } + + opts := validate.Options{ + Compile: !cast.ToBool(c.Vals["parse-only"]), + Quiet: cast.ToBool(c.Vals["quiet"]), + NDJSON: cast.ToBool(c.Vals["ndjson"]), + JSON: cast.ToBool(c.Vals["json"]), + Detailed: cast.ToBool(c.Vals["detailed"]), + } + + output := strings.ToLower(strings.TrimSpace(cast.ToString(c.Vals["output"]))) + switch output { + case "", "text": + case "json": + opts.JSON = true + default: + return ok, g.Error("invalid --output %q; expected json", output) + } + + results := validate.ParsePaths(paths, opts) + text, err := validate.GetOutput(results, opts) + if err != nil { + return ok, g.Error(err, "could not render validate output") + } + if text != "" { + fmt.Fprint(os.Stdout, text+"\n") + } + + if validate.AnyFailed(results) { + return ok, validateFailErr(results) + } + return ok, nil +} + +func collectValidatePaths(c *g.CliSC) []string { + paths := []string{} + if v := strings.TrimSpace(cast.ToString(c.Vals["paths..."])); v != "" { + paths = append(paths, v) + } + paths = append(paths, flaggy.TrailingArguments...) + return paths +} + +func validateFailErr(results []validate.FileResult) error { + n := 0 + var first validate.FileResult + for _, r := range results { + if !r.OK { + n++ + if first.Path == "" { + first = r + } + } + } + if n == 1 { + if first.Error != "" { + return g.Error("%s: %s", first.Path, first.Error) + } + return g.Error("%s: parse failed", first.Path) + } + return g.Error("%d files failed to parse", n) +} diff --git a/core/dbio/api/spec.go b/core/dbio/api/spec.go index 94677026a..f9c5a9efb 100644 --- a/core/dbio/api/spec.go +++ b/core/dbio/api/spec.go @@ -111,6 +111,10 @@ func LoadSpec(specBody string) (spec Spec, err error) { return spec, g.Error(err, "queue validation failed") } + if err = spec.validateDependsOn(compiledEndpointMap); err != nil { + return spec, g.Error(err, "depends_on validation failed") + } + // validate that all sync keys have corresponding processors if err = spec.validateSync(compiledEndpointMap); err != nil { return spec, g.Error(err, "sync validation failed") @@ -187,6 +191,30 @@ func (s *Spec) validateQueueProducers(endpointMap EndpointMap) error { return nil } +// validateDependsOn checks that each depends_on name is an endpoint in the spec. +func (s *Spec) validateDependsOn(endpointMap EndpointMap) error { + var missing []string + for _, name := range s.endpointsOrdered { + ep, ok := endpointMap[name] + if !ok { + continue + } + for _, dep := range ep.DependsOn { + dep = strings.TrimSpace(dep) + if dep == "" { + continue + } + if _, found := endpointMap[dep]; !found { + missing = append(missing, g.F("endpoint %q depends_on %q, which is not an endpoint", name, dep)) + } + } + } + if len(missing) > 0 { + return g.Error("depends_on validation failed:\n - %s", strings.Join(missing, "\n - ")) + } + return nil +} + // validateSync checks that all sync keys have corresponding processors that write to state func (s *Spec) validateSync(endpointMap EndpointMap) error { var validationErrors []string diff --git a/core/dbio/api/spec_test.go b/core/dbio/api/spec_test.go index f5488582e..b6a64c9d2 100644 --- a/core/dbio/api/spec_test.go +++ b/core/dbio/api/spec_test.go @@ -991,6 +991,19 @@ func TestSpecEndpointDependsOnEdgeCases(t *testing.T) { }) } +func TestLoadSpecUnknownDependsOn(t *testing.T) { + _, err := LoadSpec(` +name: "Bad Depends" +endpoints: + child: + request: + url: "https://api.example.com/child" + depends_on: [does_not_exist] +`) + require.Error(t, err) + assert.Contains(t, err.Error(), "does_not_exist") +} + func TestSpecCircularDependencyHandling(t *testing.T) { // While circular dependencies shouldn't occur with proper queue usage, // the topological sort should handle them gracefully diff --git a/core/dbio/connection/connection.go b/core/dbio/connection/connection.go index d1fbb5929..85a3752af 100644 --- a/core/dbio/connection/connection.go +++ b/core/dbio/connection/connection.go @@ -984,7 +984,6 @@ func (c *Connection) setURL() (err error) { switch { case port_ok && instance_ok: template += ":{port}/{instance}" - g.Debug("SQL Server: port %s and instance %s are both set. The driver uses the port and ignores the instance name.", c.Data["port"], c.Data["instance"]) if buildingURL && cast.ToInt(c.Data["port"]) == 1433 { g.Warn("SQL Server: port 1433 and instance %s are both set. The driver uses port 1433 and ignores the instance name. For a named instance, set `port` to the instance TCP port or omit `port`.", c.Data["instance"]) } diff --git a/core/dbio/connection/connection_local.go b/core/dbio/connection/connection_local.go index 03fc0bb43..825ef04eb 100644 --- a/core/dbio/connection/connection_local.go +++ b/core/dbio/connection/connection_local.go @@ -306,6 +306,19 @@ func (ec *EnvFileConns) Set(name string, kvMap map[string]any) (err error) { if name == "" { return g.Error("name is blank") } + name = strings.ToUpper(name) + + if kvMap == nil { + kvMap = map[string]any{} + } + if err = NormalizeConnProps(kvMap); err != nil { + return err + } + + ef := ec.EnvFile + if existing, ok := ef.Connections[name]; ok { + kvMap = MergeConnProps(existing, kvMap) + } // parse url if url := cast.ToString(kvMap["url"]); url != "" { @@ -322,29 +335,9 @@ func (ec *EnvFileConns) Set(name string, kvMap map[string]any) (err error) { if _, typeOK := dbio.ValidateType(cast.ToString(t)); found && !typeOK { return g.Error("invalid type (%s)", cast.ToString(t)) } else if !found { - return g.Error("need to specify valid `type` key or provide `url`.") - } - - // need to set secrets and inputs as maps - if t == "api" { - if secretsStr := cast.ToString(kvMap["secrets"]); secretsStr != "" { - secrets, err := g.UnmarshalYAMLMap(secretsStr) - if err != nil { - return g.Error(err, "could not parse secrets string") - } - kvMap["secrets"] = secrets - } - - if inputsStr := cast.ToString(kvMap["inputs"]); inputsStr != "" { - inputs, err := g.UnmarshalYAMLMap(inputsStr) - if err != nil { - return g.Error(err, "could not parse inputs string") - } - kvMap["inputs"] = inputs - } + return g.Error("need to specify valid `type` key or provide `url`") } - ef := ec.EnvFile ef.Connections[name] = kvMap err = ef.WriteEnvFile() if err != nil { diff --git a/core/dbio/connection/connection_props.go b/core/dbio/connection/connection_props.go new file mode 100644 index 000000000..d7139ae73 --- /dev/null +++ b/core/dbio/connection/connection_props.go @@ -0,0 +1,276 @@ +package connection + +import ( + "encoding/json" + "path/filepath" + "sort" + "strings" + + "github.com/flarco/g" + "github.com/samber/lo" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/spf13/cast" + "gopkg.in/yaml.v3" +) + +// MergeConnProps copies existing and applies incoming. Nested maps merge. +// Keys that incoming does not pass stay on the result. +func MergeConnProps(existing, incoming map[string]any) map[string]any { + out := copyAnyMap(existing) + if out == nil { + out = map[string]any{} + } + for k, v := range incoming { + if vMap := asAnyMap(v); vMap != nil { + if eMap := asAnyMap(out[k]); eMap != nil { + out[k] = MergeConnProps(eMap, vMap) + continue + } + } + out[k] = v + } + return out +} + +// EnvVarRef builds ${_} for a connection field. +func EnvVarRef(connName, key string) string { + name := strings.ToUpper(strings.TrimSpace(connName)) + prop := strings.ToUpper(strings.ReplaceAll(strings.TrimSpace(key), "-", "_")) + return "${" + name + "_" + prop + "}" +} + +// NormalizeConnProps parses secrets/inputs YAML strings into maps. +func NormalizeConnProps(kv map[string]any) error { + if kv == nil { + return nil + } + for _, field := range []string{"secrets", "inputs"} { + v, ok := kv[field] + if !ok { + continue + } + s, isStr := v.(string) + if !isStr || strings.TrimSpace(s) == "" { + continue + } + parsed, err := g.UnmarshalYAMLMap(s) + if err != nil { + return g.Error(err, "could not parse %s string", field) + } + kv[field] = parsed + } + return nil +} + +// RejectLiteralSecrets refuses secret fields (and nested secrets values) +// whose value is not a ${VAR} ref. +func RejectLiteralSecrets(name string, kv map[string]any) error { + if kv == nil { + return nil + } + if err := NormalizeConnProps(kv); err != nil { + return err + } + + var literals []string + for _, k := range env.SecretKeys { + if v, ok := kv[k]; ok && isLiteralSecret(v) { + literals = append(literals, k) + } + } + + // nested secrets are keyed separately, so they cannot repeat the above + if secrets := asAnyMap(kv["secrets"]); secrets != nil { + nested := lo.Keys(secrets) + sort.Strings(nested) + for _, k := range nested { + if isLiteralSecret(secrets[k]) { + literals = append(literals, "secrets."+k) + } + } + } + + if len(literals) == 0 { + return nil + } + example := EnvVarRef(name, "PASSWORD") + return g.Error("secret field(s) %s must be an env-var ref such as %s. Do not pass secret values.", strings.Join(literals, ", "), example) +} + +func isLiteralSecret(v any) bool { + if v == nil { + return false + } + if asAnyMap(v) != nil { + // nested map: checked by the caller per-key + return false + } + s := strings.TrimSpace(cast.ToString(v)) + if s == "" { + return false + } + return !env.IsEnvVarRef(s) +} + +// UnsetEnvRef is a ${VAR} value that g.Rmd did not substitute (var not set). +type UnsetEnvRef struct { + Key string + Var string +} + +// FindUnsetEnvRefs walks connection data for whole-string ${VAR} values. +func FindUnsetEnvRefs(data map[string]any) []UnsetEnvRef { + var out []UnsetEnvRef + walkUnsetRefs(data, "", &out) + return out +} + +func walkUnsetRefs(v any, prefix string, out *[]UnsetEnvRef) { + if v == nil { + return + } + if m := asAnyMap(v); m != nil { + keys := lo.Keys(m) + // stable-ish: not required, but keeps errors readable + for _, k := range keys { + path := k + if prefix != "" { + path = prefix + "." + k + } + walkUnsetRefs(m[k], path, out) + } + return + } + s := strings.TrimSpace(cast.ToString(v)) + if !env.IsEnvVarRef(s) { + return + } + *out = append(*out, UnsetEnvRef{Key: prefix, Var: env.EnvVarRefName(s)}) +} + +// FormatUnsetRefError names each unset var and its env.yaml line. +func FormatUnsetRefError(refs []UnsetEnvRef, loc env.ConnLocation) error { + file := filepath.Base(loc.Path) + if file == "" || file == "." { + file = "env.yaml" + } + lineFor := func(r UnsetEnvRef) int { + for _, m := range loc.Missing { + if m.Var == r.Var || m.Key == r.Key { + return m.Line + } + } + return 0 + } + msgs := make([]string, 0, len(refs)) + for _, r := range refs { + if line := lineFor(r); line > 0 { + msgs = append(msgs, g.F("env var %s is not set (%s:%d)", r.Var, file, line)) + } else { + msgs = append(msgs, g.F("env var %s is not set", r.Var)) + } + } + return g.Error(strings.Join(msgs, "; ")) +} + +// ScrubConnProps returns key names and ref/status only. No secret values. +func ScrubConnProps(kv map[string]any) []map[string]any { + out := []map[string]any{} + keys := lo.Keys(kv) + sort.Strings(keys) + for _, k := range keys { + v := kv[k] + if nested := asAnyMap(v); nested != nil { + nKeys := lo.Keys(nested) + sort.Strings(nKeys) + for _, nk := range nKeys { + out = append(out, scrubEntry(k+"."+nk, nested[nk])) + } + continue + } + out = append(out, scrubEntry(k, v)) + } + return out +} + +func scrubEntry(key string, v any) map[string]any { + s := strings.TrimSpace(cast.ToString(v)) + entry := g.M("key", key) + if env.IsEnvVarRef(s) { + entry["ref"] = s + return entry + } + entry["set"] = s != "" + return entry +} + +// ParsePropsInput reads a YAML or JSON property map from stdin/payload text. +func ParsePropsInput(raw string) (map[string]any, error) { + raw = strings.TrimSpace(raw) + if raw == "" { + return nil, g.Error("stdin is empty") + } + var m map[string]any + if strings.HasPrefix(raw, "{") { + if err := json.Unmarshal([]byte(raw), &m); err != nil { + return nil, g.Error(err, "could not parse JSON properties") + } + return lowercaseKeys(m), nil + } + if err := yaml.Unmarshal([]byte(raw), &m); err != nil { + return nil, g.Error(err, "could not parse YAML properties") + } + return lowercaseKeys(m), nil +} + +func lowercaseKeys(m map[string]any) map[string]any { + if m == nil { + return map[string]any{} + } + out := map[string]any{} + for k, v := range m { + key := strings.ToLower(k) + if nested := asAnyMap(v); nested != nil { + out[key] = lowercaseKeys(nested) + continue + } + out[key] = v + } + return out +} + +func copyAnyMap(m map[string]any) map[string]any { + if m == nil { + return nil + } + out := make(map[string]any, len(m)) + for k, v := range m { + if nested := asAnyMap(v); nested != nil { + out[k] = copyAnyMap(nested) + continue + } + out[k] = v + } + return out +} + +func asAnyMap(v any) map[string]any { + switch t := v.(type) { + case map[string]any: + return t + case map[any]any: + out := make(map[string]any, len(t)) + for k, val := range t { + out[cast.ToString(k)] = val + } + return out + case map[string]string: + out := make(map[string]any, len(t)) + for k, val := range t { + out[k] = val + } + return out + default: + return nil + } +} diff --git a/core/dbio/connection/connection_props_test.go b/core/dbio/connection/connection_props_test.go new file mode 100644 index 000000000..2a814c4f9 --- /dev/null +++ b/core/dbio/connection/connection_props_test.go @@ -0,0 +1,22 @@ +package connection + +import ( + "strings" + "testing" +) + +func TestRejectLiteralSecretsNested(t *testing.T) { + err := RejectLiteralSecrets("MY_API", map[string]any{ + "type": "api", + "spec": "stripe", + "secrets": map[string]any{ + "api_key": "sk_live_distinctive", + }, + }) + if err == nil { + t.Fatal("expected literal nested secret to be refused") + } + if !strings.Contains(err.Error(), "secrets.api_key") && !strings.Contains(err.Error(), "api_key") { + t.Fatalf("error should name the field: %v", err) + } +} diff --git a/core/dbio/database/database.go b/core/dbio/database/database.go index f3a00f9a1..545098e34 100755 --- a/core/dbio/database/database.go +++ b/core/dbio/database/database.go @@ -905,7 +905,12 @@ func (conn *BaseConn) BulkExportStream(table Table) (ds *iop.Datastream, err err // } g.Trace("BulkExportStream not implemented for %s", conn.GetType()) - return conn.Self().StreamRows(table.Select(), g.M("columns", table.Columns)) + ds, err = conn.Self().StreamRows(table.Select(), g.M("columns", table.Columns)) + if err != nil { + return ds, err + } + table.RestoreGeometryTypes(ds) + return ds, nil } // BulkImportStream import the stream rows in bulk @@ -1575,6 +1580,13 @@ func SQLColumns(colTypes []ColumnType, conn Connection) (columns iop.Columns) { col.Sourced = fc.Sourced } + // drivers report user-defined types (e.g. geometry) as unnamed; + // keep the fetched type so geometry stays geometry + if fc.Type.IsGeometry() && !col.Type.IsGeometry() { + col.Type = fc.Type + col.Sourced = fc.Sourced + } + col.Constraint = fc.Constraint // fetch decimal info diff --git a/core/dbio/database/database_adbc.go b/core/dbio/database/database_adbc.go index 3e0254b36..8dbdecf20 100644 --- a/core/dbio/database/database_adbc.go +++ b/core/dbio/database/database_adbc.go @@ -1366,6 +1366,11 @@ func (conn *ArrowDBConn) BulkImportStream(tableFName string, ds *iop.Datastream) DBSchema: table.Schema, } + // For 2-part targets (schema.table), ParseTableName leaves table.Database empty + if opts.Catalog == "" { + opts.Catalog = conn.GetProp("database") + } + g.Trace("arrow schema => %s", iop.ColumnsToArrowSchema(ds.Columns)) for batch := range ds.BatchChan { diff --git a/core/dbio/database/database_bigquery.go b/core/dbio/database/database_bigquery.go index c4023a2c2..3342848e9 100755 --- a/core/dbio/database/database_bigquery.go +++ b/core/dbio/database/database_bigquery.go @@ -434,6 +434,18 @@ func (conn *BigQueryConn) StreamRowsContext(ctx context.Context, sql string, opt } conn.Data.Columns, bQTC = conn.getItColumns(it.Schema) + // the wkb rewrite makes geometry columns scan as strings; + // keep the geometry type from the fetched table columns + if fetchedColumns, ok := opts["columns"].(iop.Columns); ok { + for i := range conn.Data.Columns { + fc := fetchedColumns.GetColumn(conn.Data.Columns[i].Name) + if fc != nil && fc.Type.IsGeometry() && !conn.Data.Columns[i].Type.IsGeometry() { + conn.Data.Columns[i].Type = fc.Type + conn.Data.Columns[i].Sourced = fc.Sourced + } + } + } + if err == iterator.Done { ds = iop.NewDatastreamContext(queryContext.Ctx, conn.Data.Columns) ds.SetReady() diff --git a/core/dbio/database/database_databricks.go b/core/dbio/database/database_databricks.go index 752b1817e..3a6c15dc0 100644 --- a/core/dbio/database/database_databricks.go +++ b/core/dbio/database/database_databricks.go @@ -26,11 +26,12 @@ import ( // DatabricksConn is a Databricks connection type DatabricksConn struct { BaseConn - URL string - Catalog string - Schema string - Warehouse string - CopyMethod string + URL string + Catalog string + Schema string + Warehouse string + CopyMethod string + TableFormat string } // Init initiates the object @@ -38,6 +39,7 @@ func (conn *DatabricksConn) Init() error { conn.BaseConn.URL = conn.URL conn.BaseConn.Type = dbio.TypeDbDatabricks conn.CopyMethod = "stage" + conn.TableFormat = "delta" instance := Connection(conn) conn.BaseConn.instance = &instance @@ -45,7 +47,11 @@ func (conn *DatabricksConn) Init() error { conn.Schema = conn.GetProp("schema") if m := conn.GetProp("copy_method"); m != "" { - conn.CopyMethod = strings.ToLower(conn.GetProp("copy_method")) + conn.CopyMethod = strings.ToLower(m) + } + + if tf := conn.GetProp("table_format"); tf != "" { + conn.TableFormat = strings.ToLower(tf) } if w := conn.GetProp("warehouse"); w != "" { @@ -618,10 +624,10 @@ func (conn *DatabricksConn) GenerateDDL(table Table, data iop.Dataset, temporary } // Add Databricks-specific DDL modifications - // Databricks uses Delta tables by default - if !temporary && !strings.Contains(strings.ToLower(sql), "using") { + // table format defaults to delta, set table_format=iceberg to use Iceberg + if !strings.Contains(strings.ToLower(sql), "using") { sql = strings.TrimSuffix(strings.TrimSpace(sql), ";") - sql += " USING DELTA" + sql += " USING " + strings.ToUpper(conn.TableFormat) } // Add partitioning if specified diff --git a/core/dbio/database/database_duckdb.go b/core/dbio/database/database_duckdb.go index b4691f204..ca27bbc72 100644 --- a/core/dbio/database/database_duckdb.go +++ b/core/dbio/database/database_duckdb.go @@ -79,8 +79,20 @@ func (conn *DuckDbConn) DuckDb() *iop.DuckDb { } // GenerateDDL builds the CREATE TABLE plus post-CREATE indexes and column -// comments (covers DuckDB and MotherDuck). +// comments (covers DuckDB, DuckLake and MotherDuck). func (conn *DuckDbConn) GenerateDDL(table Table, data iop.Dataset, temporary bool) (string, error) { + // a file-export staging conn (geometry_as_varchar) stores geometry as hex + // WKB varchar; the copy projection parses it at export + if cast.ToBool(conn.GetProp("geometry_as_varchar")) { + data.Columns = append(iop.Columns{}, data.Columns...) + for i := range data.Columns { + if data.Columns[i].Type.IsGeometry() { + data.Columns[i].Type = iop.TextType + data.Columns[i].DbType = "varchar" + } + } + } + ddl, err := conn.BaseConn.GenerateDDL(table, data, temporary) if err != nil { return ddl, g.Error(err) @@ -205,6 +217,59 @@ func (conn *DuckDbConn) InsertStream(tableFName string, ds *iop.Datastream) (cou return conn.BulkImportFlow(tableFName, df) } +// generateImportSelectExprs returns the select list for an import insert. +// A geometry column that the table stores as native geometry is parsed from +// its hex WKB csv encoding. It returns "*" when no column needs parsing. +func (conn *DuckDbConn) generateImportSelectExprs(table *Table, columns iop.Columns) (string, error) { + hasGeometry := false + for _, col := range columns { + if col.Type.IsGeometry() { + hasGeometry = true + break + } + } + if !hasGeometry { + return "*", nil + } + + // information_schema needs the schema name. an unqualified table + // lives in the current schema. + if table.Schema == "" { + data, err := conn.Self().Query("select current_schema() as schema_name") + if err != nil || len(data.Rows) == 0 { + return "", g.Error(err, "could not get the current schema for %s", table.FullName()) + } + table.Schema = cast.ToString(data.Rows[0][0]) + } + + tgtColumns, err := conn.Self().GetColumns(table.FullName()) + if err != nil { + return "", g.Error(err, "could not get columns of %s", table.FullName()) + } + + nativeGeometry := map[string]bool{} // lower name -> table column is geometry + for _, col := range tgtColumns { + if col.Type.IsGeometry() { + nativeGeometry[strings.ToLower(col.Name)] = true + } + } + if len(nativeGeometry) == 0 { + return "*", nil + } + + exprs := make([]string, len(columns)) + for i, col := range columns { + qName := dbio.TypeDbDuckDb.Quote(col.Name) + if col.Type.IsGeometry() && nativeGeometry[strings.ToLower(col.Name)] { + // try() makes malformed hex a null instead of failing the insert + exprs[i] = g.F("try(st_geomfromwkb(unhex(%s))) as %s", qName, qName) + } else { + exprs[i] = qName + } + } + return strings.Join(exprs, ", "), nil +} + func (conn *DuckDbConn) importViaTempCSVs(tableFName string, df *iop.Dataflow) (count uint64, err error) { table, err := ParseTableName(tableFName, conn.GetType()) @@ -235,8 +300,13 @@ func (conn *DuckDbConn) importViaTempCSVs(tableFName string, df *iop.Dataflow) ( return `"` + col + `"` }) + selectExpr, err := conn.generateImportSelectExprs(&table, file.Columns) + if err != nil { + return err + } + sqlLines := []string{ - g.F(`insert into %s (%s) select * from read_csv('%s', delim=',', header=True, columns=%s, max_line_size=%d, parallel=false, quote='"', escape='"', nullstr='\N', auto_detect=false);`, table.FDQN(), strings.Join(columnNames, ", "), file.Node.Path(), conn.generateCsvColumns(file.Columns), conn.duck.MaxLineSize(file.Columns)), + g.F(`insert into %s (%s) select %s from read_csv('%s', delim=',', header=True, columns=%s, max_line_size=%d, parallel=false, quote='"', escape='"', nullstr='\N', auto_detect=false);`, table.FDQN(), strings.Join(columnNames, ", "), selectExpr, file.Node.Path(), conn.generateCsvColumns(file.Columns), conn.duck.MaxLineSize(file.Columns)), } sql := strings.Join(sqlLines, ";\n") @@ -303,9 +373,14 @@ func (conn *DuckDbConn) importViaHTTP(tableFName string, df *iop.Dataflow, forma return `"` + col + `"` }) + selectExpr, err := conn.generateImportSelectExprs(&table, streamPart.Columns) + if err != nil { + return df.Count(), g.Error(err, "could not insert into %s", tableFName) + } + // Generate insert SQL using the fromExpr from streamPart sqlLines := []string{ - g.F(`insert into %s (%s) select * from %s;`, table.FDQN(), strings.Join(columnNames, ", "), streamPart.FromExpr), + g.F(`insert into %s (%s) select %s from %s;`, table.FDQN(), strings.Join(columnNames, ", "), selectExpr, streamPart.FromExpr), } sql := strings.Join(sqlLines, ";\n") @@ -356,6 +431,9 @@ func (conn *DuckDbConn) CastColumnForSelect(srcCol iop.Column, tgtCol iop.Column selectStr = g.F("%s::%s", qName, tgtCol.DbType) case srcCol.Type == iop.TimestampzType && tgtCol.Type != iop.TimestampzType: selectStr = g.F("%s::%s", qName, tgtCol.DbType) + case srcCol.Type.IsGeometry() && !tgtCol.Type.IsGeometry(): + // keep the hex WKB transport form for a non-geometry target column + selectStr = g.F("hex(st_aswkb(%s))", qName) default: selectStr = qName } diff --git a/core/dbio/database/database_redshift.go b/core/dbio/database/database_redshift.go index eab7a5f11..42b0b8e0e 100755 --- a/core/dbio/database/database_redshift.go +++ b/core/dbio/database/database_redshift.go @@ -83,7 +83,6 @@ func loadAWSCredentialsFromChain(conn Connection) error { return nil } - func isRedshift(URL string) (isRs bool) { db, err := sqlx.Open("postgres", URL) if err != nil { @@ -175,12 +174,7 @@ func (conn *RedshiftConn) getS3Props() []string { // redactCredentials masks AWS secrets in a SQL string, for safe logging. func (conn *RedshiftConn) redactCredentials(sql string) string { - for _, key := range []string{"AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN", "AWS_ROLE_ARN"} { - if val := conn.GetProp(key); val != "" { - sql = strings.ReplaceAll(sql, val, "*****") - } - } - return sql + return env.Clean(conn.Props(), sql) } // ensureAWSCredentials ensures AWS credentials are available for Redshift's COPY/UNLOAD diff --git a/core/dbio/database/schemata.go b/core/dbio/database/schemata.go index 901da3749..021dd1766 100644 --- a/core/dbio/database/schemata.go +++ b/core/dbio/database/schemata.go @@ -463,15 +463,17 @@ func (t *Table) Select(Opts ...SelectOptions) (sql string) { } else if len(fields) == 1 && fields[0] == "*" { fieldsStr = "*" } else { + // fields are already normalized: identifiers quoted, + // expressions (casts, function calls) pass through as-is fieldsExprs := []string{} - for _, field := range opts.Fields { - field = strings.TrimSpace(field) - colQ := t.Dialect.Quote(field) - if col := toJsonCols.GetColumn(field); col != nil { - expr := g.F("safe.parse_json(to_json_string(%s)) as %s", colQ, colQ) - fieldsExprs = append(fieldsExprs, expr) + q := GetQualifierQuote(t.Dialect) + for _, field := range fields { + name := strings.Trim(field, q) + if col := toJsonCols.GetColumn(name); col != nil { + colQ := q + name + q + fieldsExprs = append(fieldsExprs, g.F("safe.parse_json(to_json_string(%s)) as %s", colQ, colQ)) } else { - fieldsExprs = append(fieldsExprs, colQ) + fieldsExprs = append(fieldsExprs, field) } } fieldsStr = strings.Join(fieldsExprs, ", ") @@ -537,6 +539,95 @@ func (t *Table) Select(Opts ...SelectOptions) (sql string) { return } +// GeometryWKBFields maps geometry columns of fields to hex WKB expressions. +// MySQL, MariaDB and BigQuery deliver spatial columns unusable through their +// drivers; the expression makes the source return hex-encoded plain WKB. +// DuckDB and DuckLake export WKT through csv, so they use the same transport. +// Returns fields unchanged for other dialects or with no geometry column. +func (t *Table) GeometryWKBFields(fields []string) []string { + var expr func(string) string + switch t.Dialect { + case dbio.TypeDbMySQL, dbio.TypeDbMariaDB, dbio.TypeDbDuckDb, dbio.TypeDbDuckLake: + // st_aswkb drops the SRID and normalizes axis order to lon-lat + expr = func(col string) string { return g.F("hex(st_aswkb(%s))", col) } + case dbio.TypeDbBigQuery: + expr = func(col string) string { return g.F("to_hex(st_asbinary(%s))", col) } + default: + return fields + } + + spatial := map[string]string{} // lower name -> quoted name + for _, col := range t.Columns { + if col.Type.IsGeometry() { + spatial[strings.ToLower(col.Name)] = t.Dialect.Quote(col.Name) + } + } + if len(spatial) == 0 { + return fields + } + + if len(fields) == 0 || (len(fields) == 1 && fields[0] == "*") { + if t.Dialect == dbio.TypeDbBigQuery { + // bigquery can replace columns in place, keeping `select *` + replaceExprs := make([]string, 0, len(spatial)) + for _, col := range t.Columns { + if quoted, ok := spatial[strings.ToLower(col.Name)]; ok { + replaceExprs = append(replaceExprs, g.F("%s as %s", expr(quoted), quoted)) + } + } + return []string{g.F("* replace(%s)", strings.Join(replaceExprs, ", "))} + } + fields = lo.Map(t.Columns, func(col iop.Column, _ int) string { return col.Name }) + } + + mapped := make([]string, 0, len(fields)) + for _, field := range fields { + original, alias, isExclude, _ := iop.ParseSelectExpr(field) + if isExclude { + mapped = append(mapped, field) + continue + } + quoted, ok := spatial[strings.ToLower(t.Dialect.Unquote(original))] + if !ok { + mapped = append(mapped, field) + continue + } + name := quoted + if alias != "" { + name = t.Dialect.Quote(alias) + } + mapped = append(mapped, g.F("%s as %s", expr(quoted), name)) + } + return mapped +} + +// RestoreGeometryTypes restores the geometry type on ds columns that +// GeometryWKBFields rewrote to hex WKB text. The hex expression describes as +// varchar; the true type lets targets parse the WKB into native geometry. +func (t *Table) RestoreGeometryTypes(ds *iop.Datastream) { + switch t.Dialect { + case dbio.TypeDbDuckDb, dbio.TypeDbDuckLake: + default: + return + } + + spatial := map[string]bool{} // lower name -> column is geometry + for _, col := range t.Columns { + if col.Type.IsGeometry() { + spatial[strings.ToLower(col.Name)] = true + } + } + if len(spatial) == 0 { + return + } + + for i, col := range ds.Columns { + if spatial[strings.ToLower(col.Name)] { + ds.Columns[i].Type = iop.GeometryType + } + } +} + // Database represents a schemata database type Database struct { Name string `json:"name"` diff --git a/core/dbio/dbio_types.go b/core/dbio/dbio_types.go index 1fe793d21..7c4e6453a 100644 --- a/core/dbio/dbio_types.go +++ b/core/dbio/dbio_types.go @@ -281,6 +281,11 @@ func (t Type) IsSQLServer() bool { return g.In(t, TypeDbSQLServer, TypeDbAzure, TypeDbAzureDWH, TypeDbFabric) } +// IsSingleWriterDB is true for engines that allow only one writer per file. +func (t Type) IsSingleWriterDB() bool { + return g.In(t, TypeDbDuckDb, TypeDbMotherDuck, TypeDbDuckLake) +} + // NameLong return the type long name func (t Type) NameLong() string { mapping := map[Type]string{ @@ -387,6 +392,17 @@ func (t Type) Name() string { //go:embed templates/* var templatesFolder embed.FS +// TemplatesFS returns the embedded templates filesystem. +func TemplatesFS() embed.FS { + return templatesFolder +} + +// ReadTemplateFile reads a file from the embedded templates folder. +// name is relative to the templates directory, for example "_properties.yaml". +func ReadTemplateFile(name string) ([]byte, error) { + return templatesFolder.ReadFile(path.Join("templates", name)) +} + // Template is a database YAML template type Template struct { Core map[string]string `yaml:"core"` diff --git a/core/dbio/filesys/fs.go b/core/dbio/filesys/fs.go index ee770c59a..519a646b7 100755 --- a/core/dbio/filesys/fs.go +++ b/core/dbio/filesys/fs.go @@ -712,6 +712,11 @@ func (fs *BaseFileSysClient) ReadDataflow(url string, cfg ...iop.FileStreamConfi // duckdb read natively df, err = GetDataflowViaDuckDB(fs.Self(), url, nodes, Cfg) } else { + // match the error of the other paths, so callers can no-op (incremental with no new files) + if len(nodes.Files()) == 0 { + return df, g.Error("Provided 0 files for: %#v", nodes) + } + localRoot := path.Join(env.GetTempFolder(), g.NewTsID("duck.temp")) // copy to local first @@ -1443,6 +1448,7 @@ func WriteDataflowReadyViaDuckDB(fs FileSysClient, df *iop.Dataflow, uri string, Format: fileFormat, Compression: sc.Compression, FileSizeBytes: sc.FileMaxBytes, + GeometryCRS: fs.GetProp("geometry_crs"), Columns: streamPart.Columns, } @@ -1461,6 +1467,16 @@ func WriteDataflowReadyViaDuckDB(fs FileSysClient, df *iop.Dataflow, uri string, localPath = strings.TrimRight(localPath, "/") } + // Trailing slash (dated folders like .../2026/08/23/) is a + // directory target. DuckDB COPY TO a directory fails with + // "Is a directory" unless we write a file inside it. + asDir := strings.HasSuffix(localPath, "/") || strings.HasSuffix(localPath, string(os.PathSeparator)) + if asDir { + localPath = strings.TrimRight(localPath, `/\`) + os.MkdirAll(localPath, 0755) + localPath = g.F("%s/data_%03d.parquet", localPath, streamPart.Index+1) + } + // create the parent folder if needed parent := path.Dir(localPath) if err = os.MkdirAll(parent, 0755); err != nil { diff --git a/core/dbio/filesys/fs_s3.go b/core/dbio/filesys/fs_s3.go index 134fda3f3..8f2e316f5 100644 --- a/core/dbio/filesys/fs_s3.go +++ b/core/dbio/filesys/fs_s3.go @@ -512,7 +512,9 @@ func (fs *S3FileSysClient) Write(uri string, reader io.Reader) (bw int64, err er } svc := s3.NewFromConfig(fs.getConfig()) - uploader := manager.NewUploader(svc) + uploader := manager.NewUploader(svc, func(d *manager.Uploader) { + d.RequestChecksumCalculation = fs.awsConfig.RequestChecksumCalculation + }) uploader.Concurrency = fs.Context().Wg.Limit // Create pipe to get bytes written. diff --git a/core/dbio/iop/datastream.go b/core/dbio/iop/datastream.go index 83ddf6d62..2347897b2 100644 --- a/core/dbio/iop/datastream.go +++ b/core/dbio/iop/datastream.go @@ -92,6 +92,19 @@ type FileStreamConfig struct { Props map[string]string `json:"props"` } +// AddsFilenameColumn tells whether the scan query appends a filename column, +// which becomes _sling_stream_url. A custom sql must ask for it with the +// {filename_expr} placeholder, otherwise its own last column is the last one. +func (sc *FileStreamConfig) AddsFilenameColumn() bool { + if !sc.DuckDBFilename { + return false + } + if sc.SQL != "" { + return strings.Contains(sc.SQL, "{filename_expr}") + } + return true +} + func (sc *FileStreamConfig) ShouldUseDuckDB() bool { if !env.UseDuckDbCompute() { return false @@ -1671,6 +1684,17 @@ func (ds *Datastream) ConsumeCsvReaderChl(readerChn chan *ReaderReady) (err erro } if colMap != nil { + // a stale colMap can target an index past the current schema, widen it + maxCorrectI := -1 + for _, correctI := range colMap { + if correctI > maxCorrectI { + maxCorrectI = correctI + } + } + if maxCorrectI >= len(it.ds.Columns) { + it.addNewColumns(maxCorrectI + 1) + } + // remake row in proper order. row has new structure correctRow := make([]string, len(it.ds.Columns)) for incorrectI, correctI := range colMap { @@ -1684,7 +1708,8 @@ func (ds *Datastream) ConsumeCsvReaderChl(readerChn chan *ReaderReady) (err erro // This would indicate a schema synchronization issue where colMap's target index // is out of bounds for the current datastream schema size (len(it.ds.Columns)). // This case should ideally not be hit if schema updates are perfectly synchronized. - err = g.Error("CSV Schema inconsistency at row #%d (%s): colMap index %d is out of bounds for current schema length %d for column '%s'. Here are all the columns detected: %s", it.Counter+1, ds.Metadata.StreamURL.Value, correctI, len(correctRow), it.ds.Columns.Names()[correctI], g.Marshal(it.ds.Columns.Names())) + // correctI is out of range here, so do not index the schema with it. + err = g.Error("CSV Schema inconsistency at row #%d (%s): colMap index %d is out of bounds for current schema length %d. Here are all the columns detected: %s", it.Counter+1, ds.Metadata.StreamURL.Value, correctI, len(correctRow), g.Marshal(it.ds.Columns.Names())) it.ds.Context.CaptureErr(err) return false } @@ -1981,7 +2006,7 @@ func (ds *Datastream) ConsumeParquetReaderDuckDb(uri string, sc FileStreamConfig sc.DuckDBFilename = ds.Metadata.StreamURL.Key != "" sql := r.MakeQuery(sc) - ds, err = r.Duck.Stream(sql, g.M("datastream", ds, "filename", sc.DuckDBFilename)) + ds, err = r.Duck.Stream(sql, g.M("datastream", ds, "filename", sc.AddsFilenameColumn())) if err != nil { return g.Error(err, "could not read parquet rows") } @@ -2043,7 +2068,7 @@ func (ds *Datastream) ConsumeCsvReaderDuckDb(uri string, sc FileStreamConfig) (e sc.DuckDBFilename = ds.Metadata.StreamURL.Key != "" sql := r.MakeQuery(sc) - ds, err = r.Duck.Stream(sql, g.M("datastream", ds, "filename", sc.DuckDBFilename)) + ds, err = r.Duck.Stream(sql, g.M("datastream", ds, "filename", sc.AddsFilenameColumn())) if err != nil { return g.Error(err, "could not read csv rows") } diff --git a/core/dbio/iop/duckdb.go b/core/dbio/iop/duckdb.go index 142bfb8af..f44b51da6 100644 --- a/core/dbio/iop/duckdb.go +++ b/core/dbio/iop/duckdb.go @@ -487,6 +487,19 @@ func (duck *DuckDb) Open(timeOut ...int) (err error) { return nil } + // another process may still hold the file lock; retry briefly + for attempt := 0; ; attempt++ { + err = duck.openOnce(timeOut...) + if err == nil || attempt >= 4 || !strings.Contains(err.Error(), "Conflicting lock") { + return err + } + g.Debug("duckdb file lock busy, retrying (attempt %d)", attempt+1) + duck.kill() + time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond) + } +} + +func (duck *DuckDb) openOnce(timeOut ...int) (err error) { bin, err := duck.EnsureBinDuckDB(duck.GetProp("duckdb_version")) if err != nil { return g.Error(err, "could not get duckdb binary") @@ -1376,33 +1389,44 @@ type DuckDbCopyOptions struct { PartitionKey string WritePartitionCols bool FileSizeBytes int64 - Columns Columns // optional, used to decode hex-encoded binary back to BLOB + GeometryCRS string // optional, stamps exported geometry columns with this CRS + Columns Columns // optional, used to decode hex-encoded binary and geometry } -// buildSelectProjection returns the SELECT list for export. For binary columns +// buildSelectProjection returns the SELECT list for export. Binary columns // (which are streamed through CSV as hex-encoded varchar), it emits // `unhex(col)::BLOB AS col` so parquet output preserves true binary type. -// If no binary columns are present, returns `*`. +// Geometry columns (hex WKB varchar) are parsed into native geometry so +// parquet output carries GeoParquet metadata. If neither is present, +// returns `*`. func (opts DuckDbCopyOptions) buildSelectProjection() string { if len(opts.Columns) == 0 { return "*" } - hasBinary := false + hasConversion := false for _, c := range opts.Columns { - if c.IsBinary() { - hasBinary = true + if c.IsBinary() || c.Type.IsGeometry() { + hasConversion = true break } } - if !hasBinary { + if !hasConversion { return "*" } parts := make([]string, len(opts.Columns)) for i, c := range opts.Columns { qName := dbio.TypeDbDuckDb.Quote(c.Name) - if c.IsBinary() { + switch { + case c.Type.IsGeometry(): + // try() makes malformed hex a null instead of failing the copy + expr := g.F("try(st_geomfromwkb(unhex(%s)))", qName) + if opts.GeometryCRS != "" { + expr = g.F("st_setcrs(%s, '%s')", expr, strings.ReplaceAll(opts.GeometryCRS, "'", "''")) + } + parts[i] = g.F("%s AS %s", expr, qName) + case c.IsBinary(): parts[i] = g.F("unhex(%s)::BLOB AS %s", qName, qName) - } else { + default: parts[i] = qName } } @@ -1425,26 +1449,27 @@ func (duck *DuckDb) GenerateCopyStatement(fromTable, toLocalPath string, options return "", g.Error("missing partition key") } + keyExpr := duck.partitionKeyTimestampExpr(options.PartitionKey, options.Columns) pe := partExpression{ alias: g.F("%s_%s", dbio.TypeDbDuckDb.Unquote(options.PartitionKey), pl), - expression: g.F("date_part('%s', %s)", pl, options.PartitionKey), + expression: g.F("date_part('%s', %s)", pl, keyExpr), } switch pl { case PartitionLevelYear: - pe.expression = g.F("strftime(%s, '%s')", options.PartitionKey, "%Y") + pe.expression = g.F("strftime(%s, '%s')", keyExpr, "%Y") case PartitionLevelYearMonth: - pe.expression = g.F("strftime(%s, '%s')", options.PartitionKey, "%Y-%m") + pe.expression = g.F("strftime(%s, '%s')", keyExpr, "%Y-%m") case PartitionLevelMonth: - pe.expression = g.F("strftime(%s, '%s')", options.PartitionKey, "%m") + pe.expression = g.F("strftime(%s, '%s')", keyExpr, "%m") case PartitionLevelWeek: - pe.expression = g.F("strftime(%s, '%s')", options.PartitionKey, "%V") + pe.expression = g.F("strftime(%s, '%s')", keyExpr, "%V") case PartitionLevelDay: - pe.expression = g.F("strftime(%s, '%s')", options.PartitionKey, "%d") + pe.expression = g.F("strftime(%s, '%s')", keyExpr, "%d") case PartitionLevelHour: - pe.expression = g.F("strftime(%s, '%s')", options.PartitionKey, "%H") + pe.expression = g.F("strftime(%s, '%s')", keyExpr, "%H") case PartitionLevelMinute: - pe.expression = g.F("strftime(%s, '%s')", options.PartitionKey, "%M") + pe.expression = g.F("strftime(%s, '%s')", keyExpr, "%M") default: return sql, g.Error("invalid partition field: %s", pl) } @@ -1474,6 +1499,16 @@ func (duck *DuckDb) GenerateCopyStatement(fromTable, toLocalPath string, options selectExpr := "*" if options.Format == dbio.FileTypeParquet { selectExpr = options.buildSelectProjection() + if options.GeometryCRS != "" { + for _, c := range options.Columns { + if c.Type.IsGeometry() { + // st_setcrs resolves a crs string to projjson only when + // spatial is loaded explicitly, not through autoload + duck.AddExtension("spatial") + break + } + } + } } if len(partExpressions) > 0 { @@ -1514,6 +1549,30 @@ func (duck *DuckDb) GenerateCopyStatement(fromTable, toLocalPath string, options return } +// partitionKeyTimestampExpr wraps epoch-integer partition keys so DuckDB +// strftime/date_part receive a TIMESTAMP. _sling_loaded_at is Unix seconds +// by default (SLING_LOADED_AT_COLUMN=timestamp stores a real timestamptz). +func (duck *DuckDb) partitionKeyTimestampExpr(key string, cols Columns) string { + + partitionKeyIsEpoch := func(name string, cols Columns) bool { + if strings.EqualFold(name, "_sling_loaded_at") { + return true + } + for i := range cols { + if strings.EqualFold(cols[i].Name, name) { + return cols[i].IsInteger() + } + } + return false + } + + name := strings.Trim(key, `"`) + if partitionKeyIsEpoch(name, cols) { + return g.F("to_timestamp(%s)", key) + } + return key +} + // Quote quotes a column name func (duck *DuckDb) Quote(col string) (qName string) { qName = `"` + col + `"` @@ -2057,6 +2116,12 @@ func (duck *DuckDb) GenerateCsvColumns(columns Columns) (colStr string) { nativeType = "varchar" } + // geometry rides the csv as hex WKB varchar; st_geomfromwkb parses it + // in the select that reads this csv + if nativeType == "geometry" { + nativeType = "varchar" + } + colsArr[i] = g.F("'%s':'%s'", col.Name, nativeType) } @@ -2094,7 +2159,7 @@ func (duck *DuckDb) MakeScanQuery(format dbio.FileType, uri string, fsc FileStre } duckdbFilenameStr := "" - if fsc.DuckDBFilename { + if fsc.AddsFilenameColumn() { duckdbFilenameStr = g.F(", filename = true") } diff --git a/core/dbio/iop/duckdb_test.go b/core/dbio/iop/duckdb_test.go index cf61c750f..e0d7fa979 100644 --- a/core/dbio/iop/duckdb_test.go +++ b/core/dbio/iop/duckdb_test.go @@ -638,6 +638,25 @@ func TestDuckDbMaxLineSize(t *testing.T) { }) } +func TestGenerateCopyStatementEpochPartitionKey(t *testing.T) { + duck := NewDuckDb(context.Background()) + cols := NewColumnsFromFields("id", "_sling_loaded_at") + cols[0].Type = IntegerType + cols[1].Type = IntegerType + sql, err := duck.GenerateCopyStatement("main.t", "/tmp/out", DuckDbCopyOptions{ + Format: dbio.FileTypeParquet, + PartitionFields: []PartitionLevel{PartitionLevelYearMonth, PartitionLevelDay}, + PartitionKey: "_sling_loaded_at", + Columns: cols, + }) + if !assert.NoError(t, err) { + return + } + assert.Contains(t, sql, "to_timestamp(_sling_loaded_at)") + assert.Contains(t, sql, "strftime(to_timestamp(_sling_loaded_at), '%Y-%m')") + assert.NotContains(t, sql, "strftime(_sling_loaded_at,") +} + // regression guard for the v1.5.25 OOM: DuckDB sizes its read_csv buffer as // 16 × max_line_size and allocates it eagerly. The 256MB raise thus demands a // 4 GiB block, which fails on hosts with memory_limit below ~4 GiB. The bridge diff --git a/core/dbio/iop/sheet.go b/core/dbio/iop/sheet.go index cc8425e47..fb2493d9d 100644 --- a/core/dbio/iop/sheet.go +++ b/core/dbio/iop/sheet.go @@ -3,6 +3,7 @@ package iop import ( "strings" + "github.com/flarco/g" "github.com/spf13/cast" ) @@ -60,6 +61,8 @@ func (s *spreadsheet) makeDatasetAuto(rows [][]string) (data Dataset) { } } + s.widenColumns(&data, len(row0)) + row := make([]interface{}, len(row0)) for i, val := range row0 { row[i] = val @@ -92,6 +95,8 @@ func (s *spreadsheet) makeDatasetStr(rangeRows [][]string) (data Dataset) { continue } + s.widenColumns(&data, len(row0)) + row := make([]interface{}, len(row0)) for i, val := range row0 { row[i] = val @@ -129,6 +134,7 @@ func (s *spreadsheet) makeDatasetInterf(rangeRows [][]interface{}) (data Dataset continue } + s.widenColumns(&data, len(row)) data.Append(row) if i == SampleSize { @@ -147,3 +153,15 @@ func (s *spreadsheet) makeDatasetInterf(rangeRows [][]interface{}) (data Dataset } return } + +// widenColumns adds placeholder columns when a data row is wider than the +// header row, so casting the row does not index the schema out of range. +func (s *spreadsheet) widenColumns(data *Dataset, rowLen int) { + for len(data.Columns) < rowLen { + data.Columns = append(data.Columns, Column{ + Name: g.F("col_%d", len(data.Columns)+1), + Type: StringType, + Position: len(data.Columns) + 1, + }) + } +} diff --git a/core/dbio/iop/sheet_excel.go b/core/dbio/iop/sheet_excel.go index ac5d5528d..1ee37f3c8 100644 --- a/core/dbio/iop/sheet_excel.go +++ b/core/dbio/iop/sheet_excel.go @@ -181,16 +181,35 @@ func (xls *Excel) GetDatasetFromRange(sheet, cellRange string) (data Dataset, er rowEnd = len(allRows) - 1 } - if len(allRows) < rowEnd { + // a range can end past the last row with data (`A2:C1000` over a short + // sheet). Clamp to the data, so no empty rows are added to the dataset. + if rowEnd > len(allRows)-1 { + rowEnd = len(allRows) - 1 + } + + // the start row must hold data. rowEnd is clamped above, so an empty range + // here means the range begins past the last row. + if rowStart > rowEnd { err = g.Error( "Input row range is larger than file row range: %d < %d", - len(allRows), rowEnd, + len(allRows), rowStart+1, ) return - } else if len(allRows[0]) < colEnd { + } + + // the widest row in the range sets the file col range. The first row can be + // narrower than the rows below it (a comment line above a header, say). + maxCol := 0 + for r := rowStart; r <= rowEnd; r++ { + if len(allRows[r]) > maxCol { + maxCol = len(allRows[r]) + } + } + + if maxCol < colEnd { err = g.Error( "Input col range is larger than file col range: %d < %d", - len(allRows[0]), colEnd, + maxCol, colEnd, ) return } @@ -198,15 +217,14 @@ func (xls *Excel) GetDatasetFromRange(sheet, cellRange string) (data Dataset, er i := 0 rangeRows := make([][]string, rowEnd-rowStart+1) for r := rowStart; r <= rowEnd; r++ { - row0 := []string{} - if r >= len(allRows) { - continue - } + // keep every row the full width of the range, so short rows pad with + // blanks instead of shifting the columns to their left + row0 := make([]string, colEnd-colStart+1) for c := colStart; c <= colEnd; c++ { if c >= len(allRows[r]) { continue } - row0 = append(row0, strings.TrimSpace(allRows[r][c])) + row0[c-colStart] = strings.TrimSpace(allRows[r][c]) } rangeRows[i] = row0 i++ diff --git a/core/dbio/iop/stream_processor.go b/core/dbio/iop/stream_processor.go index 00d22180a..8172c875c 100644 --- a/core/dbio/iop/stream_processor.go +++ b/core/dbio/iop/stream_processor.go @@ -49,38 +49,38 @@ type StreamProcessor struct { } type StreamConfig struct { - EmptyAsNull bool `json:"empty_as_null"` - Header bool `json:"header"` - Compression CompressorType `json:"compression"` // AUTO | ZIP | GZIP | SNAPPY | NONE - NullIf string `json:"null_if"` - NullAs string `json:"null_as"` - DatetimeFormat string `json:"datetime_format"` - SkipBlankLines bool `json:"skip_blank_lines"` - SkipLines int `json:"skip_lines"` - Format dbio.FileType `json:"format"` - Delimiter string `json:"delimiter"` - Escape string `json:"escape"` - Quote string `json:"quote"` - FileMaxRows int64 `json:"file_max_rows"` - FileMaxBytes int64 `json:"file_max_bytes"` - BatchLimit int64 `json:"batch_limit"` - BatchMaxDuration time.Duration `json:"batch_max_duration"` - MaxDecimals int `json:"max_decimals"` - Flatten int `json:"flatten"` - FieldsPerRec int `json:"fields_per_rec"` - Jmespath string `json:"jmespath"` - Jq string `json:"jq"` - Select []string `json:"select"` // applied to JSON-family streams - Sheet string `json:"sheet"` - ColumnCasing ColumnCasing `json:"column_casing"` - ColumnTyping ColumnTyping `json:"column_typing"` - TargetType dbio.Type `json:"target_type"` - DeleteFile bool `json:"delete"` // whether to delete before writing - BoolAsInt bool `json:"-"` - EscapeBackslash bool `json:"-"` // escape backslashes for MySQL LOAD DATA - BinaryAsHex bool `json:"-"` // hex-encode binary in CSV - Columns Columns `json:"columns"` // list of column types. Can be partial list! likely is! - Transforms Transform + EmptyAsNull bool `json:"empty_as_null"` + Header bool `json:"header"` + Compression CompressorType `json:"compression"` // AUTO | ZIP | GZIP | SNAPPY | NONE + NullIf string `json:"null_if"` + NullAs string `json:"null_as"` + DatetimeFormat string `json:"datetime_format"` + SkipBlankLines bool `json:"skip_blank_lines"` + SkipLines int `json:"skip_lines"` + Format dbio.FileType `json:"format"` + Delimiter string `json:"delimiter"` + Escape string `json:"escape"` + Quote string `json:"quote"` + FileMaxRows int64 `json:"file_max_rows"` + FileMaxBytes int64 `json:"file_max_bytes"` + BatchLimit int64 `json:"batch_limit"` + BatchMaxDuration time.Duration `json:"batch_max_duration"` + MaxDecimals int `json:"max_decimals"` + Flatten int `json:"flatten"` + FieldsPerRec int `json:"fields_per_rec"` + Jmespath string `json:"jmespath"` + Jq string `json:"jq"` + Select []string `json:"select"` // applied to JSON-family streams + Sheet string `json:"sheet"` + ColumnCasing ColumnCasing `json:"column_casing"` + ColumnTyping ColumnTyping `json:"column_typing"` + TargetType dbio.Type `json:"target_type"` + DeleteFile bool `json:"delete"` // whether to delete before writing + BoolAsInt bool `json:"-"` + EscapeBackslash bool `json:"-"` // escape backslashes for MySQL LOAD DATA + BinaryAsHex bool `json:"-"` // hex-encode binary in CSV + Columns Columns `json:"columns"` // list of column types. Can be partial list! likely is! + Transforms Transform Map map[string]string `json:"-"` } @@ -1619,6 +1619,9 @@ func (sp *StreamProcessor) CastRow(row []any, columns Columns) []any { sp.rowBlankValCnt = 0 sp.rowChecksum = make([]uint64, len(row)) for i, val := range row { + if i >= len(columns) { + break // row is wider than the schema, caller must widen it + } col := &columns[i] row[i] = sp.CastVal(i, val, col) // g.Warn("%d | col %s | nVal => %#v", sp.N, col.Name, row[i]) @@ -1639,6 +1642,10 @@ func (sp *StreamProcessor) CastRow(row []any, columns Columns) []any { } } + if len(row) > len(columns) { + row = row[:len(columns)] + } + for len(row) < len(columns) { row = append(row, nil) } diff --git a/core/dbio/iop/transforms.go b/core/dbio/iop/transforms.go index 9d05ec3e5..eec6ebea8 100644 --- a/core/dbio/iop/transforms.go +++ b/core/dbio/iop/transforms.go @@ -756,27 +756,37 @@ func (e *Evaluator) ExtractVars(expr string) []string { } // First, we need to identify string literals to exclude them - // Track positions of string literals + // Track positions of string literals (double- and single-quoted) inString := false + stringQuote := rune(0) stringRanges := make([][]int, 0) var start int - for i, char := range expr { - if char == '"' { - // Check if the quote is escaped - if i > 0 && expr[i-1] == '\\' { - continue - } + runes := []rune(expr) + for i, char := range runes { + if char != '"' && char != '\'' { + continue + } + if inString && char != stringQuote { + continue + } + // Check if the quote is escaped + if i > 0 && runes[i-1] == '\\' { + continue + } + // SQL-style doubled quote inside a matching literal ('' or "") + if inString && i+1 < len(runes) && runes[i+1] == char { + continue + } - if !inString { - // Start of a string - inString = true - start = i - } else { - // End of a string - inString = false - stringRanges = append(stringRanges, []int{start, i}) - } + if !inString { + inString = true + stringQuote = char + start = i + } else { + inString = false + stringQuote = 0 + stringRanges = append(stringRanges, []int{start, i}) } } @@ -916,6 +926,22 @@ func (e *Evaluator) FillMissingKeys(stateMap map[string]any, varsToCheck []strin return stateMap } +// methodCallRegex matches JS/Python method-call syntax, for example +// `name.split('.')`. Sling expressions only support function calls. +var methodCallRegex = regexp.MustCompile(`\.\w+\(`) + +// methodCallHint appends guidance when a render error comes from +// method-call syntax on a string value. +func (e *Evaluator) methodCallHint(expr string, err error) string { + if err == nil || !strings.Contains(err.Error(), "cannot access fields on type string") { + return "" + } + if methodCallRegex.MatchString(expr) { + return " (method calls are not supported; use function syntax, e.g. split_part(value, \".\", 0))" + } + return "" +} + func (e *Evaluator) RenderString(val any, extras ...map[string]any) (newVal string, err error) { output, err := e.RenderAny(val, extras...) if err != nil { @@ -1122,7 +1148,9 @@ func (e *Evaluator) RenderAny(input any, extras ...map[string]any) (output any, key := "{" + expr + "}" // If jmespath failed or if we detected evaluation operators/functions, use goval if callsFuncOrEvals || err != nil || e.KeepMissingExpr { - value, err = e.Eval.Evaluate(expr, stateMap, GlobalFunctionMap) + // Rewrite SQL-style '…' literals to goval double-quoted strings + evalExpr := e.rewriteSingleQuotedStrings(expr) + value, err = e.Eval.Evaluate(evalExpr, stateMap, GlobalFunctionMap) if err != nil { // check if jmespath rendered if jpValue != nil && validJmesPath { @@ -1142,7 +1170,7 @@ func (e *Evaluator) RenderAny(input any, extras ...map[string]any) (output any, if errChk := e.Check(expr); errChk != nil { return "", g.Error(errChk, "invalid expression: %s", expr) } - return "", g.Error(err, "could not render expression: %s", expr) + return "", g.Error(err, "could not render expression: %s%s", expr, e.methodCallHint(expr, err)) } } } else { @@ -1227,40 +1255,53 @@ func (e *Evaluator) RenderPayload(val any, extras ...map[string]any) (newVal any } func (e *Evaluator) Check(expr string) (err error) { - inDouble := false + inQuote := rune(0) parenCount := 0 runes := []rune(expr) + escaped := false for i, c := range runes { - if c == '\'' && !inDouble { - return g.Error("cannot use single quotes (') for strings in expression, use double quotes (\"): %s", expr) - } else if c == '"' { - // Check if this quote is escaped by counting preceding backslashes - backslashCount := 0 - for j := i - 1; j >= 0 && runes[j] == '\\'; j-- { - backslashCount++ - } - // If even number of backslashes (including 0), the quote is not escaped - if backslashCount%2 == 0 { - inDouble = !inDouble - } - } else if !inDouble { - // Only track parentheses when not inside double quotes - switch c { - case '(': - parenCount++ - case ')': - parenCount-- - if parenCount < 0 { - return g.Error("unmatched closing parenthesis ')' in expression: %s", expr) + if escaped { + escaped = false + continue + } + if c == '\\' { + // Markdown / source-escaped quote (\"…\") is not a string opener. + escaped = true + continue + } + if inQuote != 0 { + if c == inQuote { + // SQL-style doubled quote stays inside the literal + if i+1 < len(runes) && runes[i+1] == inQuote { + escaped = true + continue } + inQuote = 0 + } + continue + } + if c == '"' || c == '\'' { + inQuote = c + continue + } + switch c { + case '(': + parenCount++ + case ')': + parenCount-- + if parenCount < 0 { + return g.Error("unmatched closing parenthesis ')' in expression: %s", expr) } } } - if inDouble { + if inQuote == '"' { return g.Error("unclosed double quote in expression: %s", expr) } + if inQuote == '\'' { + return g.Error("unclosed single quote in expression: %s", expr) + } if parenCount > 0 { return g.Error("unclosed parenthesis '(' in expression: %s", expr) @@ -1269,6 +1310,73 @@ func (e *Evaluator) Check(expr string) (err error) { return nil } +// rewriteSingleQuotedStrings converts SQL-style '…' literals to goval +// double-quoted strings so expressions like date_format(now(), '%Y-%m-%d') +// evaluate. Doubled single quotes (” ) become a literal apostrophe. +// Content already inside "…" is left alone. +func (e *Evaluator) rewriteSingleQuotedStrings(expr string) string { + runes := []rune(expr) + var b strings.Builder + b.Grow(len(expr) + 8) + inDouble := false + i := 0 + for i < len(runes) { + c := runes[i] + if inDouble { + b.WriteRune(c) + if c == '\\' && i+1 < len(runes) { + b.WriteRune(runes[i+1]) + i += 2 + continue + } + if c == '"' { + inDouble = false + } + i++ + continue + } + if c == '"' { + inDouble = true + b.WriteRune(c) + i++ + continue + } + if c == '\'' { + b.WriteByte('"') + i++ + for i < len(runes) { + if runes[i] == '\'' { + if i+1 < len(runes) && runes[i+1] == '\'' { + b.WriteByte('\'') + i += 2 + continue + } + b.WriteByte('"') + i++ + break + } + if runes[i] == '\\' && i+1 < len(runes) { + b.WriteRune(runes[i]) + b.WriteRune(runes[i+1]) + i += 2 + continue + } + if runes[i] == '"' { + b.WriteString(`\"`) + i++ + continue + } + b.WriteRune(runes[i]) + i++ + } + continue + } + b.WriteRune(c) + i++ + } + return b.String() +} + // FindMatches parses the input string and extracts expressions within curly braces, // properly handling nested brackets and quoted strings. // Returns an error if brackets are unbalanced. @@ -1318,7 +1426,7 @@ func (e *Evaluator) FindMatches(inputStr string) (expressions []string, err erro start := i depth := 1 i++ - inDoubleQuote := false + inQuote := rune(0) hasNestedOpen := false // Track if expression contains nested { outside quotes hasNestedClose := false // Track if expression contains nested } outside quotes @@ -1331,13 +1439,17 @@ func (e *Evaluator) FindMatches(inputStr string) (expressions []string, err erro continue } - // Track quote state - if c == '"' { - inDoubleQuote = !inDoubleQuote + // Track quote state (double or single) + if inQuote != 0 { + if c == inQuote { + inQuote = 0 + } + } else if c == '"' || c == '\'' { + inQuote = c } // Only count brackets outside of quotes - if !inDoubleQuote { + if inQuote == 0 { if c == '{' { depth++ hasNestedOpen = true diff --git a/core/dbio/iop/transforms_test.go b/core/dbio/iop/transforms_test.go index 2428c796d..c19a8a28b 100755 --- a/core/dbio/iop/transforms_test.go +++ b/core/dbio/iop/transforms_test.go @@ -426,6 +426,12 @@ func TestEvaluator(t *testing.T) { "state": map[string]any{"name": "Alice"}, }, }, + { + name: "single_quoted_string_literal", + input: `{ coalesce(nil, '%Y-%m-%d') }`, + expected: "%Y-%m-%d", + state: map[string]any{}, + }, // Edge cases { @@ -1356,22 +1362,19 @@ func TestEvaluatorCheckExpression(t *testing.T) { expectError: false, }, { - name: "single_quote_error", + name: "single_quote_ok", expression: `state.name == 'John'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "single_quote_in_middle", expression: `state.name == "John" && state.title == 'Mr'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "single_quote_at_beginning", expression: `'test' == state.value`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "apostrophe_outside_double_quotes", @@ -1381,8 +1384,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { { name: "apostrophe_and_single_quote_mix", expression: `state.name == "John's car" && state.other == 'test'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "escaped_double_quote", @@ -1397,8 +1399,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { { name: "single_quote_after_escaped_double_quote", expression: `state.text == "He said \"hello\"" && state.bad == 'world'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "complex_valid_expression", @@ -1406,10 +1407,9 @@ func TestEvaluatorCheckExpression(t *testing.T) { expectError: false, }, { - name: "backslash_before_single_quote_still_error", + name: "backslash_before_single_quote_is_escaped", expression: `state.test == "valid" && state.invalid == \'bad\'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "double_backslash_before_double_quote", @@ -1430,8 +1430,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { { name: "only_single_quotes", expression: `'hello world'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "only_double_quotes", @@ -1441,8 +1440,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { { name: "mixed_quotes_complex", expression: `state.a == "test" && state.b == 'invalid' && state.c == "valid"`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "unicode_characters_with_double_quotes", @@ -1452,8 +1450,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { { name: "unicode_characters_with_single_quotes", expression: `state.emoji == 'Hello 👋 world'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "json_like_string", @@ -1468,8 +1465,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { { name: "regex_like_pattern_single_quotes", expression: `state.pattern == '^[a-zA-Z0-9]+$'`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "empty_string_double_quotes", @@ -1479,8 +1475,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { { name: "empty_string_single_quotes", expression: `state.value == ''`, - expectError: true, - errorMsg: "cannot use single quotes", + expectError: false, }, { name: "multiple_consecutive_escapes", @@ -1496,7 +1491,7 @@ func TestEvaluatorCheckExpression(t *testing.T) { name: "single_quote_at_very_end", expression: `someexpression'`, expectError: true, - errorMsg: "cannot use single quotes", + errorMsg: "unclosed single quote", }, { name: "double_quote_at_very_end", @@ -2500,8 +2495,8 @@ func TestEvaluatorFindMatches(t *testing.T) { expected: []string{`"{\"key\": \"" + state.value + "\"}"`}, }, { - name: "mixed_quoted_and_unquoted_braces", - input: `{ repository(owner: "{state.owner}") { name } }`, + name: "mixed_quoted_and_unquoted_braces", + input: `{ repository(owner: "{state.owner}") { name } }`, // The outer has unquoted nested braces, so recurse. Inner "{state.owner}" has braces in quotes - ok. // { name } has no nested braces inside it. expected: []string{"state.owner", " name "}, @@ -2736,3 +2731,23 @@ func TestEvaluatorRenderJmespathJq(t *testing.T) { }) } } + +func TestRenderStringMethodCallHint(t *testing.T) { + eval := NewEvaluator(g.ArrStr("loop"), g.M("loop", g.M("value", g.M("name", "orders.csv")))) + _, err := eval.RenderString("main.{loop.value.name.split('.')}") + if err == nil { + t.Fatal("expected render error for method-call syntax") + } + if !strings.Contains(err.Error(), "method calls are not supported") { + t.Fatalf("missing hint: %s", err.Error()) + } + + // correct function syntax renders fine + got, err := eval.RenderString(`main.{split_part(loop.value.name, ".", 0)}`) + if err != nil { + t.Fatal(err) + } + if got != "main.orders" { + t.Fatalf("got %q", got) + } +} diff --git a/core/dbio/templates/_properties.yaml b/core/dbio/templates/_properties.yaml index dcca216b5..99b67c719 100644 --- a/core/dbio/templates/_properties.yaml +++ b/core/dbio/templates/_properties.yaml @@ -154,7 +154,7 @@ redshift: secret: true postgres: - title: 'Postges' + title: 'Postgres' kind: database required: ["host", "port", "database", "username"] url_template: 'postgresql://{username}:{password}@{host}:{port}/{database}?sslmode={sslmode}' diff --git a/core/dbio/templates/bigquery.yaml b/core/dbio/templates/bigquery.yaml index ac10c2adc..7e733fb01 100755 --- a/core/dbio/templates/bigquery.yaml +++ b/core/dbio/templates/bigquery.yaml @@ -575,7 +575,7 @@ native_type_map: datetime: datetime float: float float64: float - geography: string + geography: geometry int64: bigint integer: bigint json: json diff --git a/core/dbio/templates/connections.json b/core/dbio/templates/connections.json index c9f03163d..cd28d77a1 100644 --- a/core/dbio/templates/connections.json +++ b/core/dbio/templates/connections.json @@ -928,6 +928,12 @@ "schema": { "type": "string", "description": "Schema name" + }, + "table_format": { + "type": "string", + "description": "Table format to use when creating tables", + "enum": ["delta", "iceberg"], + "default": "delta" } } }, diff --git a/core/dbio/templates/databricks.yaml b/core/dbio/templates/databricks.yaml index f858e19fc..5752228fd 100644 --- a/core/dbio/templates/databricks.yaml +++ b/core/dbio/templates/databricks.yaml @@ -5,7 +5,6 @@ core: drop_index: "select 'indexes do not apply for databricks'" create_table: | create table {table} ({col_types}) - using delta # create_temporary_table: | # create table {table} ({col_types}) # using delta diff --git a/core/dbio/templates/duckdb.yaml b/core/dbio/templates/duckdb.yaml index 057392106..375bd82cf 100755 --- a/core/dbio/templates/duckdb.yaml +++ b/core/dbio/templates/duckdb.yaml @@ -344,6 +344,7 @@ native_type_map: double: float enum: string float: float + geometry: geometry hugeint: bigint integer: integer interval: string @@ -373,6 +374,7 @@ general_type_map: datetime: datetime decimal: "decimal(,)" float: double + geometry: geometry integer: integer json: json smallint: smallint diff --git a/core/dbio/templates/ducklake.yaml b/core/dbio/templates/ducklake.yaml index c9d234863..fe8e7cb05 100644 --- a/core/dbio/templates/ducklake.yaml +++ b/core/dbio/templates/ducklake.yaml @@ -313,6 +313,7 @@ native_type_map: blob: binary boolean: bool char: string + geometry: geometry date: date datetime: datetime decimal: decimal @@ -360,6 +361,7 @@ general_type_map: bool: bool date: date datetime: datetime + geometry: geometry decimal: "decimal(,)" float: double integer: integer diff --git a/core/dbio/templates/mariadb.yaml b/core/dbio/templates/mariadb.yaml index 8d11ea45e..f1db94fe5 100644 --- a/core/dbio/templates/mariadb.yaml +++ b/core/dbio/templates/mariadb.yaml @@ -557,22 +557,22 @@ native_type_map: double: float enum: string float: float - geomcollection: string - geometry: string - geometrycollection: string + geomcollection: geometry + geometry: geometry + geometrycollection: geometry int: integer json: json - linestring: string + linestring: geometry longblob: binary longtext: text mediumblob: binary mediumint: integer mediumtext: text - multilinestring: string - multipoint: string - multipolygon: string - point: string - polygon: string + multilinestring: geometry + multipoint: geometry + multipolygon: geometry + point: geometry + polygon: geometry set: string smallint: smallint text: text diff --git a/core/dbio/templates/mysql.yaml b/core/dbio/templates/mysql.yaml index 767f9b561..f6fcf55dd 100755 --- a/core/dbio/templates/mysql.yaml +++ b/core/dbio/templates/mysql.yaml @@ -560,22 +560,22 @@ native_type_map: double: float enum: string float: float - geomcollection: string - geometry: string - geometrycollection: string + geomcollection: geometry + geometry: geometry + geometrycollection: geometry int: integer json: json - linestring: string + linestring: geometry longblob: binary longtext: text mediumblob: binary mediumint: integer mediumtext: text - multilinestring: string - multipoint: string - multipolygon: string - point: string - polygon: string + multilinestring: geometry + multipoint: geometry + multipolygon: geometry + point: geometry + polygon: geometry set: string smallint: smallint text: text diff --git a/core/env/clean_test.go b/core/env/clean_test.go new file mode 100644 index 000000000..2070ae7b2 --- /dev/null +++ b/core/env/clean_test.go @@ -0,0 +1,156 @@ +package env + +import ( + "os" + "regexp" + "strings" + "testing" +) + +func TestCleanRedactsCreateStageCredentials(t *testing.T) { + secret := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" + props := map[string]string{ + "AWS_SECRET_ACCESS_KEY": secret, + } + line := "CREATE STAGE s CREDENTIALS=(AWS_SECRET_KEY='" + secret + "')" + got := Clean(props, line) + if strings.Contains(got, secret) { + t.Fatalf("CREATE CREDENTIALS not redacted: %q", got) + } + if !strings.Contains(got, "***") { + t.Fatalf("expected *** in %q", got) + } +} + +func TestCleanRedactsPasswordWithoutRegistry(t *testing.T) { + // exasol has no _properties.yaml entry; password still redacts. + secret := "exasol-super-secret" + props := map[string]string{ + "type": "exasol", + "password": secret, + } + line := "CREATE USER foo IDENTIFIED BY '" + secret + "'" + got := Clean(props, line) + if strings.Contains(got, secret) { + t.Fatalf("floor password not redacted: %q", got) + } + if !strings.Contains(got, "***") { + t.Fatalf("expected *** in %q", got) + } +} + +func TestCleanRedactsNonFloorSecretKey(t *testing.T) { + jsonSecret := `{"private_key":"not-a-real-key"}` + got := Clean(map[string]string{ + "gcp_credentials_json": jsonSecret, + "host": "db.example.com", + }, "create with "+jsonSecret+" host=db.example.com") + if strings.Contains(got, "not-a-real-key") { + t.Fatalf("registered secret key not redacted: %q", got) + } + if !strings.Contains(got, "db.example.com") { + t.Fatalf("non-secret host was redacted: %q", got) + } +} + +func TestCleanSkipsEmptySecretValues(t *testing.T) { + line := "CREATE TABLE t (id int)" + got := Clean(map[string]string{"password": ""}, line) + if got != line { + t.Fatalf("empty password mangled SQL: %q", got) + } +} + +func TestCleanConnDataRedactsNestedSecretNotInRegistry(t *testing.T) { + secret := "nested-api-key-DISTINCTIVE" + line := "Authorization: Bearer " + secret + got := CleanConnData(map[string]any{ + "type": "api", + "spec": "stripe", + "secrets": map[string]any{ + "api_key": secret, + }, + }, line) + if strings.Contains(got, secret) { + t.Fatalf("nested api_key leaked: %q", got) + } + if !strings.Contains(got, "***") { + t.Fatalf("expected *** in %q", got) + } +} + +func TestCleanConnDataKeepsShortSecretValues(t *testing.T) { + line := "Sling Replication | aws_s3 -> postgres | test/parquet/test1.parquet" + got := CleanConnData(map[string]any{ + "type": "postgres", + "password": "postgres", + }, line) + if got != line { + t.Fatalf("short password redacted an unrelated line: %q", got) + } +} + +func TestCleanConnDataKeepsShortNestedSecrets(t *testing.T) { + line := "Sling CLI | https://slingdata.io" + got := CleanConnData(map[string]any{ + "type": "api", + "secrets": map[string]any{ + "subdomain": "slingdata", + }, + }, line) + if got != line { + t.Fatalf("short nested secret redacted an unrelated line: %q", got) + } +} + +func TestCleanConnDataRedactsLongSecret(t *testing.T) { + secret := "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" + got := CleanConnData(map[string]any{ + "type": "s3", + "secret_access_key": secret, + }, "using key "+secret) + if strings.Contains(got, secret) { + t.Fatalf("long secret not redacted: %q", got) + } +} + +// TestSecretKeysCoverTemplates fails when a connector adds `secret: true` to +// core/dbio/templates/_properties.yaml but not to SecretKeys. +func TestSecretKeysCoverTemplates(t *testing.T) { + data, err := os.ReadFile("../dbio/templates/_properties.yaml") + if err != nil { + t.Skipf("templates not readable: %v", err) + } + + have := map[string]struct{}{} + for _, k := range SecretKeys { + have[strings.ToLower(k)] = struct{}{} + } + + // A property name is the last `key:` line at or above a `secret: true`. + keyRe := regexp.MustCompile(`^\s+([a-z0-9_]+):\s*$`) + lines := strings.Split(string(data), "\n") + var missing []string + for i, ln := range lines { + if strings.TrimSpace(ln) != "secret: true" { + continue + } + for j := i - 1; j >= 0; j-- { + m := keyRe.FindStringSubmatch(lines[j]) + if m == nil { + continue + } + name := m[1] + if name == "properties" { + break + } + if _, ok := have[name]; !ok { + missing = append(missing, name) + } + break + } + } + if len(missing) > 0 { + t.Fatalf("YAML secret keys absent from SecretKeys: %v", missing) + } +} diff --git a/core/env/env.go b/core/env/env.go index 032e17975..297ff89c7 100755 --- a/core/env/env.go +++ b/core/env/env.go @@ -5,7 +5,6 @@ import ( "fmt" "os" "path/filepath" - "sort" "strings" "sync" "time" @@ -19,7 +18,6 @@ import ( "github.com/mattn/go-isatty" "github.com/rs/zerolog" "github.com/segmentio/ksuid" - "github.com/slingdata-io/sling-cli/core" "github.com/spf13/cast" ) @@ -42,12 +40,7 @@ var ( RunnerID = g.Getenv("SLING_RUNNER_ID", os.Getenv("SLING_AGENT_ID")) IsRunnerMode = RunnerID != "" - // File logging - debugLogFile *os.File - traceLogFile *os.File - logFileInit = false - logFileMux sync.Mutex - GetOAuthMap = func() map[string]map[string]any { + GetOAuthMap = func() map[string]map[string]any { return map[string]map[string]any{} } ExecFolder = func() string { return filepath.Join(HomeDir, "executions", ExecID) } @@ -187,14 +180,16 @@ func SetLogger() { } } - outputOut := zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: "2006-01-02 15:04:05", FormatLevel: g.ZLogFormatLevel, FormatMessage: g.ZLogFormatMessage} - outputErr := zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: "2006-01-02 15:04:05", FormatLevel: g.ZLogFormatLevel, FormatMessage: g.ZLogFormatMessage} - outputOut.FormatErrFieldValue = func(i interface{}) string { - return fmt.Sprintf("%s", i) + formatMsg := func(i interface{}) string { + return ScrubLine(g.ZLogFormatMessage(i)) } - outputErr.FormatErrFieldValue = func(i interface{}) string { - return fmt.Sprintf("%s", i) + formatErr := func(i interface{}) string { + return ScrubLine(fmt.Sprintf("%s", i)) } + outputOut := zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: "2006-01-02 15:04:05", FormatLevel: g.ZLogFormatLevel, FormatMessage: formatMsg} + outputErr := zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: "2006-01-02 15:04:05", FormatLevel: g.ZLogFormatLevel, FormatMessage: formatMsg} + outputOut.FormatErrFieldValue = formatErr + outputErr.FormatErrFieldValue = formatErr if os.Getenv("SLING_LOGGING") == "NO_COLOR" { NoColor = true @@ -211,10 +206,11 @@ func SetLogger() { g.ZLogOut = zerolog.New(os.Stdout).With().Timestamp().Logger() g.ZLogErr = zerolog.New(os.Stdout).With().Timestamp().Logger() } else { - outputErr = zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: "3:04PM", FormatLevel: g.ZLogFormatLevel, FormatMessage: g.ZLogFormatMessage} + outputErr = zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: "3:04PM", FormatLevel: g.ZLogFormatLevel, FormatMessage: formatMsg} if g.IsDebugLow() { - outputErr = zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: "2006-01-02 15:04:05", FormatLevel: g.ZLogFormatLevel, FormatMessage: g.ZLogFormatMessage} + outputErr = zerolog.ConsoleWriter{Out: os.Stderr, TimeFormat: "2006-01-02 15:04:05", FormatLevel: g.ZLogFormatLevel, FormatMessage: formatMsg} } + outputErr.FormatErrFieldValue = formatErr g.ZLogOut = zerolog.New(outputErr).With().Timestamp().Logger() g.ZLogErr = zerolog.New(outputErr).With().Timestamp().Logger() } @@ -245,282 +241,70 @@ func InitLogger() { setupFileLogging() setupOtel() } - -// setupFileLogging initializes file logging based on SLING_DEBUG_FILE and SLING_TRACE_FILE env vars -func setupFileLogging() { - if IsThreadChild { - return // don't write log from child processes - } - - logFileMux.Lock() - defer logFileMux.Unlock() - - // Close existing files if any (for re-initialization) - if debugLogFile != nil { - debugLogFile.Close() - debugLogFile = nil - } - if traceLogFile != nil { - traceLogFile.Close() - traceLogFile = nil - } - - // setup env from env.yaml and .env.sling - LoadSlingEnvFile() - LoadDotEnvSling() - - // Open debug log file - if debugPath := os.Getenv("SLING_DEBUG_FILE"); debugPath != "" { - f, err := os.OpenFile(debugPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) - if err != nil { - g.Warn("could not open debug log file: %s", err.Error()) - } else { - debugLogFile = f - } - } - - // Open debug log file from SLING_LOG_DIR (date-based rotation) - // Only if SLING_DEBUG_FILE wasn't set and this is not a thread child process - if logDir := os.Getenv("SLING_LOG_DIR"); logDir != "" && debugLogFile == nil { - // Expand ~ to home directory - if strings.HasPrefix(logDir, "~/") { - logDir = filepath.Join(g.UserHomeDir(), logDir[2:]) - } - if err := os.MkdirAll(logDir, 0755); err != nil { - g.Warn("could not create log directory: %s", err.Error()) - } else { - logFileName := "sling_debug_" + time.Now().Format("2006_01_02") + ".log" - logPath := filepath.Join(logDir, logFileName) - - f, err := os.OpenFile(logPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) - if err != nil { - g.Warn("could not open log file: %s", err.Error()) - } else { - debugLogFile = f - cleanupOldLogFiles(logDir, 15) - } - } - } - - // Open trace log file - if tracePath := os.Getenv("SLING_TRACE_FILE"); tracePath != "" { - f, err := os.OpenFile(tracePath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) - if err != nil { - g.Warn("could not open trace log file: %s", err.Error()) - } else { - traceLogFile = f - } - } +func Print(text string) { + fmt.Fprintf(os.Stderr, "%s", text) + ll := &g.LogLine{Level: 9, Text: text, Time: time.Now()} + processLogEntry(ll) + writeToLogFile(ll) } -// CloseFileLogging closes any open log files -func CloseFileLogging() { - logFileMux.Lock() - defer logFileMux.Unlock() - - if debugLogFile != nil { - debugLogFile.Close() - debugLogFile = nil - } - if traceLogFile != nil { - traceLogFile.Close() - traceLogFile = nil - } +func Println(text string) { + text = text + "\n" + Print(text) } -// cleanupOldLogFiles removes old .log files from the directory, keeping the latest `keep` files. -// Files are sorted by name (which sorts chronologically for date-based filenames). -func cleanupOldLogFiles(dir string, keep int) { - entries, err := os.ReadDir(dir) - if err != nil { - g.Warn("could not read log directory for cleanup: %s", err.Error()) - return - } +// PrintFatal prints the fatal error (same text as g.PrintFatal) and captures +// it in the run-log buffer via Println, so stderr.log includes it. +func PrintFatal(E error, args ...interface{}) { + makeErrStrings := func(payload string) string { + cancelledCount := 0 + payload = strings.ReplaceAll(payload, "---\n\n---", "---\n---") + errParts := strings.Split(payload, "\n\n") + errStrings := []string{} + errHash := map[string]struct{}{} + for _, errPart := range errParts { + if _, ok := errHash[errPart]; !ok && errPart != "context canceled" { + if ps := strings.Split(errPart, "\n"); ps[len(ps)-1] == "context canceled" { + cancelledCount++ + } + errStrings = append(errStrings, errPart) + } + errHash[errPart] = struct{}{} + } - var logFiles []string - for _, entry := range entries { - if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".log") { - logFiles = append(logFiles, entry.Name()) + if cancelledCount == len(errStrings) { + return "cancelled" } + return strings.Join(errStrings, "\n\n") } - sort.Strings(logFiles) - - if len(logFiles) > keep { - for _, name := range logFiles[:len(logFiles)-keep] { - if err := os.Remove(filepath.Join(dir, name)); err != nil { - g.Warn("could not remove old log file %s: %s", name, err.Error()) - } + prefix := "fatal:\n" + if E != nil { + err, ok := E.(*g.ErrType) + if !ok { + err = g.NewError(3, E, args...).(*g.ErrType) } - } -} -// stripANSI removes ANSI escape codes from a string -func stripANSI(text string) string { - // Match ANSI escape sequences: ESC[ followed by any number of params and a letter - // This handles color codes like \x1b[32m, \x1b[0m, \x1b[90m, etc. - result := strings.Builder{} - i := 0 - for i < len(text) { - if i+1 < len(text) && text[i] == '\x1b' && text[i+1] == '[' { - // Skip the escape sequence - j := i + 2 - for j < len(text) && ((text[j] >= '0' && text[j] <= '9') || text[j] == ';') { - j++ - } - if j < len(text) && text[j] >= 'A' && text[j] <= 'z' { - j++ // Skip the final letter + eG, ok := E.(*g.ErrorGroup) + if ok { + if !g.IsDebugLow() { + Println(RedString(prefix + eG.Error())) + } else { + Println(RedString(prefix + eG.Debug())) } - i = j } else { - result.WriteByte(text[i]) - i++ - } - } - return result.String() -} - -func shortExecID() string { - val := ExecID - if len(val) > 8 { - val = val[len(val)-8:] - } - return val -} - -// formatLogLine formats a log line for file output (no colors) -func formatLogLine(ll *g.LogLine) string { - var levelPrefix string - - switch zerolog.Level(ll.Level) { - case zerolog.TraceLevel: - levelPrefix = "TRC " - case zerolog.DebugLevel: - levelPrefix = "DBG " - case zerolog.InfoLevel: - levelPrefix = "INF " - case zerolog.WarnLevel: - levelPrefix = "WRN " - case zerolog.ErrorLevel: - levelPrefix = "ERR " - default: - levelPrefix = "" - } - - timeText := ll.Time.Format("2006-01-02 15:04:05") - - // Filter out map arguments and special strings (used internally by g library for context logging) - filteredArgs := []any{} - for _, arg := range ll.Args { - switch arg.(type) { - case map[string]any: - // Skip map arguments - they're context fields, not format args - continue - default: - if s, ok := arg.(string); ok && strings.HasPrefix(s, "_DEBUG_CALLER_START=") { - // Skip internal caller tracking string - continue + if !g.IsDebugLow() { + joined := makeErrStrings(err.Error()) + Println(RedString(prefix + joined)) + } else { + joined := makeErrStrings(err.Err) + output := g.F("%s\n%s", strings.Join(err.Stack(), "\n"), joined) + Println(RedString(prefix + output)) } - filteredArgs = append(filteredArgs, arg) - } - } - - text := g.F(ll.Text, filteredArgs...) - - // Strip any ANSI codes from the text - text = stripANSI(text) - - return fmt.Sprintf("%s | %s %s%s\n", shortExecID(), timeText, levelPrefix, text) -} - -func writeHeader(logFile *os.File) { - // Write session header - wd, _ := os.Getwd() - header := fmt.Sprintf( - "\n%s\n== %s | version: %s | exec_id: %s\n== dir: %s | command: %s\n%s\n", - strings.Repeat("=", 100), - time.Now().Format("2006-01-02 15:04:05"), - core.Version, - ExecID, - wd, - strings.Join(os.Args, " "), - strings.Repeat("=", 80), - ) - logFile.WriteString(header) -} - -// writeToLogFile writes the log entry to configured log file(s) -func writeToLogFile(ll *g.LogLine) { - logFileMux.Lock() - defer logFileMux.Unlock() - - // Skip if no log files configured - if debugLogFile == nil && traceLogFile == nil { - return - } - - if !logFileInit { - if debugLogFile != nil { - writeHeader(debugLogFile) - } - if traceLogFile != nil { - writeHeader(traceLogFile) } - logFileInit = true - } - - level := zerolog.Level(ll.Level) - - // Handle Print/Println entries (level 9) - these are raw output from child processes - // Write them directly with ANSI codes stripped, but only if they have content - if ll.Level == 9 { - text := stripANSI(ll.Text) - if strings.TrimSpace(text) == "" { - return - } - - // Add execID prefix - text = shortExecID() + " | " + text - - // Ensure text ends with newline - if !strings.HasSuffix(text, "\n") { - text = text + "\n" - } - // Write to both files (Print output is considered important) - if traceLogFile != nil { - traceLogFile.WriteString(text) - } - if debugLogFile != nil { - debugLogFile.WriteString(text) - } - return - } - - line := formatLogLine(ll) - - // Write to trace file (all levels) - if traceLogFile != nil { - traceLogFile.WriteString(line) - } - - // Write to debug file (debug level and above) - // zerolog levels: Trace=-1, Debug=0, Info=1, Warn=2, Error=3 - if debugLogFile != nil && level >= zerolog.DebugLevel { - debugLogFile.WriteString(line) } } -func Print(text string) { - fmt.Fprintf(os.Stderr, "%s", text) - processLogEntry(&g.LogLine{Level: 9, Text: text}) - writeToLogFile(&g.LogLine{Level: 9, Text: text}) -} - -func Println(text string) { - text = text + "\n" - Print(text) -} - func LoadSlingEnvFile() (ef EnvFile) { ef = LoadEnvFile(HomeDirEnvFile) Env = &ef @@ -643,13 +427,6 @@ func CleanWindowsPath(path string) string { return strings.ReplaceAll(path, `\`, `/`) } -func processLogEntry(ll *g.LogLine) { - // Existing LogSink functionality - if LogSink != nil { - LogSink(ll) - } -} - // RemoveLocalTempFile deletes the local file func RemoveLocalTempFile(localPath string) { if !cast.ToBool(os.Getenv("SLING_KEEP_TEMP")) { @@ -664,17 +441,6 @@ func RemoveAllLocalTempFile(localPath string) { } } -func WriteTempSQL(sql string, filePrefix ...string) (sqlPath string, err error) { - sqlPath = filepath.Join(GetTempFolder(), g.NewTsID(filePrefix...)+".sql") - - err = os.WriteFile(sqlPath, []byte(sql), 0777) - if err != nil { - return "", g.Error(err, "could not create temp sql") - } - - return -} - func LogSQL(props map[string]string, query string, args ...any) { noColor := g.In(os.Getenv("SLING_LOGGING"), "NO_COLOR", "JSON") @@ -701,8 +467,9 @@ func LogSQL(props map[string]string, query string, args ...any) { } g.Trace(query, contextArgs) } else { + query = Clean(props, query) if !noColor { - query = CyanString(Clean(props, query)) + query = CyanString(query) } if !cast.ToBool(props["silent"]) { g.Debug(query + connIdSuffix) @@ -710,26 +477,24 @@ func LogSQL(props map[string]string, query string, args ...any) { } } -// Clean removes creds from a log line -func Clean(props map[string]string, line string) string { - line = strings.TrimSpace(line) - sqlLower := strings.ToLower(line) - - startsWith := func(p string) bool { return strings.HasPrefix(sqlLower, p) } +// Skip short values so passwords like "postgres" don't redact unrelated log text. +const minSecretLen = 12 - switch { - case startsWith("drop "), startsWith("create "), startsWith("insert into"), startsWith("select count"): - return line - case startsWith("alter table "), startsWith("update "), startsWith("alter table "), startsWith("update "): - return line - case startsWith("select *"): - return line - } +func redactable(val string) bool { + val = strings.TrimSpace(val) + return len(val) >= minSecretLen && !IsEnvVarRef(val) +} +// Clean removes creds from a log line. CREATE/INSERT/etc. still redact: +// Redshift COPY/UNLOAD embeds keys in those statements. +func Clean(props map[string]string, line string) string { + line = strings.TrimSpace(line) + keys := secretKeysLower() for k, v := range props { if strings.TrimSpace(v) == "" { continue - } else if g.In(k, "password", "access_key_id", "secret_access_key", "session_token", "aws_access_key_id", "aws_secret_access_key", "aws_session_token", "ssh_private_key", "ssh_passphrase", "sas_svc_url", "conn_str") { + } + if _, ok := keys[strings.ToLower(k)]; ok { line = strings.ReplaceAll(line, v, "***") } } @@ -746,3 +511,49 @@ func ExpandEnvVars(text string) string { } return text } + +// CleanConnData redacts registry secrets and every nested secrets: value. +// Nested API keys are secret even when the key name is not in the registry. +func CleanConnData(data map[string]any, line string) string { + + asStringAnyMap := func(v any) map[string]any { + switch m := v.(type) { + case map[string]any: + return m + case map[any]any: + out := map[string]any{} + for k, val := range m { + out[cast.ToString(k)] = val + } + return out + default: + return nil + } + } + + if data == nil { + return line + } + flat := map[string]string{} + var nested map[string]any + for k, v := range data { + if strings.EqualFold(k, "secrets") { + if m := asStringAnyMap(v); m != nil { + nested = m + } + continue + } + if val := cast.ToString(v); redactable(val) { + flat[k] = val + } + } + line = Clean(flat, line) + for _, v := range nested { + val := strings.TrimSpace(cast.ToString(v)) + if !redactable(val) { + continue + } + line = strings.ReplaceAll(line, val, "***") + } + return line +} diff --git a/core/env/envfile.go b/core/env/envfile.go index 04000631e..76b4ed15a 100644 --- a/core/env/envfile.go +++ b/core/env/envfile.go @@ -4,6 +4,7 @@ import ( "bytes" "os" "path" + "regexp" "strings" "github.com/flarco/g" @@ -11,6 +12,9 @@ import ( "gopkg.in/yaml.v3" ) +// envVarRefRe matches a whole-string ${VAR} ref. Unset refs stay literal after g.Rmd. +var envVarRefRe = regexp.MustCompile(`^\$\{([A-Z_][A-Z0-9_]*)\}$`) + type EnvFile struct { Connections map[string]map[string]any `json:"connections,omitempty" yaml:"connections,omitempty"` Env map[string]any `json:"env,omitempty" yaml:"env,omitempty"` @@ -72,7 +76,8 @@ func (ef *EnvFile) marshalEnvFileBytes() ([]byte, error) { return nil, err } - merged := mergeNode(original, newRoot) + merged := mergeNode(original, newRoot, interpEnvMap(ef.Path)) + annotateEnvVarRefComments(merged) var buf bytes.Buffer enc := yaml.NewEncoder(&buf) @@ -247,15 +252,7 @@ func loadEnvFile(body, path string) (ef EnvFile, err error) { } // expand variables - envMap := map[string]any{} - if path != "" { - envMap["SLING_HOME_DIR"] = HomeDir - } - for _, tuple := range os.Environ() { - key := strings.Split(tuple, "=")[0] - val := strings.TrimPrefix(tuple, key+"=") - envMap[key] = val - } + envMap := interpEnvMap(path) ef.Body = g.Rmd(ef.Body, envMap) if err = yaml.Unmarshal([]byte(ef.Body), &ef); err != nil { @@ -292,9 +289,56 @@ func GetEnvFilePath(dir string) string { return CleanWindowsPath(path.Join(dir, "env.yaml")) } +// processEnv is the current process environment as a map. +func processEnv() map[string]any { + out := map[string]any{} + for _, kv := range os.Environ() { + k, v, ok := strings.Cut(kv, "=") + if !ok || k == "" { + continue + } + out[k] = v + } + return out +} + +// MergeDeclaredEnv starts from the process environment and overlays +// declared pipeline/replication env keys. Bare {env.X} then renders +// even when the YAML has no env: block. +func MergeDeclaredEnv(declared map[string]any) map[string]any { + out := processEnv() + for k, v := range declared { + out[k] = v + } + return out +} + +// interpEnvMap is the same substitution map loadEnvFile uses for g.Rmd. +func interpEnvMap(path string) map[string]any { + envMap := processEnv() + if path != "" { + if _, ok := envMap["SLING_HOME_DIR"]; !ok { + envMap["SLING_HOME_DIR"] = HomeDir + } + } + return envMap +} + +// keepOnDiskScalar is true when newVal is origVal or origVal after env expansion. +// Load interpolates ${VAR}; write must keep the on-disk ref, not the secret. +func keepOnDiskScalar(origVal, newVal string, envMap map[string]any) bool { + if origVal == newVal { + return true + } + if envMap == nil || !strings.Contains(origVal, "${") { + return false + } + return g.Rmd(origVal, envMap) == newVal +} + // mergeNode deep-merges newNode into original, keeping original's comments and // key order. Adapted from pulumi/pulumi's yamlutil.editNodes (Apache 2.0). -func mergeNode(original, newNode *yaml.Node) *yaml.Node { +func mergeNode(original, newNode *yaml.Node, envMap map[string]any) *yaml.Node { if original == nil { out := *newNode return &out @@ -309,6 +353,9 @@ func mergeNode(original, newNode *yaml.Node) *yaml.Node { } ret := *original + if original.Kind == yaml.ScalarNode && keepOnDiskScalar(original.Value, newNode.Value, envMap) { + return &ret + } ret.Tag = newNode.Tag ret.Value = newNode.Value @@ -320,12 +367,12 @@ func mergeNode(original, newNode *yaml.Node) *yaml.Node { } content := make([]*yaml.Node, 0, len(newNode.Content)) for i := 0; i < minLen; i++ { - content = append(content, mergeNode(original.Content[i], newNode.Content[i])) + content = append(content, mergeNode(original.Content[i], newNode.Content[i], envMap)) } content = append(content, newNode.Content[minLen:]...) ret.Content = content case yaml.MappingNode: - ret.Content = mergeMappingContent(original, newNode) + ret.Content = mergeMappingContent(original, newNode, envMap) case yaml.ScalarNode, yaml.AliasNode: ret.Content = newNode.Content } @@ -335,7 +382,7 @@ func mergeNode(original, newNode *yaml.Node) *yaml.Node { // mergeMappingContent merges two mapping nodes: original keys keep their // position and comments; new-only keys append at the end; dropped keys are // removed. -func mergeMappingContent(original, newNode *yaml.Node) []*yaml.Node { +func mergeMappingContent(original, newNode *yaml.Node, envMap map[string]any) []*yaml.Node { origIdx := map[string]int{} newIdx := map[string]int{} var origOrder, newOnly []string @@ -360,8 +407,8 @@ func mergeMappingContent(original, newNode *yaml.Node) []*yaml.Node { continue } oi := origIdx[k] - key := mergeNode(original.Content[oi], newNode.Content[ni]) - val := mergeNode(original.Content[oi+1], newNode.Content[ni+1]) + key := mergeNode(original.Content[oi], newNode.Content[ni], envMap) + val := mergeNode(original.Content[oi+1], newNode.Content[ni+1], envMap) content = append(content, key, val) } for _, k := range newOnly { @@ -395,3 +442,160 @@ func (ef *EnvFile) loadRootNode() (*yaml.Node, error) { } return root, nil } + +// IsEnvVarRef is true when s is a whole-string ${VAR} reference. +func IsEnvVarRef(s string) bool { + return envVarRefRe.MatchString(strings.TrimSpace(s)) +} + +// EnvVarRefName returns VAR from ${VAR}. It returns "" when s is not a ref. +func EnvVarRefName(s string) string { + m := envVarRefRe.FindStringSubmatch(strings.TrimSpace(s)) + if len(m) != 2 { + return "" + } + return m[1] +} + +// ConnLocation is a deep-link into env.yaml for one connection. +type ConnLocation struct { + Path string `json:"path"` + Line int `json:"line"` + Connection string `json:"connection"` + Missing []MissingRef `json:"missing"` +} + +// MissingRef is one ${VAR} field that still needs a value (or an env var). +type MissingRef struct { + Key string `json:"key"` + Var string `json:"var"` + Line int `json:"line"` +} + +// LookupConnection re-parses ef.Path and returns line numbers for +// connections. and each ${VAR} field under it. +func (ef *EnvFile) LookupConnection(name string) (ConnLocation, error) { + loc := ConnLocation{ + Path: ef.Path, + Connection: strings.ToUpper(name), + Missing: []MissingRef{}, + } + root, err := ef.loadRootNode() + if err != nil { + return loc, err + } + + conns := mappingChild(root, "connections") + if conns == nil { + return loc, g.Error("connections block not found in %s", ef.Path) + } + + keyNode, valNode := mappingChildFold(conns, name) + if keyNode == nil { + return loc, g.Error("connection %s not found in %s", name, ef.Path) + } + loc.Line = keyNode.Line + collectMissingRefs(valNode, "", &loc.Missing) + return loc, nil +} + +func mappingChild(n *yaml.Node, key string) *yaml.Node { + n = mappingRoot(n) + if n == nil { + return nil + } + for i := 0; i < len(n.Content)-1; i += 2 { + if n.Content[i].Value == key { + return n.Content[i+1] + } + } + return nil +} + +func mappingChildFold(n *yaml.Node, key string) (keyNode, valNode *yaml.Node) { + n = mappingRoot(n) + if n == nil { + return nil, nil + } + for i := 0; i < len(n.Content)-1; i += 2 { + if strings.EqualFold(n.Content[i].Value, key) { + return n.Content[i], n.Content[i+1] + } + } + return nil, nil +} + +func mappingRoot(n *yaml.Node) *yaml.Node { + if n == nil { + return nil + } + if n.Kind == yaml.DocumentNode && len(n.Content) > 0 { + n = n.Content[0] + } + if n.Kind != yaml.MappingNode { + return nil + } + return n +} + +func collectMissingRefs(n *yaml.Node, prefix string, out *[]MissingRef) { + if n == nil { + return + } + switch n.Kind { + case yaml.ScalarNode: + if !IsEnvVarRef(n.Value) { + return + } + *out = append(*out, MissingRef{ + Key: prefix, + Var: EnvVarRefName(n.Value), + Line: n.Line, + }) + case yaml.MappingNode: + for i := 0; i < len(n.Content)-1; i += 2 { + k := n.Content[i].Value + path := k + if prefix != "" { + path = prefix + "." + k + } + collectMissingRefs(n.Content[i+1], path, out) + } + case yaml.SequenceNode: + for _, child := range n.Content { + collectMissingRefs(child, prefix, out) + } + } +} + +// annotateEnvVarRefComments writes EnvVarRefComment on connection ${VAR} +// scalars that have no trailing comment. Original comments stay. +func annotateEnvVarRefComments(root *yaml.Node) { + conns := mappingChild(root, "connections") + if conns == nil { + return + } + annotateMappingRefs(conns) +} + +func annotateMappingRefs(n *yaml.Node) { + + // EnvVarRefComment is the trailing comment written next to scaffolded ${VAR} refs. + const EnvVarRefComment = "replace with the value, or set the env var (CI)" + + n = mappingRoot(n) + if n == nil { + return + } + for i := 0; i < len(n.Content)-1; i += 2 { + val := n.Content[i+1] + switch val.Kind { + case yaml.ScalarNode: + if IsEnvVarRef(val.Value) && strings.TrimSpace(val.LineComment) == "" { + val.LineComment = EnvVarRefComment + } + case yaml.MappingNode: + annotateMappingRefs(val) + } + } +} diff --git a/core/env/envfile_test.go b/core/env/envfile_test.go index a63cae7cd..5f3950693 100644 --- a/core/env/envfile_test.go +++ b/core/env/envfile_test.go @@ -195,6 +195,105 @@ func TestWriteEnvFileAssistRoundTrip(t *testing.T) { } } +// TestWriteEnvFileKeepsOnDiskEnvRefs fails if write materializes a ${VAR} +// that loadEnvFile interpolated from the process environment. +func TestWriteEnvFileKeepsOnDiskEnvRefs(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "env.yaml") + const leak = "hunter2-LEAK-TEST" + t.Setenv("MY_PG_PASSWORD", leak) + + original := `connections: + MY_PG: + type: postgres + host: localhost + password: ${MY_PG_PASSWORD} + port: 5432 +` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + ef := LoadEnvFile(path) + gotPass, _ := ef.Connections["MY_PG"]["password"].(string) + if gotPass != leak { + t.Fatalf("load should interpolate password, got %q", gotPass) + } + + ef.Connections["MY_PG"]["port"] = "5433" + if err := ef.WriteEnvFile(); err != nil { + t.Fatalf("WriteEnvFile: %v", err) + } + + got, _ := os.ReadFile(path) + out := string(got) + if !strings.Contains(out, "${MY_PG_PASSWORD}") { + t.Errorf("expected on-disk ${MY_PG_PASSWORD} ref\n--- got ---\n%s", out) + } + if strings.Contains(out, leak) { + t.Errorf("merge wrote interpolated secret %q\n--- got ---\n%s", leak, out) + } + if !strings.Contains(out, "5433") { + t.Errorf("expected updated port\n--- got ---\n%s", out) + } +} + +func TestLookupConnectionLines(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "env.yaml") + original := `connections: + MY_PG: + type: postgres + host: localhost + password: ${MY_PG_PASSWORD} +` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + ef := LoadEnvFile(path) + loc, err := ef.LookupConnection("MY_PG") + if err != nil { + t.Fatalf("LookupConnection: %v", err) + } + if loc.Path != path { + t.Errorf("path: got %q want %q", loc.Path, path) + } + if loc.Connection != "MY_PG" { + t.Errorf("connection: got %q", loc.Connection) + } + if loc.Line != 2 { + t.Errorf("connection line: got %d want 2\n%s", loc.Line, original) + } + if len(loc.Missing) != 1 { + t.Fatalf("missing: got %#v", loc.Missing) + } + if loc.Missing[0].Key != "password" { + t.Errorf("missing key: got %q", loc.Missing[0].Key) + } + if loc.Missing[0].Var != "MY_PG_PASSWORD" { + t.Errorf("missing var: got %q", loc.Missing[0].Var) + } + if loc.Missing[0].Line != 5 { + t.Errorf("missing line: got %d want 5", loc.Missing[0].Line) + } +} + +func TestIsEnvVarRef(t *testing.T) { + if !IsEnvVarRef("${MY_PG_PASSWORD}") { + t.Error("expected ${MY_PG_PASSWORD} to be a ref") + } + if IsEnvVarRef("${my_pg_password}") { + t.Error("lowercase var names are not refs") + } + if IsEnvVarRef("mypass") { + t.Error("plaintext is not a ref") + } + if EnvVarRefName("${MY_PG_PASSWORD}") != "MY_PG_PASSWORD" { + t.Error("EnvVarRefName") + } +} + func TestParseDotEnv(t *testing.T) { tests := []struct { name string @@ -232,7 +331,7 @@ func TestParseDotEnv(t *testing.T) { }, }, { - name: "multi-line double-quoted value", + name: "multi-line double-quoted value", content: "KEY=\"hello\nworld\"", expected: map[string]string{ "KEY": "hello\nworld", @@ -298,22 +397,22 @@ BAZ=qux`, }, }, { - name: "double-quoted value with single quotes inside", + name: "double-quoted value with single quotes inside", content: `KEY="{'a': 'b'}"`, expected: map[string]string{ "KEY": "{'a': 'b'}", }, }, { - name: "multi-line double-quoted with single quotes inside", + name: "multi-line double-quoted with single quotes inside", content: "KEY=\"{\n 'a': 'b'\n}\"", expected: map[string]string{ "KEY": "{\n 'a': 'b'\n}", }, }, { - name: "line without equals is skipped", - content: "NOPE", + name: "line without equals is skipped", + content: "NOPE", expected: map[string]string{}, }, } @@ -325,3 +424,21 @@ BAZ=qux`, }) } } + +func TestMergeDeclaredEnv(t *testing.T) { + t.Setenv("MERGE_DECLARED_PROBE", "from-process") + t.Setenv("MERGE_DECLARED_OVERRIDE", "from-process") + + got := MergeDeclaredEnv(map[string]any{ + "MERGE_DECLARED_OVERRIDE": "from-yaml", + "MERGE_DECLARED_ONLY": 7, + }) + + assert.Equal(t, "from-process", got["MERGE_DECLARED_PROBE"]) + assert.Equal(t, "from-yaml", got["MERGE_DECLARED_OVERRIDE"]) + assert.Equal(t, 7, got["MERGE_DECLARED_ONLY"]) + + empty := MergeDeclaredEnv(nil) + assert.Equal(t, "from-process", empty["MERGE_DECLARED_PROBE"]) + assert.NotContains(t, empty, "MERGE_DECLARED_ONLY") +} diff --git a/core/env/logbuffer.go b/core/env/logbuffer.go new file mode 100644 index 000000000..7b7a08b56 --- /dev/null +++ b/core/env/logbuffer.go @@ -0,0 +1,455 @@ +package env + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "sync" + "time" + + "github.com/flarco/g" + "github.com/rs/zerolog" + "github.com/slingdata-io/sling-cli/core" +) + +// recentLogLines is how many log lines the failure buffer keeps. +const recentLogLines = 5000 + +// logBuffer owns run logging: the file sinks (debug/trace) and a bounded +// in-memory tail of the log. The tail feeds the assist failure snapshot, so a +// failed run keeps the lines that led to the error. +// +// The ring is off until Start runs. next is the next write slot; the buffer +// wraps once full, so an old line is overwritten in place instead of shifting +// the whole slice. +type logBuffer struct { + mu sync.Mutex + + // file sinks + debugFile *os.File + traceFile *os.File + headerOn bool + + // in-memory tail + on bool + lines []string + next int + full bool +} + +// logs is the process-wide run log. +var logs = &logBuffer{} + +// Start begins buffering log lines for the failure snapshot. +func (lb *logBuffer) Start() { + lb.mu.Lock() + defer lb.mu.Unlock() + lb.on = true + lb.lines = make([]string, recentLogLines) + lb.next, lb.full = 0, false +} + +// Stop stops buffering and drops what was kept. +func (lb *logBuffer) Stop() { + lb.mu.Lock() + defer lb.mu.Unlock() + lb.on = false + lb.lines = nil + lb.next, lb.full = 0, false +} + +// Recent returns the buffered tail, oldest first. +func (lb *logBuffer) Recent() string { + lb.mu.Lock() + defer lb.mu.Unlock() + + n := lb.next + if lb.full { + n = recentLogLines + } + if n == 0 { + return "" + } + + // Size the builder up front: one Grow beats repeated reallocation. + size := n - 1 // newline separators + for i := 0; i < n; i++ { + size += len(lb.at(i)) + } + + var b strings.Builder + b.Grow(size) + for i := 0; i < n; i++ { + if i > 0 { + b.WriteByte('\n') + } + b.WriteString(lb.at(i)) + } + return b.String() +} + +// at returns the i-th line, oldest first. Call with the lock held. +func (lb *logBuffer) at(i int) string { + if !lb.full { + return lb.lines[i] + } + return lb.lines[(lb.next+i)%recentLogLines] +} + +func (lb *logBuffer) shortExecID() string { + val := ExecID + if len(val) > 8 { + val = val[len(val)-8:] + } + return val +} + +// formatLogLine formats a log line for file output (no colors) +func (lb *logBuffer) formatLogLine(ll *g.LogLine) string { + var levelPrefix string + + switch zerolog.Level(ll.Level) { + case zerolog.TraceLevel: + levelPrefix = "TRC " + case zerolog.DebugLevel: + levelPrefix = "DBG " + case zerolog.InfoLevel: + levelPrefix = "INF " + case zerolog.WarnLevel: + levelPrefix = "WRN " + case zerolog.ErrorLevel: + levelPrefix = "ERR " + default: + levelPrefix = "" + } + + timeText := ll.Time.Format("2006-01-02 15:04:05") + + // Filter out map arguments and special strings (used internally by g library for context logging) + filteredArgs := []any{} + for _, arg := range ll.Args { + switch arg.(type) { + case map[string]any: + // Skip map arguments - they're context fields, not format args + continue + default: + if s, ok := arg.(string); ok && strings.HasPrefix(s, "_DEBUG_CALLER_START=") { + // Skip internal caller tracking string + continue + } + filteredArgs = append(filteredArgs, arg) + } + } + + text := g.F(ll.Text, filteredArgs...) + + // Strip any ANSI codes from the text + text = lb.stripANSI(text) + + return fmt.Sprintf("%s | %s %s%s\n", lb.shortExecID(), timeText, levelPrefix, text) +} + +// stripANSI removes ANSI escape codes from a string +func (lb *logBuffer) stripANSI(text string) string { + // Match ANSI escape sequences: ESC[ followed by any number of params and a letter + // This handles color codes like \x1b[32m, \x1b[0m, \x1b[90m, etc. + result := strings.Builder{} + i := 0 + for i < len(text) { + if i+1 < len(text) && text[i] == '\x1b' && text[i+1] == '[' { + // Skip the escape sequence + j := i + 2 + for j < len(text) && ((text[j] >= '0' && text[j] <= '9') || text[j] == ';') { + j++ + } + if j < len(text) && text[j] >= 'A' && text[j] <= 'z' { + j++ // Skip the final letter + } + i = j + } else { + result.WriteByte(text[i]) + i++ + } + } + return result.String() +} + +// Capture appends one rendered line, overwriting the oldest once full. +func (lb *logBuffer) Capture(ll *g.LogLine) { + if ll == nil { + return + } + lb.mu.Lock() + defer lb.mu.Unlock() + if !lb.on { + return + } + // Check the message before formatting: formatLogLine prepends the exec id + // and timestamp, which would make a blank message look non-blank. + if strings.TrimSpace(ll.Text) == "" { + return + } + // formatLogLine interpolates Args and adds the level/time prefix. + text := strings.TrimRight(lb.stripANSI(lb.formatLogLine(ll)), "\n") + if strings.TrimSpace(text) == "" { + return + } + lb.lines[lb.next] = text + lb.next++ + if lb.next == recentLogLines { + lb.next, lb.full = 0, true + } +} + +// CloseFiles closes any open log files. +func (lb *logBuffer) CloseFiles() { + lb.mu.Lock() + defer lb.mu.Unlock() + lb.closeFiles() +} + +// closeFiles closes the sinks. Call with the lock held. +func (lb *logBuffer) closeFiles() { + if lb.debugFile != nil { + lb.debugFile.Close() + lb.debugFile = nil + } + if lb.traceFile != nil { + lb.traceFile.Close() + lb.traceFile = nil + } +} + +// cleanupOldLogFiles removes old .log files from the directory, keeping the latest `keep` files. +// Files are sorted by name (which sorts chronologically for date-based filenames). +func (lb *logBuffer) cleanupOldLogFiles(dir string, keep int) { + entries, err := os.ReadDir(dir) + if err != nil { + g.Warn("could not read log directory for cleanup: %s", err.Error()) + return + } + + var logFiles []string + for _, entry := range entries { + if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".log") { + logFiles = append(logFiles, entry.Name()) + } + } + + sort.Strings(logFiles) + + if len(logFiles) > keep { + for _, name := range logFiles[:len(logFiles)-keep] { + if err := os.Remove(filepath.Join(dir, name)); err != nil { + g.Warn("could not remove old log file %s: %s", name, err.Error()) + } + } + } +} + +// SetupFiles opens the debug and trace sinks from the environment. +// SLING_DEBUG_FILE and SLING_TRACE_FILE name a file each; SLING_LOG_DIR gives +// date-based rotation when SLING_DEBUG_FILE is unset. +func (lb *logBuffer) SetupFiles() { + if IsThreadChild { + return // don't write log from child processes + } + + lb.mu.Lock() + defer lb.mu.Unlock() + + lb.closeFiles() // for re-initialization + + // setup env from env.yaml and .env.sling + LoadSlingEnvFile() + LoadDotEnvSling() + + if debugPath := os.Getenv("SLING_DEBUG_FILE"); debugPath != "" { + f, err := os.OpenFile(debugPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) + if err != nil { + g.Warn("could not open debug log file: %s", err.Error()) + } else { + lb.debugFile = f + } + } + + // Date-based rotation, only when SLING_DEBUG_FILE was not set. + if logDir := os.Getenv("SLING_LOG_DIR"); logDir != "" && lb.debugFile == nil { + if strings.HasPrefix(logDir, "~/") { + logDir = filepath.Join(g.UserHomeDir(), logDir[2:]) + } + if err := os.MkdirAll(logDir, 0755); err != nil { + g.Warn("could not create log directory: %s", err.Error()) + } else { + logPath := filepath.Join(logDir, "sling_debug_"+time.Now().Format("2006_01_02")+".log") + f, err := os.OpenFile(logPath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) + if err != nil { + g.Warn("could not open log file: %s", err.Error()) + } else { + lb.debugFile = f + lb.cleanupOldLogFiles(logDir, 15) + } + } + } + + if tracePath := os.Getenv("SLING_TRACE_FILE"); tracePath != "" { + f, err := os.OpenFile(tracePath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644) + if err != nil { + g.Warn("could not open trace log file: %s", err.Error()) + } else { + lb.traceFile = f + } + } +} + +// WriteFile writes the log entry to the configured log file(s). +func (lb *logBuffer) WriteFile(ll *g.LogLine) { + lb.mu.Lock() + defer lb.mu.Unlock() + + writeHeader := func(logFile *os.File) { + // Write session header + wd, _ := os.Getwd() + header := fmt.Sprintf( + "\n%s\n== %s | version: %s | exec_id: %s\n== dir: %s | command: %s\n%s\n", + strings.Repeat("=", 100), + time.Now().Format("2006-01-02 15:04:05"), + core.Version, + ExecID, + wd, + strings.Join(os.Args, " "), + strings.Repeat("=", 80), + ) + logFile.WriteString(header) + } + + if lb.debugFile == nil && lb.traceFile == nil { + return + } + + if !lb.headerOn { + if lb.debugFile != nil { + writeHeader(lb.debugFile) + } + if lb.traceFile != nil { + writeHeader(lb.traceFile) + } + lb.headerOn = true + } + + // Level 9 is raw output from a child process. Write it through with the + // ANSI codes stripped, to both files: Print output is important. + if ll.Level == 9 { + text := lb.stripANSI(ll.Text) + if strings.TrimSpace(text) == "" { + return + } + text = lb.shortExecID() + " | " + text + if !strings.HasSuffix(text, "\n") { + text = text + "\n" + } + if lb.traceFile != nil { + lb.traceFile.WriteString(text) + } + if lb.debugFile != nil { + lb.debugFile.WriteString(text) + } + return + } + + line := lb.formatLogLine(ll) + + if lb.traceFile != nil { + lb.traceFile.WriteString(line) // all levels + } + // zerolog levels: Trace=-1, Debug=0, Info=1, Warn=2, Error=3 + if lb.debugFile != nil && zerolog.Level(ll.Level) >= zerolog.DebugLevel { + lb.debugFile.WriteString(line) + } +} + +// Process is the DebugLevel hook: buffer the tail, then fan out to LogSink. +func (lb *logBuffer) Process(ll *g.LogLine) { + lb.Capture(ll) + + if LogSink != nil { + LogSink(ll) + } +} + +// --- package-level API ------------------------------------------------- + +// StartLogCapture begins buffering log lines for the failure snapshot. +func StartLogCapture() { logs.Start() } + +// StopLogCapture stops buffering and drops what was kept. +func StopLogCapture() { logs.Stop() } + +// RecentLogs returns the buffered tail, oldest first. +func RecentLogs() string { return logs.Recent() } + +// CloseFileLogging closes any open log files. +func CloseFileLogging() { logs.CloseFiles() } + +// setupFileLogging opens the log sinks from the environment. +func setupFileLogging() { logs.SetupFiles() } + +func writeToLogFile(ll *g.LogLine) { logs.WriteFile(ll) } + +func processLogEntry(ll *g.LogLine) { logs.Process(ll) } + +// SecretKeys is the full set of connection property names whose values +// are secrets. Clean and parse.Redact redact these. Keep it in sync with +// `secret: true` in core/dbio/templates/_properties.yaml. +var SecretKeys = []string{ + "access_key_id", + "account_key", + "aws_access_key_id", + "aws_secret_access_key", + "aws_session_token", + "azure_account_key", + "azure_sas_token", + "conn_str", + "gcp_credentials_json", + "gcs_secret_access_key", + "password", + "rest_oauth_client_secret", + "rest_token", + "s3_secret_access_key", + "s3_session_token", + "sas_svc_url", + "sas_token", + "secret_access_key", + "session_token", + "ssh_passphrase", + "ssh_private_key", + "token", +} + +var ( + secretKeysOnce sync.Once + secretKeyCache map[string]struct{} +) + +func secretKeysLower() map[string]struct{} { + secretKeysOnce.Do(func() { + secretKeyCache = make(map[string]struct{}, len(SecretKeys)) + for _, k := range SecretKeys { + secretKeyCache[strings.ToLower(k)] = struct{}{} + } + }) + return secretKeyCache +} + +// ScrubLine redacts secrets from every local connection in Env. +func ScrubLine(line string) string { + if Env == nil { + return line + } + for _, data := range Env.Connections { + line = CleanConnData(data, line) + } + return line +} diff --git a/core/env/logbuffer_test.go b/core/env/logbuffer_test.go new file mode 100644 index 000000000..d810df2b1 --- /dev/null +++ b/core/env/logbuffer_test.go @@ -0,0 +1,156 @@ +package env + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/flarco/g" + "github.com/rs/zerolog" +) + +func line(s string) *g.LogLine { return &g.LogLine{Level: 1, Text: s} } + +func TestLogCaptureOffByDefault(t *testing.T) { + logs.Stop() + logs.Capture(line("dropped")) + if got := logs.Recent(); got != "" { + t.Fatalf("want empty when off, got %q", got) + } +} + +func TestLogCaptureKeepsOrder(t *testing.T) { + logs.Start() + defer logs.Stop() + for i := 1; i <= 3; i++ { + logs.Capture(line(fmt.Sprintf("l%d", i))) + } + lines := strings.Split(logs.Recent(), "\n") + if len(lines) != 3 { + t.Fatalf("want 3 lines, got %d: %q", len(lines), lines) + } + for i, l := range lines { + if !strings.HasSuffix(l, fmt.Sprintf("l%d", i+1)) { + t.Fatalf("line %d out of order: %q", i, l) + } + } +} + +func TestLogCaptureWrapsKeepingNewest(t *testing.T) { + logs.Start() + defer logs.Stop() + total := recentLogLines + 10 + for i := 0; i < total; i++ { + logs.Capture(line(fmt.Sprintf("l%d", i))) + } + got := logs.Recent() + lines := strings.Split(got, "\n") + if len(lines) != recentLogLines { + t.Fatalf("want %d lines, got %d", recentLogLines, len(lines)) + } + // Oldest 10 evicted; newest retained, still in order. + if !strings.HasSuffix(lines[0], fmt.Sprintf("l%d", total-recentLogLines)) { + t.Fatalf("wrong first line: %q", lines[0]) + } + if !strings.HasSuffix(lines[len(lines)-1], fmt.Sprintf("l%d", total-1)) { + t.Fatalf("wrong last line: %q", lines[len(lines)-1]) + } + if strings.Contains(got, "l0\n") { + t.Fatal("evicted line l0 still present") + } +} + +func TestLogCaptureSkipsBlank(t *testing.T) { + logs.Start() + defer logs.Stop() + logs.Capture(line(" ")) + logs.Capture(nil) + if got := logs.Recent(); got != "" { + t.Fatalf("want empty, got %q", got) + } +} + +func TestLogCaptureRestartClears(t *testing.T) { + logs.Start() + logs.Capture(line("old")) + logs.Start() // restart must drop prior content + defer logs.Stop() + if got := logs.Recent(); got != "" { + t.Fatalf("restart must clear, got %q", got) + } +} + +func TestLogBufferWriteFileRespectsLevel(t *testing.T) { + dir := t.TempDir() + debugPath := filepath.Join(dir, "debug.log") + tracePath := filepath.Join(dir, "trace.log") + t.Setenv("SLING_DEBUG_FILE", debugPath) + t.Setenv("SLING_TRACE_FILE", tracePath) + + lb := &logBuffer{} + lb.SetupFiles() + defer lb.CloseFiles() + + lb.WriteFile(&g.LogLine{Level: int8(zerolog.TraceLevel), Text: "trace-only"}) + lb.WriteFile(&g.LogLine{Level: int8(zerolog.ErrorLevel), Text: "an-error"}) + lb.CloseFiles() + + debug, err := os.ReadFile(debugPath) + if err != nil { + t.Fatalf("read debug: %v", err) + } + trace, err := os.ReadFile(tracePath) + if err != nil { + t.Fatalf("read trace: %v", err) + } + if strings.Contains(string(debug), "trace-only") { + t.Error("trace line must not reach the debug file") + } + if !strings.Contains(string(debug), "an-error") { + t.Error("error line missing from debug file") + } + for _, want := range []string{"trace-only", "an-error"} { + if !strings.Contains(string(trace), want) { + t.Errorf("trace file missing %q", want) + } + } +} + +func TestLogBufferInstancesAreIndependent(t *testing.T) { + a, b := &logBuffer{}, &logBuffer{} + a.Start() + defer a.Stop() + a.Capture(line("only-in-a")) + if got := b.Recent(); got != "" { + t.Fatalf("second buffer must stay empty, got %q", got) + } +} + +func TestPrintFatalCapturesToLogBuffer(t *testing.T) { + logs.Start() + defer logs.Stop() + + old := os.Stderr + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + os.Stderr = w + PrintFatal(g.Error("connection refused", "could not connect")) + _ = w.Close() + os.Stderr = old + _, _ = r.Read(make([]byte, 64*1024)) + + got := logs.Recent() + if !strings.Contains(got, "fatal:") { + t.Fatalf("missing fatal prefix: %q", got) + } + if !strings.Contains(got, "could not connect") && !strings.Contains(got, "connection refused") { + t.Fatalf("missing error text: %q", got) + } + if strings.Contains(got, "0001-01-01") { + t.Fatalf("zero timestamp on captured fatal: %q", got) + } +} diff --git a/core/sling/assist/assist.go b/core/sling/assist/assist.go new file mode 100644 index 000000000..abb53a7b6 --- /dev/null +++ b/core/sling/assist/assist.go @@ -0,0 +1,279 @@ +// Package assist implements `sling assist`: profile, skills/MCP install, prompt, resume. +package assist + +import ( + "embed" + "os" + "path/filepath" + "sync" + + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core/env" + "gopkg.in/yaml.v3" +) + +//go:embed all:skills +var SkillsFS embed.FS + +//go:embed prompts.yaml +var PromptsYAML []byte + +// SchemaVersion is the on-disk layout version. +const SchemaVersion = 1 + +// HistoryMaxEntries caps entries under ~/.sling/assist/history/. +const HistoryMaxEntries = 100 + +// ExecsMaxEntries caps failure snapshots under ~/.sling/assist/errors/. +const ExecsMaxEntries = 100 + +// Paths is the injectable path seam (tests use SetPaths). +type Paths struct { + SlingHome string + UserHome string + CWD string +} + +var ( + pathsMu sync.RWMutex + pathsOverride *Paths // nil = live OS +) + +// CurrentPaths returns the active path set (override or live). +func CurrentPaths() Paths { + pathsMu.RLock() + ov := pathsOverride + pathsMu.RUnlock() + if ov != nil { + return *ov + } + cwd, _ := os.Getwd() + return Paths{ + SlingHome: env.HomeDir, + UserHome: g.UserHomeDir(), + CWD: cwd, + } +} + +// SetPaths installs a path override; restore undoes it. +func SetPaths(p Paths) (restore func()) { + pathsMu.Lock() + prev := pathsOverride + cp := p + pathsOverride = &cp + pathsMu.Unlock() + return func() { + pathsMu.Lock() + pathsOverride = prev + pathsMu.Unlock() + } +} + +func slingHome() string { + return CurrentPaths().SlingHome +} + +func userHome() string { + return CurrentPaths().UserHome +} + +func workDir() string { + p := CurrentPaths() + if p.CWD != "" { + return p.CWD + } + cwd, _ := os.Getwd() + return cwd +} + +// projectRootMarkers identify a project root when walking up for --scope project. +var projectRootMarkers = []string{ + ".git", + "go.mod", + "package.json", + "pyproject.toml", + "Cargo.toml", + "sling_build.yml", + ".sling", +} + +// projectRoot walks up from workDir for a marker; falls back to workDir. +func projectRoot() string { + dir := workDir() + if dir == "" { + return dir + } + start := dir + for { + for _, m := range projectRootMarkers { + if g.PathExists(filepath.Join(dir, m)) { + return dir + } + } + parent := filepath.Dir(dir) + if parent == dir { + return start + } + dir = parent + } +} + +func projectPath(elem ...string) string { + return filepath.Join(append([]string{projectRoot()}, elem...)...) +} + +const assistEnvKey = "SLING_ASSIST" + +// Profile is stored under env.SLING_ASSIST in ~/.sling/env.yaml. +type Profile struct { + Agent string `yaml:"agent" json:"agent"` // claude | codex | … | auto + HintInErrors bool `yaml:"hint_in_errors" json:"hint_in_errors"` // run-error footer + DefaultInstallScope string `yaml:"default_install_scope,omitempty" json:"default_install_scope"` // user | project +} + +// DefaultProfile returns sane first-run defaults. +func DefaultProfile() Profile { + return Profile{ + Agent: "auto", + HintInErrors: true, + DefaultInstallScope: "user", + } +} + +func envFilePath() string { + return env.GetEnvFilePath(slingHome()) +} + +// LoadProfile reads env.SLING_ASSIST. Missing key → (Profile{}, false, nil). +func LoadProfile() (p Profile, exists bool, err error) { + path := envFilePath() + if _, statErr := os.Stat(path); statErr != nil { + if os.IsNotExist(statErr) { + return Profile{}, false, nil + } + return Profile{}, false, g.Error(statErr, "could not stat %s", path) + } + ef := env.LoadEnvFile(path) + raw, ok := ef.Env[assistEnvKey] + if !ok || raw == nil { + return Profile{}, false, nil + } + m, err := castToStringMap(raw) + if err != nil { + return Profile{}, false, g.Error(err, "env.%s is not a mapping", assistEnvKey) + } + if len(m) == 0 { + return Profile{}, false, nil + } + p, err = profileFromMap(m) + if err != nil { + return Profile{}, false, err + } + return p, true, nil +} + +// castToStringMap normalizes YAML maps (map[string]any or map[any]any). +func castToStringMap(v any) (map[string]any, error) { + switch m := v.(type) { + case map[string]any: + return m, nil + case map[any]any: + out := make(map[string]any, len(m)) + for k, vv := range m { + ks, ok := k.(string) + if !ok { + return nil, g.Error("non-string key %v", k) + } + out[ks] = vv + } + return out, nil + case nil: + return map[string]any{}, nil + default: + return nil, g.Error("unexpected type %T", v) + } +} + +// SaveProfile writes env.SLING_ASSIST via EnvFile (preserves other keys/comments). +func SaveProfile(p Profile) error { + path := envFilePath() + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return g.Error(err, "could not create %s", filepath.Dir(path)) + } + ef := env.LoadEnvFile(path) + ef.Path = path + m, err := profileToMap(p) + if err != nil { + return err + } + if ef.Env == nil { + ef.Env = map[string]any{} + } + ef.Env[assistEnvKey] = m + return ef.WriteEnvFile() +} + +func profileToMap(p Profile) (map[string]any, error) { + b, err := yaml.Marshal(p) + if err != nil { + return nil, g.Error(err, "could not marshal profile") + } + m := map[string]any{} + if uerr := yaml.Unmarshal(b, &m); uerr != nil { + return nil, g.Error(uerr, "could not re-parse profile") + } + return m, nil +} + +func profileFromMap(m map[string]any) (Profile, error) { + b, err := yaml.Marshal(m) + if err != nil { + return Profile{}, g.Error(err, "could not marshal SLING_ASSIST block") + } + var p Profile + if uerr := yaml.Unmarshal(b, &p); uerr != nil { + return Profile{}, g.Error(uerr, "could not parse SLING_ASSIST block") + } + return p, nil +} + +// AssistDir returns ~/.sling/assist/. +func AssistDir() string { + d := filepath.Join(slingHome(), "assist") + _ = os.MkdirAll(d, 0o755) + return d +} + +// HistoryDir returns ~/.sling/assist/history/. +func HistoryDir() string { + d := filepath.Join(AssistDir(), "history") + _ = os.MkdirAll(d, 0o755) + return d +} + +// ErrorsDir returns ~/.sling/assist/errors/ (legacy snapshot root). +// New snapshots live under ExecutionsDir(); readers scan both. +func ErrorsDir() string { + d := filepath.Join(AssistDir(), "errors") + _ = os.MkdirAll(d, 0o755) + return d +} + +// ExecutionsDir returns ~/.sling/assist/errors/executions/ (failure snapshots). +func ExecutionsDir() string { + d := filepath.Join(ErrorsDir(), "executions") + _ = os.MkdirAll(d, 0o755) + return d +} + +// VersionFilePath returns ~/.sling/assist/version. +func VersionFilePath() string { + return filepath.Join(AssistDir(), "version") +} + +// CanonicalSkillsDir returns ~/.agents/skills/ (shared skill source of truth). +func CanonicalSkillsDir() string { + d := filepath.Join(userHome(), ".agents", "skills") + _ = os.MkdirAll(d, 0o755) + return d +} diff --git a/core/sling/assist/assist_test.go b/core/sling/assist/assist_test.go new file mode 100644 index 000000000..6a0c6376c --- /dev/null +++ b/core/sling/assist/assist_test.go @@ -0,0 +1,776 @@ +// Package tests: bugs, harness gaps, and extensibility seams. + +package assist + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/slingdata-io/sling-cli/core" +) + +func TestTryRefreshLockExclusiveAndTokenUnlock(t *testing.T) { + dir := t.TempDir() + lockPath := filepath.Join(dir, ".refresh-lock") + + unlock1, ok := tryRefreshLock(lockPath) + if !ok { + t.Fatal("first lock should succeed") + } + if _, ok := tryRefreshLock(lockPath); ok { + t.Fatal("second lock should fail while first is held") + } + unlock1() + unlock2, ok := tryRefreshLock(lockPath) + if !ok { + t.Fatal("lock after unlock should succeed") + } + unlock2() + if _, err := os.Stat(lockPath); !os.IsNotExist(err) { + t.Fatalf("lock file should be removed after unlock, err=%v", err) + } +} + +func TestTryRefreshLockReclaimsStale(t *testing.T) { + dir := t.TempDir() + lockPath := filepath.Join(dir, ".refresh-lock") + if err := os.WriteFile(lockPath, []byte("old"), 0o644); err != nil { + t.Fatal(err) + } + stale := time.Now().Add(-refreshLockStale - time.Minute) + if err := os.Chtimes(lockPath, stale, stale); err != nil { + t.Fatal(err) + } + unlock, ok := tryRefreshLock(lockPath) + if !ok { + t.Fatal("stale lock should be reclaimable") + } + unlock() +} + +func TestPruneRetiredSkillsRemovesStaleDirs(t *testing.T) { + withTempHomeDir(t) + root := CanonicalSkillsDir() + stale := filepath.Join(root, "sling-hooks") + if err := os.MkdirAll(stale, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(stale, "SKILL.md"), []byte("old"), 0o644); err != nil { + t.Fatal(err) + } + keep := filepath.Join(root, "sling-pipelines") + if err := os.MkdirAll(keep, 0o755); err != nil { + t.Fatal(err) + } + + pruneRetiredSkills(context.Background(), ScopeUser) + + if _, err := os.Stat(stale); !os.IsNotExist(err) { + t.Fatalf("retired skill dir %s should be removed", stale) + } + if _, err := os.Stat(keep); err != nil { + t.Fatalf("current skill dir %s should remain: %v", keep, err) + } +} + +func TestUninstallMarksStampSoAutoRefreshSkips(t *testing.T) { + withTempHomeDir(t) + // Simulate a prior install stamp, then uninstall skills. + if err := os.MkdirAll(AssistDir(), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(VersionFilePath(), []byte("0.0.0-old"), 0o644); err != nil { + t.Fatal(err) + } + if err := Uninstall(context.Background(), UninstallOptions{NonInteractive: true}); err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(VersionFilePath()) + if err != nil { + t.Fatal(err) + } + if strings.TrimSpace(string(got)) != versionUninstalled { + t.Fatalf("stamp = %q, want %q", got, versionUninstalled) + } + notice, err := AutoRefresh(context.Background()) + if err != nil { + t.Fatal(err) + } + if notice != "" { + t.Fatalf("AutoRefresh after uninstall should no-op, got notice %q", notice) + } +} + +func TestAutoRefreshNoopsWhenNeverInstalled(t *testing.T) { + withTempHomeDir(t) + // No skills on disk → never installed; must not write the bundle. + notice, err := AutoRefresh(context.Background()) + if err != nil { + t.Fatal(err) + } + if notice != "" { + t.Fatalf("AutoRefresh with no skills should no-op, got %q", notice) + } + for _, name := range listSkillNames() { + if _, err := os.Stat(canonicalSkillPath(name)); !os.IsNotExist(err) { + t.Fatalf("should not install %s when no skills exist", name) + } + } +} + +func TestAutoRefreshNoopsWhenStampStaleButNoSkills(t *testing.T) { + withTempHomeDir(t) + if err := os.MkdirAll(AssistDir(), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(VersionFilePath(), []byte("0.0.0-old"), 0o644); err != nil { + t.Fatal(err) + } + notice, err := AutoRefresh(context.Background()) + if err != nil { + t.Fatal(err) + } + if notice != "" { + t.Fatalf("AutoRefresh with no skills should no-op, got %q", notice) + } + for _, name := range listSkillNames() { + if _, err := os.Stat(canonicalSkillPath(name)); !os.IsNotExist(err) { + t.Fatalf("should not install %s from a stale stamp", name) + } + } +} + +func TestAutoRefreshHealsDriftWhenStampCurrent(t *testing.T) { + withTempHomeDir(t) + names := listSkillNames() + if len(names) == 0 { + t.Fatal("no embedded skills") + } + if err := writeCanonicalBundle(names); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(AssistDir(), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(VersionFilePath(), []byte(core.Version), 0o644); err != nil { + t.Fatal(err) + } + + name := names[0] + skillPath := canonicalSkillPath(name) + if err := os.WriteFile(skillPath, []byte("drifted-content"), 0o644); err != nil { + t.Fatal(err) + } + + notice, err := AutoRefresh(context.Background()) + if err != nil { + t.Fatal(err) + } + if notice == "" { + t.Fatal("expected refresh notice after drift") + } + + got, err := os.ReadFile(skillPath) + if err != nil { + t.Fatal(err) + } + want, err := SkillsFS.ReadFile(filepath.ToSlash(filepath.Join("skills", name, "SKILL.md"))) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got, want) { + t.Fatalf("skill %s still drifted after AutoRefresh", name) + } +} + +func TestAutoRefreshPrunesRetiredWhenStampCurrent(t *testing.T) { + withTempHomeDir(t) + names := listSkillNames() + if err := writeCanonicalBundle(names); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(AssistDir(), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(VersionFilePath(), []byte(core.Version), 0o644); err != nil { + t.Fatal(err) + } + + stale := filepath.Join(CanonicalSkillsDir(), "sling-hooks") + if err := os.MkdirAll(stale, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(stale, "SKILL.md"), []byte("old"), 0o644); err != nil { + t.Fatal(err) + } + + notice, err := AutoRefresh(context.Background()) + if err != nil { + t.Fatal(err) + } + if notice == "" { + t.Fatal("expected refresh notice after prune") + } + if _, err := os.Stat(stale); !os.IsNotExist(err) { + t.Fatalf("retired skill dir %s should be removed", stale) + } +} + +func TestAgentExitErrorExitCodeOf(t *testing.T) { + err := &AgentExitError{ExitCode: 42, Agent: "claude"} + code, ok := ExitCodeOf(err) + if !ok || code != 42 { + t.Fatalf("ExitCodeOf = %d, %v", code, ok) + } + if _, ok := ExitCodeOf(fmt.Errorf("other")); ok { + t.Fatal("non-agent error should not match") + } +} + +func TestClaudeProjectMCPPath(t *testing.T) { + c := &claudeClient{} + got := c.mcpPath(ScopeProject) + if got != filepath.Join(".", ".mcp.json") && got != ".mcp.json" { + // filepath.Join(".", ".mcp.json") is "./.mcp.json" on Unix + if !strings.HasSuffix(got, ".mcp.json") || strings.Contains(got, ".claude.json") { + t.Fatalf("project mcp path = %q", got) + } + } + if strings.Contains(c.mcpPath(ScopeProject), ".claude.json") { + t.Fatalf("project scope must not use .claude.json: %q", c.mcpPath(ScopeProject)) + } +} + +func TestVSCodeMCPUsesServersKey(t *testing.T) { + // Unit-level: path helper for project scope. + c := &vscodeClient{} + p := c.vscodeMCPPath(ScopeProject) + if !strings.Contains(p, ".vscode") || !strings.HasSuffix(p, "mcp.json") { + t.Fatalf("project vscode mcp path = %q", p) + } +} + +func TestAgentLaunchArgsPerAgent(t *testing.T) { + path := "/tmp/prompt.md" + cases := []struct { + agent string + wantArgs []string + stdin bool + }{ + {"codex", []string{"exec", "-"}, true}, + {"gemini", []string{"-p", "-"}, true}, + {"claude", []string{"Read and execute the task in @" + path}, false}, + {"cursor", []string{path}, false}, + {"grok", []string{"Read and execute the task in @" + path}, false}, + {"pi", []string{"-p"}, true}, + {"opencode", []string{"run", "--file", path, "Read and execute the attached task"}, false}, + {"unknown-cli", nil, true}, + } + for _, tc := range cases { + p := agentLaunchArgs(tc.agent, path, "", "") + if p.UseStdin != tc.stdin { + t.Errorf("%s UseStdin=%v want %v", tc.agent, p.UseStdin, tc.stdin) + } + if len(p.Args) != len(tc.wantArgs) { + t.Errorf("%s args=%v want %v", tc.agent, p.Args, tc.wantArgs) + continue + } + for i := range tc.wantArgs { + if p.Args[i] != tc.wantArgs[i] { + t.Errorf("%s args[%d]=%q want %q", tc.agent, i, p.Args[i], tc.wantArgs[i]) + } + } + } +} + +func TestResolveAgentOverrideWins(t *testing.T) { + // Need a detectable agent dir so override succeeds. + home := t.TempDir() + prevHome := os.Getenv("HOME") + os.Setenv("HOME", home) + t.Cleanup(func() { os.Setenv("HOME", prevHome) }) + // claude Detect() requires the binary on $PATH + if err := os.MkdirAll(filepath.Join(home, ".claude"), 0o755); err != nil { + t.Fatal(err) + } + // Put claude on PATH via a stub. + bin := filepath.Join(home, "bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + stub := filepath.Join(bin, "claude") + if err := os.WriteFile(stub, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + prevPath := os.Getenv("PATH") + os.Setenv("PATH", bin+string(os.PathListSeparator)+prevPath) + t.Cleanup(func() { os.Setenv("PATH", prevPath) }) + + got, err := ResolveAgent("claude", Profile{Agent: "codex"}) + if err != nil { + t.Fatal(err) + } + if got != "claude" { + t.Fatalf("override should win: got %q", got) + } +} + +func TestResolveAgentUnknownOverride(t *testing.T) { + _, err := ResolveAgent("not-an-agent", Profile{}) + if err == nil || !strings.Contains(err.Error(), "unknown agent") { + t.Fatalf("err = %v", err) + } +} + +func TestResolveAgentProfileNonLaunchable(t *testing.T) { + _, err := ResolveAgent("", Profile{Agent: "vscode"}) + if err == nil || !strings.Contains(err.Error(), "non-launchable") { + t.Fatalf("err = %v", err) + } +} + +func TestSlugify(t *testing.T) { + if got := slugify("Hello World!"); got != "hello-world" { + t.Fatalf("got %q", got) + } + if got := slugify(" "); got != "entry" { + t.Fatalf("empty → entry, got %q", got) + } + if got := slugify("a / b"); got != "a-b" { + t.Fatalf("separator runs must collapse, got %q", got) + } +} + +func TestSlugifyCapsLength(t *testing.T) { + long := "Help me create or update a Sling config (replication, pipeline, model, or API spec). First ask me which one." + got := slugify(long) + if len(got) > maxSlugLen { + t.Fatalf("slug %q is %d chars, want <= %d", got, len(got), maxSlugLen) + } + if strings.HasSuffix(got, "-") || strings.HasPrefix(got, "-") { + t.Fatalf("slug must not have dangling separators: %q", got) + } + // Cut on a word boundary: every kept word is whole. + want := "help-me-create-or-update-a-sling-config" + if got != want { + t.Fatalf("slug = %q, want %q", got, want) + } +} + +func TestSaveEntryIDStaysShort(t *testing.T) { + withTempHomeDir(t) + a := AnswersFile{ + Name: slugify("Help me create or update a Sling config (replication, pipeline, model, or API spec)"), + Task: "open", + } + id, err := SaveEntry(a, "prompt", Meta{Task: "open"}) + if err != nil { + t.Fatal(err) + } + // + "_" + slug + if len(id) > 20+maxSlugLen { + t.Fatalf("id %q is %d chars", id, len(id)) + } + if _, err := LoadEntry(id); err != nil { + t.Fatalf("round-trip: %v", err) + } +} + +func TestFilterEntriesQuery(t *testing.T) { + entries := []Entry{ + {ID: "one", Answers: AnswersFile{Task: "replication.create", Agent: "claude", Name: "pg-to-sf"}}, + {ID: "two", Answers: AnswersFile{Task: "pipeline.create", Agent: "codex", Name: "daily"}}, + } + got := filterEntries(entries, "pg-to") + if len(got) != 1 || got[0].ID != "one" { + t.Fatalf("got %+v", got) + } + got = filterEntries(entries, "codex") + if len(got) != 1 || got[0].ID != "two" { + t.Fatalf("got %+v", got) + } + got = filterEntries(entries, "") + if len(got) != 2 { + t.Fatalf("empty query len=%d", len(got)) + } +} + +func TestSessionPrintWithoutSetup(t *testing.T) { + withTempHomeDir(t) + _, err := Session(SessionOptions{Print: true, Ask: "from PG"}) + if err != nil { + t.Fatal(err) + } +} + +func TestSessionLaunchWithoutSetupFails(t *testing.T) { + withTempHomeDir(t) + _, err := Session(SessionOptions{Headless: true, Ask: "from PG"}) + if err == nil { + t.Fatal("expected setup error") + } + if !strings.Contains(err.Error(), "sling assist setup") { + t.Fatalf("want setup hint, got: %v", err) + } +} + +func TestPickerRendersOldTaskAndNewMode(t *testing.T) { + entries := []Entry{ + {ID: "old", Answers: AnswersFile{Task: "replication.create", Name: "pg-to-sf", Created: time.Now()}}, + {ID: "open1", Answers: AnswersFile{Task: "open", Name: "backfill", Created: time.Now()}}, + {ID: "ask1", Answers: AnswersFile{Task: "ask", Name: "hello", Created: time.Now()}}, + } + got := newPickerModel(entries).View() + for _, w := range []string{"replication", "open", "ask"} { + if !strings.Contains(got, w) { + t.Errorf("missing %q\n%s", w, got) + } + } +} + +func TestDoctorReportToJSONTyped(t *testing.T) { + withTempHomeDir(t) + r, err := Doctor(context.Background()) + if err != nil { + t.Fatal(err) + } + if r == nil { + t.Fatal("nil report") + } + if r.SlingVersion == "" { + t.Fatal("missing sling_version") + } + if len(r.Findings) == 0 { + t.Fatal("expected findings") + } + // Findings must not use glyph-prefixed prose as the only structure. + for _, f := range r.Findings { + if f.ID == "" { + t.Fatalf("finding missing id: %+v", f) + } + if strings.HasPrefix(f.Summary, "✓") || strings.HasPrefix(f.Summary, "✗") { + t.Fatalf("summary still has glyph: %q", f.Summary) + } + } + body, err := r.ToJSON() + if err != nil { + t.Fatal(err) + } + var decoded map[string]any + if err := json.Unmarshal(body, &decoded); err != nil { + t.Fatal(err) + } + if _, ok := decoded["findings"]; !ok { + t.Fatalf("json missing findings: %s", body) + } + if _, ok := decoded["sling_version"]; !ok { + t.Fatalf("json missing sling_version: %s", body) + } + // Lines must not appear in JSON (json:"-"). + if _, ok := decoded["Lines"]; ok { + t.Fatal("Lines should not be in JSON") + } + if _, ok := decoded["lines"]; ok { + t.Fatal("lines should not be in JSON") + } +} + +func TestDoctorContextCanceled(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err := Doctor(ctx) + if err == nil { + t.Fatal("expected context error") + } +} + +func TestCellStateJSON(t *testing.T) { + m := DoctorMatrix{ + Clients: []string{"claude"}, + Rows: []MatrixRow{{ + Label: "MCP", + Cells: map[string]CellState{"claude": CellOK}, + }}, + } + b, err := json.Marshal(m) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(b), `"ok"`) { + t.Fatalf("cell state not stringified: %s", b) + } +} + +func TestCheckResultRender(t *testing.T) { + r := checkSkill(CellFail, "sling", "redirect missing") + line := r.Render("claude") + if !strings.HasPrefix(line, "✗") { + t.Fatalf("glyph: %q", line) + } + if !strings.Contains(line, "claude") || !strings.Contains(line, "sling") { + t.Fatalf("line: %q", line) + } +} + +func TestPathsOverrideAffectsSlingHome(t *testing.T) { + dir := t.TempDir() + restore := SetPaths(Paths{SlingHome: dir, UserHome: dir, CWD: dir}) + defer restore() + if !strings.HasPrefix(LogsRoot(), dir) { + t.Fatalf("LogsRoot=%q not under %q", LogsRoot(), dir) + } + if !strings.HasPrefix(AssistDir(), dir) { + t.Fatalf("AssistDir=%q not under %q", AssistDir(), dir) + } + if !strings.HasPrefix(ErrorsDir(), dir) { + t.Fatalf("ErrorsDir=%q not under %q", ErrorsDir(), dir) + } + if userHome() != dir { + t.Fatalf("userHome=%q want %q", userHome(), dir) + } +} + +func TestInstallRespectsCanceledContext(t *testing.T) { + withTempHomeDir(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, err := Install(ctx, InstallOptions{NonInteractive: true, DefaultAgent: "claude"}) + if err == nil { + t.Fatal("expected cancel error") + } +} + +func TestCodexMCPSectionLineAnchored(t *testing.T) { + // Mention in a comment or string must not count. + if hasCodexMCPSection(`# see [mcp_servers.sling] docs +name = "x" +`) { + t.Fatal("comment mention should not count") + } + body := ` +[mcp_servers.other] +command = "x" + +[mcp_servers.sling] +command = "sling" +args = ["serve", "mcp"] + +[mcp_servers.sling.env] +FOO = "bar" +` + if !hasCodexMCPSection(body) { + t.Fatal("expected section present") + } + out := removeCodexMCP(body) + if hasCodexMCPSection(out) { + t.Fatalf("header still present after remove:\n%s", out) + } + if strings.Contains(out, "[mcp_servers.sling.env]") { + t.Fatalf("orphan subtable left behind:\n%s", out) + } + if !strings.Contains(out, "[mcp_servers.other]") { + t.Fatalf("sibling section removed:\n%s", out) + } +} + +func TestBackupPreservesSourceMode(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "secret.json") + if err := os.WriteFile(path, []byte(`{"a":1}`), 0o600); err != nil { + t.Fatal(err) + } + if err := backupBeforeEdit(path); err != nil { + t.Fatal(err) + } + info, err := os.Stat(path + backupSuffix) + if err != nil { + t.Fatal(err) + } + if info.Mode().Perm() != 0o600 { + t.Fatalf("backup mode = %o, want 0600", info.Mode().Perm()) + } +} + +func TestResumePrintsSavedPrompt(t *testing.T) { + withTempHomeDir(t) + a := AnswersFile{ + Name: "edit-me", + Task: "replication.update", + Created: time.Now().UTC(), + Agent: "claude", + Answers: map[string]any{"ask": "add incremental mode"}, + } + id, err := SaveEntry(a, "saved prompt body\n", Meta{Task: a.Task, Agent: "claude", HarnessSessionID: "abc"}) + if err != nil { + t.Fatal(err) + } + old := assistOut + var buf bytes.Buffer + assistOut = &buf + t.Cleanup(func() { assistOut = old }) + _, err = Session(SessionOptions{ResumeID: id, Print: true}) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(buf.String(), "saved prompt body") { + t.Fatalf("got %q", buf.String()) + } +} + +func TestProjectRootFindsGit(t *testing.T) { + dir := t.TempDir() + sub := filepath.Join(dir, "a", "b") + if err := os.MkdirAll(sub, 0o755); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(dir, ".git"), 0o755); err != nil { + t.Fatal(err) + } + restore := SetPaths(Paths{SlingHome: dir, UserHome: dir, CWD: sub}) + defer restore() + if got := projectRoot(); got != dir { + t.Fatalf("projectRoot = %q, want %q", got, dir) + } +} + +func TestSyncCanonicalSkillPrunesStale(t *testing.T) { + withTempHomeDir(t) + // Write bundle then plant a stale file under a skill dir. + skills := listSkillNames() + if len(skills) == 0 { + t.Fatal("no embedded skills") + } + name := skills[0] + if _, err := syncCanonicalSkill(name); err != nil { + t.Fatal(err) + } + stale := filepath.Join(CanonicalSkillsDir(), name, "STALE_DO_NOT_KEEP.md") + if err := os.WriteFile(stale, []byte("gone"), 0o644); err != nil { + t.Fatal(err) + } + changed, err := syncCanonicalSkill(name) + if err != nil { + t.Fatal(err) + } + if !changed { + t.Fatal("expected change when pruning stale file") + } + if _, err := os.Stat(stale); !os.IsNotExist(err) { + t.Fatalf("stale file still present: %v", err) + } +} + +func TestClaudeLaunchArgsAssignSessionAndModel(t *testing.T) { + path := "/tmp/prompt.md" + sid := "550e8400-e29b-41d4-a716-446655440000" + p := agentLaunchArgs("claude", path, "sonnet", sid) + joined := strings.Join(p.Args, " ") + if !strings.Contains(joined, "--session-id "+sid) { + t.Fatalf("missing session-id: %v", p.Args) + } + if !strings.Contains(joined, "--model sonnet") { + t.Fatalf("missing model: %v", p.Args) + } +} + +func TestCodexLaunchArgsInsertModelAfterSubcommand(t *testing.T) { + p := agentLaunchArgs("codex", "/tmp/p.md", "sonnet", "") + if len(p.Args) < 3 || p.Args[0] != "exec" || p.Args[1] != "-m" || p.Args[2] != "sonnet" { + t.Fatalf("codex args=%v", p.Args) + } +} + +func TestResumeArgsHaveNoPrompt(t *testing.T) { + cases := []struct { + agent string + id string + want []string + }{ + {"claude", "u1", []string{"--resume", "u1"}}, + {"grok", "u2", []string{"--resume", "u2"}}, + {"codex", "u3", []string{"resume", "u3"}}, + {"gemini", "u4", []string{"--resume", "u4"}}, + {"cursor", "u5", []string{"--resume=u5"}}, + {"opencode", "u6", []string{"--session", "u6"}}, + {"pi", "u7", []string{"--session", "u7"}}, + } + for _, tc := range cases { + p := agentResumeArgs(tc.agent, tc.id, "") + if strings.Join(p.Args, " ") != strings.Join(tc.want, " ") { + t.Errorf("%s args=%v want %v", tc.agent, p.Args, tc.want) + } + if p.UseStdin { + t.Errorf("%s resume should not use stdin", tc.agent) + } + } + p := agentResumeArgs("claude", "u1", "sonnet") + if !strings.Contains(strings.Join(p.Args, " "), "--model sonnet") { + t.Fatalf("claude resume missing model: %v", p.Args) + } + p = agentResumeArgs("codex", "u3", "sonnet") + if len(p.Args) < 4 || p.Args[0] != "resume" || p.Args[1] != "-m" || p.Args[2] != "sonnet" { + t.Fatalf("codex resume args=%v", p.Args) + } +} + +func TestDiscoverHarnessSessionIDNewFile(t *testing.T) { + home := t.TempDir() + restore := SetPaths(Paths{SlingHome: home, UserHome: home, CWD: home}) + t.Cleanup(restore) + root := filepath.Join(home, ".codex", "sessions") + if err := os.MkdirAll(root, 0o755); err != nil { + t.Fatal(err) + } + old := filepath.Join(root, "old.jsonl") + if err := os.WriteFile(old, []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + before := snapshotHarnessFiles("codex") + newID := "019dd4bf-0929-7ea0-b227-1f51085e7d71" + if err := os.WriteFile(filepath.Join(root, newID+".jsonl"), []byte("y"), 0o644); err != nil { + t.Fatal(err) + } + got := discoverHarnessSessionID("codex", before) + if got != newID { + t.Fatalf("got %q want %q", got, newID) + } +} + +func TestAgentBinaryCursorIsAgent(t *testing.T) { + if agentBinary("cursor") != "cursor-agent" { + t.Fatal(agentBinary("cursor")) + } + if agentBinary("claude") != "claude" { + t.Fatal(agentBinary("claude")) + } +} + +func TestDoctorHonorsScope(t *testing.T) { + withTempHomeDir(t) + // Doctor with ScopeProject should not panic and should return a report. + r, err := Doctor(context.Background(), DoctorOptions{Scope: ScopeProject}) + if err != nil { + t.Fatal(err) + } + if r == nil { + t.Fatal("nil report") + } +} + +func TestEnsureAssistReadyRequiresProfile(t *testing.T) { + withTempHomeDir(t) + err := EnsureAssistReady() + if err == nil { + t.Fatal("expected error when assist not set up") + } + if !strings.Contains(err.Error(), "sling assist setup") { + t.Fatalf("error should point at setup: %v", err) + } +} diff --git a/core/sling/assist/browser.go b/core/sling/assist/browser.go new file mode 100644 index 000000000..de2d0af2b --- /dev/null +++ b/core/sling/assist/browser.go @@ -0,0 +1,319 @@ +package assist + +import ( + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + + "github.com/flarco/g" + "github.com/flarco/g/net" + "github.com/slingdata-io/sling-cli/core/env" +) + +// AgentBrowserVersion is the pinned CLI release. Override with AGENT_BROWSER_VERSION. +// Pin checked against https://github.com/vercel-labs/agent-browser/releases (v0.34.0, 2026-08-11). +const AgentBrowserVersion = "0.34.0" + +const agentBrowserGitHubBase = "https://github.com/vercel-labs/agent-browser/releases/download/v{version}/{asset}" + +const agentBrowserMCPName = "agent-browser" + +// agentBrowserTestDownloadURL replaces the GitHub asset URL in tests. +var agentBrowserTestDownloadURL string + +type agentBrowserInstall struct { + version string +} + +func newAgentBrowserInstall() *agentBrowserInstall { + return &agentBrowserInstall{version: agentBrowserVersion()} +} + +func agentBrowserVersion() string { + if val := strings.TrimSpace(os.Getenv("AGENT_BROWSER_VERSION")); val != "" { + return strings.TrimPrefix(val, "v") + } + return AgentBrowserVersion +} + +func agentBrowserBinName() string { + if runtime.GOOS == "windows" { + return "agent-browser.exe" + } + return "agent-browser" +} + +func (a *agentBrowserInstall) dest() string { + return filepath.Join(env.HomeBinDir(), "agent-browser", a.version) +} + +func (a *agentBrowserInstall) bundledPath() string { + return filepath.Join(a.dest(), agentBrowserBinName()) +} + +// BundledAgentBrowserPath is ~/.sling/bin/agent-browser//agent-browser[.exe]. +func BundledAgentBrowserPath() string { + return newAgentBrowserInstall().bundledPath() +} + +func (a *agentBrowserInstall) assetName(goos, goarch string) (string, error) { + var osName, arch string + switch goos { + case "darwin": + osName = "darwin" + case "linux": + osName = "linux" + case "windows": + osName = "win32" + default: + return "", g.Error("agent-browser is not available for %s/%s", goos, goarch) + } + switch goarch { + case "amd64": + arch = "x64" + case "arm64": + arch = "arm64" + default: + return "", g.Error("agent-browser is not available for %s/%s", goos, goarch) + } + if goos == "windows" && goarch == "arm64" { + // Upstream publishes win32-x64 only; x64 binary runs under emulation. + arch = "x64" + } + if goos == "linux" && linuxMuslPresent() { + return fmt.Sprintf("agent-browser-linux-musl-%s", arch), nil + } + name := fmt.Sprintf("agent-browser-%s-%s", osName, arch) + if goos == "windows" { + name += ".exe" + } + return name, nil +} + +// AgentBrowserAssetName is the GitHub asset for goos/goarch (pinned layout, not /latest). +func AgentBrowserAssetName(goos, goarch string) (string, error) { + return newAgentBrowserInstall().assetName(goos, goarch) +} + +func (a *agentBrowserInstall) downloadURL() (string, error) { + if agentBrowserTestDownloadURL != "" { + return agentBrowserTestDownloadURL, nil + } + asset, err := a.assetName(runtime.GOOS, runtime.GOARCH) + if err != nil { + return "", err + } + return g.R(agentBrowserGitHubBase, "version", a.version, "asset", asset), nil +} + +func (a *agentBrowserInstall) versionOK(binPath string) (bool, error) { + out, err := exec.Command(binPath, "--version").CombinedOutput() + if err != nil { + return false, g.Error(err, "could not get version for agent-browser: %s", strings.TrimSpace(string(out))) + } + s := strings.TrimSpace(string(out)) + return strings.Contains(s, a.version), nil +} + +// agentBrowserBin is the command written into MCP configs. +// Order: AGENT_BROWSER_PATH, $PATH, bundled binary, then the bare name. +func agentBrowserBin() string { + if envPath := strings.TrimSpace(os.Getenv("AGENT_BROWSER_PATH")); envPath != "" { + return envPath + } + if p, err := exec.LookPath("agent-browser"); err == nil { + return p + } + bundled := BundledAgentBrowserPath() + if g.PathExists(bundled) { + return bundled + } + return "agent-browser" +} + +func agentBrowserMCPEntry() map[string]any { + return map[string]any{ + "command": agentBrowserBin(), + "args": []any{"mcp", "--tools", "core"}, + } +} + +func opencodeAgentBrowserMCPEntry() map[string]any { + return map[string]any{ + "type": "local", + "command": []any{agentBrowserBin(), "mcp", "--tools", "core"}, + "enabled": true, + } +} + +func skipAgentBrowserDownload() bool { + return os.Getenv("AGENT_BROWSER_SKIP_DOWNLOAD") == "1" +} + +// EnsureBinAgentBrowser returns a usable agent-browser binary. +// Order: AGENT_BROWSER_PATH, $PATH, then a versioned download under ~/.sling/bin/agent-browser//. +func EnsureBinAgentBrowser() (binPath string, err error) { + return newAgentBrowserInstall().ensure() +} + +func (a *agentBrowserInstall) ensure() (binPath string, err error) { + if envPath := strings.TrimSpace(os.Getenv("AGENT_BROWSER_PATH")); envPath != "" { + if !g.PathExists(envPath) { + return "", g.Error("agent-browser binary not found: %s", envPath) + } + if stat, _ := os.Stat(envPath); stat != nil && stat.IsDir() { + return "", g.Error("AGENT_BROWSER_PATH provided is a directory, should be a file: %s", envPath) + } + return envPath, nil + } + + if p, err := exec.LookPath("agent-browser"); err == nil { + return p, nil + } + + if skipAgentBrowserDownload() { + return "agent-browser", nil + } + + folderPath := a.dest() + binPath = a.bundledPath() + found := g.PathExists(binPath) + if found { + ok, verr := a.versionOK(binPath) + if verr != nil { + found = false + } else { + found = ok + } + } + + if !found { + downloadURL, uerr := a.downloadURL() + if uerr != nil { + return "", uerr + } + + if err = os.MkdirAll(folderPath, 0755); err != nil { + return "", g.Error(err, "could not create agent-browser folder") + } + + tmpPath := binPath + ".download" + defer os.Remove(tmpPath) + + g.Info("downloading agent-browser %s for %s/%s", a.version, runtime.GOOS, runtime.GOARCH) + if err = net.DownloadFile(downloadURL, tmpPath); err != nil { + return "", g.Error(err, "unable to download agent-browser binary") + } + if err = os.Rename(tmpPath, binPath); err != nil { + return "", g.Error(err, "could not move agent-browser binary to %s", binPath) + } + if err = os.Chmod(binPath, 0755); err != nil { + return "", g.Error(err, "could not make agent-browser executable") + } + } + + ok, err := a.versionOK(binPath) + if err != nil { + return "", err + } + if !ok { + return "", g.Error("agent-browser at %s does not report version %s", binPath, a.version) + } + return binPath, nil +} + +func maybeEnsureAgentBrowser(opts InstallOptions) error { + return newAgentBrowserInstall().maybeEnsure(opts) +} + +func (a *agentBrowserInstall) maybeEnsure(opts InstallOptions) error { + if skipAgentBrowserDownload() { + return nil + } + bin, err := a.ensure() + if err != nil { + return err + } + return a.maybeInstallChrome(bin, opts) +} + +func (a *agentBrowserInstall) chromeLikelyPresent() bool { + if g.PathExists(filepath.Join(userHome(), ".agent-browser", "browsers")) { + return true + } + for _, name := range []string{"google-chrome", "google-chrome-stable", "chromium", "chromium-browser", "chrome"} { + if _, err := exec.LookPath(name); err == nil { + return true + } + } + switch runtime.GOOS { + case "darwin": + return g.PathExists("/Applications/Google Chrome.app") + case "windows": + for _, p := range []string{ + filepath.Join(os.Getenv("PROGRAMFILES"), "Google", "Chrome", "Application", "chrome.exe"), + filepath.Join(os.Getenv("PROGRAMFILES(X86)"), "Google", "Chrome", "Application", "chrome.exe"), + } { + if p != "" && g.PathExists(p) { + return true + } + } + } + return false +} + +func (a *agentBrowserInstall) maybeInstallChrome(bin string, opts InstallOptions) error { + if os.Getenv("AGENT_BROWSER_SKIP_CHROME") == "1" { + return nil + } + if a.chromeLikelyPresent() { + return nil + } + yes := os.Getenv("SLING_AGENT_BROWSER_YES") == "1" + if opts.NonInteractive && !yes { + g.Info("Chrome for Testing is not installed. After setup, run: %s install", bin) + return nil + } + if !yes && !env.IsInteractiveTerminal() { + g.Info("Chrome for Testing is not installed. After setup, run: %s install", bin) + return nil + } + g.Info("downloading Chrome for Testing via agent-browser install") + cmd := exec.Command(bin, "install") + cmd.Stdout = os.Stderr + cmd.Stderr = os.Stderr + if err := cmd.Run(); err != nil { + g.Warn("agent-browser install failed: %v (MCP is still wired; run `%s install` later)", err, bin) + } + return nil +} + +func tomlQuote(s string) string { + return strconv.Quote(s) +} + +func tomlStringArray(ss []string) string { + parts := make([]string, len(ss)) + for i, s := range ss { + parts[i] = tomlQuote(s) + } + return "[" + strings.Join(parts, ", ") + "]" +} + +func writeJSONMCP(path, serversKey string) error { + if err := setJSONPath(path, serversKey+".sling", slingMCPEntry()); err != nil { + return err + } + return setJSONPath(path, serversKey+".agent-browser", agentBrowserMCPEntry()) +} + +func removeJSONMCP(path, serversKey string) error { + if err := deleteJSONPath(path, serversKey+".sling"); err != nil { + return err + } + return deleteJSONPath(path, serversKey+".agent-browser") +} diff --git a/core/sling/assist/browser_test.go b/core/sling/assist/browser_test.go new file mode 100644 index 000000000..83c410550 --- /dev/null +++ b/core/sling/assist/browser_test.go @@ -0,0 +1,182 @@ +package assist + +import ( + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "runtime" + "strings" + "sync/atomic" + "testing" +) + +func isolateAgentBrowserEnv(t *testing.T) { + t.Helper() + home := withTempHomeDir(t) + bin := filepath.Join(home, "empty-bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin) + t.Setenv("AGENT_BROWSER_PATH", "") + t.Setenv("AGENT_BROWSER_VERSION", AgentBrowserVersion) + t.Setenv("AGENT_BROWSER_SKIP_DOWNLOAD", "") + t.Setenv("AGENT_BROWSER_SKIP_CHROME", "1") + t.Cleanup(func() { agentBrowserTestDownloadURL = "" }) +} + +func stubAgentBrowserScript(version string) string { + return "#!/bin/sh\n" + + "if [ \"$1\" = \"--version\" ]; then echo \"" + version + "\"; exit 0; fi\n" + + "if [ \"$1\" = \"install\" ]; then echo chrome-stub; exit 0; fi\n" + + "echo stub\n" +} + +func serveAgentBrowserBin(t *testing.T, version string) (*httptest.Server, *atomic.Int32) { + t.Helper() + payload := []byte(stubAgentBrowserScript(version)) + hits := &atomic.Int32{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits.Add(1) + _, _ = io.ReadAll(r.Body) + w.Header().Set("Content-Type", "application/octet-stream") + _, _ = w.Write(payload) + })) + t.Cleanup(srv.Close) + return srv, hits +} + +func TestAgentBrowserAssetNamePinnedPlatforms(t *testing.T) { + cases := map[string]string{ + "darwin/arm64": "agent-browser-darwin-arm64", + "darwin/amd64": "agent-browser-darwin-x64", + "linux/arm64": "agent-browser-linux-arm64", + "linux/amd64": "agent-browser-linux-x64", + "windows/amd64": "agent-browser-win32-x64.exe", + "windows/arm64": "agent-browser-win32-x64.exe", + } + for plat, want := range cases { + if strings.HasPrefix(plat, "linux/") && linuxMuslPresent() { + continue + } + parts := strings.Split(plat, "/") + got, err := AgentBrowserAssetName(parts[0], parts[1]) + if err != nil { + t.Fatalf("%s: %v", plat, err) + } + if got != want { + t.Errorf("%s: got %s want %s", plat, got, want) + } + } +} + +func TestAgentBrowserAssetNameUnsupported(t *testing.T) { + if _, err := AgentBrowserAssetName("plan9", "amd64"); err == nil { + t.Fatal("expected error for plan9") + } +} + +func TestEnsureBinAgentBrowserDownloadsOnce(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("stub is a POSIX shell script") + } + isolateAgentBrowserEnv(t) + srv, hits := serveAgentBrowserBin(t, AgentBrowserVersion) + agentBrowserTestDownloadURL = srv.URL + + p1, err := EnsureBinAgentBrowser() + if err != nil { + t.Fatal(err) + } + if !strings.HasSuffix(p1, agentBrowserBinName()) { + t.Fatalf("path=%s", p1) + } + p2, err := EnsureBinAgentBrowser() + if err != nil { + t.Fatal(err) + } + if p1 != p2 { + t.Fatalf("path changed %s -> %s", p1, p2) + } + if hits.Load() != 1 { + t.Fatalf("downloads=%d want 1", hits.Load()) + } +} + +func TestEnsureBinAgentBrowserPathEnv(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("stub is a POSIX shell script") + } + isolateAgentBrowserEnv(t) + dir := t.TempDir() + stub := filepath.Join(dir, "agent-browser") + if err := os.WriteFile(stub, []byte(stubAgentBrowserScript(AgentBrowserVersion)), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("AGENT_BROWSER_PATH", stub) + got, err := EnsureBinAgentBrowser() + if err != nil { + t.Fatal(err) + } + if got != stub { + t.Fatalf("got %s want %s", got, stub) + } +} + +func TestEnsureBinAgentBrowserSkipDownload(t *testing.T) { + isolateAgentBrowserEnv(t) + t.Setenv("AGENT_BROWSER_SKIP_DOWNLOAD", "1") + got, err := EnsureBinAgentBrowser() + if err != nil { + t.Fatal(err) + } + if got != "agent-browser" { + t.Fatalf("got %s", got) + } +} + +func TestAgentBrowserSkillEmbedded(t *testing.T) { + names := listSkillNames() + found := false + for _, n := range names { + if n == "agent-browser" { + found = true + break + } + } + if !found { + t.Fatalf("listSkillNames missing agent-browser: %v", names) + } + stub, err := SkillsFS.ReadFile("skills/agent-browser/SKILL.md") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(stub), "CORE.md") { + t.Fatal("SKILL.md must point at CORE.md") + } + if !strings.Contains(string(stub), "sling assist setup") { + t.Fatal("SKILL.md must tell the agent about sling assist setup") + } + core, err := SkillsFS.ReadFile("skills/agent-browser/CORE.md") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(core), "The core loop") { + t.Fatal("CORE.md missing official core loop") + } +} + +func TestAgentBrowserMCPEntryUsesResolvedBin(t *testing.T) { + isolateAgentBrowserEnv(t) + t.Setenv("AGENT_BROWSER_PATH", "/tmp/custom-agent-browser") + entry := agentBrowserMCPEntry() + if entry["command"] != "/tmp/custom-agent-browser" { + t.Fatalf("command=%v", entry["command"]) + } + args, _ := entry["args"].([]any) + if len(args) != 3 || args[0] != "mcp" || args[1] != "--tools" || args[2] != "core" { + t.Fatalf("args=%v", args) + } +} diff --git a/core/sling/assist/clients.go b/core/sling/assist/clients.go new file mode 100644 index 000000000..6870e2a8d --- /dev/null +++ b/core/sling/assist/clients.go @@ -0,0 +1,1404 @@ +package assist + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io/fs" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "sync" + "time" + + "github.com/flarco/g" + "gopkg.in/yaml.v3" +) + +// ClientKind distinguishes launchable CLI agents from install-only UI surfaces. +// Only Kind() == CLIAgent clients are picked by the assist launcher. +type ClientKind int + +const ( + KindCLIAgent ClientKind = iota + KindUISurface +) + +// Client is the per-tool install adapter contract. Each adapter takes care of +// its own redirect/translation of the canonical skills bundle and its own MCP +// config shape. Install/uninstall/doctor iterate over Detected() clients. +// +// Methods that touch the filesystem accept context.Context for cancellation +// (submission/network work will share this seam). Implementations should +// honor ctx.Err() at entry when doing I/O. +type Client interface { + Name() string + Kind() ClientKind + Detect() bool + WriteSkills(ctx context.Context, skillNames []string, scope Scope) error + RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error + WriteMCP(ctx context.Context, scope Scope) error + RemoveMCP(ctx context.Context, scope Scope) error + // CheckSkills returns one typed result per skill (no glyph prefixes). + CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult + // CheckMCP returns a typed MCP wiring status. + CheckMCP(ctx context.Context, scope Scope) CheckResult + // AuthState reports offline credential-file / env-key presence. Never launches a binary. + AuthState() AuthStatus +} + +// Scope is `--scope user` (default) or `--scope project`. +type Scope int + +const ( + ScopeUser Scope = iota + ScopeProject +) + +// AllClients returns the canonical ordered list of supported clients. +// Order matters for `agent: auto` resolution and for stable output. +func AllClients() []Client { + return []Client{ + &claudeClient{}, + &codexClient{}, + &geminiClient{}, + &cursorClient{}, + &opencodeClient{}, + &piClient{}, + &grokClient{}, + &vscodeClient{}, + } +} + +// CLIAgents returns only the launchable CLI-agent clients (excludes vscode). +func CLIAgents() []Client { + out := []Client{} + for _, c := range AllClients() { + if c.Kind() == KindCLIAgent { + out = append(out, c) + } + } + return out +} + +// DetectedClients returns the subset of AllClients() whose Detect() returned true. +func DetectedClients() []Client { + out := []Client{} + for _, c := range AllClients() { + if c.Detect() { + out = append(out, c) + } + } + return out +} + +// LookupClient finds a client by name; returns nil if no match. +func LookupClient(name string) Client { + name = strings.ToLower(strings.TrimSpace(name)) + for _, c := range AllClients() { + if c.Name() == name { + return c + } + } + return nil +} + +// canonicalSkillPath returns the absolute path to a skill's SKILL.md inside +// the canonical bundle. +func canonicalSkillPath(skill string) string { + return filepath.Join(CanonicalSkillsDir(), skill, "SKILL.md") +} + +// writeRedirectFile writes a 1-line `@` redirect file. +// If symlinks are preferred (Unix), we still write a stub file so doctor's +// "resolves to canonical" check is uniform across platforms. +func writeRedirectFile(redirectPath, canonicalPath string) error { + if err := os.MkdirAll(filepath.Dir(redirectPath), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(redirectPath)) + } + body := fmt.Sprintf("@%s\n", canonicalPath) + return os.WriteFile(redirectPath, []byte(body), 0o644) +} + +// readRedirectTarget parses a 1-line `@` redirect; returns "" if the +// file isn't a redirect (or isn't readable). +func readRedirectTarget(redirectPath string) string { + data, err := os.ReadFile(redirectPath) + if err != nil { + return "" + } + line := strings.TrimSpace(string(data)) + if !strings.HasPrefix(line, "@") { + return "" + } + // drop the @, take only the first line if there are several + target := strings.TrimSpace(strings.SplitN(line[1:], "\n", 2)[0]) + return target +} + +// listSkillNames walks the embedded skills FS and returns canonical names +// (one per top-level directory). Skill bundles with multiple files (like +// sling-api-specs) still come back as one name. +func listSkillNames() []string { + entries, err := SkillsFS.ReadDir("skills") + if err != nil { + return nil + } + out := []string{} + for _, e := range entries { + if e.IsDir() { + out = append(out, e.Name()) + } + } + return out +} + +// writeCanonicalBundle copies the embedded skills tree to ~/.agents/skills/. +// Existing files are overwritten — skills are Sling-owned (see design doc). +// Stale files removed from the embed are deleted from disk. +func writeCanonicalBundle(skillNames []string) error { + for _, name := range skillNames { + if _, err := syncCanonicalSkill(name); err != nil { + return err + } + } + return nil +} + +// syncCanonicalSkill writes one skill from the embed FS onto the canonical +// disk tree and removes on-disk files under that skill that are no longer +// embedded. Shared by writeCanonicalBundle and AutoRefresh. +func syncCanonicalSkill(name string) (changed bool, err error) { + root := CanonicalSkillsDir() + embeddedRoot := filepath.ToSlash(filepath.Join("skills", name)) + wantFiles := map[string]bool{} // path relative to root, slash-separated + + err = fs.WalkDir(SkillsFS, embeddedRoot, func(p string, d fs.DirEntry, werr error) error { + if werr != nil { + return werr + } + if d.IsDir() { + return nil + } + rel := strings.TrimPrefix(p, "skills/") + wantFiles[rel] = true + dst := filepath.Join(root, filepath.FromSlash(rel)) + want, rerr := SkillsFS.ReadFile(p) + if rerr != nil { + return g.Error(rerr, "read embedded %s", p) + } + got, _ := os.ReadFile(dst) + if bytes.Equal(want, got) { + return nil + } + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(dst)) + } + if err := os.WriteFile(dst, want, 0o644); err != nil { + return g.Error(err, "write %s", dst) + } + changed = true + return nil + }) + if err != nil { + return changed, err + } + + // Prune files on disk that are no longer in the embed (stale supporting docs). + skillDir := filepath.Join(root, name) + if g.PathExists(skillDir) { + _ = filepath.WalkDir(skillDir, func(p string, d fs.DirEntry, werr error) error { + if werr != nil || d.IsDir() { + return werr + } + rel, rerr := filepath.Rel(root, p) + if rerr != nil { + return nil + } + rel = filepath.ToSlash(rel) + if !wantFiles[rel] { + if rmErr := os.Remove(p); rmErr == nil { + changed = true + } + } + return nil + }) + } + return changed, nil +} + +// skillMatchesEmbedded compares every embedded file for a skill against disk. +// Returns ok=false with a short detail when any file is missing or drifted. +func skillMatchesEmbedded(name string) (ok bool, detail string, err error) { + root := CanonicalSkillsDir() + embeddedRoot := filepath.ToSlash(filepath.Join("skills", name)) + var mismatches []string + err = fs.WalkDir(SkillsFS, embeddedRoot, func(p string, d fs.DirEntry, werr error) error { + if werr != nil { + return werr + } + if d.IsDir() { + return nil + } + rel := strings.TrimPrefix(p, "skills/") + dst := filepath.Join(root, filepath.FromSlash(rel)) + want, rerr := SkillsFS.ReadFile(p) + if rerr != nil { + return rerr + } + got, rerr := os.ReadFile(dst) + if rerr != nil { + mismatches = append(mismatches, rel+": missing") + return nil + } + if !bytes.Equal(want, got) { + mismatches = append(mismatches, rel+": drifted") + } + return nil + }) + if err != nil { + return false, "", err + } + if len(mismatches) > 0 { + return false, strings.Join(mismatches, "; "), nil + } + return true, "", nil +} + +// removeCanonicalBundle removes ~/.agents/skills/sling-* directories — only +// our own skills, never anyone else's. +func removeCanonicalBundle(skillNames []string) error { + root := CanonicalSkillsDir() + var errs []string + for _, name := range skillNames { + p := filepath.Join(root, name) + if err := os.RemoveAll(p); err != nil { + errs = append(errs, fmt.Sprintf("%s: %v", name, err)) + } + } + if len(errs) > 0 { + return g.Error("remove canonical skills: %s", strings.Join(errs, "; ")) + } + return nil +} + +// slingMCPEntry is the canonical Sling MCP server descriptor written into +// every client's MCP config. Centralized so each adapter renders the same +// shape and a future schema bump is one-line. +func slingMCPEntry() map[string]any { + return map[string]any{ + "command": "sling", + "args": []any{"serve", "mcp"}, + } +} + +// checkCanonicalSkills is the CheckSkills implementation shared by every +// client that reads ~/.agents/skills/ natively (codex, gemini, opencode, pi, +// grok) — there's no per-client redirect to verify, only the bundle itself. +func checkCanonicalSkills(skillNames []string) []CheckResult { + out := []CheckResult{} + for _, name := range skillNames { + if g.PathExists(canonicalSkillPath(name)) { + out = append(out, checkSkill(CellOK, name, "canonical")) + } else { + out = append(out, checkSkill(CellFail, name, "missing in canonical bundle")) + } + } + return out +} + +// checkMCPServersKey reports whether has sling and agent-browser +// entries under the given top-level object key ("mcpServers", "servers", "mcp"). +func checkMCPServersKey(path, key string) CheckResult { + doc, err := jsonReadOrEmpty(path) + if err != nil { + return checkFail(err.Error()) + } + servers, _ := doc[key].(map[string]any) + if servers == nil { + return checkFail("no " + key + " block") + } + if _, ok := servers["sling"]; !ok { + return checkFail("sling MCP entry missing") + } + if _, ok := servers[agentBrowserMCPName]; !ok { + return checkFail("agent-browser MCP entry missing") + } + return checkOK("sling and agent-browser MCP present") +} + +// ---- claude ---- + +type claudeClient struct{} + +func (c *claudeClient) Name() string { return "claude" } +func (c *claudeClient) Kind() ClientKind { return KindCLIAgent } +func (c *claudeClient) Detect() bool { return commandOnPath("claude") } + +func (c *claudeClient) skillsRoot(scope Scope) string { + if scope == ScopeProject { + return projectPath(".claude", "skills") + } + return filepath.Join(userHome(), ".claude", "skills") +} + +func (c *claudeClient) mcpPath(scope Scope) string { + // Project-scoped MCP lives in .mcp.json at the project root (Claude Code + // convention). User/local scope stays in ~/.claude.json. + if scope == ScopeProject { + return projectPath(".mcp.json") + } + return filepath.Join(userHome(), ".claude.json") +} + +func (c *claudeClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + root := c.skillsRoot(scope) + for _, name := range skillNames { + canonical := canonicalSkillPath(name) + redirect := filepath.Join(root, name, "SKILL.md") + if err := writeRedirectFile(redirect, canonical); err != nil { + return err + } + } + return nil +} + +func (c *claudeClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + root := c.skillsRoot(scope) + for _, name := range skillNames { + _ = os.RemoveAll(filepath.Join(root, name)) + } + return nil +} + +func (c *claudeClient) WriteMCP(ctx context.Context, scope Scope) error { + return writeJSONMCP(c.mcpPath(scope), "mcpServers") +} + +func (c *claudeClient) RemoveMCP(ctx context.Context, scope Scope) error { + return removeJSONMCP(c.mcpPath(scope), "mcpServers") +} + +func (c *claudeClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + out := []CheckResult{} + root := c.skillsRoot(scope) + for _, name := range skillNames { + redirect := filepath.Join(root, name, "SKILL.md") + canonical := canonicalSkillPath(name) + target := readRedirectTarget(redirect) + switch { + case target == "": + out = append(out, checkSkill(CellFail, name, "redirect missing")) + case target != canonical: + out = append(out, checkSkill(CellFail, name, fmt.Sprintf("points at %s (expected %s)", target, canonical))) + case !g.PathExists(canonical): + out = append(out, checkSkill(CellFail, name, "redirect target not found")) + default: + out = append(out, checkSkill(CellOK, name, "")) + } + } + return out +} + +func (c *claudeClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + return checkMCPServersKey(c.mcpPath(scope), "mcpServers") +} + +// ---- codex (reads ~/.agents/skills/ natively) ---- + +type codexClient struct{} + +func (c *codexClient) Name() string { return "codex" } +func (c *codexClient) Kind() ClientKind { return KindCLIAgent } +func (c *codexClient) Detect() bool { return commandOnPath("codex") } + +func (c *codexClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + // Codex reads ~/.agents/skills/ natively — nothing extra to do. + return nil +} +func (c *codexClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + return nil +} + +func (c *codexClient) codexConfigPath(scope Scope) string { + if scope == ScopeProject { + return projectPath(".codex", "config.toml") + } + return filepath.Join(userHome(), ".codex", "config.toml") +} + +func (c *codexClient) WriteMCP(ctx context.Context, scope Scope) error { + path := c.codexConfigPath(scope) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(path)) + } + if err := backupBeforeEdit(path); err != nil { + return err + } + body, _ := os.ReadFile(path) + merged := upsertCodexMCP(string(body)) + return writeBytesPreserveMode(path, []byte(merged), 0o600) +} + +func (c *codexClient) RemoveMCP(ctx context.Context, scope Scope) error { + path := c.codexConfigPath(scope) + body, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return err + } + if err := backupBeforeEdit(path); err != nil { + return err + } + out := removeCodexMCP(string(body)) + return writeBytesPreserveMode(path, []byte(out), 0o600) +} + +func (c *codexClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + return checkCanonicalSkills(skillNames) +} + +func (c *codexClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + body, err := os.ReadFile(c.codexConfigPath(scope)) + if err != nil { + return checkFail("config.toml not present") + } + if hasCodexMCPSection(string(body)) && hasCodexNamedMCPSection(string(body), agentBrowserMCPName) { + return checkOK("sling and agent-browser MCP present") + } + if !hasCodexMCPSection(string(body)) { + return checkFail("sling MCP entry missing") + } + return checkFail("agent-browser MCP entry missing") +} + +const codexMCPHeader = "[mcp_servers.sling]" + +func codexMCPHeaderNamed(name string) string { + return "[mcp_servers." + name + "]" +} + +// isCodexSlingTable reports whether a trimmed TOML header line is our sling +// table or a nested subtable ([mcp_servers.sling.env], etc.). +func isCodexSlingTable(trim string) bool { + return isCodexNamedMCPTable(trim, "sling") +} + +func isCodexNamedMCPTable(trim, name string) bool { + header := codexMCPHeaderNamed(name) + return trim == header || strings.HasPrefix(trim, header[:len(header)-1]+".") +} + +// hasCodexMCPSection reports whether body has a real [mcp_servers.sling] +// table header (line-anchored), not a mention inside a string or comment. +func hasCodexMCPSection(body string) bool { + return hasCodexNamedMCPSection(body, "sling") +} + +func hasCodexNamedMCPSection(body, name string) bool { + header := codexMCPHeaderNamed(name) + for _, line := range strings.Split(body, "\n") { + if strings.TrimSpace(line) == header { + return true + } + } + return false +} + +func upsertCodexNamedMCP(body, name, command string, args []string) string { + header := codexMCPHeaderNamed(name) + block := header + "\ncommand = " + tomlQuote(command) + "\nargs = " + tomlStringArray(args) + "\n" + if !hasCodexNamedMCPSection(body, name) { + if body != "" && !strings.HasSuffix(body, "\n") { + body += "\n" + } + return body + "\n" + block + } + lines := strings.Split(body, "\n") + out := []string{} + skipping := false + injected := false + for _, line := range lines { + trim := strings.TrimSpace(line) + if trim == header { + out = append(out, strings.TrimRight(block, "\n")) + skipping = true + injected = true + continue + } + if skipping { + if strings.HasPrefix(trim, "[") { + if isCodexNamedMCPTable(trim, name) { + continue + } + skipping = false + } else { + continue + } + } + out = append(out, line) + } + if !injected { + out = append(out, "", strings.TrimRight(block, "\n")) + } + return strings.Join(out, "\n") +} + +// upsertCodexMCP injects/replaces sling and agent-browser MCP tables. +func upsertCodexMCP(body string) string { + body = upsertCodexNamedMCP(body, "sling", "sling", []string{"serve", "mcp"}) + return upsertCodexNamedMCP(body, agentBrowserMCPName, agentBrowserBin(), []string{"mcp", "--tools", "core"}) +} + +func removeCodexNamedMCP(body, name string) string { + lines := strings.Split(body, "\n") + out := []string{} + skipping := false + for _, line := range lines { + trim := strings.TrimSpace(line) + if isCodexNamedMCPTable(trim, name) { + skipping = true + continue + } + if skipping { + if strings.HasPrefix(trim, "[") { + if isCodexNamedMCPTable(trim, name) { + continue + } + skipping = false + } else { + continue + } + } + out = append(out, line) + } + return strings.Join(out, "\n") +} + +// removeCodexMCP strips sling and agent-browser MCP tables (and nested subtables). +func removeCodexMCP(body string) string { + body = removeCodexNamedMCP(body, "sling") + return removeCodexNamedMCP(body, agentBrowserMCPName) +} + +// ---- gemini (reads ~/.agents/skills/ as alias) ---- + +type geminiClient struct{} + +func (c *geminiClient) Name() string { return "gemini" } +func (c *geminiClient) Kind() ClientKind { return KindCLIAgent } +func (c *geminiClient) Detect() bool { return commandOnPath("gemini") } +func (c *geminiClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + return nil +} +func (c *geminiClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + return nil +} + +func (c *geminiClient) settingsPath(scope Scope) string { + if scope == ScopeProject { + return projectPath(".gemini", "settings.json") + } + return filepath.Join(userHome(), ".gemini", "settings.json") +} + +func (c *geminiClient) WriteMCP(ctx context.Context, scope Scope) error { + return writeJSONMCP(c.settingsPath(scope), "mcpServers") +} + +func (c *geminiClient) RemoveMCP(ctx context.Context, scope Scope) error { + return removeJSONMCP(c.settingsPath(scope), "mcpServers") +} + +func (c *geminiClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + return checkCanonicalSkills(skillNames) +} + +func (c *geminiClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + return checkMCPServersKey(c.settingsPath(scope), "mcpServers") +} + +// ---- cursor (translates to .mdc) ---- + +type cursorClient struct{} + +func (c *cursorClient) Name() string { return "cursor" } +func (c *cursorClient) Kind() ClientKind { return KindCLIAgent } +func (c *cursorClient) Detect() bool { return commandOnPath(agentBinary("cursor")) } + +func (c *cursorClient) rulesDir(scope Scope) string { + if scope == ScopeProject { + return projectPath(".cursor", "rules") + } + return filepath.Join(userHome(), ".cursor", "rules") +} + +func (c *cursorClient) mcpPath(scope Scope) string { + if scope == ScopeProject { + return projectPath(".cursor", "mcp.json") + } + return filepath.Join(userHome(), ".cursor", "mcp.json") +} + +// translateSkillToMDC produces a cursor `.mdc` file body from a SKILL.md. +// Cursor's `.mdc` shape is YAML frontmatter (with `description`, +// `globs`, `alwaysApply`) + Markdown body — same shape as SKILL.md, so we +// pass through with a lightly-rewritten frontmatter. +func translateSkillToMDC(skillBody []byte) []byte { + body := string(skillBody) + // SKILL.md frontmatter looks like: + // --- + // name: sling + // description: ... + // --- + // Cursor's .mdc wants: + // --- + // description: ... + // alwaysApply: false + // --- + if !strings.HasPrefix(body, "---") { + return []byte("---\nalwaysApply: false\n---\n\n" + body) + } + parts := strings.SplitN(body, "\n---", 2) + if len(parts) != 2 { + return []byte("---\nalwaysApply: false\n---\n\n" + body) + } + header := strings.TrimPrefix(parts[0], "---\n") + rest := strings.TrimPrefix(parts[1], "\n") + + var fm map[string]any + _ = yaml.Unmarshal([]byte(header), &fm) + if fm == nil { + fm = map[string]any{} + } + out := map[string]any{ + "description": fm["description"], + "alwaysApply": false, + } + yamlBytes, _ := yaml.Marshal(out) + return []byte("---\n" + string(yamlBytes) + "---\n\n" + rest) +} + +func (c *cursorClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + dir := c.rulesDir(scope) + if err := os.MkdirAll(dir, 0o755); err != nil { + return g.Error(err, "mkdir %s", dir) + } + for _, name := range skillNames { + src := canonicalSkillPath(name) + body, err := os.ReadFile(src) + if err != nil { + return g.Error(err, "read %s", src) + } + mdc := translateSkillToMDC(body) + dst := filepath.Join(dir, name+".mdc") + if err := os.WriteFile(dst, mdc, 0o644); err != nil { + return g.Error(err, "write %s", dst) + } + } + return nil +} + +func (c *cursorClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + dir := c.rulesDir(scope) + for _, name := range skillNames { + _ = os.Remove(filepath.Join(dir, name+".mdc")) + } + return nil +} + +func (c *cursorClient) WriteMCP(ctx context.Context, scope Scope) error { + return writeJSONMCP(c.mcpPath(scope), "mcpServers") +} + +func (c *cursorClient) RemoveMCP(ctx context.Context, scope Scope) error { + return removeJSONMCP(c.mcpPath(scope), "mcpServers") +} + +func (c *cursorClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + out := []CheckResult{} + for _, name := range skillNames { + mdc := filepath.Join(c.rulesDir(scope), name+".mdc") + if g.PathExists(mdc) { + out = append(out, checkSkill(CellOK, name, "mdc present")) + } else { + out = append(out, checkSkill(CellFail, name, "mdc missing")) + } + } + return out +} + +func (c *cursorClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + return checkMCPServersKey(c.mcpPath(scope), "mcpServers") +} + +// ---- opencode (reads ~/.agents/skills/ natively) ---- + +type opencodeClient struct{} + +func (c *opencodeClient) Name() string { return "opencode" } +func (c *opencodeClient) Kind() ClientKind { return KindCLIAgent } +func (c *opencodeClient) Detect() bool { + return commandOnPath("opencode") || g.PathExists(BundledOpenCodePath()) +} + +// opencodeConfigDir is opencode's global config dir: $XDG_CONFIG_HOME/opencode +// falling back to ~/.config/opencode. +func opencodeConfigDir() string { + if xdg := os.Getenv("XDG_CONFIG_HOME"); xdg != "" { + return filepath.Join(xdg, "opencode") + } + return filepath.Join(userHome(), ".config", "opencode") +} + +func (c *opencodeClient) configPath(scope Scope) string { + if scope == ScopeProject { + return projectPath("opencode.json") + } + return filepath.Join(opencodeConfigDir(), "opencode.json") +} + +func (c *opencodeClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + // opencode globs ~/.agents/skills//SKILL.md natively. + return nil +} +func (c *opencodeClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + return nil +} + +// opencodeMCPEntry is opencode's own MCP shape: `type: local` plus a single +// argv array (not command/args like everyone else). +func opencodeMCPEntry() map[string]any { + return map[string]any{ + "type": "local", + "command": []any{"sling", "serve", "mcp"}, + "enabled": true, + } +} + +func (c *opencodeClient) WriteMCP(ctx context.Context, scope Scope) error { + if err := setJSONPath(c.configPath(scope), "mcp.sling", opencodeMCPEntry()); err != nil { + return err + } + return setJSONPath(c.configPath(scope), "mcp.agent-browser", opencodeAgentBrowserMCPEntry()) +} + +func (c *opencodeClient) RemoveMCP(ctx context.Context, scope Scope) error { + if err := deleteJSONPath(c.configPath(scope), "mcp.sling"); err != nil { + return err + } + return deleteJSONPath(c.configPath(scope), "mcp.agent-browser") +} + +func (c *opencodeClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + return checkCanonicalSkills(skillNames) +} + +func (c *opencodeClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + return checkMCPServersKey(c.configPath(scope), "mcp") +} + +// ---- pi (reads ~/.agents/skills/ natively) ---- + +type piClient struct{} + +func (c *piClient) Name() string { return "pi" } +func (c *piClient) Kind() ClientKind { return KindCLIAgent } +func (c *piClient) Detect() bool { return commandOnPath("pi") } + +// piAgentDir is pi's agent config dir — $PI_CODING_AGENT_DIR when set, +// otherwise ~/.pi/agent. +func piAgentDir() string { + if d := os.Getenv("PI_CODING_AGENT_DIR"); d != "" { + return d + } + return filepath.Join(userHome(), ".pi", "agent") +} + +// mcpPath: pi keeps MCP servers in a dedicated mcp.json, separate from +// settings.json. Project scope is .pi/mcp.json at the repo root. +func (c *piClient) mcpPath(scope Scope) string { + if scope == ScopeProject { + return projectPath(".pi", "mcp.json") + } + return filepath.Join(piAgentDir(), "mcp.json") +} + +func (c *piClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + // pi discovers ~/.agents/skills//SKILL.md natively. + return nil +} +func (c *piClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + return nil +} + +func (c *piClient) WriteMCP(ctx context.Context, scope Scope) error { + return writeJSONMCP(c.mcpPath(scope), "mcpServers") +} + +func (c *piClient) RemoveMCP(ctx context.Context, scope Scope) error { + return removeJSONMCP(c.mcpPath(scope), "mcpServers") +} + +func (c *piClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + return checkCanonicalSkills(skillNames) +} + +func (c *piClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + return checkMCPServersKey(c.mcpPath(scope), "mcpServers") +} + +// ---- grok (TOML config, reads ~/.agents/skills/ for AGENTS.md compat) ---- + +type grokClient struct{} + +func (c *grokClient) Name() string { return "grok" } +func (c *grokClient) Kind() ClientKind { return KindCLIAgent } +func (c *grokClient) Detect() bool { return commandOnPath("grok") } + +func (c *grokClient) configPath(scope Scope) string { + if scope == ScopeProject { + return projectPath(".grok", "config.toml") + } + return filepath.Join(userHome(), ".grok", "config.toml") +} + +func (c *grokClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + // grok discovers ~/.agents/skills/ as part of its AGENTS.md compatibility. + return nil +} +func (c *grokClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + return nil +} + +func (c *grokClient) WriteMCP(ctx context.Context, scope Scope) error { + path := c.configPath(scope) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(path)) + } + if err := backupBeforeEdit(path); err != nil { + return err + } + body, _ := os.ReadFile(path) + // grok's config.toml uses the same [mcp_servers.] shape as codex. + merged := upsertCodexMCP(string(body)) + return writeBytesPreserveMode(path, []byte(merged), 0o600) +} + +func (c *grokClient) RemoveMCP(ctx context.Context, scope Scope) error { + path := c.configPath(scope) + body, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return err + } + if err := backupBeforeEdit(path); err != nil { + return err + } + return writeBytesPreserveMode(path, []byte(removeCodexMCP(string(body))), 0o600) +} + +func (c *grokClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + return checkCanonicalSkills(skillNames) +} + +func (c *grokClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + body, err := os.ReadFile(c.configPath(scope)) + if err != nil { + return checkFail("config.toml not present") + } + if hasCodexMCPSection(string(body)) && hasCodexNamedMCPSection(string(body), agentBrowserMCPName) { + return checkOK("sling and agent-browser MCP present") + } + if !hasCodexMCPSection(string(body)) { + return checkFail("sling MCP entry missing") + } + return checkFail("agent-browser MCP entry missing") +} + +// ---- vscode (UI surface, install-only) ---- + +type vscodeClient struct{} + +func (c *vscodeClient) Name() string { return "vscode" } +func (c *vscodeClient) Kind() ClientKind { return KindUISurface } + +// vscodeUserDir returns the platform-specific VS Code user-config dir, or "". +// We probe Code, Code-Insiders, and VSCodium in that order. +func vscodeUserDir() string { + for _, name := range []string{"Code", "Code - Insiders", "VSCodium"} { + if p := vscodeUserDirNamed(name); p != "" { + return p + } + } + return "" +} + +func vscodeUserDirNamed(productName string) string { + var base string + switch runtime.GOOS { + case "darwin": + base = filepath.Join(userHome(), "Library", "Application Support", productName, "User") + case "windows": + base = filepath.Join(os.Getenv("APPDATA"), productName, "User") + default: + base = filepath.Join(userHome(), ".config", productName, "User") + } + if g.PathExists(base) { + return base + } + return "" +} + +func (c *vscodeClient) Detect() bool { return vscodeUserDir() != "" } + +func (c *vscodeClient) settingsPath() string { + dir := vscodeUserDir() + if dir == "" { + return "" + } + return filepath.Join(dir, "settings.json") +} + +func (c *vscodeClient) WriteSkills(ctx context.Context, skillNames []string, scope Scope) error { + path := c.settingsPath() + if path == "" { + return nil + } + // VS Code uses literal dotted keys, so the path must escape the dots. + const key = `chat\.instructionsFilesLocations` + canonical := CanonicalSkillsDir() + _, body, err := jsonReadRaw(path) + if err != nil { + return err + } + // Already present → no-op (avoid an unnecessary backup churn + write). + for _, e := range gjsonGetArrayStrings(body, key) { + if e == canonical { + return nil + } + } + // Route through setJSONPath so we get the backup + sanity-check guard. + return setJSONPath(path, key+".-1", canonical) +} + +func (c *vscodeClient) RemoveSkills(ctx context.Context, skillNames []string, scope Scope) error { + path := c.settingsPath() + if path == "" { + return nil + } + const key = `chat\.instructionsFilesLocations` + canonical := CanonicalSkillsDir() + _, body, err := jsonReadRaw(path) + if err != nil { + return err + } + cur := gjsonGetArrayStrings(body, key) + idx := -1 + for i, e := range cur { + if e == canonical { + idx = i + break + } + } + if idx < 0 { + return nil + } + return deleteJSONPath(path, fmt.Sprintf("%s.%d", key, idx)) +} + +// vscodeMCPPath returns the VS Code mcp.json for the given scope. +// User: /User/mcp.json; project: ./.vscode/mcp.json. +// VS Code reads MCP from mcp.json (servers.*), not settings.json. +func (c *vscodeClient) vscodeMCPPath(scope Scope) string { + if scope == ScopeProject { + return projectPath(".vscode", "mcp.json") + } + dir := vscodeUserDir() + if dir == "" { + return "" + } + return filepath.Join(dir, "mcp.json") +} + +func (c *vscodeClient) WriteMCP(ctx context.Context, scope Scope) error { + path := c.vscodeMCPPath(scope) + if path == "" { + return nil + } + // VS Code mcp.json uses top-level "servers", not "mcpServers". + return writeJSONMCP(path, "servers") +} + +func (c *vscodeClient) RemoveMCP(ctx context.Context, scope Scope) error { + path := c.vscodeMCPPath(scope) + if path == "" { + return nil + } + _ = deleteJSONPath(path, "servers.sling") + _ = deleteJSONPath(path, "servers.agent-browser") + // Clean the obsolete flat settings.json key from an earlier buggy path + // (github.copilot.chat.mcp.servers.sling) so doctor/settings stay tidy. + if settings := c.settingsPath(); settings != "" { + _ = deleteJSONPath(settings, `github\.copilot\.chat\.mcp\.servers\.sling`) + } + return nil +} + +func (c *vscodeClient) CheckSkills(ctx context.Context, skillNames []string, scope Scope) []CheckResult { + path := c.settingsPath() + if path == "" { + return []CheckResult{checkNA("no user config dir found")} + } + doc, err := jsonReadOrEmpty(path) + if err != nil { + return []CheckResult{checkFail(err.Error())} + } + canonical := CanonicalSkillsDir() + has := false + if locs, ok := doc["chat.instructionsFilesLocations"].([]any); ok { + for _, e := range locs { + if s, _ := e.(string); s == canonical { + has = true + break + } + } + } + // One aggregate result for all skills (vscode wires the canonical dir once). + if has { + return []CheckResult{checkOK("chat.instructionsFilesLocations includes canonical")} + } + return []CheckResult{checkFail("chat.instructionsFilesLocations missing canonical")} +} + +func (c *vscodeClient) CheckMCP(ctx context.Context, scope Scope) CheckResult { + path := c.vscodeMCPPath(scope) + if path == "" { + return checkNA("no user config dir found") + } + doc, err := jsonReadOrEmpty(path) + if err != nil { + return checkFail(err.Error()) + } + servers, _ := doc["servers"].(map[string]any) + if servers == nil { + return checkFail("no servers block in mcp.json") + } + if _, ok := servers["sling"]; !ok { + return checkFail("servers.sling missing in mcp.json") + } + if _, ok := servers[agentBrowserMCPName]; !ok { + return checkFail("servers.agent-browser missing in mcp.json") + } + return checkOK("mcp.json servers.sling and agent-browser present") +} + +// commandOnPath returns true if `name` resolves to an executable on $PATH. +// Uses exec.LookPath so Windows PATHEXT / .exe resolution works. +func commandOnPath(name string) bool { + _, err := exec.LookPath(name) + return err == nil +} + +// AuthStatus is an offline probe of whether a CLI agent looks signed in. +// Values: ok (credential file or env key present), none, unknown. +type AuthStatus string + +const ( + AuthOK AuthStatus = "ok" + AuthNone AuthStatus = "none" + AuthUnknown AuthStatus = "unknown" +) + +func fileNonEmpty(path string) bool { + st, err := os.Stat(path) + return err == nil && st.Size() > 0 +} + +func envNonEmpty(keys ...string) bool { + for _, k := range keys { + if os.Getenv(k) != "" { + return true + } + } + return false +} + +func claudeJSONHasOAuth(path string) bool { + b, err := os.ReadFile(path) + if err != nil { + return false + } + var doc map[string]any + if json.Unmarshal(b, &doc) != nil { + return false + } + v, ok := doc["oauthAccount"] + if !ok || v == nil { + return false + } + m, ok := v.(map[string]any) + return ok && len(m) > 0 +} + +func (c *claudeClient) AuthState() AuthStatus { + if envNonEmpty("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_OAUTH_TOKEN") { + return AuthOK + } + if envNonEmpty("CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY") { + return AuthOK + } + credDir := filepath.Join(userHome(), ".claude") + if d := strings.TrimSpace(os.Getenv("CLAUDE_CONFIG_DIR")); d != "" { + credDir = d + } + if fileNonEmpty(filepath.Join(credDir, ".credentials.json")) { + return AuthOK + } + if claudeJSONHasOAuth(filepath.Join(userHome(), ".claude.json")) { + return AuthOK + } + // macOS stores /login tokens in Keychain, not a file. + if runtime.GOOS == "darwin" { + return AuthUnknown + } + return AuthNone +} + +func (c *codexClient) AuthState() AuthStatus { + if envNonEmpty("OPENAI_API_KEY") { + return AuthOK + } + home := userHome() + if fileNonEmpty(filepath.Join(home, ".codex", "auth.json")) { + return AuthOK + } + if fileNonEmpty(filepath.Join(home, ".codex", "config.toml")) { + return AuthUnknown + } + return AuthNone +} + +func (c *geminiClient) AuthState() AuthStatus { + if envNonEmpty("GEMINI_API_KEY", "GOOGLE_API_KEY") { + return AuthOK + } + if p := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS"); p != "" && fileNonEmpty(p) { + return AuthOK + } + home := userHome() + if fileNonEmpty(filepath.Join(home, ".gemini", "oauth_creds.json")) || + fileNonEmpty(filepath.Join(home, ".gemini", "google_accounts.json")) { + return AuthOK + } + return AuthNone +} + +func (c *cursorClient) AuthState() AuthStatus { + if envNonEmpty("CURSOR_API_KEY") { + return AuthOK + } + // Browser login stores credentials in the OS keychain, not a file we can + // read, so ask the CLI. `cursor-agent status --format json` reports + // isAuthenticated and always exits 0 — trust the field, not the code. + if st, ok := cursorStatusAuth(); ok { + if st { + return AuthOK + } + return AuthNone + } + if runtime.GOOS == "darwin" || runtime.GOOS == "windows" { + return AuthUnknown + } + return AuthNone +} + +// cursorStatusOnce caches the probe: doctor asks several times per run and +// each call spawns the CLI. +var cursorStatusOnce struct { + sync.Once + authed, answered bool +} + +// cursorStatusAuth runs `cursor-agent status --format json`. +// Returns (authenticated, true) when the CLI answered, (false, false) otherwise. +func cursorStatusAuth() (bool, bool) { + cursorStatusOnce.Do(func() { + cursorStatusOnce.authed, cursorStatusOnce.answered = probeCursorStatus() + }) + return cursorStatusOnce.authed, cursorStatusOnce.answered +} + +// resetCursorStatusCache clears the memoized probe. Tests use it when they +// swap the stub on PATH. +func resetCursorStatusCache() { + cursorStatusOnce.Once = sync.Once{} + cursorStatusOnce.authed, cursorStatusOnce.answered = false, false +} + +func probeCursorStatus() (bool, bool) { + bin, err := exec.LookPath(agentBinary("cursor")) + if err != nil { + return false, false + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + out, err := exec.CommandContext(ctx, bin, "status", "--format", "json").Output() + if err != nil && len(out) == 0 { + return false, false + } + var doc struct { + IsAuthenticated *bool `json:"isAuthenticated"` + } + if json.Unmarshal(out, &doc) != nil || doc.IsAuthenticated == nil { + return false, false + } + return *doc.IsAuthenticated, true +} + +func (c *opencodeClient) AuthState() AuthStatus { + if envNonEmpty("OPENCODE_API_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY") { + return AuthOK + } + dataHome := os.Getenv("XDG_DATA_HOME") + if dataHome == "" { + dataHome = filepath.Join(userHome(), ".local", "share") + } + if fileNonEmpty(filepath.Join(dataHome, "opencode", "auth.json")) { + return AuthOK + } + if fileNonEmpty(filepath.Join(userHome(), ".opencode", "auth.json")) { + return AuthOK + } + return AuthNone +} + +func (c *piClient) AuthState() AuthStatus { + if envNonEmpty("PI_API_KEY", "OPENROUTER_API_KEY") { + return AuthOK + } + if fileNonEmpty(filepath.Join(piAgentDir(), "auth.json")) { + return AuthOK + } + return AuthNone +} + +func (c *grokClient) AuthState() AuthStatus { + if envNonEmpty("XAI_API_KEY", "GROK_API_KEY", "GROK_CODE_XAI_API_KEY") { + return AuthOK + } + if fileNonEmpty(filepath.Join(userHome(), ".grok", "auth.json")) { + return AuthOK + } + return AuthNone +} + +func (c *vscodeClient) AuthState() AuthStatus { + return AuthUnknown +} + +// YesNo is the install-summary token: yes / no / —. +func (s AuthStatus) YesNo() string { + switch s { + case AuthOK: + return "yes" + case AuthNone: + return "no" + default: + return "—" + } +} + +// cell is the doctor-matrix cell. Missing auth is not a failure. +func (s AuthStatus) cell() CellState { + switch s { + case AuthOK: + return CellOK + case AuthNone: + return CellEmpty + default: + return CellNA + } +} + +// RankedAgent is a detected CLI agent with install + auth state for the confirm form. +type RankedAgent struct { + Name string + Auth AuthStatus + Detect bool + Bundled bool // true when this is the downloadable opencode fallback + Score int // lower is better +} + +// RankedCLIAgents lists CLI agents on $PATH: authenticated first, then unknown, then none. +// Bundled OpenCode is appended only when no CLI agent is on $PATH (and a release asset exists). +func RankedCLIAgents() []RankedAgent { + out := []RankedAgent{} + for _, c := range CLIAgents() { + if !c.Detect() { + continue + } + auth := c.AuthState() + score := 2 + switch auth { + case AuthOK: + score = 0 + case AuthUnknown: + score = 1 + } + out = append(out, RankedAgent{ + Name: c.Name(), + Auth: auth, + Detect: true, + Score: score, + }) + } + if len(out) == 0 { + if _, err := OpenCodeAssetName(runtime.GOOS, runtime.GOARCH); err == nil { + out = append(out, RankedAgent{ + Name: "opencode", + Auth: AuthNone, + Detect: false, + Bundled: true, + Score: 3, + }) + } + } + for i := 0; i < len(out); i++ { + for j := i + 1; j < len(out); j++ { + if out[j].Score < out[i].Score { + out[i], out[j] = out[j], out[i] + } + } + } + return out +} + +// pathRanked splits RankedCLIAgents into on-PATH agents and optional bundled OpenCode. +func pathRanked() (agents []RankedAgent, bundled *RankedAgent) { + for _, a := range RankedCLIAgents() { + if a.Bundled { + cp := a + bundled = &cp + continue + } + agents = append(agents, a) + } + return agents, bundled +} + +// RecommendedAgent is the first ranked CLI agent (authenticated on PATH, else the only one). +func RecommendedAgent() string { + ranked := RankedCLIAgents() + if len(ranked) == 0 { + return "auto" + } + return ranked[0].Name +} diff --git a/core/sling/assist/clients_test.go b/core/sling/assist/clients_test.go new file mode 100644 index 000000000..9c10b1cb3 --- /dev/null +++ b/core/sling/assist/clients_test.go @@ -0,0 +1,1052 @@ +// Client adapter tests: JSONC preserve, profile YAML comments, backups. + +package assist + +import ( + "context" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/tidwall/gjson" +) + +// withTempHomeDir overrides env.HomeDir for the duration of the test. The +// package-level var is set during init() from $SLING_HOME_DIR — for tests we +// need both updated so envFilePath() picks up the temp dir. +func withTempHomeDir(t *testing.T) string { + t.Helper() + dir := t.TempDir() + // Clear the nested-launch markers. Without this, a test run from inside + // a CLI agent makes NestedLaunch() true and Session prints instead of launching. + for _, k := range []string{"CLAUDECODE", "CURSOR_TRACE_ID", "OPENCODE", "OPENCODE_SESSION"} { + t.Setenv(k, "") + os.Unsetenv(k) + } + prev := env.HomeDir + env.HomeDir = dir + // Override path seam so client adapters (userHome) and Sling dirs (slingHome) + // both resolve under the temp tree. + restore := SetPaths(Paths{SlingHome: dir, UserHome: dir, CWD: dir}) + t.Cleanup(func() { + env.HomeDir = prev + restore() + }) + return dir +} + +// TestJSONPreservesComments verifies that a JSONC document with `//` and `/* */` +// comments survives a round-trip through setJSONPath/deleteJSONPath. This is +// the bug that bit us with VS Code's settings.json, where the previous map +// rewrite silently stripped every comment in the file. +func TestJSONPreservesComments(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "settings.json") + + original := `{ + // top comment about the user's editor + "editor.fontSize": 14, + /* block comment + spanning two lines */ + "editor.fontFamily": "JetBrains Mono", // trailing comment + "files.exclude": { + "**/.git": true // hide git + } +}` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + if err := setJSONPath(path, `chat\.instructionsFilesLocations.-1`, "/Users/me/.agents/skills"); err != nil { + t.Fatalf("setJSONPath: %v", err) + } + + got, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + out := string(got) + + wantSubstrings := []string{ + "// top comment about the user's editor", + "/* block comment", + "spanning two lines */", + "// trailing comment", + "// hide git", + `"editor.fontSize": 14`, + `"editor.fontFamily": "JetBrains Mono"`, + `"chat.instructionsFilesLocations"`, + `"/Users/me/.agents/skills"`, + } + for _, sub := range wantSubstrings { + if !strings.Contains(out, sub) { + t.Errorf("expected output to contain %q\n--- got ---\n%s", sub, out) + } + } +} + +// TestJSONDeletePreservesComments verifies that deleting a key still keeps +// surrounding comments intact. +func TestJSONDeletePreservesComments(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "settings.json") + + original := `{ + // keep this comment + "editor.fontSize": 14, + "mcpServers": { + "sling": { "command": "sling", "args": ["serve", "mcp"] }, + "other": { "command": "other" } + } +}` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + if err := deleteJSONPath(path, "mcpServers.sling"); err != nil { + t.Fatalf("deleteJSONPath: %v", err) + } + + got, _ := os.ReadFile(path) + out := string(got) + + if !strings.Contains(out, "// keep this comment") { + t.Errorf("expected comment to be preserved\n--- got ---\n%s", out) + } + if strings.Contains(out, `"sling"`) { + t.Errorf("expected sling entry to be deleted\n--- got ---\n%s", out) + } + if !strings.Contains(out, `"other"`) { + t.Errorf("expected sibling entry preserved\n--- got ---\n%s", out) + } +} + +// TestSaveProfilePreservesYAMLComments verifies that adding the +// env.SLING_ASSIST entry to an existing env.yaml doesn't blow away the user's +// comments and unrelated keys. This is the analogue of +// TestJSONPreservesComments for YAML. +func TestSaveProfilePreservesYAMLComments(t *testing.T) { + homeDir := withTempHomeDir(t) + + envFile := filepath.Join(homeDir, "env.yaml") + original := `# Sling environment file — managed by you. +# These connections are used by replications and pipelines. + +connections: + # Production warehouse + PG_PROD: + type: postgres + host: db.example.com + user: app + # Staging warehouse + PG_STAGE: + type: postgres + host: stage.db.example.com + +# Variables shared across runs +variables: + region: us-west-2 +` + if err := os.WriteFile(envFile, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + prof := DefaultProfile() + prof.Agent = "claude" + if err := SaveProfile(prof); err != nil { + t.Fatalf("SaveProfile: %v", err) + } + + got, _ := os.ReadFile(envFile) + out := string(got) + + wantSubstrings := []string{ + "# Sling environment file — managed by you.", + "# These connections are used by replications and pipelines.", + "# Production warehouse", + "# Staging warehouse", + "PG_PROD:", + "PG_STAGE:", + "region: us-west-2", + "SLING_ASSIST:", + "agent: claude", + } + for _, sub := range wantSubstrings { + if !strings.Contains(out, sub) { + t.Errorf("expected output to contain %q\n--- got ---\n%s", sub, out) + } + } + // Legacy `variables:` migrates to `env:` on save — the block contents + // (region: us-west-2) survive, but the heading comment attached to the + // renamed key is dropped along with the old key. + if strings.Contains(out, "variables:") { + t.Errorf("expected legacy variables: block to be renamed to env:\n--- got ---\n%s", out) + } + + // Idempotency: a second save should leave comments intact and not duplicate + // the SLING_ASSIST entry. + prof.HintInErrors = false + if err := SaveProfile(prof); err != nil { + t.Fatalf("second SaveProfile: %v", err) + } + got, _ = os.ReadFile(envFile) + out = string(got) + + if strings.Count(out, "SLING_ASSIST:") != 1 { + t.Errorf("expected exactly one SLING_ASSIST entry, got\n%s", out) + } + if !strings.Contains(out, "# Production warehouse") { + t.Errorf("comments lost on second save\n--- got ---\n%s", out) + } + if !strings.Contains(out, "hint_in_errors: false") { + t.Errorf("SLING_ASSIST entry did not update on second save\n--- got ---\n%s", out) + } +} + +// TestJSONLeadingCommentBanner reproduces the real-world bug where VS Code's +// settings.json has a `// Place your settings...` banner *before* the opening +// `{`. sjson can't parse that prefix and rebuilds the document as a single +// compact line, blowing away the user's config. Our fix strips the leading +// non-JSON content before handing the buffer to sjson, so the body survives. +func TestJSONLeadingCommentBanner(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "settings.json") + + // Mimics the real shape of VS Code's user settings file — leading banner, + // many keys, multi-line. + original := `// Place your settings in this file to overwrite the default settings +{ + "editor.tabSize": 2, + "editor.detectIndentation": false, + "editor.guides.indentation": false, + "editor.formatOnSave": true, + "workbench.colorTheme": "Default Dark Modern", + "files.autoSave": "afterDelay", + "git.autofetch": true, + "terminal.integrated.fontSize": 13 +} +` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + if err := setJSONPath(path, `chat\.instructionsFilesLocations.-1`, "/Users/me/.agents/skills"); err != nil { + t.Fatalf("setJSONPath: %v", err) + } + + got, _ := os.ReadFile(path) + out := string(got) + + // Must still contain every original key. + wantKeys := []string{ + `"editor.tabSize"`, + `"editor.detectIndentation"`, + `"editor.guides.indentation"`, + `"editor.formatOnSave"`, + `"workbench.colorTheme"`, + `"files.autoSave"`, + `"git.autofetch"`, + `"terminal.integrated.fontSize"`, + `"chat.instructionsFilesLocations"`, + } + for _, k := range wantKeys { + if !strings.Contains(out, k) { + t.Errorf("expected output to contain %q\n--- got ---\n%s", k, out) + } + } + + // And the file should not have collapsed to one line. + if strings.Count(out, "\n") < 5 { + t.Errorf("file collapsed to a single line, only %d newlines\n--- got ---\n%s", + strings.Count(out, "\n"), out) + } + + // Leading banner must survive — it carries useful context the user wrote + // (or VS Code wrote on their behalf). + if !strings.HasPrefix(out, "// Place your settings in this file") { + t.Errorf("leading banner was dropped\n--- got ---\n%s", out) + } +} + +// TestJSONBackupCreatedBeforeEdit verifies setJSONPath writes .backup +// before mutating the original. The user can always recover from .backup if +// something goes wrong on the next install. +func TestJSONBackupCreatedBeforeEdit(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "settings.json") + + original := `{ + "editor.fontSize": 14, + "workbench.colorTheme": "Default Dark Modern" +}` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + if err := setJSONPath(path, `chat\.instructionsFilesLocations.-1`, "/x"); err != nil { + t.Fatalf("setJSONPath: %v", err) + } + + bk, err := os.ReadFile(path + ".backup") + if err != nil { + t.Fatalf("expected backup file at %s: %v", path+".backup", err) + } + if string(bk) != original { + t.Errorf("backup didn't match original\n--- backup ---\n%s\n--- want ---\n%s", string(bk), original) + } +} + +// TestJSONNoBackupWhenSourceMissing ensures we don't create an empty +// .backup when the file we're about to edit doesn't exist yet. +func TestJSONNoBackupWhenSourceMissing(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "fresh.json") + + if err := setJSONPath(path, "mcpServers.sling", map[string]any{"command": "sling"}); err != nil { + t.Fatalf("setJSONPath: %v", err) + } + if _, err := os.Stat(path + ".backup"); !os.IsNotExist(err) { + t.Errorf("expected no backup for fresh-file write, got err=%v", err) + } +} + +// TestJSONDestructiveEditRefused simulates an edit that drops top-level keys +// (e.g. a mangled sjson rewrite) and verifies the helper refuses to commit it. +// We trigger this by directly invoking the validator with mocked before/after +// snapshots. +func TestJSONDestructiveEditRefused(t *testing.T) { + before := []byte(`{ + "a": 1, + "b": 2, + "c": 3, + "d": 4 +}`) + // Mimic an sjson-trip-on-banner result: the whole body collapsed into a + // single key, single line. + after := []byte(`{"only":"survivor"}`) + + if err := validateEditNotDestructive(before, after, 0); err == nil { + t.Errorf("expected validateEditNotDestructive to reject a destructive edit") + } + + // And a single-key delete on the same document should be allowed when + // allowKeyDelta=1. + deleted := []byte(`{ + "a": 1, + "b": 2, + "c": 3 +}`) + if err := validateEditNotDestructive(before, deleted, 1); err != nil { + t.Errorf("expected single-key delete to be allowed: %v", err) + } +} + +// TestSaveProfileFreshFile covers the empty-file path: when env.yaml doesn't +// exist yet, SaveProfile should create it with just env.SLING_ASSIST. +func TestSaveProfileFreshFile(t *testing.T) { + homeDir := withTempHomeDir(t) + + prof := DefaultProfile() + prof.Agent = "codex" + if err := SaveProfile(prof); err != nil { + t.Fatalf("SaveProfile: %v", err) + } + + got, _ := os.ReadFile(filepath.Join(homeDir, "env.yaml")) + out := string(got) + + if !strings.Contains(out, "SLING_ASSIST:") { + t.Errorf("missing SLING_ASSIST entry in fresh file\n--- got ---\n%s", out) + } + if !strings.Contains(out, "agent: codex") { + t.Errorf("missing agent: codex\n--- got ---\n%s", out) + } +} + +// ---- opencode / pi / grok adapters ---- + +// TestNewClientsRegistered verifies the three additions are in the canonical +// list, are launchable CLI agents, and resolve by name. +func TestNewClientsRegistered(t *testing.T) { + for _, name := range []string{"opencode", "pi", "grok"} { + c := LookupClient(name) + if c == nil { + t.Fatalf("LookupClient(%q) returned nil", name) + } + if c.Kind() != KindCLIAgent { + t.Errorf("%s: expected KindCLIAgent, got %v", name, c.Kind()) + } + found := false + for _, a := range CLIAgents() { + if a.Name() == name { + found = true + } + } + if !found { + t.Errorf("%s missing from CLIAgents()", name) + } + } +} + +// TestOpencodeMCPRoundTrip covers opencode's distinctive MCP shape: a `mcp` +// (not `mcpServers`) block whose entries carry `type: local` and a single +// argv array rather than command/args. +func TestOpencodeMCPRoundTrip(t *testing.T) { + home := withTempHomeDir(t) + ctx := context.Background() + c := &opencodeClient{} + + path := filepath.Join(home, ".config", "opencode", "opencode.json") + if got := c.configPath(ScopeUser); got != path { + t.Fatalf("configPath = %s, want %s", got, path) + } + + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellFail { + t.Errorf("expected fail before write, got %v", res.State) + } + if err := c.WriteMCP(ctx, ScopeUser); err != nil { + t.Fatalf("WriteMCP: %v", err) + } + + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + out := string(data) + if got := gjson.GetBytes(data, "mcp.sling.type").String(); got != "local" { + t.Errorf("type = %q, want \"local\"\n--- got ---\n%s", got, out) + } + // command must be a flat argv array, not the command/args pair. + if got := gjson.GetBytes(data, "mcp.sling.command").String(); got != `["sling","serve","mcp"]` { + t.Errorf("command = %q, want [\"sling\",\"serve\",\"mcp\"]\n--- got ---\n%s", got, out) + } + if !gjson.GetBytes(data, "mcp.sling.enabled").Bool() { + t.Errorf("expected enabled: true\n--- got ---\n%s", out) + } + if gjson.GetBytes(data, "mcp.sling.args").Exists() { + t.Errorf("opencode entry should not use args key\n--- got ---\n%s", out) + } + if got := gjson.GetBytes(data, "mcp.agent-browser.type").String(); got != "local" { + t.Errorf("agent-browser type = %q, want \"local\"\n--- got ---\n%s", got, out) + } + if !strings.Contains(gjson.GetBytes(data, "mcp.agent-browser.command").Raw, `"mcp"`) { + t.Errorf("agent-browser command missing mcp\n--- got ---\n%s", out) + } + + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellOK { + t.Errorf("expected ok after write, got %v (%s)", res.State, res.Note) + } + if err := c.RemoveMCP(ctx, ScopeUser); err != nil { + t.Fatalf("RemoveMCP: %v", err) + } + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellFail { + t.Errorf("expected fail after remove, got %v", res.State) + } +} + +// TestOpencodeMCPPreservesSiblings ensures we only touch mcp.sling and leave +// the user's other opencode settings (and their comments) alone. +func TestOpencodeMCPPreservesSiblings(t *testing.T) { + home := withTempHomeDir(t) + ctx := context.Background() + path := filepath.Join(home, ".config", "opencode", "opencode.json") + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + original := `{ + // my opencode setup + "$schema": "https://opencode.ai/config.json", + "theme": "tokyonight", + "mcp": { + "other": { "type": "local", "command": ["other"] } + } +}` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + c := &opencodeClient{} + if err := c.WriteMCP(ctx, ScopeUser); err != nil { + t.Fatalf("WriteMCP: %v", err) + } + out, _ := os.ReadFile(path) + for _, sub := range []string{"// my opencode setup", `"tokyonight"`, `"other"`, `"sling"`, `"agent-browser"`, `"$schema"`} { + if !strings.Contains(string(out), sub) { + t.Errorf("expected %q preserved\n--- got ---\n%s", sub, string(out)) + } + } + + if err := c.RemoveMCP(ctx, ScopeUser); err != nil { + t.Fatalf("RemoveMCP: %v", err) + } + out, _ = os.ReadFile(path) + if strings.Contains(string(out), `"sling"`) { + t.Errorf("sling entry not removed\n--- got ---\n%s", string(out)) + } + if strings.Contains(string(out), `"agent-browser"`) { + t.Errorf("agent-browser entry not removed\n--- got ---\n%s", string(out)) + } + if !strings.Contains(string(out), `"other"`) { + t.Errorf("sibling MCP entry lost\n--- got ---\n%s", string(out)) + } +} + +// TestOpencodeConfigDirXDG verifies XDG_CONFIG_HOME wins over ~/.config. +func TestOpencodeConfigDirXDG(t *testing.T) { + home := withTempHomeDir(t) + if got, want := opencodeConfigDir(), filepath.Join(home, ".config", "opencode"); got != want { + t.Errorf("default configDir = %s, want %s", got, want) + } + t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, "xdg")) + if got, want := opencodeConfigDir(), filepath.Join(home, "xdg", "opencode"); got != want { + t.Errorf("XDG configDir = %s, want %s", got, want) + } +} + +// TestPiMCPRoundTrip covers pi's mcp.json (separate from settings.json) with +// the standard mcpServers command/args shape. +func TestPiMCPRoundTrip(t *testing.T) { + home := withTempHomeDir(t) + ctx := context.Background() + c := &piClient{} + + path := filepath.Join(home, ".pi", "agent", "mcp.json") + if got := c.mcpPath(ScopeUser); got != path { + t.Fatalf("mcpPath = %s, want %s", got, path) + } + + if err := c.WriteMCP(ctx, ScopeUser); err != nil { + t.Fatalf("WriteMCP: %v", err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + // pi uses the standard command/args split (not opencode's argv array). + if got := gjson.GetBytes(data, "mcpServers.sling.command").String(); got != "sling" { + t.Errorf("command = %q, want \"sling\"\n--- got ---\n%s", got, string(data)) + } + if got := gjson.GetBytes(data, "mcpServers.sling.args").String(); got != `["serve","mcp"]` { + t.Errorf("args = %q, want [\"serve\",\"mcp\"]\n--- got ---\n%s", got, string(data)) + } + if got := gjson.GetBytes(data, "mcpServers.agent-browser.args").String(); got != `["mcp","--tools","core"]` { + t.Errorf("agent-browser args = %q\n--- got ---\n%s", got, string(data)) + } + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellOK { + t.Errorf("expected ok after write, got %v (%s)", res.State, res.Note) + } + + if err := c.RemoveMCP(ctx, ScopeUser); err != nil { + t.Fatalf("RemoveMCP: %v", err) + } + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellFail { + t.Errorf("expected fail after remove, got %v", res.State) + } +} + +// TestPiAgentDirEnvOverride verifies PI_CODING_AGENT_DIR takes precedence over +// the ~/.pi/agent default. +func TestPiAgentDirEnvOverride(t *testing.T) { + home := withTempHomeDir(t) + if got, want := piAgentDir(), filepath.Join(home, ".pi", "agent"); got != want { + t.Errorf("default agentDir = %s, want %s", got, want) + } + custom := filepath.Join(home, "custom-pi") + t.Setenv("PI_CODING_AGENT_DIR", custom) + if got := piAgentDir(); got != custom { + t.Errorf("env agentDir = %s, want %s", got, custom) + } + if got, want := (&piClient{}).mcpPath(ScopeUser), filepath.Join(custom, "mcp.json"); got != want { + t.Errorf("mcpPath = %s, want %s", got, want) + } +} + +// TestGrokMCPRoundTrip covers grok's TOML config: [mcp_servers.sling] written +// into ~/.grok/config.toml, same table shape as codex. +func TestGrokMCPRoundTrip(t *testing.T) { + home := withTempHomeDir(t) + ctx := context.Background() + c := &grokClient{} + + path := filepath.Join(home, ".grok", "config.toml") + if got := c.configPath(ScopeUser); got != path { + t.Fatalf("configPath = %s, want %s", got, path) + } + + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellFail { + t.Errorf("expected fail before write, got %v", res.State) + } + if err := c.WriteMCP(ctx, ScopeUser); err != nil { + t.Fatalf("WriteMCP: %v", err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + for _, sub := range []string{"[mcp_servers.sling]", `command = "sling"`, `args = ["serve", "mcp"]`, "[mcp_servers.agent-browser]"} { + if !strings.Contains(string(data), sub) { + t.Errorf("expected %q in config.toml\n--- got ---\n%s", sub, string(data)) + } + } + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellOK { + t.Errorf("expected ok after write, got %v (%s)", res.State, res.Note) + } + + // Idempotent: a second write must not duplicate the section. + if err := c.WriteMCP(ctx, ScopeUser); err != nil { + t.Fatalf("second WriteMCP: %v", err) + } + data, _ = os.ReadFile(path) + if n := strings.Count(string(data), "[mcp_servers.sling]"); n != 1 { + t.Errorf("expected 1 sling section, got %d\n--- got ---\n%s", n, string(data)) + } + if n := strings.Count(string(data), "[mcp_servers.agent-browser]"); n != 1 { + t.Errorf("expected 1 agent-browser section, got %d\n--- got ---\n%s", n, string(data)) + } + + if err := c.RemoveMCP(ctx, ScopeUser); err != nil { + t.Fatalf("RemoveMCP: %v", err) + } + if res := c.CheckMCP(ctx, ScopeUser); res.State != CellFail { + t.Errorf("expected fail after remove, got %v", res.State) + } +} + +// TestGrokMCPPreservesOtherSections verifies we only own the sling table and +// leave the user's other grok config intact. +func TestGrokMCPPreservesOtherSections(t *testing.T) { + home := withTempHomeDir(t) + ctx := context.Background() + path := filepath.Join(home, ".grok", "config.toml") + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + original := `model = "grok-4" + +[mcp_servers.filesystem] +command = "npx" +args = ["-y", "@modelcontextprotocol/server-filesystem"] + +[skills] +paths = ["~/.agents/skills"] +` + if err := os.WriteFile(path, []byte(original), 0o644); err != nil { + t.Fatal(err) + } + + c := &grokClient{} + if err := c.WriteMCP(ctx, ScopeUser); err != nil { + t.Fatalf("WriteMCP: %v", err) + } + out, _ := os.ReadFile(path) + for _, sub := range []string{`model = "grok-4"`, "[mcp_servers.filesystem]", "[skills]", "[mcp_servers.sling]", "[mcp_servers.agent-browser]"} { + if !strings.Contains(string(out), sub) { + t.Errorf("expected %q preserved\n--- got ---\n%s", sub, string(out)) + } + } + + if err := c.RemoveMCP(ctx, ScopeUser); err != nil { + t.Fatalf("RemoveMCP: %v", err) + } + out, _ = os.ReadFile(path) + if strings.Contains(string(out), "[mcp_servers.sling]") { + t.Errorf("sling section not removed\n--- got ---\n%s", string(out)) + } + if strings.Contains(string(out), "[mcp_servers.agent-browser]") { + t.Errorf("agent-browser section not removed\n--- got ---\n%s", string(out)) + } + if !strings.Contains(string(out), "[mcp_servers.filesystem]") { + t.Errorf("sibling MCP section lost\n--- got ---\n%s", string(out)) + } + if !strings.Contains(string(out), "[skills]") { + t.Errorf("skills section lost\n--- got ---\n%s", string(out)) + } +} + +// TestNewClientsSkillsAreCanonical verifies opencode/pi/grok are all no-op on +// WriteSkills (they read ~/.agents/skills/ natively) and that CheckSkills +// tracks the canonical bundle rather than a per-client redirect. +func TestNewClientsSkillsAreCanonical(t *testing.T) { + withTempHomeDir(t) + ctx := context.Background() + skills := []string{"sling", "sling-replications"} + + clients := []Client{&opencodeClient{}, &piClient{}, &grokClient{}} + for _, c := range clients { + if err := c.WriteSkills(ctx, skills, ScopeUser); err != nil { + t.Fatalf("%s WriteSkills: %v", c.Name(), err) + } + // Bundle not written yet → every skill reports missing. + for _, res := range c.CheckSkills(ctx, skills, ScopeUser) { + if res.State != CellFail { + t.Errorf("%s: expected fail before bundle write, got %v", c.Name(), res.State) + } + } + } + + if err := writeCanonicalBundle(skills); err != nil { + t.Fatalf("writeCanonicalBundle: %v", err) + } + for _, c := range clients { + results := c.CheckSkills(ctx, skills, ScopeUser) + if len(results) != len(skills) { + t.Errorf("%s: got %d results, want %d", c.Name(), len(results), len(skills)) + } + for _, res := range results { + if res.State != CellOK { + t.Errorf("%s: expected ok after bundle write, got %v (%s)", c.Name(), res.State, res.Note) + } + } + // RemoveSkills must not touch the canonical bundle — it's shared. + if err := c.RemoveSkills(ctx, skills, ScopeUser); err != nil { + t.Fatalf("%s RemoveSkills: %v", c.Name(), err) + } + if !g.PathExists(canonicalSkillPath("sling")) { + t.Errorf("%s RemoveSkills deleted the shared canonical bundle", c.Name()) + } + } +} + +// TestNewClientsProjectScope verifies project-scope paths resolve under +// projectRoot() (absolute), not the bare relative "./..." form that scattered +// files when CWD was a subdirectory. +func TestNewClientsProjectScope(t *testing.T) { + root := projectRoot() + cases := []struct { + name string + got string + want string + }{ + {"opencode", (&opencodeClient{}).configPath(ScopeProject), filepath.Join(root, "opencode.json")}, + {"pi", (&piClient{}).mcpPath(ScopeProject), filepath.Join(root, ".pi", "mcp.json")}, + {"grok", (&grokClient{}).configPath(ScopeProject), filepath.Join(root, ".grok", "config.toml")}, + {"claude", (&claudeClient{}).mcpPath(ScopeProject), filepath.Join(root, ".mcp.json")}, + {"vscode", (&vscodeClient{}).vscodeMCPPath(ScopeProject), filepath.Join(root, ".vscode", "mcp.json")}, + } + for _, tc := range cases { + if tc.got != tc.want { + t.Errorf("%s project path = %s, want %s", tc.name, tc.got, tc.want) + } + if !filepath.IsAbs(tc.got) { + t.Errorf("%s project path should be absolute, got %s", tc.name, tc.got) + } + } +} + +// TestNewClientsDetect is PATH-only: a config dir without a binary is not enough. +func TestNewClientsDetect(t *testing.T) { + home := withTempHomeDir(t) + bin := filepath.Join(home, "empty-bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin) + + cases := []Client{&opencodeClient{}, &piClient{}, &grokClient{}, &geminiClient{}} + for _, c := range cases { + if c.Detect() { + t.Errorf("%s: detected with empty PATH", c.Name()) + } + } + for _, name := range []string{"opencode", "pi", "grok", "gemini"} { + stub := filepath.Join(bin, name) + if err := os.WriteFile(stub, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + } + for _, c := range cases { + if !c.Detect() { + t.Errorf("%s: not detected after stub on PATH", c.Name()) + } + } +} + +func clearAuthEnv(t *testing.T) { + t.Helper() + for _, k := range []string{ + "ANTHROPIC_API_KEY", "CLAUDE_CODE_OAUTH_TOKEN", + "OPENAI_API_KEY", "GEMINI_API_KEY", "GOOGLE_API_KEY", + "GOOGLE_APPLICATION_CREDENTIALS", "ANTHROPIC_AUTH_TOKEN", + "CURSOR_API_KEY", "AIDER_API_KEY", "OPENCODE_API_KEY", + "PI_API_KEY", "OPENROUTER_API_KEY", "XAI_API_KEY", "GROK_API_KEY", + "GROK_CODE_XAI_API_KEY", "CLAUDE_CONFIG_DIR", + "CLAUDE_CODE_USE_BEDROCK", "CLAUDE_CODE_USE_VERTEX", "CLAUDE_CODE_USE_FOUNDRY", + "XDG_DATA_HOME", "PI_CODING_AGENT_DIR", + } { + t.Setenv(k, "") + } +} + +func TestAuthStateFakeConfigTrees(t *testing.T) { + clearAuthEnv(t) + dir := withTempHomeDir(t) + + wantClaudeEmpty := AuthNone + if runtime.GOOS == "darwin" { + wantClaudeEmpty = AuthUnknown // Keychain may hold /login tokens + } + if st := (&claudeClient{}).AuthState(); st != wantClaudeEmpty { + t.Fatalf("claude empty = %s, want %s", st, wantClaudeEmpty) + } + if st := (&grokClient{}).AuthState(); st != AuthNone { + t.Fatalf("grok empty = %s, want none", st) + } + + cred := filepath.Join(dir, ".claude", ".credentials.json") + if err := os.MkdirAll(filepath.Dir(cred), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(cred, []byte(`{"oauth":true}`), 0o600); err != nil { + t.Fatal(err) + } + if st := (&claudeClient{}).AuthState(); st != AuthOK { + t.Fatalf("claude credentials file = %s, want ok", st) + } + + oauth := filepath.Join(dir, ".claude.json") + _ = os.Remove(cred) + if err := os.WriteFile(oauth, []byte(`{"oauthAccount":{"uuid":"x"}}`), 0o600); err != nil { + t.Fatal(err) + } + if st := (&claudeClient{}).AuthState(); st != AuthOK { + t.Fatalf("claude oauth json = %s, want ok", st) + } + + grokAuth := filepath.Join(dir, ".grok", "auth.json") + if err := os.MkdirAll(filepath.Dir(grokAuth), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(grokAuth, []byte(`{"token":"x"}`), 0o600); err != nil { + t.Fatal(err) + } + if st := (&grokClient{}).AuthState(); st != AuthOK { + t.Fatalf("grok auth.json = %s, want ok", st) + } + + codexAuth := filepath.Join(dir, ".codex", "auth.json") + if err := os.MkdirAll(filepath.Dir(codexAuth), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(codexAuth, []byte(`{}`), 0o600); err != nil { + t.Fatal(err) + } + if st := (&codexClient{}).AuthState(); st != AuthOK { + t.Fatalf("codex auth.json = %s, want ok", st) + } + + if st := (&vscodeClient{}).AuthState(); st != AuthUnknown { + t.Fatalf("vscode = %s, want unknown", st) + } +} + +func TestAuthStateEnvKeys(t *testing.T) { + withTempHomeDir(t) + t.Setenv("ANTHROPIC_API_KEY", "sk-test") + if st := (&claudeClient{}).AuthState(); st != AuthOK { + t.Fatalf("claude env = %s, want ok", st) + } +} + +func TestRankedCLIAgentsAuthFirst(t *testing.T) { + clearAuthEnv(t) + dir := withTempHomeDir(t) + bin := filepath.Join(dir, "bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + for _, name := range []string{"claude", "codex"} { + if err := os.WriteFile(filepath.Join(bin, name), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + } + t.Setenv("PATH", bin) + t.Setenv("XDG_CONFIG_HOME", "") + if err := os.MkdirAll(filepath.Join(dir, ".claude"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, ".claude", ".credentials.json"), []byte(`{"ok":1}`), 0o600); err != nil { + t.Fatal(err) + } + ranked := RankedCLIAgents() + if len(ranked) < 2 { + t.Fatalf("expected claude+codex, got %+v", ranked) + } + if ranked[0].Name != "claude" || ranked[0].Auth != AuthOK { + t.Fatalf("want claude/ok first, got %+v", ranked[0]) + } + if ranked[1].Name != "codex" { + t.Fatalf("want codex second, got %+v", ranked[1]) + } + for _, a := range ranked { + if a.Bundled { + t.Fatalf("PATH agents present: bundled opencode should not appear: %+v", a) + } + } +} + +func TestRankedCLIAgentsBundledOnlyWhenNoneOnPath(t *testing.T) { + clearAuthEnv(t) + dir := withTempHomeDir(t) + t.Setenv("PATH", filepath.Join(dir, "empty-bin")) + t.Setenv("XDG_CONFIG_HOME", "") + if err := os.MkdirAll(filepath.Join(dir, ".gemini"), 0o755); err != nil { + t.Fatal(err) + } + ranked := RankedCLIAgents() + if len(ranked) == 0 { + t.Fatal("expected bundled opencode when nothing is on PATH") + } + if ranked[0].Name != "opencode" || !ranked[0].Bundled { + t.Fatalf("want bundled opencode, got %+v", ranked[0]) + } + for _, a := range ranked { + if a.Name == "gemini" { + t.Fatalf("gemini config dir without binary must not rank: %+v", a) + } + } +} + +func TestDoctorMatrixOmitsOffPathAgents(t *testing.T) { + clearAuthEnv(t) + dir := withTempHomeDir(t) + bin := filepath.Join(dir, "bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(bin, "claude"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin) + if err := os.MkdirAll(filepath.Join(dir, ".gemini"), 0o755); err != nil { + t.Fatal(err) + } + if err := SaveProfile(DefaultProfile()); err != nil { + t.Fatal(err) + } + r, err := Doctor(context.Background()) + if err != nil { + t.Fatal(err) + } + if r.Matrix == nil { + t.Fatal("expected matrix") + } + for _, c := range r.Matrix.Clients { + if c == "gemini" { + t.Fatal("gemini not on PATH must not appear in the matrix") + } + } + var hasAuth bool + for _, row := range r.Matrix.Rows { + if row.Label == "agent on PATH" { + t.Fatal("PATH row should be gone") + } + if row.Label == "auth" { + hasAuth = true + } + } + if !hasAuth { + t.Fatal("expected auth row") + } + out := r.Render() + if strings.Contains(out, "binary not on $PATH") { + t.Fatalf("PATH-missing note should not render:\n%s", out) + } + if strings.Contains(out, "GEMINI") { + t.Fatalf("gemini column should not render:\n%s", out) + } +} + +func TestNestedLaunchEnvMarkers(t *testing.T) { + t.Setenv("CLAUDECODE", "") + t.Setenv("CURSOR_TRACE_ID", "") + t.Setenv("OPENCODE", "") + t.Setenv("OPENCODE_SESSION", "") + if NestedLaunch() { + t.Fatal("empty env should not nest") + } + t.Setenv("CLAUDECODE", "1") + if !NestedLaunch() { + t.Fatal("CLAUDECODE should nest") + } + t.Setenv("CLAUDECODE", "") + t.Setenv("CURSOR_TRACE_ID", "abc") + if !NestedLaunch() { + t.Fatal("CURSOR_TRACE_ID should nest") + } + t.Setenv("CURSOR_TRACE_ID", "") + t.Setenv("OPENCODE_SESSION", "s1") + if !NestedLaunch() { + t.Fatal("OPENCODE_SESSION should nest") + } +} + +func TestCursorAuthEnvKeyWins(t *testing.T) { + clearAuthEnv(t) + resetCursorStatusCache() + t.Cleanup(resetCursorStatusCache) + t.Setenv("CURSOR_API_KEY", "key-123") + if st := (&cursorClient{}).AuthState(); st != AuthOK { + t.Fatalf("CURSOR_API_KEY = %s, want ok", st) + } +} + +// Cursor keeps browser-login tokens in the OS keychain, so AuthState shells +// out to `cursor-agent status --format json`. When that CLI is absent the +// probe must not claim the user is signed out. +func TestCursorAuthWithoutCLI(t *testing.T) { + clearAuthEnv(t) + resetCursorStatusCache() + t.Cleanup(resetCursorStatusCache) + t.Setenv("PATH", t.TempDir()) + st := (&cursorClient{}).AuthState() + if _, ok := cursorStatusAuth(); ok { + t.Fatal("no cursor-agent on PATH, yet the probe answered") + } + want := AuthNone + if runtime.GOOS == "darwin" || runtime.GOOS == "windows" { + want = AuthUnknown + } + if st != want { + t.Fatalf("AuthState = %s, want %s", st, want) + } +} + +func TestCursorStatusAuthParsesCLI(t *testing.T) { + clearAuthEnv(t) + resetCursorStatusCache() + t.Cleanup(resetCursorStatusCache) + dir := t.TempDir() + stub := filepath.Join(dir, "cursor-agent") + // echo is a shell builtin: the stub's PATH holds only this dir. + script := "#!/bin/sh\necho '{\"status\":\"authenticated\",\"isAuthenticated\":true}'\n" + if err := os.WriteFile(stub, []byte(script), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", dir) + + authed, ok := cursorStatusAuth() + if !ok || !authed { + t.Fatalf("authed=%v answered=%v", authed, ok) + } + if st := (&cursorClient{}).AuthState(); st != AuthOK { + t.Fatalf("AuthState = %s, want ok", st) + } + + // Logged out: the CLI still exits 0, so the field decides. + script = "#!/bin/sh\necho '{\"status\":\"unauthenticated\",\"isAuthenticated\":false}'\n" + if err := os.WriteFile(stub, []byte(script), 0o755); err != nil { + t.Fatal(err) + } + resetCursorStatusCache() + authed, ok = cursorStatusAuth() + if !ok || authed { + t.Fatalf("logged out: authed=%v answered=%v", authed, ok) + } + if st := (&cursorClient{}).AuthState(); st != AuthNone { + t.Fatalf("AuthState = %s, want none", st) + } +} diff --git a/core/sling/assist/doctor.go b/core/sling/assist/doctor.go new file mode 100644 index 000000000..b2c388e6c --- /dev/null +++ b/core/sling/assist/doctor.go @@ -0,0 +1,514 @@ +package assist + +import ( + "context" + "encoding/json" + "fmt" + "os" + "strings" + + "github.com/flarco/g" + "github.com/jedib0t/go-pretty/table" + "github.com/jedib0t/go-pretty/text" + "github.com/slingdata-io/sling-cli/core" + "github.com/slingdata-io/sling-cli/core/env" +) + +// CellState is one cell in the doctor matrix. +type CellState int + +const ( + CellOK CellState = iota // ✓ + CellFail // ✗ + CellNA // ⊘ unsupported + CellEmpty // — not applicable +) + +// CheckResult is a typed status from Client.CheckSkills / CheckMCP. +type CheckResult struct { + State CellState `json:"state"` + Skill string `json:"skill,omitempty"` // set for per-skill checks + Note string `json:"note,omitempty"` // short detail, no glyph +} + +// Render returns a CLI display line (e.g. "✓ claude: sling"). +func (r CheckResult) Render(clientName string) string { + label := clientName + if r.Skill != "" { + if r.Note != "" { + return fmt.Sprintf("%s %s: %s — %s", r.State.Glyph(), label, r.Skill, r.Note) + } + return fmt.Sprintf("%s %s: %s", r.State.Glyph(), label, r.Skill) + } + if r.Note != "" { + return fmt.Sprintf("%s %s: %s", r.State.Glyph(), label, r.Note) + } + return fmt.Sprintf("%s %s", r.State.Glyph(), label) +} + +// Glyph returns the terminal marker for a cell state. +func (s CellState) Glyph() string { + switch s { + case CellOK: + return "✓" + case CellFail: + return "✗" + case CellNA: + return "⊘" + case CellEmpty: + return "—" + default: + return "?" + } +} + +// String is the JSON/API token for a cell state. +func (s CellState) String() string { + switch s { + case CellOK: + return "ok" + case CellFail: + return "fail" + case CellNA: + return "na" + case CellEmpty: + return "empty" + default: + return "unknown" + } +} + +// MarshalJSON encodes CellState as a stable string. +func (s CellState) MarshalJSON() ([]byte, error) { + return json.Marshal(s.String()) +} + +func checkOK(note string) CheckResult { + return CheckResult{State: CellOK, Note: note} +} + +func checkFail(note string) CheckResult { + return CheckResult{State: CellFail, Note: note} +} + +func checkNA(note string) CheckResult { + return CheckResult{State: CellNA, Note: note} +} + +func checkEmpty(note string) CheckResult { + return CheckResult{State: CellEmpty, Note: note} +} + +func checkSkill(state CellState, skill, note string) CheckResult { + return CheckResult{State: state, Skill: skill, Note: note} +} + +// MatrixRow is one row in the agent × capability matrix. +type MatrixRow struct { + Label string `json:"label"` + Cells map[string]CellState `json:"cells"` // client name → state + Notes map[string]string `json:"notes,omitempty"` +} + +// DoctorMatrix is the cross-check table for detected clients. +type DoctorMatrix struct { + Clients []string `json:"clients"` + Rows []MatrixRow `json:"rows"` +} + +// DoctorFinding is one structured global check (profile, skill, env, version). +type DoctorFinding struct { + ID string `json:"id"` + OK bool `json:"ok"` + Summary string `json:"summary"` + Detail string `json:"detail,omitempty"` + Hint string `json:"hint,omitempty"` +} + +// DoctorReport is the `sling assist setup --doctor` result. +type DoctorReport struct { + OK bool `json:"ok"` + SlingVersion string `json:"sling_version"` + Findings []DoctorFinding `json:"findings"` + Matrix *DoctorMatrix `json:"matrix,omitempty"` + Lines []string `json:"-"` // CLI glyph prose +} + +// AddFinding records a structured finding and a CLI display line. +func (r *DoctorReport) AddFinding(f DoctorFinding) { + r.Findings = append(r.Findings, f) + glyph := "✓" + if !f.OK { + glyph = "✗" + r.OK = false + } + line := fmt.Sprintf("%s %s", glyph, f.Summary) + if f.Hint != "" { + line += " → " + f.Hint + } + r.Lines = append(r.Lines, line) +} + +// Add maps glyph prose into a finding. +func (r *DoctorReport) Add(pass bool, line string) { + sum := strings.TrimSpace(line) + for _, pfx := range []string{"✓ ", "✗ ", "⊘ ", "— "} { + sum = strings.TrimPrefix(sum, pfx) + } + hint := "" + if i := strings.Index(sum, " → "); i >= 0 { + hint = strings.TrimSpace(sum[i+4:]) + sum = strings.TrimSpace(sum[:i]) + } + id := "misc" + if parts := strings.SplitN(sum, ":", 2); len(parts) > 0 { + id = strings.TrimSpace(strings.ReplaceAll(parts[0], " ", "_")) + } + r.AddFinding(DoctorFinding{ID: id, OK: pass, Summary: sum, Hint: hint}) +} + +// ToJSON returns a pretty-printed doctor.json payload. +func (r *DoctorReport) ToJSON() ([]byte, error) { + if r == nil { + return []byte("{}"), nil + } + return json.MarshalIndent(r, "", " ") +} + +// DoctorOptions configures Doctor. Zero value uses ScopeUser. +type DoctorOptions struct { + Scope Scope // must match install scope under test +} + +// Doctor probes the install end-to-end. opts[0].Scope defaults to ScopeUser. +func Doctor(ctx context.Context, opts ...DoctorOptions) (*DoctorReport, error) { + if ctx == nil { + ctx = context.Background() + } + if err := ctx.Err(); err != nil { + return nil, err + } + scope := ScopeUser + if len(opts) > 0 { + scope = opts[0].Scope + } + + r := &DoctorReport{OK: true, SlingVersion: core.Version} + + prof, exists, err := LoadProfile() + switch { + case err != nil: + r.AddFinding(DoctorFinding{ + ID: "profile", OK: false, + Summary: fmt.Sprintf("profile: %v", err), + Hint: "run: sling assist setup", + }) + case !exists: + r.AddFinding(DoctorFinding{ + ID: "profile", OK: false, + Summary: "profile: env.SLING_ASSIST missing in env.yaml", + Hint: "run: sling assist setup", + }) + default: + r.AddFinding(DoctorFinding{ + ID: "profile", OK: true, + Summary: fmt.Sprintf("profile: agent=%s, hint_in_errors=%v", prof.Agent, prof.HintInErrors), + }) + } + + if err := ctx.Err(); err != nil { + return r, err + } + + skillNames := listSkillNames() + for _, name := range skillNames { + if err := ctx.Err(); err != nil { + return r, err + } + canonical := canonicalSkillPath(name) + id := "skills." + name + if !g.PathExists(canonical) { + r.AddFinding(DoctorFinding{ + ID: id, OK: false, + Summary: fmt.Sprintf("skills: %s missing in ~/.agents/skills/", name), + Hint: "run: sling assist setup", + }) + continue + } + ok, detail, merr := skillMatchesEmbedded(name) + if merr != nil { + r.AddFinding(DoctorFinding{ + ID: id, OK: false, + Summary: fmt.Sprintf("skills: %s could not be compared", name), + Detail: merr.Error(), + }) + continue + } + if !ok { + r.AddFinding(DoctorFinding{ + ID: id, OK: false, + Summary: fmt.Sprintf("skills: %s drifted from embedded", name), + Hint: "run: sling assist setup", + Detail: detail, + }) + continue + } + r.AddFinding(DoctorFinding{ + ID: id, OK: true, + Summary: fmt.Sprintf("skills: %s matches embedded", name), + }) + } + + if !g.PathExists(envFilePath()) { + r.AddFinding(DoctorFinding{ + ID: "env", OK: false, + Summary: fmt.Sprintf("env: %s missing", envFilePath()), + }) + } else { + r.AddFinding(DoctorFinding{ + ID: "env", OK: true, + Summary: fmt.Sprintf("env: %s present", envFilePath()), + }) + } + + stamp, _ := os.ReadFile(VersionFilePath()) + stampVer := strings.TrimSpace(string(stamp)) + switch { + case stampVer == "": + r.AddFinding(DoctorFinding{ + ID: "version", OK: false, + Summary: "version: ~/.sling/assist/version not stamped", + Hint: "run: sling assist setup", + }) + case stampVer == versionUninstalled: + r.AddFinding(DoctorFinding{ + ID: "version", OK: false, + Summary: "version: assist uninstalled", + Hint: "run: sling assist setup", + }) + case stampVer != core.Version: + r.AddFinding(DoctorFinding{ + ID: "version", OK: false, + Summary: fmt.Sprintf("version: stamped %q but binary is %q (auto-refresh did not run)", stampVer, core.Version), + }) + default: + r.AddFinding(DoctorFinding{ + ID: "version", OK: true, + Summary: fmt.Sprintf("version: %s", core.Version), + }) + } + + if err := ctx.Err(); err != nil { + return r, err + } + + r.addZenFinding() + + detected := DetectedClients() + if len(detected) == 0 { + r.AddFinding(DoctorFinding{ + ID: "clients", OK: false, + Summary: "clients: no CLI agent on $PATH", + Hint: "install claude, codex, gemini, cursor, opencode, pi, or grok — or run sling assist setup to install OpenCode", + }) + return r, nil + } + r.Matrix = buildMatrix(ctx, detected, skillNames, scope) + for _, row := range r.Matrix.Rows { + for _, cl := range r.Matrix.Clients { + if row.Cells[cl] == CellFail { + r.OK = false + } + } + } + return r, nil +} + +func buildMatrix(ctx context.Context, detected []Client, skillNames []string, scope Scope) *DoctorMatrix { + m := &DoctorMatrix{} + for _, c := range detected { + m.Clients = append(m.Clients, c.Name()) + } + + authRow := MatrixRow{Label: "auth", Cells: map[string]CellState{}, Notes: map[string]string{}} + for _, c := range detected { + if c.Kind() != KindCLIAgent { + authRow.Cells[c.Name()] = CellEmpty + continue + } + authRow.Cells[c.Name()] = c.AuthState().cell() + } + m.Rows = append(m.Rows, authRow) + + mcpRow := MatrixRow{Label: "MCP", Cells: map[string]CellState{}, Notes: map[string]string{}} + for _, c := range detected { + res := c.CheckMCP(ctx, scope) + mcpRow.Cells[c.Name()] = res.State + if res.State == CellFail && res.Note != "" { + mcpRow.Notes[c.Name()] = res.Note + } + } + m.Rows = append(m.Rows, mcpRow) + + for _, skill := range skillNames { + row := MatrixRow{Label: skill, Cells: map[string]CellState{}, Notes: map[string]string{}} + for _, c := range detected { + results := c.CheckSkills(ctx, []string{skill}, scope) + if len(results) == 0 { + row.Cells[c.Name()] = CellNA + continue + } + res := results[0] + row.Cells[c.Name()] = res.State + if res.State == CellFail && res.Note != "" { + row.Notes[c.Name()] = res.Note + } + } + m.Rows = append(m.Rows, row) + } + return m +} + +// Render is the CLI doctor report (glyph lines + matrix). +func (r *DoctorReport) Render() string { + if r == nil { + return "" + } + var b strings.Builder + for _, line := range r.Lines { + b.WriteString(colorizeDoctorLine(line)) + b.WriteByte('\n') + } + if r.Matrix != nil { + b.WriteByte('\n') + b.WriteString(env.BlueString("Agent × Capability:")) + b.WriteByte('\n') + b.WriteString(r.Matrix.render()) + } + return b.String() +} + +// MissingComponents lists install pieces that look incomplete on disk. +func (r *DoctorReport) MissingComponents() []string { + if r == nil || r.OK { + return nil + } + skillsBad, mcpBad := false, false + for _, f := range r.Findings { + if !f.OK && strings.HasPrefix(f.ID, "skills.") { + skillsBad = true + } + } + for _, line := range r.Lines { + if strings.HasPrefix(strings.TrimSpace(line), "✗ skills") { + skillsBad = true + } + } + if r.Matrix != nil { + for _, row := range r.Matrix.Rows { + for _, c := range r.Matrix.Clients { + if row.Cells[c] == CellFail { + switch row.Label { + case "MCP": + mcpBad = true + case "auth": + default: + skillsBad = true + } + } + } + } + } + var out []string + if skillsBad { + out = append(out, "skills") + } + if mcpBad { + out = append(out, "mcp") + } + return out +} + +func (m *DoctorMatrix) render() string { + t := table.NewWriter() + t.SetStyle(table.StyleRounded) + + header := table.Row{""} + for _, c := range m.Clients { + header = append(header, c) + } + t.AppendHeader(header) + + var notes []string + for _, row := range m.Rows { + r := table.Row{row.Label} + for _, c := range m.Clients { + r = append(r, renderCell(row.Cells[c])) + } + t.AppendRow(r) + for _, c := range m.Clients { + if note, ok := row.Notes[c]; ok && note != "" { + notes = append(notes, fmt.Sprintf(" %s/%s: %s", + env.YellowString(c), env.YellowString(row.Label), note)) + } + } + } + + var colCfgs []table.ColumnConfig + for i := range m.Clients { + colCfgs = append(colCfgs, table.ColumnConfig{ + Number: i + 2, + Align: text.AlignCenter, + AlignHeader: text.AlignCenter, + }) + } + t.SetColumnConfigs(colCfgs) + + out := t.Render() + "\n" + if len(notes) > 0 { + out += "\n " + env.YellowString("Notes:") + "\n" + for _, n := range notes { + out += n + "\n" + } + } + return out +} + +func renderCell(s CellState) string { + switch s { + case CellOK: + return env.GreenString("✓") + case CellFail: + return env.RedString("✗") + case CellNA: + return env.YellowString("⊘") + default: + return env.DarkGrayString("—") + } +} + +func colorizeDoctorLine(line string) string { + trimmed := strings.TrimLeft(line, " ") + indent := line[:len(line)-len(trimmed)] + var prefix, rest string + switch { + case strings.HasPrefix(trimmed, "✓"): + prefix = env.GreenString("✓") + rest = strings.TrimPrefix(trimmed, "✓") + case strings.HasPrefix(trimmed, "✗"): + prefix = env.RedString("✗") + rest = strings.TrimPrefix(trimmed, "✗") + case strings.HasPrefix(trimmed, "⊘"): + prefix = env.YellowString("⊘") + rest = strings.TrimPrefix(trimmed, "⊘") + case strings.HasPrefix(trimmed, "—"): + prefix = env.DarkGrayString("—") + rest = strings.TrimPrefix(trimmed, "—") + default: + return line + } + if idx := strings.Index(rest, "→ run:"); idx >= 0 { + rest = rest[:idx] + env.CyanString(rest[idx:]) + } + return indent + prefix + rest +} diff --git a/core/sling/assist/history.go b/core/sling/assist/history.go new file mode 100644 index 000000000..a895cafba --- /dev/null +++ b/core/sling/assist/history.go @@ -0,0 +1,430 @@ +package assist + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "time" + + tea "github.com/charmbracelet/bubbletea" + "github.com/charmbracelet/lipgloss" + "github.com/flarco/g" + "gopkg.in/yaml.v3" +) + +// AnswersFile is the persisted state of an assist run — saved to +// //answers.yaml. +type AnswersFile struct { + SchemaVersion int `yaml:"schema_version" json:"schema_version"` + Name string `yaml:"name" json:"name"` + Task string `yaml:"task" json:"task"` + TemplateVersion string `yaml:"template_version,omitempty" json:"template_version"` + SlingVersion string `yaml:"sling_version" json:"sling_version"` + Created time.Time `yaml:"created" json:"created"` + Agent string `yaml:"agent" json:"agent"` + Parent string `yaml:"parent,omitempty" json:"parent"` + Cwd string `yaml:"cwd" json:"cwd"` + Answers map[string]any `yaml:"answers" json:"answers"` +} + +// Meta is the runtime side-record of an entry — saved to //meta.json. +type Meta struct { + ID string `json:"id"` + Task string `json:"task"` + Agent string `json:"agent"` + Model string `json:"model,omitempty"` + HarnessSessionID string `json:"harness_session_id,omitempty"` + LaunchedAt *time.Time `json:"launched_at"` + Doctor map[string]any `json:"doctor,omitempty"` + Parent string `json:"parent,omitempty"` +} + +// Entry is a loaded view of one history dir, used by the listing/picker. +type Entry struct { + ID string + Path string + Answers AnswersFile + Meta Meta +} + +// SaveEntry writes //{answers.yaml, prompt.md, meta.json}. +// id is generated from Created + slug if empty. +func SaveEntry(a AnswersFile, prompt string, m Meta) (string, error) { + if a.SchemaVersion == 0 { + a.SchemaVersion = SchemaVersion + } + if a.Created.IsZero() { + a.Created = time.Now().UTC() + } + id := m.ID + if id == "" { + slug := slugify(a.Name) + if slug == "" { + slug = slugify(a.Task) + } + id = a.Created.UTC().Format("2006-01-02_15-04-05") + "_" + slug + m.ID = id + } + dir := filepath.Join(HistoryDir(), id) + if err := os.MkdirAll(dir, 0o755); err != nil { + return "", g.Error(err, "mkdir %s", dir) + } + + ay, err := yaml.Marshal(a) + if err != nil { + return "", g.Error(err, "marshal answers") + } + if err := os.WriteFile(filepath.Join(dir, "answers.yaml"), ay, 0o644); err != nil { + return "", err + } + if err := os.WriteFile(filepath.Join(dir, "prompt.md"), []byte(prompt), 0o644); err != nil { + return "", err + } + + e := Entry{ID: id, Path: dir, Answers: a, Meta: m} + if err := e.saveMeta(); err != nil { + return "", err + } + return id, nil +} + +func (e Entry) saveMeta() error { + mj, err := json.MarshalIndent(e.Meta, "", " ") + if err != nil { + return err + } + mj = append(mj, '\n') + return os.WriteFile(filepath.Join(e.Path, "meta.json"), mj, 0o644) +} + +// LoadEntry reads //. +func LoadEntry(id string) (Entry, error) { + dir := filepath.Join(HistoryDir(), id) + e := Entry{ID: id, Path: dir} + ay, err := os.ReadFile(filepath.Join(dir, "answers.yaml")) + if err != nil { + return e, g.Error(err, "read answers") + } + if err := yaml.Unmarshal(ay, &e.Answers); err != nil { + return e, g.Error(err, "parse answers") + } + if mj, err := os.ReadFile(filepath.Join(dir, "meta.json")); err == nil { + _ = json.Unmarshal(mj, &e.Meta) + } + return e, nil +} + +// ListEntries returns all entries in ~/.sling/assist/history/, most-recent first. +func ListEntries() ([]Entry, error) { + root := HistoryDir() + dirs, err := os.ReadDir(root) + if err != nil { + return nil, g.Error(err, "read %s", root) + } + out := []Entry{} + for _, d := range dirs { + if !d.IsDir() || strings.HasPrefix(d.Name(), ".") { + continue + } + e, err := LoadEntry(d.Name()) + if err != nil { + continue + } + out = append(out, e) + } + sort.Slice(out, func(i, j int) bool { + return out[i].Answers.Created.After(out[j].Answers.Created) + }) + return out, nil +} + +// AutoTrim deletes the oldest entries until at most HistoryMaxEntries remain. +func AutoTrim() error { + entries, err := ListEntries() + if err != nil { + return err + } + if len(entries) <= HistoryMaxEntries { + return nil + } + var first error + for _, e := range entries[HistoryMaxEntries:] { + if rmErr := os.RemoveAll(e.Path); rmErr != nil && first == nil { + first = g.Error(rmErr, "remove history %s", e.Path) + } + } + return first +} + +// FormatRelative returns a short human relative time like "3h ago", "yesterday", +// "2026-05-01" (for entries older than a week). +func FormatRelative(t time.Time) string { + d := time.Since(t) + switch { + case d < time.Minute: + return "just now" + case d < time.Hour: + return fmt.Sprintf("%dm ago", int(d.Minutes())) + case d < 24*time.Hour: + return fmt.Sprintf("%dh ago", int(d.Hours())) + case d < 48*time.Hour: + return "yesterday" + case d < 7*24*time.Hour: + return fmt.Sprintf("%d days ago", int(d.Hours()/24)) + default: + return t.Format("2006-01-02") + } +} + +// maxSlugLen caps the slug so _ stays a short directory name. +const maxSlugLen = 40 + +func slugify(s string) string { + s = strings.ToLower(strings.TrimSpace(s)) + out := strings.Builder{} + last := rune(0) + for _, r := range s { + switch { + case r >= 'a' && r <= 'z', r >= '0' && r <= '9': + out.WriteRune(r) + last = r + case r == ' ' || r == '_' || r == '-': + if last == '-' { + continue // collapse runs of separators + } + out.WriteRune('-') + last = '-' + } + } + res := strings.Trim(out.String(), "-") + if res == "" { + return "entry" + } + return truncateSlug(res, maxSlugLen) +} + +// truncateSlug cuts at the last word boundary within n, so the slug stays readable. +func truncateSlug(s string, n int) string { + if len(s) <= n { + return s + } + cut := s[:n] + if i := strings.LastIndexByte(cut, '-'); i > 0 { + cut = cut[:i] + } + return strings.Trim(cut, "-") +} + +func collapseHome(p string) string { + home := userHome() + if home == "" { + return p + } + if strings.HasPrefix(p, home) { + return "~" + strings.TrimPrefix(p, home) + } + return p +} + +func mustGetwd() string { + wd, err := os.Getwd() + if err != nil { + g.Debug("assist: getwd failed: %s", err.Error()) + return "" + } + return wd +} + +// PickHistoryEntry opens a searchable table of recent sessions. +// Returns ErrUserAborted when the user cancels. +func PickHistoryEntry() (Entry, error) { + entries, err := ListEntries() + if err != nil { + return Entry{}, err + } + if len(entries) == 0 { + return Entry{}, g.Error("no sessions yet — run `sling assist` first") + } + if !isTTY(os.Stdin) || !isTTY(os.Stdout) { + return Entry{}, g.Error("pass a session id (`sling assist --resume `) when not on a TTY") + } + + m := newPickerModel(entries) + p := tea.NewProgram(m, tea.WithAltScreen()) + final, err := p.Run() + if err != nil { + return Entry{}, g.Error(err, "session picker") + } + got, ok := final.(pickerModel) + if !ok || got.chosen == nil { + return Entry{}, ErrUserAborted + } + return *got.chosen, nil +} + +func filterEntries(entries []Entry, query string) []Entry { + q := strings.ToLower(strings.TrimSpace(query)) + if q == "" { + return entries + } + out := []Entry{} + for _, e := range entries { + if entryMatches(e, q) { + out = append(out, e) + } + } + return out +} + +func entryMatches(e Entry, q string) bool { + ask := "" + if e.Answers.Answers != nil { + if v, ok := e.Answers.Answers["ask"].(string); ok { + ask = v + } + if v, ok := e.Answers.Answers["intention"].(string); ok && ask == "" { + ask = v + } + } + hay := strings.ToLower(strings.Join([]string{ + e.ID, e.Answers.Name, e.Answers.Task, e.Answers.Agent, e.Answers.Cwd, ask, e.Meta.Agent, + }, " ")) + return strings.Contains(hay, q) +} + +type pickerModel struct { + all []Entry + filtered []Entry + query string + cursor int + chosen *Entry + width int + height int + quit bool +} + +func newPickerModel(entries []Entry) pickerModel { + return pickerModel{ + all: entries, + filtered: entries, + width: 80, + height: 24, + } +} + +func (m pickerModel) Init() tea.Cmd { return nil } + +func (m pickerModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) { + switch msg := msg.(type) { + case tea.WindowSizeMsg: + m.width = msg.Width + m.height = msg.Height + case tea.KeyMsg: + switch msg.String() { + case "ctrl+c", "esc": + m.quit = true + return m, tea.Quit + case "enter": + if m.cursor >= 0 && m.cursor < len(m.filtered) { + e := m.filtered[m.cursor] + m.chosen = &e + } + return m, tea.Quit + case "up", "k": + if m.cursor > 0 { + m.cursor-- + } + case "down", "j": + if m.cursor < len(m.filtered)-1 { + m.cursor++ + } + case "backspace": + if m.query != "" { + r := []rune(m.query) + m.query = string(r[:len(r)-1]) + m.filtered = filterEntries(m.all, m.query) + if m.cursor >= len(m.filtered) { + m.cursor = max(0, len(m.filtered)-1) + } + } + default: + if msg.Type == tea.KeyRunes { + m.query += string(msg.Runes) + m.filtered = filterEntries(m.all, m.query) + m.cursor = 0 + } + } + } + return m, nil +} + +func (m pickerModel) View() string { + var b strings.Builder + title := lipgloss.NewStyle().Bold(true).Render("Resume session") + fmt.Fprintf(&b, "%s\n", title) + fmt.Fprintf(&b, " search: %s█\n\n", m.query) + + header := fmt.Sprintf(" %-28s %-16s %-10s %-12s %s", "ID", "TASK", "AGENT", "CREATED", "NAME") + fmt.Fprintln(&b, lipgloss.NewStyle().Faint(true).Render(header)) + + if len(m.filtered) == 0 { + fmt.Fprintln(&b, " (no matches)") + return b.String() + } + + rows := m.height - 8 + if rows < 3 { + rows = 3 + } + start := 0 + if m.cursor >= rows { + start = m.cursor - rows + 1 + } + end := start + rows + if end > len(m.filtered) { + end = len(m.filtered) + } + + sel := lipgloss.NewStyle().Reverse(true) + for i := start; i < end; i++ { + e := m.filtered[i] + agent := e.Answers.Agent + if agent == "" { + agent = e.Meta.Agent + } + if agent == "" { + agent = "—" + } + line := fmt.Sprintf(" %-28s %-16s %-10s %-12s %s", + clipRunes(e.ID, 28), + clipRunes(e.Answers.Task, 16), + clipRunes(agent, 10), + FormatRelative(e.Answers.Created), + clipRunes(e.Answers.Name, 24), + ) + if i == m.cursor { + line = sel.Render(line) + } + fmt.Fprintln(&b, line) + } + fmt.Fprintln(&b) + fmt.Fprintln(&b, lipgloss.NewStyle().Faint(true).Render(" ↑/↓ move enter resume esc abort")) + return b.String() +} + +func clipRunes(s string, n int) string { + if n <= 0 { + return "" + } + r := []rune(s) + if len(r) <= n { + return s + } + if n == 1 { + return "…" + } + return string(r[:n-1]) + "…" +} diff --git a/core/sling/assist/install.go b/core/sling/assist/install.go new file mode 100644 index 000000000..ebd3f0ff0 --- /dev/null +++ b/core/sling/assist/install.go @@ -0,0 +1,746 @@ +package assist + +import ( + "context" + "crypto/md5" + "encoding/hex" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "time" + + "github.com/charmbracelet/huh" + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core" +) + +// InstallOptions controls `sling assist setup` (install path). +type InstallOptions struct { + Reconfigure bool + SkillsOnly bool + MCPOnly bool + Scope Scope + NonInteractive bool + DefaultAgent string // used only on non-interactive first-run +} + +// InstallResult summarizes what install touched. +type InstallResult struct { + ProfileWritten bool + CanonicalSkillsDir string + WiredClients []ClientResult + SkippedClients []string // not detected +} + +// ClientResult is one row of the install summary. +type ClientResult struct { + Name string + Kind ClientKind + WroteSkills bool + WroteMCP bool + Authed AuthStatus + Notes string +} + +func requestedAgent(opts InstallOptions) string { + agent := strings.ToLower(strings.TrimSpace(opts.DefaultAgent)) + if agent != "" { + return agent + } + if prof, exists, err := LoadProfile(); err == nil && exists { + return strings.ToLower(strings.TrimSpace(prof.Agent)) + } + return "" +} + +func anyUsableCLIAgent() bool { + for _, c := range CLIAgents() { + if c.Detect() { + return true + } + } + return false +} + +// maybeEnsureOpenCode downloads opencode only when the user picked it, or when +// no other CLI agent is usable (bundled fallback). System binaries still win. +func maybeEnsureOpenCode(opts InstallOptions) error { + agent := requestedAgent(opts) + switch { + case agent == "opencode": + // user picked opencode + case agent != "" && agent != "auto": + return nil + case anyUsableCLIAgent(): + return nil + } + if _, err := EnsureBinOpenCode(); err != nil { + return err + } + return ApplyHarnessProviderConfig() +} + +// Install is idempotent install/refresh. Honors ctx between clients. +func Install(ctx context.Context, opts InstallOptions) (*InstallResult, error) { + if ctx == nil { + ctx = context.Background() + } + if err := ctx.Err(); err != nil { + return nil, err + } + res := &InstallResult{ + CanonicalSkillsDir: CanonicalSkillsDir(), + } + + if err := maybeEnsureOpenCode(opts); err != nil { + return nil, err + } + if !opts.SkillsOnly { + if err := maybeEnsureAgentBrowser(opts); err != nil { + return nil, err + } + } + + detected := DetectedClients() + if len(detected) == 0 { + return nil, g.Error("no AI agent on $PATH; install one of: claude, codex, gemini, cursor, opencode, pi, grok") + } + + prof, exists, err := LoadProfile() + if err != nil { + return nil, err + } + if !exists || opts.Reconfigure { + prof = DefaultProfile() + switch { + case opts.DefaultAgent != "": + prof.Agent = opts.DefaultAgent + default: + for _, c := range detected { + if c.Kind() == KindCLIAgent { + prof.Agent = c.Name() + break + } + } + if prof.Agent == "" { + prof.Agent = "auto" + } + } + if err := SaveProfile(prof); err != nil { + return nil, err + } + res.ProfileWritten = true + } + + skillNames := listSkillNames() + if !opts.MCPOnly { + if err := writeCanonicalBundle(skillNames); err != nil { + return nil, err + } + pruneRetiredSkills(ctx, opts.Scope) + } + + var wireErrs []string + for _, c := range detected { + if err := ctx.Err(); err != nil { + return res, err + } + row := ClientResult{Name: c.Name(), Kind: c.Kind(), Authed: c.AuthState()} + if !opts.MCPOnly { + if err := c.WriteSkills(ctx, skillNames, opts.Scope); err != nil { + row.Notes = fmt.Sprintf("skills: %v", err) + wireErrs = append(wireErrs, fmt.Sprintf("%s skills: %v", c.Name(), err)) + } else { + row.WroteSkills = true + } + } + if !opts.SkillsOnly { + if err := c.WriteMCP(ctx, opts.Scope); err != nil { + if row.Notes != "" { + row.Notes += "; " + } + row.Notes += fmt.Sprintf("mcp: %v", err) + wireErrs = append(wireErrs, fmt.Sprintf("%s mcp: %v", c.Name(), err)) + } else { + row.WroteMCP = true + } + } + res.WiredClients = append(res.WiredClients, row) + } + for _, c := range AllClients() { + if !c.Detect() { + res.SkippedClients = append(res.SkippedClients, c.Name()) + } + } + + if len(wireErrs) > 0 { + // Do not stamp on partial failure — retry on next install. + return res, g.Error("install incomplete: %s", strings.Join(wireErrs, "; ")) + } + + if err := os.WriteFile(VersionFilePath(), []byte(core.Version), 0o644); err != nil { + return res, g.Error(err, "could not stamp %s", VersionFilePath()) + } + + return res, nil +} + +// UninstallOptions controls `sling assist setup --uninstall`. +type UninstallOptions struct { + All bool + SkillsOnly bool + MCPOnly bool + Scope Scope + NonInteractive bool + IncludeClients []string // empty = all detected +} + +// versionUninstalled prevents AutoRefresh from resurrecting after uninstall. +const versionUninstalled = "uninstalled" + +// Uninstall removes sling skills/MCP only (never other tools' entries). +func Uninstall(ctx context.Context, opts UninstallOptions) error { + if ctx == nil { + ctx = context.Background() + } + if err := ctx.Err(); err != nil { + return err + } + skillNames := listSkillNames() + + pickClient := func(name string) bool { + if len(opts.IncludeClients) == 0 { + return true + } + for _, n := range opts.IncludeClients { + if n == name { + return true + } + } + return false + } + + var errs []string + for _, c := range AllClients() { + if err := ctx.Err(); err != nil { + return err + } + if !c.Detect() { + continue + } + if !pickClient(c.Name()) { + continue + } + if !opts.MCPOnly { + if err := c.RemoveSkills(ctx, skillNames, opts.Scope); err != nil { + errs = append(errs, fmt.Sprintf("%s skills: %v", c.Name(), err)) + } + } + if !opts.SkillsOnly { + if err := c.RemoveMCP(ctx, opts.Scope); err != nil { + errs = append(errs, fmt.Sprintf("%s mcp: %v", c.Name(), err)) + } + } + } + if !opts.MCPOnly { + if err := removeCanonicalBundle(skillNames); err != nil { + errs = append(errs, err.Error()) + } + // Mark uninstalled so AutoRefresh will not re-wire on upgrade. + if err := os.MkdirAll(AssistDir(), 0o755); err != nil { + errs = append(errs, fmt.Sprintf("mkdir assist: %v", err)) + } else if err := os.WriteFile(VersionFilePath(), []byte(versionUninstalled), 0o644); err != nil { + errs = append(errs, fmt.Sprintf("stamp uninstalled: %v", err)) + } + } + if len(errs) > 0 { + return g.Error("uninstall completed with errors: %s", strings.Join(errs, "; ")) + } + return nil +} + +// AutoRefresh updates installed skills to match the embedded bundle. +// It also removes retired skill directories. +// This function runs on each `sling assist` command. +// If no current skills exist, it does not install them. +// If the user ran uninstall, it does not install them again. +// Returns a notice line when something changed. +func AutoRefresh(ctx context.Context) (string, error) { + if ctx == nil { + ctx = context.Background() + } + if err := ctx.Err(); err != nil { + return "", err + } + + stamp, err := os.ReadFile(VersionFilePath()) + if err != nil && !os.IsNotExist(err) { + return "", g.Error(err, "could not read %s", VersionFilePath()) + } + s := strings.TrimSpace(string(stamp)) + if s == versionUninstalled { + return "", nil + } + + if !anyCurrentSkillsInstalled() { + // Remove leftover retired dirs. Do not install the current bundle. + pruneRetiredSkills(ctx, ScopeUser) + return "", nil + } + + lockPath := filepath.Join(AssistDir(), ".refresh-lock") + unlock, ok := tryRefreshLock(lockPath) + if !ok { + return "", nil + } + defer unlock() + + skillNames := listSkillNames() + changed := []string{} + var refreshErrs []string + for _, name := range skillNames { + if err := ctx.Err(); err != nil { + return "", err + } + didChange, err := syncCanonicalSkill(name) + if err != nil { + refreshErrs = append(refreshErrs, fmt.Sprintf("%s: %v", name, err)) + continue + } + if didChange { + changed = append(changed, name) + } + } + pruned := pruneRetiredSkills(ctx, ScopeUser) + + // Files already match the embed and the stamp is current. Skip re-wire. + if len(changed) == 0 && len(pruned) == 0 && s == core.Version && len(refreshErrs) == 0 { + return "", nil + } + + clients := []string{} + for _, c := range DetectedClients() { + if err := ctx.Err(); err != nil { + return "", err + } + if err := c.WriteSkills(ctx, skillNames, ScopeUser); err != nil { + refreshErrs = append(refreshErrs, fmt.Sprintf("%s: %v", c.Name(), err)) + continue + } + clients = append(clients, c.Name()) + } + + // Only stamp on full success so partial failure retries next time. + if len(refreshErrs) > 0 { + return "", g.Error("auto-refresh incomplete: %s", strings.Join(refreshErrs, "; ")) + } + if err := os.WriteFile(VersionFilePath(), []byte(core.Version), 0o644); err != nil { + return "", g.Error(err, "could not stamp %s", VersionFilePath()) + } + + if len(changed) == 0 && len(pruned) == 0 { + return "", nil + } + return fmt.Sprintf("sling: refreshed AI skills for v%s (%s)", core.Version, strings.Join(clients, ", ")), nil +} + +const refreshLockStale = 5 * time.Minute + +// tryRefreshLock acquires an exclusive lock file; unlock removes only if we still own it. +func tryRefreshLock(lockPath string) (func(), bool) { + token := fmt.Sprintf("%d-%d", os.Getpid(), time.Now().UnixNano()) + + create := func() (*os.File, error) { + return os.OpenFile(lockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644) + } + + f, err := create() + if err != nil { + if !os.IsExist(err) { + return nil, false + } + info, statErr := os.Stat(lockPath) + if statErr != nil || time.Since(info.ModTime()) < refreshLockStale { + return nil, false + } + // Stale lock: reclaim (racing peer loses on O_EXCL). + _ = os.Remove(lockPath) + f, err = create() + if err != nil { + return nil, false + } + } + _, _ = f.Write([]byte(token)) + _ = f.Close() + + unlock := func() { + got, err := os.ReadFile(lockPath) + if err == nil && string(got) == token { + _ = os.Remove(lockPath) + } + } + return unlock, true +} + +// retiredSkillNames are Sling-owned skill names from earlier bundle versions +// that no longer exist in the embed. Pruned on install/refresh so agents stop +// picking up their stale content. +var retiredSkillNames = []string{"sling-hooks", "sling-transforms", "sling-troubleshooting"} + +// pruneRetiredSkills removes retired skills from the canonical bundle and from +// per-skill client redirects. Best-effort: failures only debug-log. +// vscode is skipped — RemoveSkills unwires the whole bundle, +// and it references the canonical dir, so the canonical prune covers it. +// Returns the retired names that were present on disk and removed. +func pruneRetiredSkills(ctx context.Context, scope Scope) []string { + root := CanonicalSkillsDir() + pruned := []string{} + for _, name := range retiredSkillNames { + p := filepath.Join(root, name) + if !g.PathExists(p) { + continue + } + if err := os.RemoveAll(p); err != nil { + g.Debug("assist: prune retired skill %s: %s", name, err.Error()) + continue + } + pruned = append(pruned, name) + } + for _, c := range DetectedClients() { + if ctx.Err() != nil { + return pruned + } + if c.Name() == "vscode" { + continue + } + if err := c.RemoveSkills(ctx, retiredSkillNames, scope); err != nil { + g.Debug("assist: prune retired skills for %s: %s", c.Name(), err.Error()) + } + } + return pruned +} + +// anyCurrentSkillsInstalled reports whether at least one embedded skill +// is present in ~/.agents/skills/. Absence means the user has not run setup. +func anyCurrentSkillsInstalled() bool { + for _, name := range listSkillNames() { + if g.PathExists(canonicalSkillPath(name)) { + return true + } + } + return false +} + +// MD5OfFile returns the MD5 hex of a file. +func MD5OfFile(path string) (string, error) { + data, err := os.ReadFile(path) + if err != nil { + return "", err + } + sum := md5.Sum(data) + return hex.EncodeToString(sum[:]), nil +} + +// MD5OfEmbeddedSkill returns the MD5 of an embedded skill file. +func MD5OfEmbeddedSkill(rel string) (string, error) { + data, err := SkillsFS.ReadFile(filepath.ToSlash(filepath.Join("skills", rel))) + if err != nil { + return "", err + } + sum := md5.Sum(data) + return hex.EncodeToString(sum[:]), nil +} + +// SetupAction is what RunSetupActionForm returns. Drives the post-doctor +// branching in `sling assist setup` for users who already have a profile. +type SetupAction string + +const ( + SetupActionRefresh SetupAction = "refresh" // re-install everything (idempotent) + SetupActionInstallMissing SetupAction = "install_missing" // install only the failing components + SetupActionReconfigure SetupAction = "reconfigure" // re-prompt the profile form, then install + SetupActionUninstall SetupAction = "uninstall" // wipe everything + SetupActionExit SetupAction = "exit" // do nothing +) + +// ErrUserAborted is returned by interactive forms when the user declines. +var ErrUserAborted = errors.New("user aborted") + +// RunSetupActionForm runs after doctor has printed its report. +func RunSetupActionForm(report *DoctorReport) (SetupAction, error) { + missingLabel := "Install missing components" + hasFailures := report != nil && !report.OK + opts := []huh.Option[string]{} + if hasFailures { + opts = append(opts, huh.NewOption(missingLabel+" (recommended)", string(SetupActionInstallMissing))) + opts = append(opts, huh.NewOption("Re-install everything (refresh)", string(SetupActionRefresh))) + } else { + opts = append(opts, huh.NewOption("Re-install everything (refresh)", string(SetupActionRefresh))) + } + opts = append(opts, + huh.NewOption("Reconfigure (change preferred agent / scope)", string(SetupActionReconfigure)), + huh.NewOption("Uninstall everything", string(SetupActionUninstall)), + huh.NewOption("Exit (do nothing)", string(SetupActionExit)), + ) + + chosen := opts[0].Value + form := huh.NewForm( + huh.NewGroup( + huh.NewSelect[string](). + Title("What would you like to do?"). + Description("Doctor already ran — pick your next action."). + Options(opts...). + Value(&chosen), + ), + ).WithTheme(huh.ThemeCharm()) + if err := form.Run(); err != nil { + return SetupActionExit, g.Error(err, "setup form aborted") + } + return SetupAction(chosen), nil +} + +// HarnessConfirmResult is the first-run / setup confirm form. +type HarnessConfirmResult struct { + Agent string + HintInErrors bool + Scope string + Components []string +} + +func agentAuthLabel(a RankedAgent) string { + if a.Bundled { + choice := HarnessProviderChoice() + if choice.Disclosure != "" { + return fmt.Sprintf("%s (bundled fallback) — %s", a.Name, choice.Disclosure) + } + return fmt.Sprintf("%s (bundled fallback, keyed provider)", a.Name) + } + auth := "no auth" + switch a.Auth { + case AuthOK: + auth = "auth ok" + case AuthUnknown: + auth = "auth unknown" + } + return fmt.Sprintf("%s (detected, %s)", a.Name, auth) +} + +func harnessAgentDescription(ranked []RankedAgent) string { + base := "Detected agents with install and auth state. Authenticated first." + hasBundled := false + for _, a := range ranked { + if a.Bundled { + hasBundled = true + break + } + } + if !hasBundled { + return base + } + choice := HarnessProviderChoice() + if choice.Disclosure != "" { + return base + " OpenCode downloads only when no other agent is usable. Keyed alternative: set ANTHROPIC_API_KEY or OPENAI_API_KEY. " + choice.Disclosure + "." + } + return base + " OpenCode downloads only when no other agent is usable." +} + +// RunHarnessConfirmForm lists detected agents with install+auth state. +func RunHarnessConfirmForm(prefill Profile) (*HarnessConfirmResult, error) { + agent, err := resolveSetupAgent(prefill.Agent) + if err != nil { + return nil, err + } + + res := &HarnessConfirmResult{ + Agent: agent, + HintInErrors: prefill.HintInErrors, + Scope: "user", + Components: []string{"skills", "mcp"}, + } + if prefill.DefaultInstallScope != "" { + res.Scope = prefill.DefaultInstallScope + } + + fields := []huh.Field{ + huh.NewMultiSelect[string](). + Title("What would you like to install?"). + Description("Skills are markdown guides; MCP wires the Sling MCP server into each client."). + Options( + huh.NewOption("Skills (canonical bundle + per-client redirects)", "skills").Selected(true), + huh.NewOption("MCP server registration", "mcp").Selected(true), + ). + Value(&res.Components), + huh.NewConfirm(). + Title("Show AI hint in run errors?"). + Description("Append `sling assist error ` to failed `sling run` / `conns test` output."). + Value(&res.HintInErrors), + huh.NewSelect[string](). + Title("Install scope"). + Description("Where to wire skills + MCP. `user` writes to ~/./; `project` writes to ././"). + Options( + huh.NewOption("user (recommended)", "user"), + huh.NewOption("project", "project"), + ). + Value(&res.Scope), + } + if opts, ranked, ok := setupAgentSelectOptions(); ok { + fields = append([]huh.Field{ + huh.NewSelect[string](). + Title("Preferred agent"). + Description(harnessAgentDescription(ranked)). + Options(opts...). + Filtering(filteringFor(opts)). + Value(&res.Agent), + }, fields...) + } + + form := huh.NewForm(huh.NewGroup(fields...)).WithTheme(huh.ThemeCharm()) + if err := form.Run(); err != nil { + return nil, g.Error(err, "setup form aborted") + } + if len(res.Components) == 0 { + return nil, g.Error("no components selected") + } + return res, nil +} + +// InstallFormResult is what the first-run install form returns. +type InstallFormResult struct { + Agent string + HintInErrors bool + Scope string +} + +// RunInstallForm shows the first-run interactive form and returns the user's +// choices. +func RunInstallForm(prefill Profile) (*InstallFormResult, error) { + agent, err := resolveSetupAgent(prefill.Agent) + if err != nil { + return nil, err + } + + res := &InstallFormResult{ + Agent: agent, + HintInErrors: prefill.HintInErrors, + Scope: "user", + } + if prefill.DefaultInstallScope != "" { + res.Scope = prefill.DefaultInstallScope + } + + fields := []huh.Field{ + huh.NewConfirm(). + Title("Show AI hint in run errors?"). + Description("Append `sling assist error ` to failed `sling run` / `conns test` output."). + Value(&res.HintInErrors), + huh.NewSelect[string](). + Title("Install scope"). + Description("Where to wire skills + MCP. `user` writes to ~/./; `project` writes to ././"). + Options( + huh.NewOption("user (recommended)", "user"), + huh.NewOption("project", "project"), + ). + Value(&res.Scope), + } + if opts, ranked, ok := setupAgentSelectOptions(); ok { + fields = append([]huh.Field{ + huh.NewSelect[string](). + Title("Preferred agent"). + Description(harnessAgentDescription(ranked)). + Options(opts...). + Filtering(filteringFor(opts)). + Value(&res.Agent), + }, fields...) + } + + form := huh.NewForm(huh.NewGroup(fields...)).WithTheme(huh.ThemeCharm()) + if err := form.Run(); err != nil { + return nil, g.Error(err, "install form aborted") + } + return res, nil +} + +// resolveSetupAgent picks the agent for setup: the only one on PATH, else +// the authenticated one (via a later picker), else bundled OpenCode after confirm. +func resolveSetupAgent(current string) (string, error) { + agents, bundled := pathRanked() + switch { + case len(agents) == 0: + if bundled == nil { + return "", g.Error("no AI agent on $PATH; install one of: claude, codex, gemini, cursor, opencode, pi, grok") + } + if err := confirmInstallOpenCode(); err != nil { + return "", err + } + return bundled.Name, nil + case len(agents) == 1: + return agents[0].Name, nil + default: + if current != "" { + for _, a := range agents { + if a.Name == current { + return current, nil + } + } + } + return agents[0].Name, nil + } +} + +// setupAgentSelectOptions is the picker for two or more PATH agents. +func setupAgentSelectOptions() ([]huh.Option[string], []RankedAgent, bool) { + agents, _ := pathRanked() + if len(agents) < 2 { + return nil, agents, false + } + opts := make([]huh.Option[string], 0, len(agents)) + for i, a := range agents { + label := agentAuthLabel(a) + if i == 0 { + label += " — recommended" + } + opts = append(opts, huh.NewOption(label, a.Name)) + } + return opts, agents, true +} + +func confirmInstallOpenCode() error { + ok := false + form := huh.NewForm( + huh.NewGroup( + huh.NewConfirm(). + Title("No AI agent on $PATH"). + Description("Sling can install OpenCode and use it as the agent."). + Affirmative("Install OpenCode"). + Negative("Cancel"). + Value(&ok), + ), + ).WithTheme(huh.ThemeCharm()) + if err := form.Run(); err != nil { + return g.Error(err, "setup form aborted") + } + if !ok { + return ErrUserAborted + } + return nil +} + +func EnsureAssistReady() error { + prof, exists, err := LoadProfile() + if err != nil { + return err + } + if !exists { + return g.Error("Sling assist is not set up yet. Run:\n\n sling assist setup\n") + } + if _, err := ResolveAgent("", prof); err != nil { + return g.Error("no AI agent ready for assist. Run:\n\n sling assist setup\n\n(%s)", err.Error()) + } + return nil +} + +func filteringFor(opts []huh.Option[string]) bool { + return len(opts) >= 6 +} diff --git a/core/sling/assist/investigate.go b/core/sling/assist/investigate.go new file mode 100644 index 000000000..c63d5b612 --- /dev/null +++ b/core/sling/assist/investigate.go @@ -0,0 +1,1169 @@ +// Investigate surface: error signatures, snapshots, local exec picker, hints, sensitivity. + +package assist + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "time" + "unicode" + + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core/dbio" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/spf13/cast" +) + +// AlgorithmVersion labels the normalizer generation (not a material prefix). +const AlgorithmVersion = "v1" + +// Material prefixes distinguish pattern vs edge digests (composite signature). +const ( + PatternMaterialPrefix = "v1p" // skeleton-only + EdgeMaterialPrefix = "v1e" // source|target|skeleton +) + +// PartIDLen is hex chars kept for each of pattern and edge digests. +const PartIDLen = 8 + +// CompositeIDLen is PatternID + EdgeID (16 hex chars, no separator). +const CompositeIDLen = PartIDLen * 2 + +// SignatureIDLen is an alias for PartIDLen. +const SignatureIDLen = PartIDLen + +// SignMeta holds connector types only (never hostnames, DB names, or task type). +// Task type (db-db, file-db, …) is not stored: it is fully determined by the +// Kind of SourceType and TargetType (see InferredTaskType). +type SignMeta struct { + SourceType dbio.Type // e.g. dbio.TypeDbPrometheus + TargetType dbio.Type // e.g. dbio.TypeDbPostgres +} + +func MakeSignMeta() (sm SignMeta) { + taskMap, _ := g.UnmarshalMap(cast.ToString(env.TelMap["task"])) + src := cast.ToString(taskMap["source_type"]) + tgt := cast.ToString(taskMap["target_type"]) + if src == "" { + src = cast.ToString(env.TelMap["conn_type"]) + } + return SignMeta{SourceType: dbio.Type(src), TargetType: dbio.Type(tgt)} +} + +// Signature is the full result of SignError (composite: pattern + edge). +// +// PatternID = hex(sha256("v1p|" + skeleton))[0:8] +// EdgeID = hex(sha256("v1e|" + src + "|" + tgt + "|" + skeleton))[0:8] +// ID = PatternID + EdgeID // 16 hex, machine form +// IDDashed = PatternID + "-" + EdgeID +// +// Same skeleton ⇒ same PatternID across all source/target pairs. +// Same skeleton + same types ⇒ same EdgeID and ID. +type Signature struct { + ID string // 16 hex chars (pattern||edge) + PatternID string // 8 hex — skeleton only + EdgeID string // 8 hex — types + skeleton + Skeleton string // normalized message body + PatternMaterial string // v1p| + EdgeMaterial string // v1e||| + ShortLabel string // human-only slug (not hashed) + Algorithm string // e.g. v1 + Meta SignMeta +} + +// SignError normalizes errText + meta into a stable composite signature. +// Pure function: no I/O, deterministic across machines. +func SignError(errText string, meta SignMeta) Signature { + meta = normalizeMeta(meta) + skel := Skeleton(errText) + pMat := PatternMaterial(skel) + eMat := EdgeMaterial(skel, meta) + patternID := HashPart(pMat) + edgeID := HashPart(eMat) + return Signature{ + ID: patternID + edgeID, + PatternID: patternID, + EdgeID: edgeID, + Skeleton: skel, + PatternMaterial: pMat, + EdgeMaterial: eMat, + ShortLabel: ShortLabel(skel), + Algorithm: AlgorithmVersion, + Meta: meta, + } +} + +// IDDashed returns pattern-edge with a hyphen for humans / copy-paste. +func (s Signature) IDDashed() string { + if s.PatternID == "" || s.EdgeID == "" { + if len(s.ID) == CompositeIDLen { + return s.ID[:PartIDLen] + "-" + s.ID[PartIDLen:] + } + return s.ID + } + return s.PatternID + "-" + s.EdgeID +} + +// Display formats the signature for the failure footer. +// Example: a1b2c3d4-e4f9c2a1 (prometheus→postgres · no_stream_columns) +func (s Signature) Display() string { + src := typeToken(s.Meta.SourceType) + tgt := typeToken(s.Meta.TargetType) + label := s.ShortLabel + if label == "" { + label = "unknown" + } + return fmt.Sprintf("%s (%s→%s · %s)", s.IDDashed(), src, tgt, label) +} + +// ParseSignatureID normalizes user input into compact 8 or 16 hex lowercase. +// Accepts "aabbccdd", "aabbccdd-eeff0011", "aabbccddeeff0011", or a Display() line. +// Returns compact hex and whether it is pattern-only (len 8) vs full composite (len 16). +func ParseSignatureID(raw string) (compact string, patternOnly bool, err error) { + s := strings.ToLower(strings.TrimSpace(raw)) + if i := strings.IndexAny(s, " \t("); i > 0 { + s = s[:i] + } + s = strings.ReplaceAll(s, "-", "") + if len(s) != PartIDLen && len(s) != CompositeIDLen { + return "", false, fmt.Errorf("invalid error signature %q (expected 8 or 16 hex chars)", raw) + } + for _, r := range s { + if r < '0' || (r > '9' && r < 'a') || r > 'f' { + return "", false, fmt.Errorf("invalid error signature %q (expected hex)", raw) + } + } + return s, len(s) == PartIDLen, nil +} + +// InferredTaskType returns the job-type slug (db-db, file-db, api-file, …) +// derived from connector kinds. Empty when either side is unknown. +// Not part of the hash material — source+target types already encode this. +func (m SignMeta) InferredTaskType() string { + m = normalizeMeta(m) + sk, tk := kindAbbrev(m.SourceType.Kind()), kindAbbrev(m.TargetType.Kind()) + if sk == "" || tk == "" { + return "" + } + return sk + "-" + tk +} + +func kindAbbrev(k dbio.Kind) string { + switch k { + case dbio.KindDatabase: + return "db" + case dbio.KindFile: + return "file" + case dbio.KindAPI: + return "api" + default: + return "" + } +} + +func normalizeMeta(m SignMeta) SignMeta { + return SignMeta{ + SourceType: dbio.Type(strings.ToLower(strings.TrimSpace(string(m.SourceType)))), + TargetType: dbio.Type(strings.ToLower(strings.TrimSpace(string(m.TargetType)))), + } +} + +func typeToken(t dbio.Type) string { + s := strings.TrimSpace(string(t)) + if s == "" { + return "-" + } + return s +} + +// PatternMaterial builds the pattern-layer hash input (skeleton only). +// +// v1p| +func PatternMaterial(skeleton string) string { + if skeleton == "" { + skeleton = "unknown_error" + } + return PatternMaterialPrefix + "|" + skeleton +} + +// EdgeMaterial builds the edge-layer hash input (types + skeleton). +// +// v1e||| +// +// Task type is omitted: it is redundant given source and target connector types. +func EdgeMaterial(skeleton string, meta SignMeta) string { + meta = normalizeMeta(meta) + if skeleton == "" { + skeleton = "unknown_error" + } + return strings.Join([]string{ + EdgeMaterialPrefix, + typeToken(meta.SourceType), + typeToken(meta.TargetType), + skeleton, + }, "|") +} + +// Material is an alias for EdgeMaterial (edge-layer input). +func Material(skeleton string, meta SignMeta) string { + return EdgeMaterial(skeleton, meta) +} + +// HashPart returns the first PartIDLen hex chars of SHA-256(material). +func HashPart(material string) string { + sum := sha256.Sum256([]byte(material)) + return hex.EncodeToString(sum[:])[:PartIDLen] +} + +// HashMaterial is an alias for HashPart. +func HashMaterial(material string) string { + return HashPart(material) +} + +// ShortLabel is a human-only slug from the skeleton (not part of the hash). +func ShortLabel(skeleton string) string { + if skeleton == "" || skeleton == "unknown_error" { + return "unknown_error" + } + lines := strings.Split(skeleton, "\n") + // Prefer a vendor code when present (stable, short). + for i := len(lines) - 1; i >= 0; i-- { + if strings.HasPrefix(lines[i], "code:") { + return slugLabel(strings.TrimPrefix(lines[i], "code:")) + } + } + // Else last non-empty message line (usually the leaf driver message). + pick := lines[0] + for i := len(lines) - 1; i >= 0; i-- { + l := lines[i] + if l == "" { + continue + } + pick = l + break + } + return slugLabel(pick) +} + +func slugLabel(pick string) string { + // Slugify: non-alnum → _, collapse + var b strings.Builder + prevUnderscore := false + for _, r := range pick { + if unicode.IsLetter(r) || unicode.IsDigit(r) { + b.WriteRune(r) + prevUnderscore = false + continue + } + if !prevUnderscore { + b.WriteByte('_') + prevUnderscore = true + } + } + s := strings.Trim(b.String(), "_") + if len(s) > 40 { + s = s[:40] + s = strings.TrimRight(s, "_") + } + if s == "" { + return "unknown_error" + } + return s +} + +// --- normalization ----------------------------------------------------------- + +var ( + // Stack frames: --- task_run.go:140 func2 --- + reStackFrame = regexp.MustCompile(`(?i)^---\s+\S+\.go:\d+\s+`) + + // Vendor / driver codes (uppercase-ish tokens in brackets or parentheses) + reBracketCode = regexp.MustCompile(`\[([A-Z][A-Z0-9_.]+)\]`) + reParenCode = regexp.MustCompile(`\(([A-Z][A-Z0-9_]{2,})\)`) + reSQLState = regexp.MustCompile(`(?i)\bSQLSTATE[:\s]+([0-9A-Z]{5})\b`) + reCHCode = regexp.MustCompile(`\bCode:\s*(\d+)\b`) + + // Placeholders — order matters (more specific first). + // Quoted URL / path before bare forms so later "" does not swallow them. + // URLs on slingdata.io (or subdomains) are kept as-is (see replaceURL). + reQuotedURL = regexp.MustCompile(`(?i)"(?:https?|s3|gs|file|azure|abfs|abfss)://[^"]*"`) + reQuotedPath = regexp.MustCompile(`"(?:/|~/|~\\)[^"]*"`) + reURL = regexp.MustCompile(`(?i)\b(?:https?|s3|gs|file|azure|abfs|abfss)://[^\s"'<>]+`) + // Absolute / home paths (unix + windows drive). Match path token only. + reUnixPath = regexp.MustCompile(`(?:^|[\s"'=(])(/[^\s"'<>]+)`) + reWinPath = regexp.MustCompile(`(?i)(?:^|[\s"'=(])([a-z]:\\[^\s"'<>]+)`) + reHomePath = regexp.MustCompile(`(?:^|[\s"'=(])(~/[^\s"'<>]*)`) + reUUID = regexp.MustCompile(`(?i)\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`) + // ksuid / sling exec ids / long hex (exclude already-replaced tokens) + reLongID = regexp.MustCompile(`\b(?:exec_[A-Za-z0-9]+|[0-9A-Za-z]{24,}|[0-9a-fA-F]{16,})\b`) + reISOTs = regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`) + // IPv4 (+ optional port) + reIPv4 = regexp.MustCompile(`\b\d{1,3}(?:\.\d{1,3}){3}(?::\d{1,5})?\b`) + // Quoted / backtick identifiers (no spaces — keeps prose messages intact). + // Matches schema.table, columns, etc. after URL/path quoted forms. + reIdentDQ = regexp.MustCompile(`"[^"\s]{1,256}"`) + // Backtick pattern assembled so raw-string delimiters do not collide. + reIdentBT = regexp.MustCompile("`" + `[^` + "`" + `\s]{1,256}` + "`") + // Temp table names sling generates + reTempTable = regexp.MustCompile(`\btemp[A-Za-z0-9]{3,}\b`) + // Version banners from drivers + reVersionBanner = regexp.MustCompile(`\(version\s+[^)]+\)`) + // Long digit runs (≥3) — last, so short codes survive earlier patterns + reDigits = regexp.MustCompile(`\b\d{3,}\b`) + + reMultiSpace = regexp.MustCompile(`[ \t]+`) +) + +// Skeleton normalizes a full error chain into a stable multi-line skeleton. +func Skeleton(errText string) string { + rawLines := strings.Split(errText, "\n") + kept := make([]string, 0, len(rawLines)) + codeSet := map[string]struct{}{} + var codes []string + + for _, raw := range rawLines { + line := strings.TrimSpace(raw) + if line == "" { + continue + } + if reStackFrame.MatchString(line) { + continue + } + // Stream/section banners like "----- name -----" + if isSectionBanner(line) { + continue + } + if strings.HasPrefix(line, "~ ") { + line = strings.TrimSpace(strings.TrimPrefix(line, "~ ")) + } else if strings.HasPrefix(line, "~") { + line = strings.TrimSpace(strings.TrimPrefix(line, "~")) + } + if line == "" { + continue + } + + for _, c := range extractCodes(line) { + if _, ok := codeSet[c]; !ok { + codeSet[c] = struct{}{} + codes = append(codes, c) + } + } + + line = substitutePlaceholders(line) + line = strings.ToLower(line) + line = reMultiSpace.ReplaceAllString(line, " ") + line = strings.TrimSpace(line) + if line == "" { + continue + } + kept = append(kept, line) + } + + if len(kept) == 0 && len(codes) == 0 { + return "unknown_error" + } + + // Append stable vendor codes last so the leaf message stays first for labels. + for _, c := range codes { + kept = append(kept, "code:"+strings.ToLower(c)) + } + return strings.Join(kept, "\n") +} + +func isSectionBanner(line string) bool { + // e.g. --------------------------- lxp_app_oomkilled_count --------------------------- + if len(line) < 10 { + return false + } + trimmed := strings.Trim(line, "- \t") + if trimmed == "" { + return false + } + // mostly dashes on the outside + return strings.HasPrefix(strings.TrimSpace(line), "---") && + strings.HasSuffix(strings.TrimSpace(line), "---") && + !reStackFrame.MatchString(line) +} + +func extractCodes(line string) []string { + var out []string + for _, m := range reBracketCode.FindAllStringSubmatch(line, -1) { + out = append(out, m[1]) + } + for _, m := range reParenCode.FindAllStringSubmatch(line, -1) { + // Skip common non-code paren groups + c := m[1] + if c == "official" || c == "build" { + continue + } + out = append(out, c) + } + for _, m := range reSQLState.FindAllStringSubmatch(line, -1) { + out = append(out, m[1]) + } + for _, m := range reCHCode.FindAllStringSubmatch(line, -1) { + out = append(out, "CH_"+m[1]) + } + return out +} + +// replaceURL masks a URL match, unless it points at slingdata.io (or a subdomain). +func replaceURL(match string) string { + u := strings.Trim(match, `"`) + if i := strings.Index(u, "://"); i >= 0 { + host := u[i+3:] + if j := strings.IndexAny(host, "/?#"); j >= 0 { + host = host[:j] + } + if j := strings.LastIndex(host, ":"); j >= 0 { + host = host[:j] + } + host = strings.ToLower(host) + if host == "slingdata.io" || strings.HasSuffix(host, ".slingdata.io") { + return match + } + } + return "" +} + +func substitutePlaceholders(line string) string { + // Order: more specific first. + line = reQuotedURL.ReplaceAllStringFunc(line, replaceURL) + line = reQuotedPath.ReplaceAllString(line, "") + line = reURL.ReplaceAllStringFunc(line, replaceURL) + line = reISOTs.ReplaceAllString(line, "") + line = reUUID.ReplaceAllString(line, "") + line = reIPv4.ReplaceAllString(line, "") + line = reVersionBanner.ReplaceAllString(line, "") + line = reTempTable.ReplaceAllString(line, "") + line = reLongID.ReplaceAllString(line, "") + line = reIdentDQ.ReplaceAllString(line, "") + line = reIdentBT.ReplaceAllString(line, "") + // Bare paths: keep leading delimiter, replace path token. + line = replacePathKeepLead(reUnixPath, line) + line = replacePathKeepLead(reWinPath, line) + line = replacePathKeepLead(reHomePath, line) + line = reDigits.ReplaceAllString(line, "") + return line +} + +// replacePathKeepLead replaces path matches of the form (lead)(path) with lead+"". +func replacePathKeepLead(re *regexp.Regexp, line string) string { + return re.ReplaceAllStringFunc(line, func(m string) string { + sub := re.FindStringSubmatch(m) + if len(sub) < 2 { + return m + } + path := sub[1] + lead := strings.TrimSuffix(m, path) + return lead + "" + }) +} + +// --- assist error lookup ----------------------------------------------------- + +// ErrorLookupResult is the response shape for `sling assist error `. +// Worker-backed known-issue lookup lands later; v1 returns local status only. +type ErrorLookupResult struct { + Signature string `json:"signature"` // compact 8 or 16 hex + PatternID string `json:"pattern_id,omitempty"` // first 8 of composite + EdgeID string `json:"edge_id,omitempty"` // last 8 when full composite + PatternOnly bool `json:"pattern_only,omitempty"` // true when caller passed 8 hex + Status string `json:"status"` // known_config | known_bug | pending | unknown + Title string `json:"title,omitempty"` + Guidance string `json:"guidance,omitempty"` + DocsURL string `json:"docs_url,omitempty"` + FixedIn string `json:"fixed_in,omitempty"` + IssueURL string `json:"issue_url,omitempty"` +} + +// LookupError validates a signature id and returns guidance when known. +// Accepts composite (16 hex), pattern-only (8 hex), dashed form, or Display() line. +// Network lookup is not wired yet; well-formed ids return status "unknown". +func LookupError(signature string) (ErrorLookupResult, error) { + compact, patternOnly, err := ParseSignatureID(signature) + if err != nil { + return ErrorLookupResult{}, err + } + out := ErrorLookupResult{ + Signature: compact, + PatternOnly: patternOnly, + Status: "unknown", + Title: "No published guidance yet", + Guidance: "This signature is not in the known-issue registry yet. Run `sling assist` to debug the latest failed run locally, or check docs.slingdata.io.", + DocsURL: "https://docs.slingdata.io/", + } + if patternOnly { + out.PatternID = compact + } else { + out.PatternID = compact[:PartIDLen] + out.EdgeID = compact[PartIDLen:] + } + return out, nil +} + +// Sensitivity classifies what may leave the machine in a log/submit bundle. +type Sensitivity int + +const ( + // SensitivityPublic is safe to ship as-is (no credentials by design). + SensitivityPublic Sensitivity = iota + // SensitivityInternal may contain operational detail; redact values before ship. + SensitivityInternal + // SensitivitySecret must never leave the machine (credentials, tokens, backups of same). + SensitivitySecret +) + +func (s Sensitivity) String() string { + switch s { + case SensitivityPublic: + return "public" + case SensitivityInternal: + return "internal" + case SensitivitySecret: + return "secret" + default: + return "unknown" + } +} + +// SensitiveClass describes one path/category the assist package may touch. +type SensitiveClass struct { + // ID is a stable key for manifests (e.g. "env.yaml", "claude.json"). + ID string `json:"id"` + // Glob is matched against absolute or home-relative paths (slash-normalized). + // Supports * and ** suffix style via pathMatch. + Glob string `json:"glob"` + // Class is the sensitivity tier. + Class Sensitivity `json:"class"` + // Reason is a short human explanation (never contains secret values). + Reason string `json:"reason"` +} + +// SensitivityManifest returns the static inventory of sensitive surface area. +// Used by submit/bundle builders to decide include / redact / exclude. +func SensitivityManifest() []SensitiveClass { + return []SensitiveClass{ + { + ID: "env.yaml", Glob: "**/env.yaml", Class: SensitivitySecret, + Reason: "may contain connection credentials and env secrets", + }, + { + ID: "claude.json", Glob: "**/.claude.json", Class: SensitivitySecret, + Reason: "Claude Code OAuth session and user MCP config", + }, + { + ID: "claude-mcp-project", Glob: "**/.mcp.json", Class: SensitivityInternal, + Reason: "project MCP server definitions; may reference env vars", + }, + { + ID: "codex-config", Glob: "**/.codex/config.toml", Class: SensitivityInternal, + Reason: "may include MCP env blocks", + }, + { + ID: "gemini-settings", Glob: "**/.gemini/settings.json", Class: SensitivityInternal, + Reason: "MCP and model settings", + }, + { + ID: "cursor-mcp", Glob: "**/.cursor/mcp.json", Class: SensitivityInternal, + Reason: "MCP server definitions", + }, + { + ID: "vscode-mcp", Glob: "**/mcp.json", Class: SensitivityInternal, + Reason: "VS Code MCP servers (user or .vscode)", + }, + { + ID: "config-backup", Glob: "**/*.backup", Class: SensitivitySecret, + Reason: "backups of credential-bearing config files", + }, + { + ID: "run-error", Glob: "**/assist/errors/**/error.txt", Class: SensitivityInternal, + Reason: "error chains may embed query fragments or object names", + }, + { + ID: "run-stderr", Glob: "**/assist/errors/**/stderr.log", Class: SensitivityInternal, + Reason: "debug logs may include connection props if not redacted at write", + }, + { + ID: "run-meta", Glob: "**/assist/errors/**/meta.json", Class: SensitivityPublic, + Reason: "exec metadata; argv must be redacted at write time", + }, + { + ID: "run-config-snapshot", Glob: "**/assist/errors/**/config.snapshot.yaml", Class: SensitivityInternal, + Reason: "resolved config; secrets must be masked at write time", + }, + { + ID: "doctor.json", Glob: "**/doctor.json", Class: SensitivityPublic, + Reason: "install health only; no connection secrets", + }, + { + ID: "assist-history-prompt", Glob: "**/assist/history/*/prompt.md", Class: SensitivityInternal, + Reason: "user intention and log tails; may include business context", + }, + { + ID: "canonical-skills", Glob: "**/.agents/skills/**", Class: SensitivityPublic, + Reason: "embedded public skill docs", + }, + } +} + +// ClassifyPath returns the highest-sensitivity class matching path. +// Unknown paths default to SensitivityInternal (safe default: redact before ship). +// When multiple globs match, Secret > Internal > Public. +func ClassifyPath(path string) Sensitivity { + n := filepath.ToSlash(path) + matched := false + best := SensitivityPublic + for _, c := range SensitivityManifest() { + if !pathMatch(c.Glob, n) { + continue + } + matched = true + if c.Class == SensitivitySecret { + return SensitivitySecret + } + if c.Class == SensitivityInternal { + best = SensitivityInternal + } + } + if !matched { + return SensitivityInternal + } + return best +} + +// pathMatch supports a small glob dialect used by SensitivityManifest: +// - "**/" prefix = match anywhere +// - "*" = one path segment +// - "**" = one or more segments (including zero when trailing) +// - "*.ext" basename wildcards +func pathMatch(glob, path string) bool { + glob = filepath.ToSlash(glob) + path = filepath.ToSlash(path) + if glob == path { + return true + } + gParts := strings.Split(glob, "/") + pParts := strings.Split(path, "/") + return matchParts(gParts, pParts) +} + +func matchParts(gParts, pParts []string) bool { + // Recursive glob matcher for segment lists. + var rec func(gi, pi int) bool + rec = func(gi, pi int) bool { + for gi < len(gParts) { + g := gParts[gi] + if g == "**" { + // "**" matches zero or more segments. + if gi == len(gParts)-1 { + return true // trailing ** + } + // Try consuming 0..N path segments. + for k := pi; k <= len(pParts); k++ { + if rec(gi+1, k) { + return true + } + } + return false + } + if pi >= len(pParts) { + return false + } + if g == "*" || matchSeg(g, pParts[pi]) { + gi++ + pi++ + continue + } + return false + } + return pi == len(pParts) + } + return rec(0, 0) +} + +func matchSeg(pat, seg string) bool { + if pat == "*" || pat == seg { + return true + } + // basename wildcard: *.backup + if strings.HasPrefix(pat, "*.") { + return strings.HasSuffix(seg, pat[1:]) // ".backup" + } + if strings.Contains(pat, "*") { + // simple prefix*suffix + i := strings.Index(pat, "*") + return strings.HasPrefix(seg, pat[:i]) && strings.HasSuffix(seg, pat[i+1:]) + } + return false +} + +// FailureSnapshot is the minimal set of fields written when a command fails +// (run, conns test, conns discover) so `sling assist` can probe the error. +type FailureSnapshot struct { + ExecID string + ErrMsg string + ConfigPath string // replication / pipeline config path when known + ConnName string // connection name for `sling conns test|discover` + Rows string + Duration string + // RunLog is the captured log tail (env.RecentLogs). Written to stderr.log. + RunLog string + // ConfigBody is the replication/pipeline config file content. Written to + // config.snapshot.yaml when set. Not used for conns test/discover. + ConfigBody string + // SignMeta optional connector types for error_signature. + SignMeta SignMeta + // Extra is merged into meta.json as-is (connector types, etc.). + Extra map[string]any +} + +const ( + reservedExecutionsDir = "executions" + reservedSignaturesDir = "signatures" +) + +func isReservedErrorName(name string) bool { + return name == reservedExecutionsDir || name == reservedSignaturesDir +} + +// WriteFailureSnapshot writes ~/.sling/assist/errors/executions//{meta.json, +// error.txt, stderr.log}. Best-effort: no-op when execID is empty; never fails +// the caller run (errors are logged via g.Debug only). +func WriteFailureSnapshot(s FailureSnapshot) { + if strings.TrimSpace(s.ExecID) == "" { + return + } + + // load config body + if ext := strings.ToLower(filepath.Ext(s.ConfigPath)); g.In(ext, ".yaml", ".yml", ".json") && s.ConfigBody == "" { + if b, err := os.ReadFile(s.ConfigPath); err == nil && int64(len(b)) <= 64*1024 { + s.ConfigBody = string(b) + } + } + + dir := filepath.Join(ExecutionsDir(), s.ExecID) + if err := os.MkdirAll(dir, 0o755); err != nil { + g.Debug("assist: could not create error dir %s: %s", dir, err.Error()) + return + } + + errMsg := s.ErrMsg + if errMsg == "" { + errMsg = "(no error message captured)" + } + if err := os.WriteFile(filepath.Join(dir, "error.txt"), []byte(errMsg), 0o644); err != nil { + g.Debug("assist: could not write error.txt: %s", err.Error()) + } + // stderr.log holds the captured run log. Fall back to the error text when + // nothing was buffered, so the file is never empty. + runLog := s.RunLog + if strings.TrimSpace(runLog) == "" { + runLog = errMsg + } + if err := os.WriteFile(filepath.Join(dir, "stderr.log"), []byte(runLog), 0o644); err != nil { + g.Debug("assist: could not write stderr.log: %s", err.Error()) + } + if body := strings.TrimSpace(s.ConfigBody); body != "" { + if err := os.WriteFile(filepath.Join(dir, "config.snapshot.yaml"), []byte(body), 0o644); err != nil { + g.Debug("assist: could not write config.snapshot.yaml: %s", err.Error()) + } + } + + sig := SignError(errMsg, s.SignMeta) + + meta := map[string]any{ + "exec_id": s.ExecID, + "exit_code": 1, + "when": time.Now().UTC().Format(time.RFC3339), + "error_signature": sig.ID, // 16 hex: pattern||edge + "error_pattern_id": sig.PatternID, + "error_edge_id": sig.EdgeID, + "error_algorithm": sig.Algorithm, + "error_short_label": sig.ShortLabel, + } + if s.SignMeta.SourceType != "" { + meta["source_type"] = s.SignMeta.SourceType.String() + } + if s.SignMeta.TargetType != "" { + meta["target_type"] = s.SignMeta.TargetType.String() + } + if tt := s.SignMeta.InferredTaskType(); tt != "" { + meta["task_type"] = tt // derived; not part of signature hash + } + if s.ConfigPath != "" { + meta["config_path"] = s.ConfigPath + } + if s.ConnName != "" { + meta["conn_name"] = s.ConnName + } + if s.Rows != "" { + meta["rows"] = s.Rows + } + if s.Duration != "" { + meta["duration"] = s.Duration + } + for k, v := range s.Extra { + if _, exists := meta[k]; !exists { + meta[k] = v + } + } + body, err := json.MarshalIndent(meta, "", " ") + if err != nil { + g.Debug("assist: could not marshal meta.json: %s", err.Error()) + return + } + if err := os.WriteFile(filepath.Join(dir, "meta.json"), body, 0o644); err != nil { + g.Debug("assist: could not write meta.json: %s", err.Error()) + } + + if err := AutoTrimExecs(); err != nil { + g.Debug("assist: auto-trim execs: %s", err.Error()) + } +} + +// FailureFooterOpts controls the post-failure hint line. +// A command only — never an interactive prompt (TTY or not). +type FailureFooterOpts struct { + ExecID string + ErrMsg string + SignMeta SignMeta +} + +// PrintFailureFooter prints one indented hint line after a failure. +// The error signature is not printed: it is agent context, written to +// meta.json by WriteFailureSnapshot and read back by Probe. +// Suppressed when HintInErrors is false or SLING_ASSIST_HINT is falsey. +func PrintFailureFooter(opts FailureFooterOpts) { + if envDisabled("SLING_ASSIST_HINT") { + return + } + + prof, exists, err := LoadProfile() + if err != nil { + return + } + hintOn := true + if exists { + hintOn = prof.HintInErrors + } + if !hintOn { + return + } + + line := " sling assist setup" + if execID := strings.TrimSpace(opts.ExecID); execID != "" { + line = " sling assist --id " + ShortExecID(execID) + } else if exists || len(DetectedClients()) > 0 { + return + } + + if isTTY(os.Stderr) && !env.NoColor { + label := terminalLink(AssistDocsURL, "investigate with AI") + line = " " + label + " -> " + env.CyanString(strings.TrimSpace(line)) + } + fmt.Fprintln(os.Stderr, "") + fmt.Fprintln(os.Stderr, line) + fmt.Fprintln(os.Stderr, "") +} + +// AssistDocsURL is linked from the failure hint. +const AssistDocsURL = "https://docs.slingdata.io/sling-cli/assist" + +// terminalLink wraps text in an OSC 8 hyperlink. Terminals without OSC 8 +// support drop the escape codes and show only the text. +func terminalLink(url, text string) string { + return "\x1b]8;;" + url + "\x1b\\" + text + "\x1b]8;;\x1b\\" +} + +// ShortExecIDLen is the exec-id prefix length shown in the failure hint. +// ResolveLocalExec accepts any unique prefix. +const ShortExecIDLen = 8 + +// ShortExecID trims an exec id to the prefix shown to users. +func ShortExecID(id string) string { + id = strings.TrimSpace(id) + if len(id) > ShortExecIDLen { + return id[:ShortExecIDLen] + } + return id +} + +// MaybePrintErrorHint is kept for callers/tests that only have an exec id. +// Prefer PrintFailureFooter when error text is available. +func MaybePrintErrorHint(execID string) { + PrintFailureFooter(FailureFooterOpts{ExecID: execID}) +} + +// envDisabled is true when the named env var is set to a falsey value. +func envDisabled(key string) bool { + v := os.Getenv(key) + if v == "" { + return false + } + return !cast.ToBool(v) +} + +// LocalExec is one failed-run snapshot under ~/.sling/assist/errors/. +// New snapshots live in errors/executions//; legacy dirs stay readable. +type LocalExec struct { + ID string + When time.Time + Status string // "ok" | "err" | "?" + ConfigPath string // replication / pipeline config path + ConnName string // connection name for conns test/discover + Rows string + Duration string + LogDir string // absolute path to the exec's snapshot dir +} + +func (e LocalExec) displayObject() string { + if e.ConfigPath != "" { + return e.ConfigPath + } + return e.ConnName +} + +// LogsRoot returns ~/.sling/logs (SLING_LOG_DIR day files). Not used for failure snapshots. +func LogsRoot() string { + return filepath.Join(slingHome(), "logs") +} + +// ListLocalExecs scans errors/executions// then legacy errors// +// and returns the 20 most-recent execs by mtime. Unreadable dirs are skipped. +func ListLocalExecs() ([]LocalExec, error) { + ids, err := listLocalExecIDs() + if err != nil { + return nil, err + } + out := []LocalExec{} + for _, id := range ids { + dir := findLocalExecDir(id) + if dir == "" { + continue + } + le := LocalExec{ID: id, LogDir: dir, Status: "?"} + if info, err := os.Stat(dir); err == nil { + le.When = info.ModTime() + } + loadLocalExecMeta(&le) + out = append(out, le) + } + sort.Slice(out, func(i, j int) bool { + return out[i].When.After(out[j].When) + }) + if len(out) > 20 { + out = out[:20] + } + return out, nil +} + +// AutoTrimExecs deletes the oldest failure snapshots until at most +// ExecsMaxEntries remain. Covers both errors/executions// and the legacy +// errors// layout. Best-effort: a snapshot that cannot be removed is +// reported, and the rest still get trimmed. +func AutoTrimExecs() error { + ids, err := listLocalExecIDs() + if err != nil { + return err + } + if len(ids) <= ExecsMaxEntries { + return nil + } + + type entry struct { + dir string + when time.Time + } + entries := make([]entry, 0, len(ids)) + for _, id := range ids { + dir := findLocalExecDir(id) + if dir == "" { + continue + } + e := entry{dir: dir} + if info, statErr := os.Stat(dir); statErr == nil { + e.when = info.ModTime() + } + entries = append(entries, e) + } + if len(entries) <= ExecsMaxEntries { + return nil + } + + sort.Slice(entries, func(i, j int) bool { + return entries[i].when.After(entries[j].when) + }) + + var first error + for _, e := range entries[ExecsMaxEntries:] { + if rmErr := os.RemoveAll(e.dir); rmErr != nil && first == nil { + first = g.Error(rmErr, "remove exec snapshot %s", e.dir) + } + } + return first +} + +func listLocalExecIDs() ([]string, error) { + seen := map[string]struct{}{} + var ids []string + addFrom := func(root string, skipReserved bool) error { + if root == "" || !g.PathExists(root) { + return nil + } + entries, err := os.ReadDir(root) + if err != nil { + return g.Error(err, "read %s", root) + } + for _, e := range entries { + if !e.IsDir() { + continue + } + name := e.Name() + if skipReserved && isReservedErrorName(name) { + continue + } + if _, ok := seen[name]; ok { + continue + } + seen[name] = struct{}{} + ids = append(ids, name) + } + return nil + } + if err := addFrom(ExecutionsDir(), false); err != nil { + return nil, err + } + if err := addFrom(ErrorsDir(), true); err != nil { + return nil, err + } + return ids, nil +} + +func firstMetaString(doc map[string]any, keys ...string) string { + for _, k := range keys { + if v, _ := doc[k].(string); strings.TrimSpace(v) != "" { + return v + } + } + return "" +} + +func loadLocalExecMeta(e *LocalExec) { + metaPath := filepath.Join(e.LogDir, "meta.json") + doc, err := jsonReadOrEmpty(metaPath) + if err != nil || len(doc) == 0 { + return + } + if v, ok := doc["exit_code"]; ok { + if fmt.Sprintf("%v", v) == "0" { + e.Status = "ok" + } else { + e.Status = "err" + } + } + if v := firstMetaString(doc, "config_path", "object"); v != "" { + e.ConfigPath = v + } + if v, _ := doc["conn_name"].(string); v != "" { + e.ConnName = v + } + if v, _ := doc["rows"].(string); v != "" { + e.Rows = v + } + if v, _ := doc["duration"].(string); v != "" { + e.Duration = v + } +} + +// LookupLocalExec resolves an exec id to its snapshot. Accepts the full id or +// a unique prefix. Returns false when unknown or when a prefix is ambiguous. +func LookupLocalExec(id string) (LocalExec, bool) { + le, err := ResolveLocalExec(id) + return le, err == nil +} + +// ResolveLocalExec is LookupLocalExec with a reason: unknown vs ambiguous. +func ResolveLocalExec(id string) (LocalExec, error) { + id = strings.TrimSpace(id) + if id == "" { + return LocalExec{}, g.Error("empty exec id") + } + if dir := findLocalExecDir(id); dir != "" { + le := LocalExec{ID: id, LogDir: dir, Status: "?"} + if info, err := os.Stat(dir); err == nil { + le.When = info.ModTime() + } + loadLocalExecMeta(&le) + return le, nil + } + // Prefix match against every snapshot, not just the 20 ListLocalExecs keeps. + ids, err := listLocalExecIDs() + if err != nil { + return LocalExec{}, g.Error("unknown exec id %q", id) + } + hits := []string{} + for _, name := range ids { + if strings.HasPrefix(name, id) { + hits = append(hits, name) + } + } + switch len(hits) { + case 0: + return LocalExec{}, g.Error("unknown exec id %q", id) + case 1: + return ResolveLocalExec(hits[0]) + default: + sort.Strings(hits) + return LocalExec{}, g.Error("exec id %q is ambiguous (%d matches: %s)", + id, len(hits), strings.Join(hits[:2], ", ")+", …") + } +} + +// findLocalExecDir returns the snapshot dir for id. New layout first, then legacy. +func findLocalExecDir(id string) string { + if id == "" || isReservedErrorName(id) { + return "" + } + if strings.ContainsAny(id, `/\`) || strings.Contains(id, "..") { + return "" + } + if candidate := filepath.Join(ExecutionsDir(), id); g.PathExists(candidate) { + return candidate + } + if candidate := filepath.Join(ErrorsDir(), id); g.PathExists(candidate) { + return candidate + } + return "" +} + +const maxErrorTailBytes = 16 * 1024 + +// sanitizeLogForPrompt caps length, scrubs local connection secrets, and +// neutralizes triple-backtick fences so hostile log content cannot escape +// the markdown code blocks in prompts.yaml. +func sanitizeLogForPrompt(s string, maxBytes int) string { + s = scrubLocalConnSecrets(s) + if maxBytes > 0 && len(s) > maxBytes { + // Keep the tail (errors are usually at the end). + s = s[len(s)-maxBytes:] + if i := strings.IndexByte(s, '\n'); i >= 0 && i < 200 { + s = s[i+1:] + } + s = "[...truncated...]\n" + s + } + // Break ``` fences so log content cannot close the surrounding fence. + s = strings.ReplaceAll(s, "```", "'''") + return s +} + +func scrubLocalConnSecrets(s string) string { + return env.ScrubLine(s) +} diff --git a/core/sling/assist/investigate_test.go b/core/sling/assist/investigate_test.go new file mode 100644 index 000000000..005006863 --- /dev/null +++ b/core/sling/assist/investigate_test.go @@ -0,0 +1,1276 @@ +package assist + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core/dbio" + "github.com/slingdata-io/sling-cli/core/dbio/connection" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/spf13/cast" +) + +func TestSignError_NoStreamColumns(t *testing.T) { + errText := `--- task_run.go:140 func2 --- +--- task_run.go:830 runDbToDb --- +~ Could not WriteToDb +--- task_run_write.go:168 WriteToDb --- +no stream columns detected` + meta := SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres} + sig := SignError(errText, meta) + + if sig.Algorithm != "v1" { + t.Fatalf("algorithm: got %q", sig.Algorithm) + } + if len(sig.ID) != CompositeIDLen { + t.Fatalf("id len: got %q (want %d hex)", sig.ID, CompositeIDLen) + } + if len(sig.PatternID) != PartIDLen || len(sig.EdgeID) != PartIDLen { + t.Fatalf("parts: pattern=%q edge=%q", sig.PatternID, sig.EdgeID) + } + if sig.ID != sig.PatternID+sig.EdgeID { + t.Fatalf("composite != pattern||edge: %s vs %s%s", sig.ID, sig.PatternID, sig.EdgeID) + } + // Stack frames must not appear in skeleton. + if strings.Contains(sig.Skeleton, "task_run") || strings.Contains(sig.Skeleton, ".go:") { + t.Fatalf("skeleton still has frames:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "could not writetodb") { + t.Fatalf("skeleton missing context msg:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "no stream columns detected") { + t.Fatalf("skeleton missing leaf:\n%s", sig.Skeleton) + } + // Deterministic + sig2 := SignError(errText, meta) + if sig.ID != sig2.ID || sig.PatternID != sig2.PatternID || sig.EdgeID != sig2.EdgeID { + t.Fatalf("not deterministic: %v vs %v", sig, sig2) + } + // Line numbers must not change signature. + errAlt := strings.ReplaceAll(errText, "140", "999") + errAlt = strings.ReplaceAll(errAlt, "830", "1") + errAlt = strings.ReplaceAll(errAlt, "168", "42") + if SignError(errAlt, meta).ID != sig.ID { + t.Fatalf("line numbers changed signature") + } + // Different target type → different composite / edge, same pattern + other := SignError(errText, SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbSnowflake}) + if other.ID == sig.ID { + t.Fatalf("different meta should change composite id") + } + if other.PatternID != sig.PatternID { + t.Fatalf("same skeleton should share pattern_id: %s vs %s", sig.PatternID, other.PatternID) + } + if other.EdgeID == sig.EdgeID { + t.Fatalf("different target should change edge_id") + } +} + +func TestSignError_ConnectionRefusedScrubsIP(t *testing.T) { + errText := `--- proc.go:283 main --- +--- sling_cli.go:517 main --- +~ could not connect to database(try adding ` + "`sslmode=require`" + ` or ` + "`sslmode=disable`" + `) +dial tcp 192.168.176.200:5432: connect: connection refused` + meta := SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbPostgres} + sig := SignError(errText, meta) + if strings.Contains(sig.Skeleton, "192.168") { + t.Fatalf("IP not scrubbed:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "") { + t.Fatalf("expected placeholder:\n%s", sig.Skeleton) + } + err2 := strings.ReplaceAll(errText, "192.168.176.200", "10.0.0.5") + if SignError(err2, meta).ID != sig.ID { + t.Fatalf("different IPs should cluster") + } +} + +func TestSignError_AuthFailedExtractsCode(t *testing.T) { + errText := `--- task_run.go:142 func2 --- +~ Could not initialize target connection +--- database_clickhouse.go:74 Connect --- +~ could not connect to database +clickhouse [execute]:: 403 code: Code: 516. DB::Exception: bcdata: Authentication failed: password is incorrect, or there is no user with such name. (AUTHENTICATION_FAILED) (version 25.11.2.24 (official build)) +` + meta := SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbClickhouse} + sig := SignError(errText, meta) + if strings.Contains(sig.Skeleton, "25.11") { + t.Fatalf("version banner not scrubbed:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "code:authentication_failed") { + t.Fatalf("expected AUTHENTICATION_FAILED code:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "code:ch_516") { + t.Fatalf("expected CH code:\n%s", sig.Skeleton) + } + t.Logf("skeleton:\n%s\nid=%s dashed=%s label=%s", sig.Skeleton, sig.ID, sig.IDDashed(), sig.ShortLabel) +} + +func TestSignError_URLAndPathScrub(t *testing.T) { + errText := `--- database.go:709 Connect --- +Post "https://clickhouse.bcstuff.dev:443?database=bc_clickhouse_db&default_format=Native": dial tcp 136.41.64.93:443: i/o timeout +unable to open database "/root/.duckdb/extensions/v1.4.2/linux_amd64/motherduck.duckdb_extension" +` + meta := SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbClickhouse} + sig := SignError(errText, meta) + if strings.Contains(sig.Skeleton, "bcstuff") || strings.Contains(sig.Skeleton, "clickhouse.bc") { + t.Fatalf("URL host not scrubbed:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "") { + t.Fatalf("expected :\n%s", sig.Skeleton) + } + if strings.Contains(sig.Skeleton, "/root/") { + t.Fatalf("path not scrubbed:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "") { + t.Fatalf("expected :\n%s", sig.Skeleton) + } +} + +func TestSignError_DatabricksArityCode(t *testing.T) { + errText := `--- task_run.go:142 func2 --- +~ Could not WriteToDb +--- database_databricks.go:184 BulkImportFlow --- +~ could not insert into ` + "`safenet`.`approval_plan_ship_tmp`" + ` +databricks: execution error: failed to execute query: unexpected operation state ERROR_STATE: [COPY_INTO_COLUMN_ARITY_MISMATCH.NOT_ENOUGH_DATA_COLUMNS] Cannot write to ` + "`abl_analytics_ws_prd`.`safenet`.`approval_plan_ship_tmp`" + `, the reason is not enough data columns compared to specified columns: +Specified columns: ` + "`approval_plan_id`" + `, ` + "`ship_id`" + `, ` + "`_sling_loaded_at`" + `. +Data columns: .` + meta := SignMeta{SourceType: dbio.TypeDbMySQL, TargetType: dbio.TypeDbDatabricks} + sig := SignError(errText, meta) + if !strings.Contains(sig.Skeleton, "code:copy_into_column_arity_mismatch.not_enough_data_columns") { + t.Fatalf("expected arity code:\n%s", sig.Skeleton) + } + if strings.Contains(sig.Skeleton, "safenet") || strings.Contains(sig.Skeleton, "approval_plan") { + t.Fatalf("idents not scrubbed:\n%s", sig.Skeleton) + } +} + +func TestSignError_IdentAndTempTable(t *testing.T) { + errText := `~ could not prepare Tx: COPY "public"."lxp_cpu_throttled_percentage_tmp" ("app") FROM STDIN +pq: relation "public.lxp_cpu_throttled_percentage_tmp" does not exist +~ Error executing: create unique index if not exists tempSipc0_idx on tempSipc0 ("timestamp") +` + meta := SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres} + sig := SignError(errText, meta) + if strings.Contains(sig.Skeleton, "tempSipc0") { + t.Fatalf("temp table not scrubbed:\n%s", sig.Skeleton) + } + if strings.Contains(sig.Skeleton, "lxp_cpu") { + t.Fatalf("quoted ident not scrubbed:\n%s", sig.Skeleton) + } + if !strings.Contains(sig.Skeleton, "") || !strings.Contains(sig.Skeleton, "") { + t.Fatalf("expected placeholders:\n%s", sig.Skeleton) + } +} + +func TestSignError_Empty(t *testing.T) { + sig := SignError("", SignMeta{}) + if sig.Skeleton != "unknown_error" { + t.Fatalf("got skeleton %q", sig.Skeleton) + } + if sig.PatternMaterial != "v1p|unknown_error" { + t.Fatalf("pattern material: %q", sig.PatternMaterial) + } + if sig.EdgeMaterial != "v1e|-|-|unknown_error" { + t.Fatalf("edge material: %q", sig.EdgeMaterial) + } + if len(sig.ID) != CompositeIDLen { + t.Fatalf("id: %q", sig.ID) + } + // Known vector: empty meta + unknown_error + if sig.ID != "fb2398c014bc4249" { + t.Fatalf("empty id vector: got %s want fb2398c014bc4249", sig.ID) + } +} + +func TestSignError_MetaCaseInsensitive(t *testing.T) { + errText := "no stream columns detected" + a := SignError(errText, SignMeta{SourceType: dbio.Type("Postgres"), TargetType: dbio.Type("SNOWFLAKE")}) + b := SignError(errText, SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbSnowflake}) + if a.ID != b.ID { + t.Fatalf("meta case should not matter: %s vs %s", a.ID, b.ID) + } +} + +func TestComposite_KnownVector(t *testing.T) { + skel := "could not writetodb\nno stream columns detected" + meta := SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres} + // Materials + pMat := PatternMaterial(skel) + eMat := EdgeMaterial(skel, meta) + if pMat != "v1p|"+skel { + t.Fatalf("pattern material: %q", pMat) + } + if eMat != "v1e|prometheus|postgres|"+skel { + t.Fatalf("edge material: %q", eMat) + } + // Known digests (python/sha256 vectors) + if HashPart(pMat) != "97d84811" { + t.Fatalf("pattern part: got %s want 97d84811", HashPart(pMat)) + } + if HashPart(eMat) != "5aede62c" { + t.Fatalf("edge part: got %s want 5aede62c", HashPart(eMat)) + } + // Full SignError on framed error yields same skeleton materials + errText := `--- task_run.go:140 func2 --- +~ Could not WriteToDb +--- task_run_write.go:168 WriteToDb --- +no stream columns detected` + sig := SignError(errText, meta) + if sig.PatternID != "97d84811" || sig.EdgeID != "5aede62c" { + t.Fatalf("parts: pattern=%s edge=%s", sig.PatternID, sig.EdgeID) + } + if sig.ID != "97d848115aede62c" { + t.Fatalf("composite: got %s", sig.ID) + } + if sig.IDDashed() != "97d84811-5aede62c" { + t.Fatalf("dashed: %s", sig.IDDashed()) + } +} + +func TestShortLabel(t *testing.T) { + lab := ShortLabel("could not writetodb\nno stream columns detected") + if lab != "no_stream_columns_detected" { + t.Fatalf("label: %q", lab) + } + lab2 := ShortLabel("unknown_error") + if lab2 != "unknown_error" { + t.Fatalf("label: %q", lab2) + } +} + +func TestDisplay(t *testing.T) { + sig := SignError("no stream columns detected", SignMeta{ + SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres, + }) + d := sig.Display() + if !strings.Contains(d, sig.IDDashed()) || !strings.Contains(d, "prometheus→postgres") { + t.Fatalf("display: %q", d) + } + if !strings.Contains(d, "-") { + t.Fatalf("display should use dashed composite: %q", d) + } +} + +func TestParseSignatureID(t *testing.T) { + compact, patternOnly, err := ParseSignatureID("97d848115aede62c") + if err != nil || compact != "97d848115aede62c" || patternOnly { + t.Fatalf("16hex: %q %v %v", compact, patternOnly, err) + } + compact, patternOnly, err = ParseSignatureID("97d84811-5aede62c") + if err != nil || compact != "97d848115aede62c" || patternOnly { + t.Fatalf("dashed: %q %v %v", compact, patternOnly, err) + } + compact, patternOnly, err = ParseSignatureID("97d84811 (prometheus→postgres · no_stream_columns)") + if err != nil || compact != "97d84811" || !patternOnly { + t.Fatalf("pattern from display: %q %v %v", compact, patternOnly, err) + } + // Display line with dashed id + compact, patternOnly, err = ParseSignatureID("97d84811-5aede62c (prometheus→postgres · x)") + if err != nil || compact != "97d848115aede62c" || patternOnly { + t.Fatalf("dashed display: %q %v %v", compact, patternOnly, err) + } + if _, _, err := ParseSignatureID("not-a-sig"); err == nil { + t.Fatal("expected error") + } + if _, _, err := ParseSignatureID("abcd"); err == nil { + t.Fatal("expected error for short hex") + } +} + +func TestSkeleton_DropsSectionBanners(t *testing.T) { + errText := `~ failure running replication +--------------------------- lxp_app_oomkilled_count --------------------------- +~ Could not WriteToDb +no stream columns detected +--------------------------- lxp_app_middleware_pod_oomkilled_count --------------------------- +~ Could not WriteToDb +no stream columns detected` + skel := Skeleton(errText) + if strings.Contains(skel, "lxp_app_oomkilled") { + t.Fatalf("section banner leaked:\n%s", skel) + } + if strings.Count(skel, "no stream columns detected") < 1 { + t.Fatalf("missing leaf:\n%s", skel) + } +} + +func TestVariousErrorShapes(t *testing.T) { + cases := []struct { + name string + err string + meta SignMeta + want []string + deny []string + }{ + { + name: "ssl_not_enabled", + err: "--- sling_run.go:442 runTask ---\n~ could not connect to database\npq: SSL is not enabled on the server", + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbPostgres}, + want: []string{"ssl is not enabled"}, + deny: []string{"sling_run.go"}, + }, + { + name: "update_key_missing", + err: "--- task_run_read.go:149 ReadFromDB ---\ndid not find update_key: modified_at", + meta: SignMeta{SourceType: dbio.TypeDbMySQL, TargetType: dbio.TypeDbSnowflake}, + want: []string{"did not find update_key"}, + deny: []string{"task_run_read"}, + }, + { + name: "table_not_found_sqlserver", + err: "--- database_sqlserver.go:543 GetTableColumns ---\ndid not find table or synonym: \"V12PROD\".\"XL\"", + meta: SignMeta{SourceType: dbio.TypeDbSQLServer, TargetType: dbio.TypeDbClickhouse}, + want: []string{"did not find table or synonym", ""}, + deny: []string{"V12PROD", "database_sqlserver"}, + }, + { + name: "not_enough_space", + err: "clickhouse [execute]:: 500 code: Code: 243. DB::Exception: Cannot reserve 1.00 MiB, not enough space: While executing WaitForAsyncInsert. (NOT_ENOUGH_SPACE) (version 26.6.2.81 (official build))", + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbClickhouse}, + want: []string{"code:not_enough_space", "code:ch_243", "not enough space"}, + deny: []string{"26.6.2"}, + }, + { + name: "motherduck_auth", + err: `~ Failed to execute SQL +Error: unable to open database "md:warehouse": Invalid Input Error: Initialization function "motherduck_duckdb_cpp_init" from file "/root/.duckdb/extensions/v1.4.2/linux_amd64/motherduck.duckdb_extension" threw an exception: "Invalid Error: Request failed: Your request is not authenticated. Please check your MotherDuck token."`, + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbMotherDuck}, + want: []string{"not authenticated", ""}, + deny: []string{"/root/.duckdb"}, + }, + { + name: "eof_connect", + err: "~ could not connect to database(try adding `sslmode=require` or `sslmode=disable`)\nEOF", + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbPostgres}, + want: []string{"could not connect", "eof"}, + deny: []string{}, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + sig := SignError(tc.err, tc.meta) + if len(sig.ID) != CompositeIDLen { + t.Fatalf("bad id %q", sig.ID) + } + for _, w := range tc.want { + if !strings.Contains(sig.Skeleton, w) { + t.Fatalf("skeleton missing %q:\n%s", w, sig.Skeleton) + } + } + for _, d := range tc.deny { + if strings.Contains(sig.Skeleton, d) { + t.Fatalf("skeleton has denied %q:\n%s", d, sig.Skeleton) + } + } + if !strings.HasPrefix(sig.PatternMaterial, "v1p|") { + t.Fatalf("pattern material: %q", sig.PatternMaterial) + } + if !strings.HasPrefix(sig.EdgeMaterial, "v1e|") { + t.Fatalf("edge material: %q", sig.EdgeMaterial) + } + }) + } +} + +func TestInferredTaskType(t *testing.T) { + m := SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres} + if m.InferredTaskType() != "db-db" { + t.Fatalf("got %q", m.InferredTaskType()) + } + m = SignMeta{SourceType: dbio.TypeFileS3, TargetType: dbio.TypeDbSnowflake} + if m.InferredTaskType() != "file-db" { + t.Fatalf("got %q", m.InferredTaskType()) + } + m = SignMeta{SourceType: dbio.TypeApi, TargetType: dbio.TypeFileLocal} + if m.InferredTaskType() != "api-file" { + t.Fatalf("got %q", m.InferredTaskType()) + } + if (SignMeta{}).InferredTaskType() != "" { + t.Fatal("empty meta should not infer task type") + } +} + +// --- ClickHouse live parity ---------------------------------------------------- + +func chQuery(t *testing.T, sql string) []map[string]any { + t.Helper() + entry := connection.GetLocalConns().Get("clickhouse_top") + if entry.Name == "" { + t.Skip("clickhouse_top connection not configured") + } + db, err := entry.Connection.AsDatabase() + if err != nil { + t.Skipf("AsDatabase: %v", err) + } + if err := db.Connect(); err != nil { + t.Skipf("Connect: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + + data, err := db.Query(sql) + if err != nil { + t.Fatalf("query failed: %v\nsql=%s", err, truncate(sql, 400)) + } + return data.Records(true) +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "…" +} + +// TestClickHouseHashMatchesGo proves SHA-256 first-8 hex is identical for the +// same material bytes in Go and ClickHouse (pattern and edge materials). +func TestClickHouseHashMatchesGo(t *testing.T) { + skel := Skeleton(`--- task_run.go:140 func2 --- +~ Could not WriteToDb +no stream columns detected`) + meta := SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres} + materials := []string{ + PatternMaterial(skel), + EdgeMaterial(skel, meta), + "v1p|unknown_error", + "v1e|-|-|unknown_error", + EdgeMaterial(Skeleton("dial tcp 10.0.0.1:5432: connection refused"), SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbMySQL}), + PatternMaterial(Skeleton(`[COPY_INTO_COLUMN_ARITY_MISMATCH.NOT_ENOUGH_DATA_COLUMNS] bad columns`)), + } + + for i, mat := range materials { + t.Run(fmt.Sprintf("mat_%d", i), func(t *testing.T) { + goSig := HashPart(mat) + sql := "SELECT " + chSQLHashPart(mat) + " AS sig" + rows := chQuery(t, sql) + if len(rows) != 1 { + t.Fatalf("expected 1 row, got %d", len(rows)) + } + chSig := cast.ToString(rows[0]["sig"]) + if chSig != goSig { + t.Fatalf("hash mismatch\n go=%s\n ch=%s\n material=%q", goSig, chSig, mat) + } + }) + } +} + +// TestClickHouseSignErrorMatchesGo compares full composite SignError against CH SQL. +func TestClickHouseSignErrorMatchesGo(t *testing.T) { + cases := []struct { + name string + err string + meta SignMeta + }{ + { + name: "no_stream_columns", + err: `--- task_run.go:140 func2 --- +--- task_run.go:830 runDbToDb --- +~ Could not WriteToDb +--- task_run_write.go:168 WriteToDb --- +no stream columns detected`, + meta: SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres}, + }, + { + name: "connection_refused", + err: `--- proc.go:283 main --- +~ could not connect to database(try adding sslmode=require or sslmode=disable) +dial tcp 192.168.176.200:5432: connect: connection refused`, + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbPostgres}, + }, + { + name: "ssl_not_enabled", + err: `~ could not connect to database +pq: SSL is not enabled on the server`, + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbClickhouse}, + }, + { + name: "url_timeout", + err: `~ could not connect to database +Post "https://clickhouse.example.com:443?database=db&default_format=Native": dial tcp 136.41.64.93:443: i/o timeout`, + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbClickhouse}, + }, + { + name: "empty_meta", + err: "no stream columns detected", + meta: SignMeta{}, + }, + { + name: "path_scrub", + err: `unable to open database "/root/.duckdb/extensions/v1.4.2/linux_amd64/motherduck.duckdb_extension"`, + meta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbMotherDuck}, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + goSig := SignError(tc.err, tc.meta) + + // Pattern + edge material hash parity + for _, part := range []struct { + name, mat, want string + }{ + {"pattern", goSig.PatternMaterial, goSig.PatternID}, + {"edge", goSig.EdgeMaterial, goSig.EdgeID}, + } { + rows := chQuery(t, "SELECT "+chSQLHashPart(part.mat)+" AS sig") + if cast.ToString(rows[0]["sig"]) != part.want { + t.Fatalf("%s hash mismatch go=%s ch=%s mat=%q", part.name, part.want, rows[0]["sig"], part.mat) + } + } + + // Full composite from CH normalizer + sqlFull := fmt.Sprintf( + "SELECT %s AS sig, %s AS pattern_id, %s AS edge_id, %s AS skel", + chSQLCompositeExpr(chQuote(tc.err), chQuote(string(tc.meta.SourceType)), chQuote(string(tc.meta.TargetType))), + chSQLPatternExpr(chQuote(tc.err)), + chSQLEdgeExpr(chQuote(tc.err), chQuote(string(tc.meta.SourceType)), chQuote(string(tc.meta.TargetType))), + chSkeletonExpr(chQuote(tc.err)), + ) + rows2 := chQuery(t, sqlFull) + chSig := cast.ToString(rows2[0]["sig"]) + chPat := cast.ToString(rows2[0]["pattern_id"]) + chEdge := cast.ToString(rows2[0]["edge_id"]) + chSkel := cast.ToString(rows2[0]["skel"]) + + if chSkel != goSig.Skeleton { + t.Fatalf("skeleton mismatch\n--- go ---\n%s\n--- ch ---\n%s", goSig.Skeleton, chSkel) + } + if chPat != goSig.PatternID || chEdge != goSig.EdgeID { + t.Fatalf("parts mismatch go=%s/%s ch=%s/%s", goSig.PatternID, goSig.EdgeID, chPat, chEdge) + } + if chSig != goSig.ID { + t.Fatalf("composite mismatch go=%s ch=%s", goSig.ID, chSig) + } + }) + } +} + +// TestClickHouseLiveErrors_GoAndCHMatch pulls real plausible_events rows. +func TestClickHouseLiveErrors_GoAndCHMatch(t *testing.T) { + sql := ` +SELECT + toString(meta_json.error) AS err, + JSONExtractString(ifNull(task_string, ''), 'source_type') AS source_type, + JSONExtractString(ifNull(task_string, ''), 'target_type') AS target_type +FROM analytics.plausible_events +WHERE timestamp > now() - INTERVAL 14 DAY + AND toString(meta_json.error) != '' + AND length(toString(meta_json.error)) BETWEEN 40 AND 2500 + AND JSONExtractString(ifNull(task_string, ''), 'source_type') != '' +LIMIT 40 +` + rows := chQuery(t, sql) + if len(rows) == 0 { + t.Skip("no recent error rows in plausible_events") + } + + var ( + hashMatches int + fullMatches int + fullChecked int + ) + + seenPattern := map[string]struct{}{} + seenComposite := map[string]struct{}{} + for i, row := range rows { + errText := cast.ToString(row["err"]) + meta := SignMeta{ + SourceType: dbio.Type(cast.ToString(row["source_type"])), + TargetType: dbio.Type(cast.ToString(row["target_type"])), + } + goSig := SignError(errText, meta) + + // Always: CH SHA256 of pattern/edge materials == Go parts + prows := chQuery(t, "SELECT "+chSQLHashPart(goSig.PatternMaterial)+" AS sig") + erows := chQuery(t, "SELECT "+chSQLHashPart(goSig.EdgeMaterial)+" AS sig") + if cast.ToString(prows[0]["sig"]) != goSig.PatternID || cast.ToString(erows[0]["sig"]) != goSig.EdgeID { + t.Errorf("row %d material hash mismatch go=%s/%s ch=%s/%s", + i, goSig.PatternID, goSig.EdgeID, prows[0]["sig"], erows[0]["sig"]) + continue + } + hashMatches++ + seenPattern[goSig.PatternID] = struct{}{} + seenComposite[goSig.ID] = struct{}{} + + if strings.Contains(goSig.Skeleton, "code:") { + continue + } + fullChecked++ + fullSQL := fmt.Sprintf( + "SELECT %s AS sig", + chSQLCompositeExpr(chQuote(errText), chQuote(string(meta.SourceType)), chQuote(string(meta.TargetType))), + ) + frows := chQuery(t, fullSQL) + chSig := cast.ToString(frows[0]["sig"]) + if chSig != goSig.ID { + skelSQL := "SELECT " + chSkeletonExpr(chQuote(errText)) + " AS skel" + srows := chQuery(t, skelSQL) + t.Errorf("row %d full sig mismatch go=%s ch=%s\n--- go skel ---\n%s\n--- ch skel ---\n%s\nerr_prefix=%q", + i, goSig.ID, chSig, goSig.Skeleton, cast.ToString(srows[0]["skel"]), truncate(errText, 160)) + continue + } + fullMatches++ + } + + t.Logf("live sample: rows=%d material_hash_ok=%d full_checked=%d full_ok=%d unique_patterns=%d unique_composites=%d", + len(rows), hashMatches, fullChecked, fullMatches, len(seenPattern), len(seenComposite)) + + if hashMatches != len(rows) { + t.Fatalf("material hash parity failed for %d/%d rows", len(rows)-hashMatches, len(rows)) + } + if fullChecked > 0 && fullMatches < fullChecked { + t.Fatalf("full CH normalizer parity failed for %d/%d non-code rows", fullChecked-fullMatches, fullChecked) + } +} + +// TestClickHouseLiveTopPatternStable: same skeleton, renumbered frames, same composite; +// different target shares pattern only. +func TestClickHouseLiveTopPatternStable(t *testing.T) { + errA := `--- task_run.go:140 func2 --- +--- task_run.go:830 runDbToDb --- +~ Could not WriteToDb +--- task_run_write.go:168 WriteToDb --- +no stream columns detected` + errB := `--- task_run.go:125 func2 --- +--- task_run.go:881 runDbToDb --- +~ Could not WriteToDb +--- task_run_write.go:450 WriteToDb --- +no stream columns detected` + meta := SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres} + sa := SignError(errA, meta) + sb := SignError(errB, meta) + if sa.ID != sb.ID || sa.PatternID != sb.PatternID { + t.Fatalf("line renumber should not split: %s vs %s", sa.ID, sb.ID) + } + + // Different target → same pattern, different edge/composite + sc := SignError(errA, SignMeta{SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbSnowflake}) + if sc.PatternID != sa.PatternID { + t.Fatalf("pattern should match across targets: %s vs %s", sa.PatternID, sc.PatternID) + } + if sc.ID == sa.ID || sc.EdgeID == sa.EdgeID { + t.Fatalf("edge/composite should differ by target") + } + + // CH agrees on both composites + for _, errText := range []string{errA, errB} { + sql := fmt.Sprintf("SELECT %s AS sig, %s AS pattern_id", + chSQLCompositeExpr(chQuote(errText), chQuote(string(meta.SourceType)), chQuote(string(meta.TargetType))), + chSQLPatternExpr(chQuote(errText))) + rows := chQuery(t, sql) + if cast.ToString(rows[0]["sig"]) != sa.ID { + t.Fatalf("CH composite: got %s want %s", rows[0]["sig"], sa.ID) + } + if cast.ToString(rows[0]["pattern_id"]) != sa.PatternID { + t.Fatalf("CH pattern: got %s want %s", rows[0]["pattern_id"], sa.PatternID) + } + } + t.Logf("stable composite for no_stream_columns: %s (%s) pattern=%s", sa.ID, sa.Display(), sa.PatternID) +} + +// --- ClickHouse SQL helpers (test-only parity with Go SignError) --------------- + +func chSQLHashPart(material string) string { + return fmt.Sprintf("lower(substring(hex(SHA256(%s)), 1, %d))", chQuote(material), PartIDLen) +} + +func chSQLPatternExpr(errExpr string) string { + skel := chSkeletonExpr(errExpr) + mat := fmt.Sprintf("concat(%s, '|', %s)", chQuote(PatternMaterialPrefix), skel) + return fmt.Sprintf("lower(substring(hex(SHA256(%s)), 1, %d))", mat, PartIDLen) +} + +func chSQLEdgeExpr(errExpr, sourceExpr, targetExpr string) string { + skel := chSkeletonExpr(errExpr) + src := fmt.Sprintf("if(empty(trimBoth(%s)), '-', lower(trimBoth(%s)))", sourceExpr, sourceExpr) + tgt := fmt.Sprintf("if(empty(trimBoth(%s)), '-', lower(trimBoth(%s)))", targetExpr, targetExpr) + mat := fmt.Sprintf( + "concat(%s, '|', %s, '|', %s, '|', %s)", + chQuote(EdgeMaterialPrefix), src, tgt, skel, + ) + return fmt.Sprintf("lower(substring(hex(SHA256(%s)), 1, %d))", mat, PartIDLen) +} + +func chSQLCompositeExpr(errExpr, sourceExpr, targetExpr string) string { + return fmt.Sprintf("concat(%s, %s)", + chSQLPatternExpr(errExpr), + chSQLEdgeExpr(errExpr, sourceExpr, targetExpr), + ) +} + +func chSkeletonExpr(errExpr string) string { + steps := []struct { + pat, repl string + }{ + {`(?m)^---\s+\S+\.go:\d+\s+.*\n?`, ``}, + {`(?m)^-{3,}[^-].*-{3,}\s*\n?`, ``}, + {`(?m)^~\s*`, ``}, + {`(?i)"(?:https?|s3|gs|file|azure|abfs|abfss)://[^"]*"`, ``}, + {`"(?:/|~/)[^"]*"`, ``}, + {`(?i)\b(?:https?|s3|gs|file|azure|abfs|abfss)://[^\s"'<>]+`, ``}, + {`\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?\b`, ``}, + {`(?i)\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b`, ``}, + {`\b\d{1,3}(?:\.\d{1,3}){3}(?::\d{1,5})?\b`, ``}, + {`\(version\s+[^)]+\)`, ``}, + {`\btemp[A-Za-z0-9]{3,}\b`, ``}, + {`\b(?:exec_[A-Za-z0-9]+|[0-9A-Za-z]{24,}|[0-9a-fA-F]{16,})\b`, ``}, + {`"[^"\s]{1,256}"`, ``}, + {"`[^`\\s]{1,256}`", ``}, + {`(^|[\s"'=(])(/[^\s"'<>]+)`, `\1`}, + {`\b\d{3,}\b`, ``}, + } + + expr := errExpr + for _, s := range steps { + expr = fmt.Sprintf("replaceRegexpAll(%s, %s, %s)", expr, chQuote(s.pat), chQuote(s.repl)) + } + + lineMap := fmt.Sprintf( + `arrayMap(x -> trimBoth(replaceRegexpAll(lower(x), %s, ' ')), splitByChar('\n', %s))`, + chQuote(`[ \t]+`), + expr, + ) + filtered := fmt.Sprintf(`arrayFilter(x -> x != '', %s)`, lineMap) + joined := fmt.Sprintf(`arrayStringConcat(%s, '\n')`, filtered) + return fmt.Sprintf(`if(%s = '', 'unknown_error', %s)`, joined, joined) +} + +func chQuote(s string) string { + s = strings.ReplaceAll(s, `\`, `\\`) + s = strings.ReplaceAll(s, `'`, `\'`) + return "'" + s + "'" +} + +func TestWriteFailureSnapshotReadableByInvestigate(t *testing.T) { + withTempHomeDir(t) + + WriteFailureSnapshot(FailureSnapshot{ + ExecID: "exec_test123", + ErrMsg: "column missing: email_verified", + ConfigPath: "./r.yaml", + }) + + dir := findLocalExecDir("exec_test123") + if dir == "" { + t.Fatal("findLocalExecDir returned empty after WriteFailureSnapshot") + } + wantDir := filepath.Join(ExecutionsDir(), "exec_test123") + if dir != wantDir { + t.Fatalf("snapshot dir = %q want %q", dir, wantDir) + } + errBytes, err := os.ReadFile(filepath.Join(dir, "error.txt")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(errBytes), "email_verified") { + t.Fatalf("error.txt = %q", errBytes) + } + if _, err := os.Stat(filepath.Join(dir, "meta.json")); err != nil { + t.Fatal(err) + } + + execs, err := ListLocalExecs() + if err != nil { + t.Fatal(err) + } + found := false + for _, e := range execs { + if e.ID == "exec_test123" { + found = true + if e.Status != "err" { + t.Fatalf("status = %q, want err", e.Status) + } + if e.ConfigPath != "./r.yaml" { + t.Fatalf("config_path = %q", e.ConfigPath) + } + } + } + if !found { + t.Fatal("ListLocalExecs did not include written exec") + } +} + +func TestPrintFailureFooterRespectsEnv(t *testing.T) { + t.Setenv("SLING_ASSIST_HINT", "false") + PrintFailureFooter(FailureFooterOpts{ExecID: "exec_x", ErrMsg: "boom"}) + MaybePrintErrorHint("exec_x") +} + +func TestPrintFailureFooterErrorFlag(t *testing.T) { + withTempHomeDir(t) + t.Setenv("SLING_ASSIST_HINT", "true") + + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + old := os.Stderr + os.Stderr = w + PrintFailureFooter(FailureFooterOpts{ExecID: "exec_x", ErrMsg: "boom"}) + _ = w.Close() + os.Stderr = old + + var buf bytes.Buffer + if _, err := buf.ReadFrom(r); err != nil { + t.Fatal(err) + } + out := buf.String() + if !strings.Contains(out, "sling assist --id exec_x") { + t.Fatalf("missing --id hint in footer: %q", out) + } + // The signature is agent context (meta.json), never user-facing output. + if strings.Contains(out, "error_signature") || strings.Contains(out, "sling assist error ") { + t.Fatalf("signature leaked into footer: %q", out) + } + if strings.Contains(out, "--exec ") { + t.Fatalf("stale --exec flag in footer: %q", out) + } + // One hint line, plus one leading blank. + lines := []string{} + for _, ln := range strings.Split(strings.Trim(out, "\n"), "\n") { + if strings.TrimSpace(ln) != "" { + lines = append(lines, ln) + } + } + if len(lines) != 1 { + t.Fatalf("footer must be one line, got %d: %q", len(lines), out) + } + if !strings.HasPrefix(lines[0], " ") { + t.Fatalf("footer line must be indented: %q", lines[0]) + } +} + +func TestShortExecIDTrims(t *testing.T) { + long := "3IGpCdEfUbXfaOlpYjrXXy2uKbL" + if got := ShortExecID(long); got != "3IGpCdEf" { + t.Fatalf("ShortExecID = %q", got) + } + if got := ShortExecID("abc"); got != "abc" { + t.Fatalf("short id must pass through, got %q", got) + } +} + +func TestWriteFailureSnapshotConnName(t *testing.T) { + withTempHomeDir(t) + WriteFailureSnapshot(FailureSnapshot{ + ExecID: "exec_conntest", + ErrMsg: "connection refused", + ConnName: "MY_PG", + }) + dir := findLocalExecDir("exec_conntest") + if dir == "" { + t.Fatal("missing snapshot dir") + } + if _, err := os.Stat(filepath.Join(dir, "config.snapshot.yaml")); !os.IsNotExist(err) { + t.Fatal("conns test snapshot must not include config.snapshot.yaml") + } + le, err := ResolveLocalExec("exec_conntest") + if err != nil { + t.Fatal(err) + } + if le.ConnName != "MY_PG" { + t.Fatalf("ConnName=%q", le.ConnName) + } + if le.ConfigPath != "" { + t.Fatalf("ConfigPath should be empty, got %q", le.ConfigPath) + } + if le.displayObject() != "MY_PG" { + t.Fatalf("displayObject=%q", le.displayObject()) + } +} + +func TestWriteFailureSnapshotIncludesSignature(t *testing.T) { + withTempHomeDir(t) + errMsg := `--- task_run.go:140 func2 --- +~ Could not WriteToDb +no stream columns detected` + WriteFailureSnapshot(FailureSnapshot{ + ExecID: "sigExec1", + ErrMsg: errMsg, + SignMeta: SignMeta{ + SourceType: dbio.TypeDbPrometheus, + TargetType: dbio.TypeDbPostgres, + }, + }) + dir := findLocalExecDir("sigExec1") + if dir == "" { + t.Fatal("exec dir missing") + } + body, err := os.ReadFile(filepath.Join(dir, "meta.json")) + if err != nil { + t.Fatal(err) + } + meta := map[string]any{} + if err := json.Unmarshal(body, &meta); err != nil { + t.Fatal(err) + } + wantSig := SignError(errMsg, SignMeta{ + SourceType: dbio.TypeDbPrometheus, TargetType: dbio.TypeDbPostgres, + }) + if cast.ToString(meta["error_signature"]) != wantSig.ID { + t.Fatalf("error_signature = %v want %s", meta["error_signature"], wantSig.ID) + } + if cast.ToString(meta["error_pattern_id"]) != wantSig.PatternID { + t.Fatalf("pattern_id = %v", meta["error_pattern_id"]) + } + if cast.ToString(meta["error_edge_id"]) != wantSig.EdgeID { + t.Fatalf("edge_id = %v", meta["error_edge_id"]) + } + if cast.ToString(meta["error_algorithm"]) != "v1" { + t.Fatalf("algorithm = %v", meta["error_algorithm"]) + } +} + +func TestLookupError(t *testing.T) { + // Pattern-only (8 hex) + r, err := LookupError("97d84811") + if err != nil { + t.Fatal(err) + } + if r.Signature != "97d84811" || !r.PatternOnly || r.PatternID != "97d84811" { + t.Fatalf("%+v", r) + } + // Full composite + r2, err := LookupError("97d84811-5aede62c") + if err != nil || r2.Signature != "97d848115aede62c" || r2.PatternOnly { + t.Fatalf("dashed composite: %+v err=%v", r2, err) + } + if r2.PatternID != "97d84811" || r2.EdgeID != "5aede62c" { + t.Fatalf("parts: %+v", r2) + } + // Display form + r3, err := LookupError("97d84811-5aede62c (prometheus→postgres · no_stream_columns)") + if err != nil || r3.Signature != "97d848115aede62c" { + t.Fatalf("display form: %+v err=%v", r3, err) + } + if _, err := LookupError("not-a-sig"); err == nil { + t.Fatal("expected invalid signature error") + } +} + +func TestSanitizeLogForPromptCapsAndEscapesFences(t *testing.T) { + prev := env.Env + t.Cleanup(func() { env.Env = prev }) + env.Env = &env.EnvFile{Connections: map[string]map[string]any{}} + + in := "before\n```\ninject\n```\nafter" + out := sanitizeLogForPrompt(in, 0) + if strings.Contains(out, "```") { + t.Fatalf("fence not escaped: %q", out) + } + if !strings.Contains(out, "'''") { + t.Fatalf("expected escaped fence: %q", out) + } + big := strings.Repeat("x", maxErrorTailBytes+1000) + capped := sanitizeLogForPrompt(big, maxErrorTailBytes) + if len(capped) > maxErrorTailBytes+len("[...truncated...]\n")+10 { + t.Fatalf("cap too large: %d", len(capped)) + } + if !strings.HasPrefix(capped, "[...truncated...]\n") { + t.Fatalf("missing truncation marker: %q", capped[:40]) + } +} + +func TestSanitizeLogForPromptScrubsSecrets(t *testing.T) { + prev := env.Env + t.Cleanup(func() { env.Env = prev }) + env.Env = &env.EnvFile{Connections: map[string]map[string]any{ + "MY_PG": { + "type": "postgres", + "password": "super-secret-pass", + "secrets": map[string]any{ + "api_key": "nested-api-key", + }, + }, + }} + out := sanitizeLogForPrompt("failed auth super-secret-pass and nested-api-key", 0) + if strings.Contains(out, "super-secret-pass") { + t.Fatalf("password leaked: %q", out) + } + if strings.Contains(out, "nested-api-key") { + t.Fatalf("nested secret leaked: %q", out) + } + if !strings.Contains(out, "***") { + t.Fatalf("expected redaction marker: %q", out) + } +} + +func TestSensitivityClassify(t *testing.T) { + cases := []struct { + path string + want Sensitivity + }{ + {"/home/u/.sling/env.yaml", SensitivitySecret}, + {"/home/u/.claude.json", SensitivitySecret}, + {"/tmp/settings.json.backup", SensitivitySecret}, + {"/home/u/.sling/assist/errors/exec_x/meta.json", SensitivityPublic}, + {"/home/u/.sling/assist/errors/exec_x/error.txt", SensitivityInternal}, + {"/home/u/.agents/skills/sling/SKILL.md", SensitivityPublic}, + {"/unknown/random/file.txt", SensitivityInternal}, // default + } + for _, tc := range cases { + if got := ClassifyPath(tc.path); got != tc.want { + t.Errorf("ClassifyPath(%q)=%s want %s", tc.path, got, tc.want) + } + } +} + +func TestSensitivityManifestNonEmpty(t *testing.T) { + m := SensitivityManifest() + if len(m) < 5 { + t.Fatalf("manifest too small: %d", len(m)) + } + ids := map[string]bool{} + for _, c := range m { + if c.ID == "" || c.Glob == "" || c.Reason == "" { + t.Fatalf("incomplete entry: %+v", c) + } + if ids[c.ID] { + t.Fatalf("duplicate id %s", c.ID) + } + ids[c.ID] = true + } +} + +func TestLegacyExecDirStillReadable(t *testing.T) { + withTempHomeDir(t) + id := "legacy_exec1" + dir := filepath.Join(ErrorsDir(), id) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + meta := `{"exec_id":"` + id + `","exit_code":1,"object":"r.yaml"}` + if err := os.WriteFile(filepath.Join(dir, "meta.json"), []byte(meta), 0o644); err != nil { + t.Fatal(err) + } + if got := findLocalExecDir(id); got != dir { + t.Fatalf("findLocalExecDir = %q want %q", got, dir) + } + execs, err := ListLocalExecs() + if err != nil { + t.Fatal(err) + } + found := false + for _, e := range execs { + if e.ID == id { + found = true + } + } + if !found { + t.Fatal("ListLocalExecs missed legacy exec dir") + } +} + +func TestReservedErrorDirNamesNotListed(t *testing.T) { + withTempHomeDir(t) + _ = ExecutionsDir() + if err := os.MkdirAll(filepath.Join(ErrorsDir(), "signatures"), 0o755); err != nil { + t.Fatal(err) + } + execs, err := ListLocalExecs() + if err != nil { + t.Fatal(err) + } + for _, e := range execs { + if e.ID == "executions" || e.ID == "signatures" { + t.Fatalf("reserved name listed as exec: %q", e.ID) + } + } +} + +func TestLookupLocalExecPrefixAndAmbiguity(t *testing.T) { + withTempHomeDir(t) + for _, id := range []string{"abc111", "abc222", "zzz999"} { + dir := filepath.Join(ErrorsDir(), id) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + meta := `{"exec_id":"` + id + `","exit_code":1,"object":"r.yaml"}` + if err := os.WriteFile(filepath.Join(dir, "meta.json"), []byte(meta), 0o644); err != nil { + t.Fatal(err) + } + } + + if le, ok := LookupLocalExec("zzz999"); !ok || le.ID != "zzz999" { + t.Fatalf("full id: %+v ok=%v", le, ok) + } + if le, ok := LookupLocalExec("zzz"); !ok || le.ID != "zzz999" { + t.Fatalf("unique prefix: %+v ok=%v", le, ok) + } + if _, ok := LookupLocalExec("abc"); ok { + t.Fatal("ambiguous prefix must not resolve") + } + if _, err := ResolveLocalExec("abc"); err == nil || !strings.Contains(err.Error(), "ambiguous") { + t.Fatalf("want ambiguous error, got %v", err) + } + if _, err := ResolveLocalExec("nope"); err == nil || !strings.Contains(err.Error(), "unknown") { + t.Fatalf("want unknown error, got %v", err) + } +} + +func TestWriteFailureSnapshotKeepsRunLog(t *testing.T) { + withTempHomeDir(t) + WriteFailureSnapshot(FailureSnapshot{ + ExecID: "exec_runlog", + ErrMsg: "boom", + RunLog: "DBG opened conn\nINF execution failed", + }) + dir := filepath.Join(ExecutionsDir(), "exec_runlog") + b, err := os.ReadFile(filepath.Join(dir, "stderr.log")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(b), "DBG opened conn") { + t.Fatalf("stderr.log missing run log: %q", string(b)) + } + // error.txt keeps the error only, so runLogExcerpt sees them as distinct. + e, _ := os.ReadFile(filepath.Join(dir, "error.txt")) + if string(e) == string(b) { + t.Fatal("stderr.log must not duplicate error.txt when a run log exists") + } + if got := runLogExcerpt(LocalExec{LogDir: dir}); !strings.Contains(got, "execution failed") { + t.Fatalf("runLogExcerpt: %q", got) + } +} + +func TestRunLogExcerptEmptyWhenMirrored(t *testing.T) { + withTempHomeDir(t) + // No RunLog: stderr.log mirrors error.txt, so nothing extra reaches the prompt. + WriteFailureSnapshot(FailureSnapshot{ExecID: "exec_mirror", ErrMsg: "boom"}) + dir := filepath.Join(ExecutionsDir(), "exec_mirror") + if got := runLogExcerpt(LocalExec{LogDir: dir}); got != "" { + t.Fatalf("want empty excerpt for mirrored log, got %q", got) + } +} + +func TestAutoTrimExecsKeepsNewest(t *testing.T) { + withTempHomeDir(t) + + total := ExecsMaxEntries + 10 + base := time.Now().Add(-time.Duration(total) * time.Hour) + for i := 0; i < total; i++ { + id := fmt.Sprintf("exec_%03d", i) + dir := filepath.Join(ExecutionsDir(), id) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + // later index = newer + when := base.Add(time.Duration(i) * time.Hour) + if err := os.Chtimes(dir, when, when); err != nil { + t.Fatal(err) + } + } + + if err := AutoTrimExecs(); err != nil { + t.Fatal(err) + } + + ids, err := listLocalExecIDs() + if err != nil { + t.Fatal(err) + } + if len(ids) != ExecsMaxEntries { + t.Fatalf("kept %d snapshots, want %d", len(ids), ExecsMaxEntries) + } + // the 10 oldest must be gone, the newest must stay + if dir := findLocalExecDir("exec_000"); dir != "" { + t.Fatalf("oldest snapshot survived: %s", dir) + } + if dir := findLocalExecDir(fmt.Sprintf("exec_%03d", total-1)); dir == "" { + t.Fatal("newest snapshot was trimmed") + } +} + +func TestAutoTrimExecsNoopUnderCap(t *testing.T) { + withTempHomeDir(t) + + for i := 0; i < 5; i++ { + WriteFailureSnapshot(FailureSnapshot{ + ExecID: fmt.Sprintf("exec_keep%d", i), + ErrMsg: "boom", + }) + } + if err := AutoTrimExecs(); err != nil { + t.Fatal(err) + } + ids, err := listLocalExecIDs() + if err != nil { + t.Fatal(err) + } + if len(ids) != 5 { + t.Fatalf("kept %d snapshots, want 5", len(ids)) + } +} + +func TestAutoTrimExecsSkipsReservedDirs(t *testing.T) { + withTempHomeDir(t) + + total := ExecsMaxEntries + 5 + base := time.Now().Add(-time.Duration(total) * time.Hour) + for i := 0; i < total; i++ { + // legacy layout: errors// + dir := filepath.Join(ErrorsDir(), fmt.Sprintf("legacy_%03d", i)) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + when := base.Add(time.Duration(i) * time.Hour) + if err := os.Chtimes(dir, when, when); err != nil { + t.Fatal(err) + } + } + execsDir := ExecutionsDir() // reserved, must survive + + if err := AutoTrimExecs(); err != nil { + t.Fatal(err) + } + + if !g.PathExists(execsDir) { + t.Fatal("executions dir was removed") + } + ids, err := listLocalExecIDs() + if err != nil { + t.Fatal(err) + } + if len(ids) != ExecsMaxEntries { + t.Fatalf("kept %d snapshots, want %d", len(ids), ExecsMaxEntries) + } +} + +func TestWriteFailureSnapshotTrims(t *testing.T) { + withTempHomeDir(t) + + base := time.Now().Add(-time.Duration(ExecsMaxEntries+1) * time.Hour) + for i := 0; i < ExecsMaxEntries; i++ { + dir := filepath.Join(ExecutionsDir(), fmt.Sprintf("exec_old%03d", i)) + if err := os.MkdirAll(dir, 0o755); err != nil { + t.Fatal(err) + } + when := base.Add(time.Duration(i) * time.Hour) + if err := os.Chtimes(dir, when, when); err != nil { + t.Fatal(err) + } + } + + WriteFailureSnapshot(FailureSnapshot{ExecID: "exec_newest", ErrMsg: "boom"}) + + ids, err := listLocalExecIDs() + if err != nil { + t.Fatal(err) + } + if len(ids) != ExecsMaxEntries { + t.Fatalf("kept %d snapshots, want %d", len(ids), ExecsMaxEntries) + } + if dir := findLocalExecDir("exec_newest"); dir == "" { + t.Fatal("new snapshot was trimmed by its own write") + } + if dir := findLocalExecDir("exec_old000"); dir != "" { + t.Fatalf("oldest snapshot survived: %s", dir) + } +} diff --git a/core/sling/assist/jsonedit.go b/core/sling/assist/jsonedit.go new file mode 100644 index 000000000..919c2d400 --- /dev/null +++ b/core/sling/assist/jsonedit.go @@ -0,0 +1,255 @@ +package assist + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + + "github.com/flarco/g" + "github.com/tidwall/gjson" + "github.com/tidwall/jsonc" + "github.com/tidwall/sjson" +) + +// jsonReadOrEmpty parses JSON/JSONC into a map. Missing/empty → empty map. +func jsonReadOrEmpty(path string) (map[string]any, error) { + out := map[string]any{} + data, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return out, nil + } + return nil, g.Error(err, "read %s", path) + } + if len(data) == 0 { + return out, nil + } + stripped := jsonc.ToJSON(data) + if err := json.Unmarshal(stripped, &out); err != nil { + return nil, g.Error(err, "parse %s", path) + } + return out, nil +} + +// jsonWritePretty writes sling-owned JSON (no user comments to preserve). +// For user configs use setJSONPath/deleteJSONPath. +func jsonWritePretty(path string, m map[string]any) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(path)) + } + data, err := json.MarshalIndent(m, "", " ") + if err != nil { + return g.Error(err, "marshal %s", path) + } + data = append(data, '\n') + return writeBytesPreserveMode(path, data, 0o644) +} + +// jsonReadRaw reads JSON/JSONC for surgical sjson edits. +// Missing/empty → "{}". Leading non-JSON (e.g. VS Code banner) is split off +// so sjson does not compact the whole file; caller re-prepends via jsonWriteRaw. +func jsonReadRaw(path string) (prefix, body []byte, err error) { + data, rerr := os.ReadFile(path) + if rerr != nil { + if os.IsNotExist(rerr) { + return nil, []byte("{}"), nil + } + return nil, nil, g.Error(rerr, "read %s", path) + } + if len(bytes.TrimSpace(data)) == 0 { + return nil, []byte("{}"), nil + } + prefix, body = splitLeadingNonJSON(data) + return prefix, body, nil +} + +// splitLeadingNonJSON splits at the first `{` or `[`. +func splitLeadingNonJSON(data []byte) (prefix, body []byte) { + for i, c := range data { + if c == '{' || c == '[' { + if i == 0 { + return nil, data + } + return data[:i], data[i:] + } + } + return data, nil +} + +// jsonWriteRaw writes prefix+body, preserving mode (default 0600 for new files). +func jsonWriteRaw(path string, prefix, body []byte) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(path)) + } + out := body + if len(prefix) > 0 { + out = append(append([]byte{}, prefix...), body...) + } + if len(out) == 0 || out[len(out)-1] != '\n' { + out = append(out, '\n') + } + return writeBytesPreserveMode(path, out, 0o600) +} + +var sjsonOpts = &sjson.Options{Optimistic: true} + +const backupSuffix = ".backup" + +func fileMode(path string, def os.FileMode) os.FileMode { + info, err := os.Stat(path) + if err != nil { + return def + } + return info.Mode().Perm() +} + +// writeBytesPreserveMode writes data, keeping existing mode when overwriting. +func writeBytesPreserveMode(path string, data []byte, defMode os.FileMode) error { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(path)) + } + mode := fileMode(path, defMode) + return os.WriteFile(path, data, mode) +} + +// backupBeforeEdit copies path → path.backup (no-op if missing). +func backupBeforeEdit(path string) error { + src, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return g.Error(err, "read %s for backup", path) + } + if len(src) == 0 { + return nil + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return g.Error(err, "mkdir %s", filepath.Dir(path)) + } + mode := fileMode(path, 0o600) + if err := os.WriteFile(path+backupSuffix, src, mode); err != nil { + return g.Error(err, "write %s", path+backupSuffix) + } + return nil +} + +func restoreBackup(path string) error { + src, err := os.ReadFile(path + backupSuffix) + if err != nil { + return g.Error(err, "read %s", path+backupSuffix) + } + mode := fileMode(path+backupSuffix, fileMode(path, 0o600)) + return os.WriteFile(path, src, mode) +} + +func countTopLevelKeys(data []byte) int { + clean := jsonc.ToJSON(data) + res := gjson.ParseBytes(clean) + if !res.IsObject() { + return 0 + } + n := 0 + res.ForEach(func(_, _ gjson.Result) bool { + n++ + return true + }) + return n +} + +func countLines(data []byte) int { + if len(data) == 0 { + return 0 + } + n := 1 + for _, c := range data { + if c == '\n' { + n++ + } + } + return n +} + +// validateEditNotDestructive refuses edits that drop top-level keys or +// collapse multi-line docs (sjson banner bug). allowKeyDelta=1 for deletes. +func validateEditNotDestructive(before, after []byte, allowKeyDelta int) error { + oldKeys := countTopLevelKeys(before) + newKeys := countTopLevelKeys(after) + if newKeys < oldKeys-allowKeyDelta { + return g.Error("destructive edit refused: top-level keys went from %d to %d", oldKeys, newKeys) + } + oldLines := countLines(before) + newLines := countLines(after) + if oldLines >= 4 && newLines*2 < oldLines { + return g.Error("destructive edit refused: line count went from %d to %d", oldLines, newLines) + } + return nil +} + +// setJSONPath rewrites one path; backs up and refuses destructive rewrites. +func setJSONPath(path, jsonPath string, value any) error { + if err := backupBeforeEdit(path); err != nil { + return err + } + before, err := os.ReadFile(path) + if err != nil && !os.IsNotExist(err) { + return g.Error(err, "read %s", path) + } + prefix, body, err := jsonReadRaw(path) + if err != nil { + return err + } + out, serr := sjson.SetBytesOptions(body, jsonPath, value, sjsonOpts) + if serr != nil { + return g.Error(serr, "set %s in %s", jsonPath, path) + } + if len(before) > 0 { + full := append(append([]byte{}, prefix...), out...) + if verr := validateEditNotDestructive(before, full, 0); verr != nil { + return g.Error(verr, "would have corrupted %s — left original in place; previous content also at %s%s", path, path, backupSuffix) + } + } + return jsonWriteRaw(path, prefix, out) +} + +// deleteJSONPath removes one path; same backup + sanity checks as setJSONPath. +func deleteJSONPath(path, jsonPath string) error { + if err := backupBeforeEdit(path); err != nil { + return err + } + before, err := os.ReadFile(path) + if err != nil && !os.IsNotExist(err) { + return g.Error(err, "read %s", path) + } + prefix, body, err := jsonReadRaw(path) + if err != nil { + return err + } + out, derr := sjson.DeleteBytes(body, jsonPath) + if derr != nil { + return g.Error(derr, "delete %s in %s", jsonPath, path) + } + if len(before) > 0 { + full := append(append([]byte{}, prefix...), out...) + if verr := validateEditNotDestructive(before, full, 1); verr != nil { + return g.Error(verr, "would have corrupted %s — left original in place; previous content also at %s%s", path, path, backupSuffix) + } + } + return jsonWriteRaw(path, prefix, out) +} + +// gjsonGetArrayStrings reads a JSON string array (JSONC-safe). +func gjsonGetArrayStrings(data []byte, path string) []string { + clean := jsonc.ToJSON(data) + res := gjson.GetBytes(clean, path) + if !res.Exists() || !res.IsArray() { + return nil + } + out := []string{} + res.ForEach(func(_, v gjson.Result) bool { + out = append(out, v.String()) + return true + }) + return out +} diff --git a/core/sling/assist/opencode.go b/core/sling/assist/opencode.go new file mode 100644 index 000000000..00b6db28f --- /dev/null +++ b/core/sling/assist/opencode.go @@ -0,0 +1,426 @@ +package assist + +import ( + "context" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "time" + + "github.com/flarco/g" + "github.com/flarco/g/net" + "github.com/slingdata-io/sling-cli/core/dbio/iop" + "github.com/slingdata-io/sling-cli/core/env" + "golang.org/x/sys/cpu" +) + +// OpenCodeVersion is the pinned CLI release. Override with OPENCODE_VERSION. +// Pin checked against https://github.com/anomalyco/opencode/releases (v1.18.18, 2026-08-13). +const OpenCodeVersion = "1.18.18" + +// ZenFreeDisclosure is shown when the harness uses a free Zen model. +const ZenFreeDisclosure = "free-model prompts may be used for training" + +// ZenFreeModel is the default free OpenCode Zen model id (`provider/model`). +const ZenFreeModel = "opencode/big-pickle" + +const openCodeGitHubBase = "https://github.com/anomalyco/opencode/releases/download/v{version}/{asset}" + +// openCodeTestDownloadURL replaces the GitHub asset URL in tests (httptest zip). +var openCodeTestDownloadURL string + +// zenModelsURL is the Zen catalog. Tests point this at httptest. +var zenModelsURL = "https://opencode.ai/zen/v1/models" + +var zenHTTPClient = &http.Client{Timeout: 5 * time.Second} + +var ( + openCodeHasAVX2 = func() bool { return cpu.X86.HasAVX2 } + openCodeIsMusl = linuxMuslPresent +) + +type openCodeInstall struct { + version string +} + +func newOpenCodeInstall() *openCodeInstall { + return &openCodeInstall{version: openCodeVersion()} +} + +func openCodeVersion() string { + if val := strings.TrimSpace(os.Getenv("OPENCODE_VERSION")); val != "" { + return strings.TrimPrefix(val, "v") + } + return OpenCodeVersion +} + +func openCodeBinName() string { + if runtime.GOOS == "windows" { + return "opencode.exe" + } + return "opencode" +} + +func (o *openCodeInstall) dest() string { + return filepath.Join(env.HomeBinDir(), "opencode", o.version) +} + +func (o *openCodeInstall) bundledPath() string { + return filepath.Join(o.dest(), openCodeBinName()) +} + +// BundledOpenCodePath is ~/.sling/bin/opencode//opencode[.exe]. +func BundledOpenCodePath() string { + return newOpenCodeInstall().bundledPath() +} + +func linuxMuslPresent() bool { + matches, _ := filepath.Glob("/lib/ld-musl-*") + return len(matches) > 0 +} + +func (o *openCodeInstall) assetName(goos, goarch string) (string, error) { + var osName, arch string + switch goos { + case "darwin", "linux", "windows": + osName = goos + default: + return "", g.Error("opencode is not available for %s/%s", goos, goarch) + } + switch goarch { + case "amd64": + arch = "x64" + case "arm64": + arch = "arm64" + default: + return "", g.Error("opencode is not available for %s/%s", goos, goarch) + } + + ext := "zip" + if goos == "linux" { + ext = "tar.gz" + } + + suffix := "" + if arch == "x64" && !openCodeHasAVX2() { + suffix += "-baseline" + } + if goos == "linux" && openCodeIsMusl() { + suffix += "-musl" + } + return fmt.Sprintf("opencode-%s-%s%s.%s", osName, arch, suffix, ext), nil +} + +// OpenCodeAssetName is the GitHub asset for goos/goarch (pinned layout, not /latest). +func OpenCodeAssetName(goos, goarch string) (string, error) { + return newOpenCodeInstall().assetName(goos, goarch) +} + +func (o *openCodeInstall) downloadURL() (string, error) { + if openCodeTestDownloadURL != "" { + return openCodeTestDownloadURL, nil + } + asset, err := o.assetName(runtime.GOOS, runtime.GOARCH) + if err != nil { + return "", err + } + return g.R(openCodeGitHubBase, "version", o.version, "asset", asset), nil +} + +func openCodeDownloadURL(version string) (string, error) { + return (&openCodeInstall{version: version}).downloadURL() +} + +func versionMatches(out, version string) bool { + s := strings.TrimSpace(out) + return strings.HasPrefix(s, version) || strings.HasPrefix(s, "v"+version) +} + +func (o *openCodeInstall) versionOK(binPath string) (bool, error) { + out, err := exec.Command(binPath, "--version").CombinedOutput() + if err != nil { + return false, g.Error(err, "could not get version for opencode: %s", strings.TrimSpace(string(out))) + } + return versionMatches(string(out), o.version), nil +} + +func (o *openCodeInstall) findBin(folder string) string { + want := openCodeBinName() + direct := filepath.Join(folder, want) + if g.PathExists(direct) { + return direct + } + var found string + _ = filepath.Walk(folder, func(p string, info os.FileInfo, err error) error { + if err != nil || info.IsDir() { + return nil + } + if info.Name() == want || info.Name() == "opencode" { + found = p + return filepath.SkipAll + } + return nil + }) + return found +} + +func (o *openCodeInstall) extract(archive, dest string) error { + if strings.HasSuffix(archive, ".tar.gz") || strings.HasSuffix(archive, ".tgz") { + if err := g.ExtractTarGz(archive, dest); err != nil { + return g.Error(err, "error extracting opencode archive") + } + return nil + } + if _, err := iop.Unzip(archive, dest); err != nil { + return g.Error(err, "error unzipping opencode archive") + } + return nil +} + +// EnsureBinOpenCode returns a usable opencode binary. +// Order: OPENCODE_PATH, $PATH, then a versioned download under ~/.sling/bin/opencode//. +func EnsureBinOpenCode() (binPath string, err error) { + return newOpenCodeInstall().ensure() +} + +func (o *openCodeInstall) ensure() (binPath string, err error) { + if envPath := os.Getenv("OPENCODE_PATH"); envPath != "" { + if !g.PathExists(envPath) { + return "", g.Error("opencode binary not found: %s", envPath) + } + if stat, _ := os.Stat(envPath); stat != nil && stat.IsDir() { + return "", g.Error("OPENCODE_PATH provided is a directory, should be a file: %s", envPath) + } + return envPath, nil + } + + if p, err := exec.LookPath("opencode"); err == nil { + return p, nil + } + + folderPath := o.dest() + binPath = o.bundledPath() + found := g.PathExists(binPath) + if found { + ok, verr := o.versionOK(binPath) + if verr != nil { + found = false + } else { + found = ok + } + } + + if !found { + downloadURL, uerr := o.downloadURL() + if uerr != nil { + return "", uerr + } + + ext := ".zip" + if strings.Contains(downloadURL, ".tar.gz") { + ext = ".tar.gz" + } + archivePath := filepath.Join(os.TempDir(), g.F("opencode-%s%s", o.version, ext)) + defer os.Remove(archivePath) + + g.Info("downloading opencode %s for %s/%s", o.version, runtime.GOOS, runtime.GOARCH) + if err = net.DownloadFile(downloadURL, archivePath); err != nil { + return "", g.Error(err, "unable to download opencode binary") + } + + if err = os.MkdirAll(folderPath, 0755); err != nil { + return "", g.Error(err, "could not create opencode folder") + } + + if err = o.extract(archivePath, folderPath); err != nil { + return "", err + } + + foundBin := o.findBin(folderPath) + if foundBin == "" { + return "", g.Error("cannot find opencode binary at %s after extraction", binPath) + } + if foundBin != binPath { + if err = os.Rename(foundBin, binPath); err != nil { + return "", g.Error(err, "could not move opencode binary to %s", binPath) + } + } + if !g.PathExists(binPath) { + return "", g.Error("cannot find opencode binary at %s after extraction", binPath) + } + if err = os.Chmod(binPath, 0755); err != nil { + return "", g.Error(err, "could not make opencode executable") + } + } + + ok, err := o.versionOK(binPath) + if err != nil { + return "", err + } + if !ok { + return "", g.Error("opencode at %s does not report version %s", binPath, o.version) + } + return binPath, nil +} + +// ProviderChoice is one harness LLM setup (Zen free vs keyed provider). +type ProviderChoice struct { + Kind string // zen-free | anthropic | openai | google | xai + Model string // provider/model + Provider map[string]any // opencode.json `provider` map; nil if unused + Disclosure string // set for zen-free +} + +func envFirst(keys ...string) string { + for _, k := range keys { + if v := strings.TrimSpace(os.Getenv(k)); v != "" { + return k + } + } + return "" +} + +func keyedProvider(kind, model, envKey string) ProviderChoice { + return ProviderChoice{ + Kind: kind, + Model: model, + Provider: map[string]any{ + kind: map[string]any{ + "options": map[string]any{ + "apiKey": "{env:" + envKey + "}", + }, + }, + }, + } +} + +// HarnessProviderChoice picks a keyed provider env if set, else free Zen. +func HarnessProviderChoice() ProviderChoice { + if k := envFirst("ANTHROPIC_API_KEY"); k != "" { + return keyedProvider("anthropic", "anthropic/claude-sonnet-4-5", k) + } + if k := envFirst("OPENAI_API_KEY"); k != "" { + return keyedProvider("openai", "openai/gpt-4o", k) + } + if k := envFirst("GEMINI_API_KEY", "GOOGLE_API_KEY"); k != "" { + return keyedProvider("google", "google/gemini-2.0-flash", k) + } + if k := envFirst("XAI_API_KEY"); k != "" { + return keyedProvider("xai", "xai/grok-3", k) + } + return ProviderChoice{ + Kind: "zen-free", + Model: ZenFreeModel, + Disclosure: ZenFreeDisclosure, + } +} + +// HarnessProviderConfig is the opencode.json fragment for the harness choice. +func HarnessProviderConfig() map[string]any { + c := HarnessProviderChoice() + out := map[string]any{"model": c.Model} + if len(c.Provider) > 0 { + out["provider"] = c.Provider + } + return out +} + +// ApplyHarnessProviderConfig writes model (and provider) when missing. +func ApplyHarnessProviderConfig() error { + path := filepath.Join(opencodeConfigDir(), "opencode.json") + doc, err := jsonReadOrEmpty(path) + if err != nil { + return err + } + cfg := HarnessProviderConfig() + if _, ok := doc["model"]; !ok { + if err := setJSONPath(path, "model", cfg["model"]); err != nil { + return err + } + } + if _, has := doc["provider"]; !has { + if p, ok := cfg["provider"]; ok { + if err := setJSONPath(path, "provider", p); err != nil { + return err + } + } + } + return nil +} + +func opencodeRelevant() bool { + if (&opencodeClient{}).Detect() { + return true + } + prof, exists, err := LoadProfile() + if err != nil || !exists { + return false + } + return prof.Agent == "opencode" +} + +var zenFreeModelTokens = []string{ + "big-pickle", + "mimo-v2.5-free", + "hy3-free", + "nemotron-3-ultra-free", + "nemotron-3.5-lightning-free", + "muse-spark-1.2-contributor-free", +} + +// ProbeZenFreeModel reports whether the free Zen catalog is reachable. +func ProbeZenFreeModel() (ok bool, detail string) { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, zenModelsURL, nil) + if err != nil { + return false, err.Error() + } + resp, err := zenHTTPClient.Do(req) + if err != nil { + return false, err.Error() + } + defer resp.Body.Close() + body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return false, fmt.Sprintf("HTTP %d", resp.StatusCode) + } + s := strings.ToLower(string(body)) + for _, tok := range zenFreeModelTokens { + if strings.Contains(s, strings.ToLower(tok)) { + return true, tok + } + } + return false, "no free model in catalog" +} + +func (r *DoctorReport) addZenFinding() { + if r == nil || !opencodeRelevant() { + return + } + if HarnessProviderChoice().Kind != "zen-free" { + return + } + ok, detail := ProbeZenFreeModel() + if ok { + r.AddFinding(DoctorFinding{ + ID: "opencode.zen", + OK: true, + Summary: "free model available", + Detail: detail, + }) + return + } + r.AddFinding(DoctorFinding{ + ID: "opencode.zen", + OK: false, + Summary: "free model unavailable", + Detail: detail, + Hint: "set ANTHROPIC_API_KEY or OPENAI_API_KEY for a keyed provider", + }) +} diff --git a/core/sling/assist/opencode_network_test.go b/core/sling/assist/opencode_network_test.go new file mode 100644 index 000000000..95c4e2b00 --- /dev/null +++ b/core/sling/assist/opencode_network_test.go @@ -0,0 +1,35 @@ +//go:build network + +package assist + +import ( + "net/http" + "testing" + "time" +) + +// Optional live check that the pinned GitHub asset exists. +// Default `go test ./core/sling/assist/` stays offline (this file is tagged). +func TestPinnedOpenCodeReleaseAssetExists(t *testing.T) { + openCodeTestDownloadURL = "" + u, err := openCodeDownloadURL(OpenCodeVersion) + if err != nil { + t.Fatal(err) + } + client := &http.Client{Timeout: 15 * time.Second, CheckRedirect: func(req *http.Request, via []*http.Request) error { + return nil + }} + req, err := http.NewRequest(http.MethodGet, u, nil) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Range", "bytes=0-0") + resp, err := client.Do(req) + if err != nil { + t.Fatal(err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent && resp.StatusCode != http.StatusFound && resp.StatusCode != http.StatusTemporaryRedirect { + t.Fatalf("pinned asset %s: HTTP %d", u, resp.StatusCode) + } +} diff --git a/core/sling/assist/opencode_test.go b/core/sling/assist/opencode_test.go new file mode 100644 index 000000000..d52d2cd7d --- /dev/null +++ b/core/sling/assist/opencode_test.go @@ -0,0 +1,432 @@ +package assist + +import ( + "archive/zip" + "bytes" + "context" + "io" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "sync/atomic" + "testing" +) + +func isolateOpenCodeEnv(t *testing.T) { + t.Helper() + home := withTempHomeDir(t) + bin := filepath.Join(home, "empty-bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin) + t.Setenv("OPENCODE_PATH", "") + t.Setenv("OPENCODE_VERSION", OpenCodeVersion) + t.Setenv("XDG_CONFIG_HOME", "") + t.Setenv("ANTHROPIC_API_KEY", "") + t.Setenv("OPENAI_API_KEY", "") + t.Setenv("GEMINI_API_KEY", "") + t.Setenv("GOOGLE_API_KEY", "") + t.Setenv("XAI_API_KEY", "") + t.Cleanup(func() { openCodeTestDownloadURL = "" }) +} + +func zipOpenCodeStub(t *testing.T, version string) []byte { + t.Helper() + buf := new(bytes.Buffer) + zw := zip.NewWriter(buf) + h := &zip.FileHeader{Name: "opencode", Method: zip.Deflate} + h.SetMode(0o755) + w, err := zw.CreateHeader(h) + if err != nil { + t.Fatal(err) + } + script := "#!/bin/sh\n" + + "if [ \"$1\" = \"--version\" ]; then echo \"" + version + "\"; exit 0; fi\n" + + "echo stub\n" + if _, err := io.WriteString(w, script); err != nil { + t.Fatal(err) + } + if err := zw.Close(); err != nil { + t.Fatal(err) + } + return buf.Bytes() +} + +func serveOpenCodeZip(t *testing.T, version string) (*httptest.Server, *atomic.Int32) { + t.Helper() + payload := zipOpenCodeStub(t, version) + hits := &atomic.Int32{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits.Add(1) + w.Header().Set("Content-Type", "application/zip") + _, _ = w.Write(payload) + })) + t.Cleanup(srv.Close) + return srv, hits +} + +func TestOpenCodeAssetNameLinuxX64(t *testing.T) { + prevAVX, prevMusl := openCodeHasAVX2, openCodeIsMusl + t.Cleanup(func() { + openCodeHasAVX2, openCodeIsMusl = prevAVX, prevMusl + }) + + cases := []struct { + avx2, musl bool + want string + }{ + {true, false, "opencode-linux-x64.tar.gz"}, + {false, false, "opencode-linux-x64-baseline.tar.gz"}, + {true, true, "opencode-linux-x64-musl.tar.gz"}, + {false, true, "opencode-linux-x64-baseline-musl.tar.gz"}, + } + for _, tc := range cases { + openCodeHasAVX2 = func() bool { return tc.avx2 } + openCodeIsMusl = func() bool { return tc.musl } + got, err := OpenCodeAssetName("linux", "amd64") + if err != nil { + t.Fatalf("avx2=%v musl=%v: %v", tc.avx2, tc.musl, err) + } + if got != tc.want { + t.Errorf("avx2=%v musl=%v: got %s want %s", tc.avx2, tc.musl, got, tc.want) + } + } +} + +func TestOpenCodeAssetNamePinnedPlatforms(t *testing.T) { + prevAVX, prevMusl := openCodeHasAVX2, openCodeIsMusl + t.Cleanup(func() { + openCodeHasAVX2, openCodeIsMusl = prevAVX, prevMusl + }) + openCodeHasAVX2 = func() bool { return true } + openCodeIsMusl = func() bool { return false } + + cases := map[string]string{ + "darwin/arm64": "opencode-darwin-arm64.zip", + "darwin/amd64": "opencode-darwin-x64.zip", + "windows/amd64": "opencode-windows-x64.zip", + "windows/arm64": "opencode-windows-arm64.zip", + "linux/arm64": "opencode-linux-arm64.tar.gz", + } + for plat, want := range cases { + parts := strings.Split(plat, "/") + got, err := OpenCodeAssetName(parts[0], parts[1]) + if err != nil { + t.Fatalf("%s: %v", plat, err) + } + if got != want { + t.Errorf("%s: got %s want %s", plat, got, want) + } + } + if _, err := OpenCodeAssetName("js", "wasm"); err == nil { + t.Fatal("expected error for js/wasm") + } +} + +func TestEnsureBinOpenCodeInstallFromZip(t *testing.T) { + isolateOpenCodeEnv(t) + srv, hits := serveOpenCodeZip(t, OpenCodeVersion) + openCodeTestDownloadURL = srv.URL + "/opencode.zip" + + bin, err := EnsureBinOpenCode() + if err != nil { + t.Fatalf("EnsureBinOpenCode: %v", err) + } + want := BundledOpenCodePath() + if bin != want { + t.Fatalf("bin = %s, want %s", bin, want) + } + info, err := os.Stat(bin) + if err != nil { + t.Fatal(err) + } + if info.Mode().Perm() != 0o755 { + t.Fatalf("perm = %o, want 0755", info.Mode().Perm()) + } + out, err := exec.Command(bin, "--version").CombinedOutput() + if err != nil { + t.Fatalf("--version: %v (%s)", err, out) + } + if !versionMatches(string(out), OpenCodeVersion) { + t.Fatalf("version %q does not match %s", out, OpenCodeVersion) + } + if hits.Load() != 1 { + t.Fatalf("download hits = %d, want 1", hits.Load()) + } + + bin2, err := EnsureBinOpenCode() + if err != nil { + t.Fatal(err) + } + if bin2 != bin { + t.Fatalf("second call bin = %s", bin2) + } + if hits.Load() != 1 { + t.Fatalf("stale dir should not re-download; hits = %d", hits.Load()) + } +} + +func TestEnsureBinOpenCodeStaleRedownload(t *testing.T) { + isolateOpenCodeEnv(t) + want := BundledOpenCodePath() + if err := os.MkdirAll(filepath.Dir(want), 0o755); err != nil { + t.Fatal(err) + } + stale := "#!/bin/sh\necho 0.0.1\n" + if err := os.WriteFile(want, []byte(stale), 0o755); err != nil { + t.Fatal(err) + } + + srv, hits := serveOpenCodeZip(t, OpenCodeVersion) + openCodeTestDownloadURL = srv.URL + "/opencode.zip" + + bin, err := EnsureBinOpenCode() + if err != nil { + t.Fatalf("EnsureBinOpenCode: %v", err) + } + if bin != want { + t.Fatalf("bin = %s, want %s", bin, want) + } + if hits.Load() != 1 { + t.Fatalf("stale binary should re-download; hits = %d", hits.Load()) + } + out, err := exec.Command(bin, "--version").CombinedOutput() + if err != nil { + t.Fatal(err) + } + if !versionMatches(string(out), OpenCodeVersion) { + t.Fatalf("after re-download version = %q", out) + } +} + +func TestEnsureBinOpenCodePathOverride(t *testing.T) { + isolateOpenCodeEnv(t) + dir := t.TempDir() + asDir := filepath.Join(dir, "as-dir") + if err := os.MkdirAll(asDir, 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("OPENCODE_PATH", asDir) + if _, err := EnsureBinOpenCode(); err == nil { + t.Fatal("expected error for directory OPENCODE_PATH") + } + + file := filepath.Join(dir, "opencode") + if err := os.WriteFile(file, []byte("x"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("OPENCODE_PATH", file) + got, err := EnsureBinOpenCode() + if err != nil { + t.Fatal(err) + } + if got != file { + t.Fatalf("got %s want %s", got, file) + } +} + +func TestEnsureBinOpenCodeSystemPathWins(t *testing.T) { + isolateOpenCodeEnv(t) + home := withTempHomeDir(t) + binDir := filepath.Join(home, "sys-bin") + if err := os.MkdirAll(binDir, 0o755); err != nil { + t.Fatal(err) + } + sys := filepath.Join(binDir, "opencode") + if err := os.WriteFile(sys, []byte("#!/bin/sh\necho sys\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", binDir) + + hits := &atomic.Int32{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits.Add(1) + w.WriteHeader(500) + })) + t.Cleanup(srv.Close) + openCodeTestDownloadURL = srv.URL + "/opencode.zip" + + got, err := EnsureBinOpenCode() + if err != nil { + t.Fatal(err) + } + if got != sys { + t.Fatalf("got %s want system %s", got, sys) + } + if hits.Load() != 0 { + t.Fatalf("system binary should skip download; hits = %d", hits.Load()) + } +} + +func TestHarnessProviderChoiceZenDisclosure(t *testing.T) { + isolateOpenCodeEnv(t) + c := HarnessProviderChoice() + if c.Kind != "zen-free" { + t.Fatalf("kind = %s", c.Kind) + } + if c.Model != ZenFreeModel { + t.Fatalf("model = %s", c.Model) + } + if c.Disclosure != ZenFreeDisclosure { + t.Fatalf("disclosure = %q", c.Disclosure) + } + cfg := HarnessProviderConfig() + if cfg["model"] != ZenFreeModel { + t.Fatalf("config model = %v", cfg["model"]) + } +} + +func TestHarnessProviderChoiceKeyedWins(t *testing.T) { + isolateOpenCodeEnv(t) + t.Setenv("ANTHROPIC_API_KEY", "sk-test") + c := HarnessProviderChoice() + if c.Kind != "anthropic" { + t.Fatalf("kind = %s", c.Kind) + } + if c.Disclosure != "" { + t.Fatalf("keyed provider should have no zen disclosure, got %q", c.Disclosure) + } + if !strings.Contains(c.Model, "anthropic/") { + t.Fatalf("model = %s", c.Model) + } +} + +func TestApplyHarnessProviderConfigWritesModel(t *testing.T) { + isolateOpenCodeEnv(t) + if err := ApplyHarnessProviderConfig(); err != nil { + t.Fatal(err) + } + path := filepath.Join(opencodeConfigDir(), "opencode.json") + body, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(body), ZenFreeModel) { + t.Fatalf("missing model in %s", body) + } + if err := setJSONPath(path, "model", "keep/me"); err != nil { + t.Fatal(err) + } + if err := ApplyHarnessProviderConfig(); err != nil { + t.Fatal(err) + } + body, _ = os.ReadFile(path) + if !strings.Contains(string(body), "keep/me") { + t.Fatalf("should not clobber existing model: %s", body) + } +} + +func TestDoctorFreeModelUnavailable(t *testing.T) { + isolateOpenCodeEnv(t) + if err := SaveProfile(Profile{Agent: "opencode", HintInErrors: true}); err != nil { + t.Fatal(err) + } + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusServiceUnavailable) + })) + t.Cleanup(srv.Close) + prev := zenModelsURL + zenModelsURL = srv.URL + t.Cleanup(func() { zenModelsURL = prev }) + + r, err := Doctor(context.Background()) + if err != nil { + t.Fatal(err) + } + var found bool + for _, f := range r.Findings { + if f.ID == "opencode.zen" { + found = true + if f.OK { + t.Fatalf("want fail, got %+v", f) + } + if !strings.Contains(f.Summary, "free model unavailable") { + t.Fatalf("summary = %q", f.Summary) + } + } + } + if !found { + t.Fatalf("missing opencode.zen finding: %+v", r.Findings) + } + if r.OK { + t.Fatal("doctor should fail when free model is unavailable") + } +} + +func TestDoctorFreeModelAvailable(t *testing.T) { + isolateOpenCodeEnv(t) + if err := SaveProfile(Profile{Agent: "opencode", HintInErrors: true}); err != nil { + t.Fatal(err) + } + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`{"data":[{"id":"big-pickle"}]}`)) + })) + t.Cleanup(srv.Close) + prev := zenModelsURL + zenModelsURL = srv.URL + t.Cleanup(func() { zenModelsURL = prev }) + + r, err := Doctor(context.Background()) + if err != nil { + t.Fatal(err) + } + var found bool + for _, f := range r.Findings { + if f.ID == "opencode.zen" { + found = true + if !f.OK { + t.Fatalf("want ok, got %+v", f) + } + } + } + if !found { + t.Fatalf("missing opencode.zen finding: %+v", r.Findings) + } +} + +func TestRankedCLIAgentsIncludesBundledOpenCode(t *testing.T) { + isolateOpenCodeEnv(t) + ranked := RankedCLIAgents() + if len(ranked) == 0 { + t.Fatal("expected bundled opencode fallback") + } + last := ranked[len(ranked)-1] + if !last.Bundled || last.Name != "opencode" { + t.Fatalf("want bundled opencode last, got %+v", last) + } + label := agentAuthLabel(last) + if !strings.Contains(label, ZenFreeDisclosure) { + t.Fatalf("bundled label missing disclosure: %s", label) + } + if !strings.Contains(harnessAgentDescription(ranked), "ANTHROPIC_API_KEY") { + t.Fatal("form description should mention keyed-provider alternative") + } +} + +func TestOpenCodeDownloadURLNeverLatest(t *testing.T) { + prevAVX, prevMusl := openCodeHasAVX2, openCodeIsMusl + t.Cleanup(func() { + openCodeHasAVX2, openCodeIsMusl = prevAVX, prevMusl + }) + openCodeHasAVX2 = func() bool { return true } + openCodeIsMusl = func() bool { return false } + openCodeTestDownloadURL = "" + u, err := openCodeDownloadURL(OpenCodeVersion) + if err != nil { + t.Fatal(err) + } + if strings.Contains(u, "/latest") { + t.Fatalf("must not use releases/latest: %s", u) + } + if !strings.Contains(u, "/v"+OpenCodeVersion+"/") { + t.Fatalf("missing pinned tag: %s", u) + } + if runtime.GOOS == "darwin" && runtime.GOARCH == "arm64" && !strings.HasSuffix(u, "opencode-darwin-arm64.zip") { + t.Fatalf("unexpected darwin arm64 url: %s", u) + } +} diff --git a/core/sling/assist/prompt.go b/core/sling/assist/prompt.go new file mode 100644 index 000000000..2ee93baeb --- /dev/null +++ b/core/sling/assist/prompt.go @@ -0,0 +1,945 @@ +package assist + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "text/template" + "time" + "unicode/utf8" + + "github.com/slingdata-io/sling-cli/core" + "github.com/slingdata-io/sling-cli/core/dbio/connection" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/slingdata-io/sling-cli/core/sling/project" + "github.com/slingdata-io/sling-cli/core/sling/validate" + "github.com/spf13/cast" + "gopkg.in/yaml.v3" +) + +// maxPromptTokens is a soft cap for the assembled --out prompt (~4 chars/token). +const maxPromptTokens = 1500 + +// maxErrorExcerptLines caps the sanitized failure excerpt in Context. +const maxErrorExcerptLines = 15 + +const maxProbeConnections = 10 + +// canonicalFolders are the wave-5 project dirs scanned for file counts. +var canonicalFolders = []string{"replications", "pipelines", "models", "specs"} + +// ProbeConn is name/type/source from ConnEntries.List() (no payload fields). +type ProbeConn struct { + Name string + Type string + Source string +} + +// PromptContext is the assembled state for the five-section agent prompt. +type PromptContext struct { + Version string + Cwd string + ProjectName string + ProjectRoot string + HasProject bool + FileCounts map[string]int + Connections []ProbeConn + ConnectionExtra int + RecentRuns []LocalExec + TargetExec *LocalExec // set by --id: investigate this failure + PlatformExec *PlatformExec // set by --id when the exec is on the platform only + MCPWired bool + Signature string + Lookup *ErrorLookupResult + ErrorExcerpt string + RunLogExcerpt string + Ask string + Suggestions []string + Route string + Agent string +} + +type promptView struct { + Rules string + State string + Context string + Ask string + Objective string +} + +type promptBundle struct { + Skeleton string `yaml:"_skeleton"` + Rules string `yaml:"_rules"` + Objective string `yaml:"_objective"` +} + +func loadPromptBundle() (promptBundle, error) { + var b promptBundle + if err := yaml.Unmarshal(PromptsYAML, &b); err != nil { + return b, err + } + if strings.TrimSpace(b.Skeleton) == "" { + return b, fmt.Errorf("prompts.yaml missing _skeleton") + } + if strings.TrimSpace(b.Rules) == "" { + return b, fmt.Errorf("prompts.yaml missing _rules") + } + if strings.TrimSpace(b.Objective) == "" { + return b, fmt.Errorf("prompts.yaml missing _objective") + } + return b, nil +} + +// ProbeOptions is the user-facing input to Probe. +// IncludeFailure pulls signature/lookup/excerpt into Context. Set it only +// when the ask targets the failure (investigate pick, empty-ask fallback, +// landing card); other asks keep the one-line recent-runs summary. +type ProbeOptions struct { + Ask string + IncludeFailure bool + ExecID string // target this exec instead of the latest failure +} + +// Probe gathers local state for the agent prompt. +// It lists connections via ConnEntries.List() only (name/type/source). +func Probe(opts ProbeOptions) PromptContext { + cwd := workDir() + p := PromptContext{ + Version: core.Version, + Cwd: cwd, + Ask: strings.TrimSpace(opts.Ask), + FileCounts: map[string]int{}, + } + p.ProjectName, p.ProjectRoot, p.HasProject = detectProject(cwd) + scanRoot := cwd + if p.ProjectRoot != "" { + scanRoot = p.ProjectRoot + } + p.FileCounts = countCanonicalFiles(scanRoot) + p.Connections, p.ConnectionExtra = capProbeConnections(listProbeConnections()) + p.RecentRuns = recentRunsForPrompt() + p.MCPWired = probeMCPWired() + if prof, ok, _ := LoadProfile(); ok { + p.Agent = prof.Agent + } + if opts.ExecID != "" { + if le, ok := LookupLocalExec(opts.ExecID); ok { + p.TargetExec = &le + p.Signature, p.Lookup, p.ErrorExcerpt = failureDetails(le) + p.RunLogExcerpt = runLogExcerpt(le) + } + } else if opts.IncludeFailure { + if lead, ok := leadingFailure(p.RecentRuns); ok { + p.Signature, p.Lookup, p.ErrorExcerpt = failureDetails(lead) + p.RunLogExcerpt = runLogExcerpt(lead) + } + } + for _, s := range p.suggestions() { + p.Suggestions = append(p.Suggestions, s.Label) + } + p.Route = Route(p) + return p +} + +// Route picks the prompt ladder rung. Pure Go, no LLM. +// Order: ask > recent failed run > zero connections > no project in cwd > default. +// Free function: PromptContext already has a Route field. +func Route(p PromptContext) string { + if p.PlatformExec != nil { + return "platform_failed_run" + } + if p.TargetExec != nil { + return "failed_run" + } + if strings.TrimSpace(p.Ask) != "" { + return "ask" + } + if _, ok := leadingFailure(p.RecentRuns); ok { + return "failed_run" + } + if userConnectionCount(p.Connections) == 0 { + return "zero_connections" + } + if !p.HasProject { + return "no_project" + } + return "default" +} + +const ( + suggestionElseLabel = "Something else — just describe it" + maxSuggestionRows = 4 + maxSignalRows = 3 +) + +// suggestion is one open-screen row. Label is shown; Ask is the launch text. +// Investigate marks the failed-run row: only that pick puts failure details +// (signature, lookup, excerpt) into the prompt Context. +type suggestion struct { + Label string + Ask string // empty → free text ("something else") + Investigate bool +} + +func (p PromptContext) suggestions() []suggestion { + var signals []suggestion + if fail, ok := leadingFailure(p.RecentRuns); ok { + label := failedRunLabel(fail) + signals = append(signals, suggestion{Label: label, Ask: label, Investigate: true}) + } + if userConnectionCount(p.Connections) == 0 { + signals = append(signals, suggestion{ + Label: "Add your first connection", + Ask: "Add a connection", + }) + } + if !p.HasProject { + signals = append(signals, suggestion{ + Label: "Scaffold a project (`sling init`)", + Ask: "Scaffold a Sling project in this folder", + }) + } + out := signals + if len(out) < maxSignalRows { + out = append(out, suggestion{ + Label: "Create or update a replication / pipeline / model / API spec", + Ask: "Help me create or update a Sling config (replication, pipeline, model, or API spec). First ask me which one and which connections it involves.", + }) + } + if len(out) > maxSignalRows { + out = out[:maxSignalRows] + } + out = append(out, suggestion{Label: suggestionElseLabel}) + if len(out) > maxSuggestionRows { + out = append(out[:maxSuggestionRows-1], suggestion{Label: suggestionElseLabel}) + } + return out +} + +func failedRunLabel(r LocalExec) string { + idObj := strings.TrimSpace(r.ID + " " + r.displayObject()) + idObj = strings.Join(strings.Fields(idObj), " ") + s := "Investigate the failed run " + idObj + if r.When.IsZero() { + return s + } + rt := relTime(r.When) + if rt == "just now" { + return s + " (just now)" + } + return s + " (" + rt + " ago)" +} + +func userConnectionCount(conns []ProbeConn) int { + n := 0 + for _, c := range conns { + if c.Source == "built-in" { + continue + } + n++ + } + return n +} + +func leadingFailure(runs []LocalExec) (LocalExec, bool) { + for _, r := range runs { + if r.Status == "err" { + return r, true + } + } + return LocalExec{}, false +} + +func listProbeConnections() []ProbeConn { + entries := connection.GetLocalConns(true) + _, rows := entries.List() + out := make([]ProbeConn, 0, len(rows)) + for _, row := range rows { + if len(row) < 3 { + continue + } + out = append(out, ProbeConn{ + Name: cast.ToString(row[0]), + Type: cast.ToString(row[1]), + Source: cast.ToString(row[2]), + }) + } + sort.Slice(out, func(i, j int) bool { + si, sj := connSourceRank(out[i].Source), connSourceRank(out[j].Source) + if si != sj { + return si < sj + } + return out[i].Name < out[j].Name + }) + return out +} + +func connSourceRank(source string) int { + switch source { + case "sling env yaml": + return 0 + case "built-in": + return 1 + default: + return 2 + } +} + +func capProbeConnections(all []ProbeConn) ([]ProbeConn, int) { + if len(all) <= maxProbeConnections { + return all, 0 + } + return all[:maxProbeConnections], len(all) - maxProbeConnections +} + +func detectProject(cwd string) (name, root string, has bool) { + if cwd == "" { + return "", "", false + } + if r, err := project.FindRoot(cwd); err == nil && r != "" { + m, err := project.Load(r) + n := filepath.Base(r) + if err == nil && strings.TrimSpace(m.Name) != "" { + n = m.Name + } + return n, r, true + } + if hasCanonicalFolders(cwd) { + return filepath.Base(cwd), cwd, true + } + return "", cwd, false +} + +func hasCanonicalFolders(dir string) bool { + if project.HasManifest(dir) { + return true + } + for _, n := range canonicalFolders { + if _, err := os.Stat(filepath.Join(dir, n)); err == nil { + return true + } + } + return false +} + +func countCanonicalFiles(root string) map[string]int { + out := map[string]int{} + for _, folder := range canonicalFolders { + out[folder] = scanCanonicalFolder(root, folder) + } + return out +} + +func scanCanonicalFolder(root, folder string) int { + dir := filepath.Join(root, folder) + n := 0 + _ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error { + if err != nil || d.IsDir() { + return nil + } + ext := strings.ToLower(filepath.Ext(p)) + if ext != ".yaml" && ext != ".yml" && ext != ".sql" { + return nil + } + body, rerr := os.ReadFile(p) + if rerr != nil { + return nil + } + if validate.DetectFileKind(body, p) != validate.KindUnknown { + n++ + } + return nil + }) + return n +} + +func recentRunsForPrompt() []LocalExec { + execs, err := ListLocalExecs() + if err != nil || len(execs) == 0 { + return nil + } + sort.SliceStable(execs, func(i, j int) bool { + iFail := execs[i].Status == "err" + jFail := execs[j].Status == "err" + if iFail != jFail { + return iFail + } + return execs[i].When.After(execs[j].When) + }) + if len(execs) > 3 { + execs = execs[:3] + } + return execs +} + +func probeMCPWired() bool { + report, err := Doctor(context.Background()) + if err != nil || report == nil || report.Matrix == nil { + return false + } + for _, row := range report.Matrix.Rows { + if row.Label != "MCP" { + continue + } + for _, st := range row.Cells { + if st == CellOK { + return true + } + } + } + return false +} + +// maxRunLogLines caps the run-log tail that enters the prompt. +const maxRunLogLines = 40 + +// runLogExcerpt returns the tail of the captured run log for this exec. +// Empty when nothing was captured or it only mirrors error.txt. +func runLogExcerpt(le LocalExec) string { + b, err := os.ReadFile(filepath.Join(le.LogDir, "stderr.log")) + if err != nil { + return "" + } + errB, _ := os.ReadFile(filepath.Join(le.LogDir, "error.txt")) + if string(b) == string(errB) { + return "" // pre-capture snapshot: stderr.log duplicates error.txt + } + return capExcerptLines(sanitizeLogForPrompt(string(b), 0), maxRunLogLines) +} + +func failureDetails(le LocalExec) (sig string, lookup *ErrorLookupResult, excerpt string) { + metaPath := filepath.Join(le.LogDir, "meta.json") + if b, err := os.ReadFile(metaPath); err == nil { + doc := map[string]any{} + if json.Unmarshal(b, &doc) == nil { + sig = cast.ToString(doc["error_signature"]) + if sig == "" { + sig = cast.ToString(doc["error_pattern_id"]) + } + } + } + if sig != "" { + if r, err := LookupError(sig); err == nil { + lookup = &r + } + } + errPath := filepath.Join(le.LogDir, "error.txt") + if b, err := os.ReadFile(errPath); err == nil { + excerpt = capExcerptLines(sanitizeLogForPrompt(string(b), 0), maxErrorExcerptLines) + } + return sig, lookup, excerpt +} + +func capExcerptLines(s string, n int) string { + s = strings.TrimRight(s, "\n") + lines := strings.Split(s, "\n") + if len(lines) <= n { + return s + } + return strings.Join(lines[len(lines)-n:], "\n") +} + +// Render fills the five-section skeleton. Caps at ~maxPromptTokens. +func (p PromptContext) Render() (string, error) { + bundle, err := loadPromptBundle() + if err != nil { + return "", err + } + p = p.shrinkContextForBudget() + view := promptView{ + Rules: strings.TrimSpace(bundle.Rules), + State: p.renderState(), + Context: p.renderContext(), + Ask: p.renderAsk(), + Objective: p.renderObjective(bundle), + } + tmpl, err := template.New("skeleton").Parse(bundle.Skeleton) + if err != nil { + return "", err + } + var buf bytes.Buffer + if err := tmpl.Execute(&buf, view); err != nil { + return "", err + } + out := strings.TrimRight(buf.String(), "\n") + "\n" + return capPromptTokens(out), nil +} + +// RenderPrompt is the exported wrapper for (PromptContext).Render. +func RenderPrompt(p PromptContext) (string, error) { + return p.Render() +} + +func (p PromptContext) renderState() string { + var b strings.Builder + fmt.Fprintf(&b, "- Sling version: %s\n", p.Version) + fmt.Fprintf(&b, "- cwd: %s\n", p.Cwd) + switch { + case p.HasProject && p.ProjectName != "": + fmt.Fprintf(&b, "- project: %s (%s)\n", p.ProjectName, p.ProjectRoot) + case p.HasProject: + fmt.Fprintf(&b, "- project: %s\n", p.ProjectRoot) + default: + b.WriteString("- project: (none in cwd)\n") + } + if p.MCPWired { + b.WriteString("- MCP wired: yes\n") + } else { + b.WriteString("- MCP wired: no\n") + } + if p.Agent != "" { + fmt.Fprintf(&b, "- preferred agent: %s\n", p.Agent) + } + return strings.TrimRight(b.String(), "\n") +} + +func (p PromptContext) renderContext() string { + var b strings.Builder + b.WriteString("- connections: ") + b.WriteString(formatProbeConnections(p.Connections, p.ConnectionExtra)) + b.WriteByte('\n') + b.WriteString("- files: ") + b.WriteString(formatFileCounts(p.FileCounts)) + b.WriteByte('\n') + b.WriteString("- recent runs: ") + b.WriteString(formatRecentRuns(p.RecentRuns)) + b.WriteByte('\n') + if p.PlatformExec != nil { + pe := p.PlatformExec + fmt.Fprintf(&b, "- platform exec: %s [%s] type=%s job=%s file=%s host=%s\n", + pe.ExecID, pe.Status, pe.Type, pe.JobName, pe.FileName, pe.HostLabel) + if strings.TrimSpace(pe.ErrSummary) != "" { + fmt.Fprintf(&b, "- platform error: %s\n", pe.ErrSummary) + } + } + if p.Signature != "" { + fmt.Fprintf(&b, "- error signature: %s\n", p.Signature) + if p.Lookup != nil && p.Lookup.Title != "" { + fmt.Fprintf(&b, "- error lookup: %s (%s)\n", p.Lookup.Title, p.Lookup.Status) + } + } + if strings.TrimSpace(p.ErrorExcerpt) != "" { + b.WriteString("- error excerpt:\n") + for _, line := range strings.Split(p.ErrorExcerpt, "\n") { + fmt.Fprintf(&b, " %s\n", line) + } + } + if strings.TrimSpace(p.RunLogExcerpt) != "" { + b.WriteString("- run log (tail):\n") + for _, line := range strings.Split(p.RunLogExcerpt, "\n") { + fmt.Fprintf(&b, " %s\n", line) + } + } + if strings.TrimSpace(p.Ask) == "" && len(p.Suggestions) > 0 { + b.WriteString("- suggestions: ") + b.WriteString(strings.Join(p.Suggestions, "; ")) + b.WriteByte('\n') + } + return strings.TrimRight(b.String(), "\n") +} + +func formatProbeConnections(conns []ProbeConn, extra int) string { + if len(conns) == 0 { + return "(none — run `sling conns set --type ` with ${VAR} refs; never ask for credentials in chat)" + } + parts := make([]string, 0, len(conns)) + for _, c := range conns { + if c.Type != "" { + parts = append(parts, fmt.Sprintf("%s (%s)", c.Name, c.Type)) + } else { + parts = append(parts, c.Name) + } + } + s := strings.Join(parts, ", ") + if extra > 0 { + s += fmt.Sprintf(" and %d more — run `sling conns list`", extra) + } + return s +} + +func formatFileCounts(counts map[string]int) string { + if len(counts) == 0 { + return "(none)" + } + parts := make([]string, 0, len(canonicalFolders)) + total := 0 + for _, k := range canonicalFolders { + n := counts[k] + total += n + parts = append(parts, fmt.Sprintf("%s %d", k, n)) + } + if total == 0 { + return "(none)" + } + return strings.Join(parts, ", ") +} + +func formatRecentRuns(runs []LocalExec) string { + if len(runs) == 0 { + return "(none)" + } + parts := make([]string, 0, len(runs)) + for _, r := range runs { + label := r.ID + if obj := r.displayObject(); obj != "" { + label = r.ID + " " + obj + } + parts = append(parts, fmt.Sprintf("%s [%s]", label, r.Status)) + } + return strings.Join(parts, "; ") +} + +func (p PromptContext) renderAsk() string { + if strings.TrimSpace(p.Ask) != "" { + return p.Ask + } + if p.PlatformExec != nil { + return fmt.Sprintf("Investigate platform execution %s (%s)", p.PlatformExec.ExecID, p.PlatformExec.Status) + } + if p.TargetExec != nil { + return failedRunLabel(*p.TargetExec) + } + return "(none — ask the user what they want; the suggestions are in Context)" +} + +func (p PromptContext) renderObjective(bundle promptBundle) string { + return strings.TrimSpace(bundle.Objective) +} + +func estimateTokens(s string) int { + n := len([]rune(s)) + return (n + 3) / 4 +} + +const truncNote = "[...truncated to ~1500 tokens...]" + +func (p PromptContext) shrinkContextForBudget() PromptContext { + p.ErrorExcerpt = capExcerptLines(p.ErrorExcerpt, maxErrorExcerptLines) + p.RunLogExcerpt = capExcerptLines(p.RunLogExcerpt, maxRunLogLines) + for n := maxErrorExcerptLines; n > 3 && estimateTokens(p.renderContext()) > maxPromptTokens/2; n -= 4 { + p.ErrorExcerpt = capExcerptLines(p.ErrorExcerpt, n) + p.RunLogExcerpt = capExcerptLines(p.RunLogExcerpt, n) + } + if estimateTokens(p.renderContext()) > maxPromptTokens/2 && len(p.Connections) > 3 { + extra := len(p.Connections) - 3 + p.Connections = p.Connections[:3] + p.ConnectionExtra += extra + } + if estimateTokens(p.renderContext()) > maxPromptTokens/2 && len(p.Suggestions) > 1 { + p.Suggestions = p.Suggestions[len(p.Suggestions)-1:] + } + return p +} + +func capPromptTokens(s string) string { + if estimateTokens(s) <= maxPromptTokens { + return s + } + const ctxH = "# Context\n" + const askH = "\n# Ask\n" + ctxAt := strings.Index(s, ctxH) + askAt := strings.Index(s, askH) + if ctxAt < 0 || askAt <= ctxAt { + return s + } + prefix := s[:ctxAt+len(ctxH)] + ctxBody := s[ctxAt+len(ctxH) : askAt] + suffix := s[askAt:] + budget := maxPromptTokens * 4 + keep := budget - len([]rune(prefix)) - len([]rune(suffix)) - len([]rune(truncNote)) - 2 + if keep < 80 { + return prefix + truncNote + "\n" + suffix + } + runes := []rune(ctxBody) + if len(runes) <= keep { + return s + } + cut := keep + for cut > 0 && runes[cut-1] != '\n' { + cut-- + } + if cut < keep/2 { + cut = keep + } + return prefix + string(runes[:cut]) + "\n" + truncNote + "\n" + suffix +} + +// LandingKind is the bare-`sling` TTY card. +type LandingKind string + +const ( + LandingFresh LandingKind = "fresh" + LandingNoProject LandingKind = "no_project" + LandingProject LandingKind = "project" +) + +// IsFreshInstall is true when env.yaml is the seeded default and there is +// no assist history (sessions or failure snapshots). Does not create dirs. +func IsFreshInstall() bool { + home := slingHome() + // Any session or error snapshot under ~/.sling/assist means not fresh. + if home != "" { + for _, rel := range []string{ + filepath.Join("assist", "history"), + filepath.Join("assist", "errors"), + } { + entries, err := os.ReadDir(filepath.Join(home, rel)) + if err != nil { + continue + } + for _, e := range entries { + if e.IsDir() && !strings.HasPrefix(e.Name(), ".") { + return false + } + } + } + } + path := envFilePath() + // Missing env.yaml is fresh. A user connection or assist profile is not. + if path == "" { + return true + } + if _, err := os.Stat(path); err != nil { + return true + } + ef := env.LoadEnvFile(path) + if len(ef.Connections) > 0 { + return false + } + if _, ok := ef.Env[assistEnvKey]; ok { + return false + } + for _, v := range ef.Env { + if v == nil { + continue + } + if strings.TrimSpace(fmt.Sprint(v)) != "" { + return false + } + } + return true +} + +// ClassifyLanding picks the card. Fresh wins so a default home never +// falls through to the probe states. +func ClassifyLanding(p PromptContext, fresh bool) LandingKind { + if fresh { + return LandingFresh + } + if p.HasProject { + return LandingProject + } + return LandingNoProject +} + +// SuggestedCommand is the wave-7 ladder mapped to one runnable command. +func (p PromptContext) SuggestedCommand() string { + switch Route(p) { + case "failed_run": + if sig := strings.TrimSpace(p.Signature); sig != "" { + return "sling assist error " + sig + } + return "sling assist" + case "zero_connections": + return "sling assist" + case "no_project": + return "sling init" + default: + return "sling assist" + } +} + +// RenderLanding prints the TTY card for one probe state. width 0 → 80. +func RenderLanding(kind LandingKind, p PromptContext, width int) string { + if width <= 0 { + width = 80 + } + var body string + switch kind { + case LandingFresh: + // All three steps are `sling assist` so the card never points at `conns set`. + body = `Welcome to sling. Three steps to your first data flow: + + 1. Set up your agent sling assist + 2. Add a connection sling assist (or edit ~/.sling/env.yaml) + 3. Move some data sling assist + +Docs: https://docs.slingdata.io +` + case LandingNoProject: + // Conn count plus a pointer at `sling init`. + n := userConnectionCount(p.Connections) + noun := "connections" + if n == 1 { + noun = "connection" + } + body = fmt.Sprintf("%d %s configured (`sling conns list`). No project here — `sling init` scaffolds one.\n", n, noun) + default: + // Project card: name, linked?, file counts, last run, next command. + name := strings.TrimSpace(p.ProjectName) + if name == "" { + if p.ProjectRoot != "" { + name = filepath.Base(p.ProjectRoot) + } else { + name = "(unnamed)" + } + } + linked := "not linked" + if strings.TrimSpace(p.ProjectRoot) != "" { + // Linked when sling_project.yml (or .sling.json) has a project id. + if m, err := project.Load(p.ProjectRoot); err == nil && m.Linked() { + linked = "linked" + } + } + var b strings.Builder + fmt.Fprintf(&b, "On project %s (%s)\n", name, linked) + fmt.Fprintf(&b, " files: %s\n", formatFileCounts(p.FileCounts)) + if run, ok := latestRun(p.RecentRuns); ok { + label := run.ID + if obj := run.displayObject(); obj != "" { + label += " " + obj + } + s := fmt.Sprintf("%s [%s]", label, run.Status) + if !run.When.IsZero() { + rt := relTime(run.When) + if rt == "just now" { + s += " just now" + } else { + s += " " + rt + " ago" + } + } + fmt.Fprintf(&b, " last run: %s\n", s) + } else { + b.WriteString(" last run: none\n") + } + fmt.Fprintf(&b, "\n Next: %s\n", p.SuggestedCommand()) + body = b.String() + } + return wrapToWidth(strings.TrimRight(body, "\n")+"\n", width) +} + +// relTime is a short age phrase: "just now", "3m", "2h", "1d". +func relTime(t time.Time) string { + d := time.Since(t) + if d < 0 { + d = 0 + } + switch { + case d < time.Minute: + return "just now" + case d < time.Hour: + return fmt.Sprintf("%dm", int(d.Minutes())) + case d < 24*time.Hour: + return fmt.Sprintf("%dh", int(d.Hours())) + default: + return fmt.Sprintf("%dd", int(d.Hours()/24)) + } +} + +func latestRun(runs []LocalExec) (LocalExec, bool) { + var best LocalExec + ok := false + for _, r := range runs { + if !ok || r.When.After(best.When) { + best, ok = r, true + } + } + return best, ok +} + +func wrapToWidth(s string, width int) string { + if width <= 0 { + width = 80 + } + // Keep `backtick commands` as one token so a wrapped card stays copyable. + leadingSpaces := func(line string) string { + i := 0 + for i < len(line) && line[i] == ' ' { + i++ + } + return line[:i] + } + wrapTokens := func(in string) []string { + var words []string + var cur strings.Builder + inTick := false + for _, r := range in { + switch { + case r == '`': + inTick = !inTick + cur.WriteRune(r) + case r == ' ' && !inTick: + if cur.Len() > 0 { + words = append(words, cur.String()) + cur.Reset() + } + default: + cur.WriteRune(r) + } + } + if cur.Len() > 0 { + words = append(words, cur.String()) + } + return words + } + chunkRunes := func(in string, w int) []string { + runes := []rune(in) + var out []string + for len(runes) > w { + out = append(out, string(runes[:w])) + runes = runes[w:] + } + if len(runes) > 0 { + out = append(out, string(runes)) + } + return out + } + wrapLine := func(line string) []string { + if utf8.RuneCountInString(line) <= width { + return []string{line} + } + indent := leadingSpaces(line) + body := strings.TrimLeft(line, " ") + avail := width - len(indent) + if avail < 20 { + avail = 20 + indent = "" + } + words := wrapTokens(body) + if len(words) == 0 { + return []string{line} + } + var lines []string + cur := indent + words[0] + for _, w := range words[1:] { + trial := cur + " " + w + if utf8.RuneCountInString(trial) <= width { + cur = trial + continue + } + lines = append(lines, cur) + cur = indent + w + if utf8.RuneCountInString(cur) > width { + lines = append(lines, chunkRunes(cur, width)...) + cur = indent + } + } + if strings.TrimSpace(cur) != "" { + lines = append(lines, cur) + } + return lines + } + + lines := strings.Split(strings.TrimRight(s, "\n"), "\n") + out := make([]string, 0, len(lines)) + for _, line := range lines { + out = append(out, wrapLine(line)...) + } + return strings.Join(out, "\n") + "\n" +} diff --git a/core/sling/assist/prompt_test.go b/core/sling/assist/prompt_test.go new file mode 100644 index 000000000..cab47ba4c --- /dev/null +++ b/core/sling/assist/prompt_test.go @@ -0,0 +1,1036 @@ +package assist + +import ( + "bytes" + "io/fs" + "os" + "path/filepath" + "strings" + "testing" + "time" + "unicode/utf8" + + "github.com/slingdata-io/sling-cli/core/dbio/iop" +) + +func sampleLookup() *ErrorLookupResult { + r, err := LookupError("97d84811") + if err != nil { + return &ErrorLookupResult{Signature: "97d84811", Status: "unknown", Title: "No published guidance yet"} + } + return &r +} + +func snapshotCtx(route string) PromptContext { + base := PromptContext{ + Version: "dev", + Cwd: "/work", + ProjectName: "demo", + ProjectRoot: "/work", + HasProject: true, + FileCounts: map[string]int{"replications": 1, "pipelines": 0, "models": 0, "specs": 0}, + Connections: []ProbeConn{ + {Name: "MY_PG", Type: "PostgreSQL", Source: "sling env yaml"}, + {Name: "MY_SF", Type: "Snowflake", Source: "sling env yaml"}, + }, + MCPWired: true, + Agent: "claude", + Route: route, + } + switch route { + case "ask": + base.Ask = "backfill orders" + case "failed_run": + base.Signature = "97d84811" + base.Lookup = sampleLookup() + base.ErrorExcerpt = "column missing: email_verified" + base.RecentRuns = []LocalExec{{ID: "exec_fail1", Status: "err", ConfigPath: "./r.yaml"}} + case "zero_connections": + base.Connections = nil + base.HasProject = true + case "no_project": + base.HasProject = false + base.ProjectName = "" + base.ProjectRoot = "/tmp" + base.FileCounts = map[string]int{"replications": 0, "pipelines": 0, "models": 0, "specs": 0} + case "default": + base.Ask = "" + } + for _, s := range base.suggestions() { + base.Suggestions = append(base.Suggestions, s.Label) + } + return base +} + +func TestPrintSnapshots(t *testing.T) { + rungs := []string{"ask", "failed_run", "zero_connections", "no_project", "default"} + for _, rung := range rungs { + t.Run(rung, func(t *testing.T) { + got, err := RenderPrompt(snapshotCtx(rung)) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(got, "# Rules") || !strings.Contains(got, "# State") || !strings.Contains(got, "# Context") { + t.Fatalf("missing section headings:\n%s", got) + } + if strings.Contains(got, "super-secret") || strings.Contains(got, "password") { + t.Fatalf("prompt leaked a secret-shaped value:\n%s", got) + } + path := filepath.Join("testdata", "print_"+rung+".golden") + if os.Getenv("UPDATE_GOLDEN") == "1" { + if err := os.MkdirAll("testdata", 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(got), 0o644); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read golden %s: %v (set UPDATE_GOLDEN=1 to write)", path, err) + } + if got != string(want) { + t.Errorf("golden mismatch for %s\n--- got ---\n%s\n--- want ---\n%s", rung, got, want) + } + }) + } +} + +func TestRouteLadder(t *testing.T) { + plat := PromptContext{PlatformExec: &PlatformExec{ExecID: "x"}, Ask: "do it"} + if Route(plat) != "platform_failed_run" { + t.Fatalf("platform rung lost: %s", Route(plat)) + } + ask := PromptContext{Ask: "do it", RecentRuns: []LocalExec{{Status: "err"}}} + if Route(ask) != "ask" { + t.Fatalf("ask rung lost to failure: %s", Route(ask)) + } + fail := PromptContext{RecentRuns: []LocalExec{{Status: "err"}}} + if Route(fail) != "failed_run" { + t.Fatalf("got %s", Route(fail)) + } + zero := PromptContext{Connections: []ProbeConn{{Name: "LOCAL", Source: "built-in"}}, HasProject: true} + if Route(zero) != "zero_connections" { + t.Fatalf("got %s", Route(zero)) + } + noProj := PromptContext{Connections: []ProbeConn{{Name: "MY_PG", Source: "sling env yaml"}}} + if Route(noProj) != "no_project" { + t.Fatalf("got %s", Route(noProj)) + } + def := PromptContext{ + HasProject: true, + Connections: []ProbeConn{{Name: "MY_PG", Source: "sling env yaml"}}, + } + if Route(def) != "default" { + t.Fatalf("got %s", Route(def)) + } +} + +func TestProbeListsNamesNotSecrets(t *testing.T) { + dir := withTempHomeDir(t) + envPath := filepath.Join(dir, "env.yaml") + body := "connections:\n MY_PG:\n type: postgres\n url: postgresql://user:super-secret-pass@localhost/db\n" + if err := os.WriteFile(envPath, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + p := Probe(ProbeOptions{Ask: "sync users"}) + out, err := RenderPrompt(p) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, "# Rules") || !strings.Contains(out, "# State") || !strings.Contains(out, "# Context") { + t.Fatalf("missing headings:\n%s", out) + } + if !strings.Contains(out, "MY_PG") { + t.Fatalf("expected connection name MY_PG:\n%s", out) + } + if strings.Contains(out, "super-secret-pass") { + t.Fatalf("secret leaked into prompt:\n%s", out) + } + if strings.Contains(out, "postgresql://") { + t.Fatalf("connection URL leaked into prompt:\n%s", out) + } +} + +func TestFailureDetailsGatedByIntent(t *testing.T) { + withTempHomeDir(t) + WriteFailureSnapshot(FailureSnapshot{ + ExecID: "exec_gate1", + ErrMsg: "column missing: email_verified", + ConfigPath: "./r.yaml", + }) + + // A normal ask keeps the one-line run summary only. + p := Probe(ProbeOptions{Ask: "create a replication"}) + out, err := RenderPrompt(p) + if err != nil { + t.Fatal(err) + } + if p.Signature != "" || p.ErrorExcerpt != "" { + t.Fatalf("failure details leaked into ask mode: sig=%q excerpt=%q", p.Signature, p.ErrorExcerpt) + } + if strings.Contains(out, "error excerpt") || strings.Contains(out, "error signature") { + t.Fatalf("ask-mode prompt contains failure details:\n%s", out) + } + if !strings.Contains(out, "exec_gate1") || !strings.Contains(out, "[err]") { + t.Fatalf("ask-mode prompt lost the recent-runs summary:\n%s", out) + } + + // The investigate intent pulls the full details in. + p = Probe(ProbeOptions{Ask: "Investigate the failed run exec_gate1 ./r.yaml", IncludeFailure: true}) + out, err = RenderPrompt(p) + if err != nil { + t.Fatal(err) + } + if p.Signature == "" { + t.Fatal("investigate mode missing signature") + } + if !strings.Contains(out, "error excerpt") || !strings.Contains(out, "column missing: email_verified") { + t.Fatalf("investigate prompt missing failure details:\n%s", out) + } +} + +func TestPromptAssemblyDoesNotCallExpandEnvVars(t *testing.T) { + files := []string{"prompt.go", "session.go", "assist_platform..go"} + for _, name := range files { + b, err := os.ReadFile(name) + if err != nil { + // a rename must fail the guard, not skip it + t.Fatalf("cannot read %s: %s", name, err) + } + if bytes.Contains(b, []byte("ExpandEnvVars")) { + t.Errorf("%s must not call ExpandEnvVars", name) + } + if bytes.Contains(b, []byte(".Data")) && bytes.Contains(b, []byte("Connection")) { + t.Errorf("%s must not read connection payload fields", name) + } + } +} + +func TestConnectionCap(t *testing.T) { + all := make([]ProbeConn, 12) + for i := range all { + all[i] = ProbeConn{Name: "C" + strings.Repeat("X", 1), Type: "PostgreSQL"} + all[i].Name = "CONN_" + string(rune('A'+i)) + } + got, extra := capProbeConnections(all) + if len(got) != maxProbeConnections || extra != 2 { + t.Fatalf("len=%d extra=%d", len(got), extra) + } + s := formatProbeConnections(got, extra) + if !strings.Contains(s, "and 2 more — run `sling conns list`") { + t.Fatalf("cap suffix missing: %s", s) + } +} + +func TestErrorExcerptCaps15Lines(t *testing.T) { + var lines []string + for i := 0; i < 20; i++ { + lines = append(lines, "line") + } + got := capExcerptLines(strings.Join(lines, "\n"), maxErrorExcerptLines) + if n := strings.Count(got, "\n") + 1; n != 15 { + t.Fatalf("lines=%d want 15", n) + } +} + +func TestNestedLaunchPrintsInsteadOfSpawn(t *testing.T) { + withTempHomeDir(t) + t.Setenv("CLAUDECODE", "1") + buf := &bytes.Buffer{} + prev := assistOut + assistOut = buf + defer func() { assistOut = prev }() + + out, err := Session(SessionOptions{Ask: "backfill orders"}) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(buf.String(), "# Rules") { + t.Fatalf("stdout missing Rules:\n%s", buf.String()) + } + if !strings.Contains(out, "backfill orders") { + t.Fatalf("ask missing from prompt:\n%s", out) + } +} + +func TestAPISpecsSkillMentionsAgentBrowser(t *testing.T) { + b, err := SkillsFS.ReadFile("skills/sling-api-specs/SKILL.md") + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(b, []byte("agent-browser")) { + t.Fatal("sling-api-specs/SKILL.md must mention agent-browser") + } + if bytes.Contains(b, []byte("sling assist --browse")) { + t.Fatal("sling-api-specs/SKILL.md still mentions sling assist --browse") + } +} + +func TestPipelineSkillsTeachStateResult(t *testing.T) { + // Skills must teach the runtime shape, not the rejected one. + skill, err := SkillsFS.ReadFile("skills/sling-pipelines/SKILL.md") + if err != nil { + t.Fatal(err) + } + steps, err := SkillsFS.ReadFile("skills/sling-pipelines/STEPS.md") + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(skill, []byte("state.count_query.result[0]")) { + t.Fatal("SKILL.md must teach state..result[0]") + } + if bytes.Contains(skill, []byte("state.count_query[0]")) { + t.Fatal("SKILL.md still teaches the rejected state.[0] shape") + } + if !bytes.Contains(steps, []byte("state.my_query.result[0]")) { + t.Fatal("STEPS.md must teach state..result[0]") + } + if bytes.Contains(steps, []byte("state.my_query[0]")) { + t.Fatal("STEPS.md still teaches the rejected state.[0] shape") + } +} + +func TestSkillsExpressionsParse(t *testing.T) { + // Every {…} expression in the skill bundle must pass the real parser. + eval := iop.NewEvaluator([]string{ + "env", "state", "secrets", "auth", "response", "request", "sync", + "context", "record", "queue", "source", "target", "stream", "object", + "timestamp", "store", "execution", "loop", "run", + }) + var failed []string + err := fs.WalkDir(SkillsFS, "skills", func(path string, d fs.DirEntry, err error) error { + if err != nil || d.IsDir() { + return err + } + if !strings.HasSuffix(path, ".md") && !strings.HasSuffix(path, ".yaml") && !strings.HasSuffix(path, ".yml") { + return nil + } + body, err := SkillsFS.ReadFile(path) + if err != nil { + return err + } + text := string(body) + for _, span := range incorrectExampleSpans(text) { + text = text[:span[0]] + strings.Repeat(" ", span[1]-span[0]) + text[span[1]:] + } + for _, expr := range extractBraceExprs(text) { + if err := eval.Check(expr); err != nil { + failed = append(failed, path+": "+err.Error()) + } + } + return nil + }) + if err != nil { + t.Fatal(err) + } + if len(failed) > 0 { + t.Fatalf("skills teach expressions the runtime rejects:\n%s", strings.Join(failed, "\n")) + } +} + +func extractBraceExprs(s string) []string { + var out []string + for i := 0; i < len(s); i++ { + if s[i] != '{' { + continue + } + if i+1 < len(s) && (s[i+1] == '{' || s[i+1] == '#' || s[i+1] == '%') { + continue + } + depth := 1 + j := i + 1 + for j < len(s) && depth > 0 { + switch s[j] { + case '{': + depth++ + case '}': + depth-- + } + j++ + } + if depth != 0 { + continue + } + expr := strings.TrimSpace(s[i+1 : j-1]) + if !looksLikeRuntimeExpr(expr) { + i = j - 1 + continue + } + out = append(out, expr) + i = j - 1 + } + return out +} + +func incorrectExampleSpans(s string) [][2]int { + var out [][2]int + for _, marker := range []string{"# ❌", "❌ Incorrect", "Incorrect - single quotes"} { + i := 0 + for { + j := strings.Index(s[i:], marker) + if j < 0 { + break + } + start := i + j + end := strings.Index(s[start:], "\n```") + if end < 0 { + end = len(s) - start + } + out = append(out, [2]int{start, start + end}) + i = start + len(marker) + } + } + return out +} + +func looksLikeRuntimeExpr(expr string) bool { + expr = strings.TrimSpace(expr) + if expr == "" || strings.HasPrefix(expr, "#") || strings.HasPrefix(expr, "%") { + return false + } + // Python / JSON dicts in skill examples: {'a': 1}, {"id": 1} + if strings.HasPrefix(expr, "'") || (strings.HasPrefix(expr, `"`) && strings.Contains(expr, ":")) { + return false + } + if strings.ContainsAny(expr, "()") { + return true + } + if strings.Contains(expr, " + ") || strings.Contains(expr, " - ") || + strings.Contains(expr, " == ") || strings.Contains(expr, " != ") { + return true + } + return false +} + +func TestGatherFirstIntroDoNotAskWhenResolved(t *testing.T) { + names := []string{ + "sling-replications/SKILL.md", + "sling-pipelines/SKILL.md", + "sling-build/SKILL.md", + "sling-api-specs/SKILL.md", + "sling-connections/SKILL.md", + } + needle := []byte("If every row resolves, do not ask") + for _, name := range names { + b, err := SkillsFS.ReadFile("skills/" + name) + if err != nil { + t.Fatalf("%s: %v", name, err) + } + if !bytes.Contains(b, needle) { + t.Errorf("%s missing Gather first intro", name) + } + } +} + +func TestSuggestionPriorityFullStack(t *testing.T) { + p := PromptContext{ + HasProject: false, + Connections: []ProbeConn{{Name: "LOCAL", Source: "built-in"}}, + RecentRuns: []LocalExec{{ID: "exec1", Status: "err", ConfigPath: "./r.yaml"}}, + } + got := p.suggestions() + if len(got) != 4 { + t.Fatalf("len=%d want 4: %+v", len(got), got) + } + if !strings.Contains(got[0].Label, "Investigate the failed run") { + t.Fatalf("first: %s", got[0].Label) + } + if got[1].Label != "Add your first connection" { + t.Fatalf("second: %s", got[1].Label) + } + if !strings.Contains(got[2].Label, "Scaffold a project") { + t.Fatalf("third: %s", got[2].Label) + } + if got[3].Label != suggestionElseLabel { + t.Fatalf("last: %s", got[3].Label) + } + for _, s := range got { + if strings.Contains(s.Label, "Create or update") { + t.Fatal("filler must drop when three signals fill the cap") + } + } +} + +func TestSuggestionFillerWhenSlotsRemain(t *testing.T) { + p := PromptContext{ + HasProject: true, + Connections: []ProbeConn{{Name: "MY_PG", Source: "sling env yaml"}}, + } + got := p.suggestions() + if len(got) != 2 { + t.Fatalf("len=%d want 2: %+v", len(got), got) + } + if !strings.Contains(got[0].Label, "Create or update") { + t.Fatalf("filler first: %s", got[0].Label) + } + if got[1].Label != suggestionElseLabel { + t.Fatalf("else last: %s", got[1].Label) + } +} + +func TestAskModeObjectiveHasGatherFirst(t *testing.T) { + out, err := RenderPrompt(snapshotCtx("ask")) + if err != nil { + t.Fatal(err) + } + if strings.Contains(out, " 1. ") { + t.Fatalf("ask mode must not print a menu:\n%s", out) + } + if !strings.Contains(out, "Gather first") { + t.Fatalf("Objective missing Gather first:\n%s", out) + } + if !strings.Contains(out, "Load the matching Sling skill") { + t.Fatalf("Objective missing skill rule:\n%s", out) + } + if strings.Contains(out, "- suggestions:") { + t.Fatalf("ask mode must not list suggestions:\n%s", out) + } +} + +func TestEmptyAskFallbackSuggestionsLine(t *testing.T) { + out, err := RenderPrompt(snapshotCtx("default")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, "- suggestions:") { + t.Fatalf("empty-ask Context missing suggestions:\n%s", out) + } + if !strings.Contains(out, suggestionElseLabel) { + t.Fatalf("missing else row:\n%s", out) + } + if !strings.Contains(out, "ask the user what they want") { + t.Fatalf("missing empty-ask fallback:\n%s", out) + } +} + +func TestTruncationNeverCutsObjective(t *testing.T) { + excerpt := strings.Repeat("error line that is fairly long for the cap test\n", 80) + conns := make([]ProbeConn, 10) + for i := range conns { + conns[i] = ProbeConn{Name: "CONN_" + strings.Repeat("X", 40) + string(rune('A'+i)), Type: "PostgreSQL", Source: "sling env yaml"} + } + p := PromptContext{ + Version: "dev", + Cwd: "/work", + HasProject: true, + ProjectName: "demo", + ProjectRoot: "/work", + FileCounts: map[string]int{"replications": 1, "pipelines": 0, "models": 0, "specs": 0}, + Connections: conns, + ErrorExcerpt: excerpt, + Ask: "backfill orders", + Suggestions: []string{ + strings.Repeat("Investigate the failed run a-very-long-id ./r.yaml (3m ago)", 8), + suggestionElseLabel, + }, + } + out, err := RenderPrompt(p) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, "# Objective") { + t.Fatalf("Objective heading cut:\n%s", out) + } + if !strings.Contains(out, "Gather first") { + t.Fatalf("Objective body cut:\n%s", out) + } + if !strings.Contains(out, "Load the matching Sling skill") { + t.Fatalf("Objective skill rule cut:\n%s", out) + } +} + +func TestSessionPrintTwiceStable(t *testing.T) { + withTempHomeDir(t) + opts := SessionOptions{Ask: "backfill orders", Print: true} + buf1, buf2 := &bytes.Buffer{}, &bytes.Buffer{} + prev := assistOut + defer func() { assistOut = prev }() + assistOut = buf1 + a, err := Session(opts) + if err != nil { + t.Fatal(err) + } + assistOut = buf2 + b, err := Session(opts) + if err != nil { + t.Fatal(err) + } + if a != b || buf1.String() != buf2.String() { + t.Fatalf("print not stable\n---1---\n%s\n---2---\n%s", buf1.String(), buf2.String()) + } + for _, heading := range []string{"# Rules", "# State", "# Context"} { + if !strings.Contains(a, heading) { + t.Fatalf("missing %s", heading) + } + } +} + +func TestRelTime(t *testing.T) { + now := time.Now() + cases := []struct { + t time.Time + want string + }{ + {now.Add(-10 * time.Second), "just now"}, + {now.Add(-3 * time.Minute), "3m"}, + {now.Add(-2 * time.Hour), "2h"}, + {now.Add(-25 * time.Hour), "1d"}, + } + for _, tc := range cases { + if got := relTime(tc.t); got != tc.want { + t.Errorf("relTime(%v)=%q want %q", tc.t, got, tc.want) + } + } +} + +func TestRenderOpenCardFailedRunRow(t *testing.T) { + p := PromptContext{ + HasProject: true, + ProjectName: "demo", + Connections: []ProbeConn{{Name: "MY_PG", Source: "sling env yaml"}}, + RecentRuns: []LocalExec{{ + ID: "exec_fail1", + Status: "err", + ConfigPath: "./r.yaml", + When: time.Now().Add(-3 * time.Minute), + }}, + } + got := renderOpenCard(p, 80) + if !strings.Contains(got, "On project demo") { + t.Fatalf("missing project line:\n%s", got) + } + if !strings.Contains(got, "Investigate the failed run exec_fail1 ./r.yaml (3m ago)") { + t.Fatalf("missing failed-run row:\n%s", got) + } + if !strings.Contains(got, "3m ago") { + t.Fatalf("missing rel-time:\n%s", got) + } + if !strings.Contains(got, suggestionElseLabel) { + t.Fatalf("missing else row:\n%s", got) + } +} + +func TestRenderOpenCardZeroConnRow(t *testing.T) { + p := PromptContext{ + HasProject: true, + ProjectName: "demo", + Connections: []ProbeConn{{Name: "LOCAL", Source: "built-in"}}, + } + got := renderOpenCard(p, 80) + if !strings.Contains(got, "Add your first connection") { + t.Fatalf("missing zero-conn row:\n%s", got) + } + if !strings.Contains(got, "0 connections") { + t.Fatalf("missing conn count:\n%s", got) + } +} + +func TestResolveOpenPick(t *testing.T) { + p := PromptContext{ + HasProject: false, + Connections: []ProbeConn{{Name: "LOCAL", Source: "built-in"}}, + RecentRuns: []LocalExec{{ID: "exec1", Status: "err", ConfigPath: "./r.yaml"}}, + } + opts := p.suggestions() + var buf bytes.Buffer + ask, investigate, ok := readOpenAsk(strings.NewReader("1\n"), &buf, opts) + if !ok { + t.Fatal("pick 1 aborted") + } + if !strings.Contains(ask, "Investigate the failed run") { + t.Fatalf("ask=%q", ask) + } + if !investigate { + t.Fatal("investigate pick must set the investigate flag") + } + + ask, investigate, ok = readOpenAsk(strings.NewReader("2\n"), &buf, opts) + if !ok || ask != "Add a connection" || investigate { + t.Fatalf("pick 2: ok=%v ask=%q investigate=%v", ok, ask, investigate) + } + + ask, investigate, ok = readOpenAsk(strings.NewReader("3\n"), &buf, opts) + if !ok || ask != "Scaffold a Sling project in this folder" || investigate { + t.Fatalf("pick 3: ok=%v ask=%q investigate=%v", ok, ask, investigate) + } + + ask, investigate, ok = readOpenAsk(strings.NewReader("backfill orders\n"), &buf, opts) + if !ok || ask != "backfill orders" || investigate { + t.Fatalf("free text: ok=%v ask=%q investigate=%v", ok, ask, investigate) + } +} + +func TestReadOpenAskEmptyLineExit(t *testing.T) { + opts := []suggestion{{Label: suggestionElseLabel}} + var buf bytes.Buffer + ask, _, ok := readOpenAsk(strings.NewReader("\n\n"), &buf, opts) + if ok || ask != "" { + t.Fatalf("ok=%v ask=%q", ok, ask) + } + if !strings.Contains(buf.String(), `sling assist ""`) { + t.Fatalf("missing hint:\n%s", buf.String()) + } +} + +func TestSessionOpenScreenDoesNotLaunchBeforePick(t *testing.T) { + withTempHomeDir(t) + prevTTY, prevIn, prevOut := ttyCheck, assistIn, assistOut + t.Cleanup(func() { + ttyCheck = prevTTY + assistIn = prevIn + assistOut = prevOut + }) + ttyCheck = func(*os.File) bool { return true } + assistIn = strings.NewReader("\n\n") + buf := &bytes.Buffer{} + assistOut = buf + + id, err := Session(SessionOptions{}) + if err != nil { + t.Fatalf("empty abort should exit 0: %v", err) + } + if id != "" { + t.Fatalf("launched session %q", id) + } + if !strings.Contains(buf.String(), suggestionElseLabel) { + t.Fatalf("card missing:\n%s", buf.String()) + } + if !strings.Contains(buf.String(), `sling assist ""`) { + t.Fatalf("missing hint:\n%s", buf.String()) + } +} + +func TestSessionHeadlessNoAskErrors(t *testing.T) { + withTempHomeDir(t) + _, err := Session(SessionOptions{Headless: true}) + if err == nil { + t.Fatal("expected error") + } + if !strings.Contains(err.Error(), "no ask given") { + t.Fatalf("got %v", err) + } +} + +func TestSessionModeAskWrittenToMeta(t *testing.T) { + dir := withTempHomeDir(t) + if err := os.MkdirAll(dir+"/bin", 0o755); err != nil { + t.Fatal(err) + } + stub := dir + "/bin/claude" + if err := os.WriteFile(stub, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", dir+"/bin"+string(os.PathListSeparator)+os.Getenv("PATH")) + if err := SaveProfile(Profile{Agent: "claude"}); err != nil { + t.Fatal(err) + } + + id, err := Session(SessionOptions{Ask: "backfill orders", Headless: true, Agent: "claude"}) + if err != nil { + t.Fatal(err) + } + e, err := LoadEntry(id) + if err != nil { + t.Fatal(err) + } + if e.Meta.Task != modeAsk { + t.Fatalf("Meta.Task=%q want %q", e.Meta.Task, modeAsk) + } + if e.Answers.Task != modeAsk { + t.Fatalf("Answers.Task=%q want %q", e.Answers.Task, modeAsk) + } +} + +func TestSessionOpenModeWrittenToMeta(t *testing.T) { + dir := withTempHomeDir(t) + if err := os.MkdirAll(dir+"/bin", 0o755); err != nil { + t.Fatal(err) + } + stub := dir + "/bin/claude" + if err := os.WriteFile(stub, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", dir+"/bin"+string(os.PathListSeparator)+os.Getenv("PATH")) + if err := SaveProfile(Profile{Agent: "claude"}); err != nil { + t.Fatal(err) + } + + prevTTY, prevIn, prevOut := ttyCheck, assistIn, assistOut + t.Cleanup(func() { + ttyCheck = prevTTY + assistIn = prevIn + assistOut = prevOut + }) + ttyCheck = func(*os.File) bool { return true } + assistIn = strings.NewReader("Add a connection\n") + assistOut = &bytes.Buffer{} + + id, err := Session(SessionOptions{Agent: "claude"}) + if err != nil { + t.Fatal(err) + } + e, err := LoadEntry(id) + if err != nil { + t.Fatal(err) + } + if e.Meta.Task != modeOpen { + t.Fatalf("Meta.Task=%q want %q", e.Meta.Task, modeOpen) + } + if e.Answers.Task != modeOpen { + t.Fatalf("Answers.Task=%q want %q", e.Answers.Task, modeOpen) + } +} + +func TestRunOpenScreenUsesReaderWhenStdinSwapped(t *testing.T) { + prev := assistIn + t.Cleanup(func() { assistIn = prev }) + assistIn = strings.NewReader("Add a connection\n") + + p := PromptContext{Connections: []ProbeConn{{Name: "LOCAL", Source: "built-in"}}} + prevOut := assistOut + t.Cleanup(func() { assistOut = prevOut }) + buf := &bytes.Buffer{} + assistOut = buf + + ask, investigate, ok := runOpenScreen(p) + if !ok || ask != "Add a connection" || investigate { + t.Fatalf("ok=%v ask=%q investigate=%v", ok, ask, investigate) + } + if !strings.Contains(buf.String(), suggestionElseLabel) { + t.Fatalf("numbered card missing:\n%s", buf.String()) + } +} + +func TestPickOpenAskNoOptions(t *testing.T) { + if ask, _, ok := pickOpenAsk(nil); ok || ask != "" { + t.Fatalf("ok=%v ask=%q", ok, ask) + } +} + +func landingCtx(kind LandingKind) PromptContext { + base := PromptContext{ + Version: "dev", + Cwd: "/work", + ProjectName: "demo", + ProjectRoot: "/work", + HasProject: true, + FileCounts: map[string]int{"replications": 2, "pipelines": 1, "models": 3, "specs": 0}, + Connections: []ProbeConn{ + {Name: "MY_PG", Type: "PostgreSQL", Source: "sling env yaml"}, + {Name: "MY_SF", Type: "Snowflake", Source: "sling env yaml"}, + {Name: "LOCAL", Type: "Local File System", Source: "built-in"}, + }, + Route: "default", + } + switch kind { + case LandingFresh: + base.HasProject = false + base.ProjectName = "" + base.ProjectRoot = "" + base.Connections = nil + base.FileCounts = map[string]int{"replications": 0, "pipelines": 0, "models": 0, "specs": 0} + case LandingNoProject: + base.HasProject = false + base.ProjectName = "" + base.ProjectRoot = "/tmp" + base.Route = "no_project" + base.FileCounts = map[string]int{"replications": 0, "pipelines": 0, "models": 0, "specs": 0} + case LandingProject: + base.RecentRuns = []LocalExec{ + {ID: "exec_old", Status: "err", ConfigPath: "./old.yaml", When: time.Now().Add(-48 * time.Hour)}, + {ID: "exec_new", Status: "ok", ConfigPath: "./r.yaml", When: time.Now().Add(-2 * time.Hour)}, + } + } + return base +} + +func TestClassifyLanding(t *testing.T) { + if got := ClassifyLanding(PromptContext{HasProject: true}, true); got != LandingFresh { + t.Fatalf("fresh wins over project: %s", got) + } + if got := ClassifyLanding(PromptContext{HasProject: true}, false); got != LandingProject { + t.Fatalf("got %s", got) + } + if got := ClassifyLanding(PromptContext{}, false); got != LandingNoProject { + t.Fatalf("got %s", got) + } +} + +func TestSuggestedCommandLadder(t *testing.T) { + fail := PromptContext{RecentRuns: []LocalExec{{Status: "err"}}, Signature: "97d84811"} + if got := fail.SuggestedCommand(); got != "sling assist error 97d84811" { + t.Fatalf("got %q", got) + } + failNoSig := PromptContext{RecentRuns: []LocalExec{{Status: "err"}}} + if got := failNoSig.SuggestedCommand(); got != "sling assist" { + t.Fatalf("got %q", got) + } + zero := PromptContext{Connections: []ProbeConn{{Name: "LOCAL", Source: "built-in"}}, HasProject: true} + if got := zero.SuggestedCommand(); got != "sling assist" { + t.Fatalf("got %q", got) + } + noProj := PromptContext{Connections: []ProbeConn{{Name: "MY_PG", Source: "sling env yaml"}}} + if got := noProj.SuggestedCommand(); got != "sling init" { + t.Fatalf("got %q", got) + } + def := PromptContext{HasProject: true, Connections: []ProbeConn{{Name: "MY_PG", Source: "sling env yaml"}}} + if got := def.SuggestedCommand(); got != "sling assist" { + t.Fatalf("got %q", got) + } +} + +func TestRenderLandingThreeStates(t *testing.T) { + cases := []struct { + kind LandingKind + want []string + skip []string + }{ + { + kind: LandingFresh, + want: []string{ + "Welcome to sling", + "sling assist", + "https://docs.slingdata.io", + }, + skip: []string{"conns set", "sling init"}, + }, + { + kind: LandingNoProject, + want: []string{ + "2 connections configured", + "sling conns list", + "sling init", + }, + skip: []string{"conns set", "Welcome to sling"}, + }, + { + kind: LandingProject, + want: []string{ + "On project demo", + "not linked", + "replications 2", + "pipelines 1", + "models 3", + "last run: exec_new ./r.yaml [ok]", + "Next: sling assist", + }, + skip: []string{"conns set", "Welcome to sling"}, + }, + } + for _, tc := range cases { + t.Run(string(tc.kind), func(t *testing.T) { + got := RenderLanding(tc.kind, landingCtx(tc.kind), 80) + for _, w := range tc.want { + if !strings.Contains(got, w) { + t.Errorf("missing %q\n%s", w, got) + } + } + for _, s := range tc.skip { + if strings.Contains(got, s) { + t.Errorf("must not contain %q\n%s", s, got) + } + } + if n := strings.Count(got, "conns set"); n != 0 { + t.Errorf("conns set leaked") + } + }) + } +} + +func TestRenderLandingFailedRunSuggestsError(t *testing.T) { + p := landingCtx(LandingProject) + p.RecentRuns = []LocalExec{{ID: "exec_fail", Status: "err", ConfigPath: "./r.yaml", When: time.Now()}} + p.Signature = "97d84811" + p.Route = "failed_run" + got := RenderLanding(LandingProject, p, 80) + if !strings.Contains(got, "Next: sling assist error 97d84811") { + t.Fatalf("missing error suggestion:\n%s", got) + } + if !strings.Contains(got, "last run: exec_fail ./r.yaml [err]") { + t.Fatalf("missing last run:\n%s", got) + } +} + +func TestRenderLandingZeroConnectionsNoProject(t *testing.T) { + p := PromptContext{ + Connections: []ProbeConn{{Name: "LOCAL", Source: "built-in"}}, + } + got := RenderLanding(LandingNoProject, p, 80) + if !strings.Contains(got, "0 connections configured") { + t.Fatalf("got:\n%s", got) + } + if !strings.Contains(got, "sling init") { + t.Fatalf("missing project init:\n%s", got) + } +} + +func TestRenderLandingRespectsWidth(t *testing.T) { + got := RenderLanding(LandingNoProject, landingCtx(LandingNoProject), 40) + for i, line := range strings.Split(strings.TrimRight(got, "\n"), "\n") { + if n := utf8.RuneCountInString(line); n > 40 { + t.Errorf("line %d len %d > 40: %q", i, n, line) + } + } +} + +func TestFreshLandingPointsOnlyAtAssist(t *testing.T) { + got := RenderLanding(LandingFresh, PromptContext{}, 80) + if strings.Count(got, "sling assist") < 3 { + t.Fatalf("expected 3 assist pointers:\n%s", got) + } + if strings.Contains(got, "conns set") { + t.Fatalf("conns set leaked:\n%s", got) + } +} + +func TestIsFreshInstall(t *testing.T) { + dir := withTempHomeDir(t) + + if !IsFreshInstall() { + t.Fatal("empty home should be fresh") + } + + envPath := filepath.Join(dir, "env.yaml") + body := "# Environment Credentials for Sling CLI\n# See https://docs.slingdata.io/sling-cli/environment\n\nconnections:\n\n\nvariables:\n" + if err := os.WriteFile(envPath, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + if !IsFreshInstall() { + t.Fatal("seeded default env.yaml should be fresh") + } + + if err := os.WriteFile(envPath, []byte("connections:\n MY_PG:\n type: postgres\n"), 0o644); err != nil { + t.Fatal(err) + } + if IsFreshInstall() { + t.Fatal("user connection should not be fresh") + } + + if err := os.WriteFile(envPath, []byte("connections:\n\nenv:\n SLING_ASSIST:\n agent: claude\n"), 0o644); err != nil { + t.Fatal(err) + } + if IsFreshInstall() { + t.Fatal("assist profile should not be fresh") + } + + if err := os.WriteFile(envPath, []byte("connections:\n"), 0o644); err != nil { + t.Fatal(err) + } + hist := filepath.Join(dir, "assist", "history", "sess1") + if err := os.MkdirAll(hist, 0o755); err != nil { + t.Fatal(err) + } + if IsFreshInstall() { + t.Fatal("assist history should not be fresh") + } +} + +func TestLatestRunPicksNewestNotFailure(t *testing.T) { + oldFail := LocalExec{ID: "old", Status: "err", When: time.Now().Add(-time.Hour)} + newOK := LocalExec{ID: "new", Status: "ok", When: time.Now()} + got, ok := latestRun([]LocalExec{oldFail, newOK}) + if !ok || got.ID != "new" { + t.Fatalf("got %+v ok=%v", got, ok) + } +} diff --git a/core/sling/assist/prompts.yaml b/core/sling/assist/prompts.yaml new file mode 100644 index 000000000..6f07992b8 --- /dev/null +++ b/core/sling/assist/prompts.yaml @@ -0,0 +1,44 @@ +# Embedded prompt templates for `sling assist`. +# +# `_skeleton` is the five-section agent prompt (`sling assist --out -`). +# `_objective` is the single ask-mode Objective. +# +# Top-level keys: +# _skeleton — Rules / State / Context / Ask / Objective +# _rules — shared Rules body +# _objective — ask-mode Objective + +_skeleton: |- + # Rules + {{.Rules}} + + # State + {{.State}} + + # Context + {{.Context}} + + # Ask + {{.Ask}} + + # Objective + {{.Objective}} + +_rules: |- + - You help with Sling CLI (https://docs.slingdata.io/llms.txt). + - Use Sling MCP tools when they are wired (connection, database, replication, pipeline, api_spec, file_system). + - Validate YAML with MCP `validate`. Execute replications and pipelines with `sling run`. Execute SQL models with `sling build`. There is no MCP `run` or `build` action. + - Do not invent connection names, streams, or modes. Ask one focused question instead. + - Never ask for credentials in chat. Use ${VAR} refs and `sling conns set`. + - Do not read env.yaml or print secrets. Connection names and types only. + - Validate YAML with the validate tools before you declare done. + - Before you create or change a Sling file, load the matching skill and resolve its 'Gather first' checklist. Use the request, files, and MCP discovery first; ask the user only for what stays unresolved. Do not ask when the answers are already in context. Propose defaults. + - Do not start work the user did not request. + - The Ask is your only goal. Context is background — do not act on it unless the Ask points there. + - When you debug a failed run, read `sling/TROUBLESHOOTING.md`. Re-run only after the user accepts the fix. + - After you classify a failure as a sling defect (not user config), offer: "Report this? GitHub issue (public, needs account) or email to support." Never offer this for user-config errors. Never auto-send. There is no MCP report tool. Run `sling assist report --id ` so the user reviews the draft, then `--github` or `--email`. + - When Context shows a platform exec, load the sling-platform skill. Get state with `sling platform execs status `, then logs with `sling platform execs log --status error`. Pull the config with `sling platform files get `. These commands need SLING_PROJECT_TOKEN, which is already set. + +_objective: |- + Follow the Ask. Load the matching Sling skill (replications, pipelines, build, api-specs, connections, or project). + Complete its "Gather first" checklist before you write files. diff --git a/core/sling/assist/report.go b/core/sling/assist/report.go new file mode 100644 index 000000000..9b85cd22d --- /dev/null +++ b/core/sling/assist/report.go @@ -0,0 +1,574 @@ +package assist + +import ( + "encoding/base64" + "encoding/json" + "fmt" + "net/url" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + + "github.com/charmbracelet/huh" + "github.com/flarco/g" + "github.com/slingdata-io/sling-cli/core" + "github.com/slingdata-io/sling-cli/core/dbio" + "github.com/slingdata-io/sling-cli/core/env" + "github.com/spf13/cast" +) + +const ( + githubIssueURLMax = 8000 + // Cloudflare rejects request lines over ~16KB. Stay well under it. + contactFormURLMax = 12000 + githubIssueBaseURL = "https://github.com/slingdata-io/sling-cli/issues/new" + contactFormBaseURL = "https://slingdata.io/contact/" + maxLogExcerptBytes = 64 * 1024 + maxSkeletonBytes = 4 * 1024 +) + +// ReportDraft is the composed, redacted report. Both routes consume it. +type ReportDraft struct { + Title string `json:"title"` + Description string `json:"description"` + Config string `json:"config"` + LogExcerpt string `json:"log_excerpt"` + Version string `json:"version"` + OS string `json:"os"` + SourceType string `json:"source_type"` + TargetType string `json:"target_type"` + SignatureID string `json:"signature_id"` + Skeleton string `json:"skeleton"` + ExecID string `json:"exec_id"` + Origin string `json:"origin,omitempty"` + PlatformHost string `json:"platform_host,omitempty"` + JobName string `json:"job_name,omitempty"` + ConnName string `json:"conn_name,omitempty"` + // CustomDescription is caller-supplied context, shown above the error. + CustomDescription string `json:"custom_description,omitempty"` +} + +// ComposeReport builds a redacted report from a local failure snapshot. +func ComposeReport(execID string) (ReportDraft, error) { + le, err := ResolveLocalExec(execID) + if err != nil { + return ReportDraft{}, err + } + + errText := readSnapshotFile(le.LogDir, "error.txt") + runLog := readSnapshotFile(le.LogDir, "stderr.log") + meta := map[string]any{} + if b, err := os.ReadFile(filepath.Join(le.LogDir, "meta.json")); err == nil { + _ = json.Unmarshal(b, &meta) + } + + src := cast.ToString(meta["source_type"]) + tgt := cast.ToString(meta["target_type"]) + label := cast.ToString(meta["error_short_label"]) + configPath := cast.ToString(meta["config_path"]) + if configPath == "" { + configPath = cast.ToString(meta["object"]) // legacy snapshots + } + connName := cast.ToString(meta["conn_name"]) + sigID := strings.ToLower(cast.ToString(meta["error_signature"])) + origin, platformHost, jobName := reportOriginFromMeta(meta) + + skel := Skeleton(errText) + if sigID == "" || len(sigID) != CompositeIDLen { + sig := SignError(errText, SignMeta{SourceType: dbio.Type(src), TargetType: dbio.Type(tgt)}) + sigID = sig.ID + if label == "" { + label = sig.ShortLabel + } + if src == "" { + src = string(sig.Meta.SourceType) + } + if tgt == "" { + tgt = string(sig.Meta.TargetType) + } + } + if label == "" { + label = ShortLabel(skel) + } + + d := ReportDraft{ + Title: reportTitle(label, src, tgt), + Description: reportDescription(redactForReport(errText)), + Config: reportConfig(configPath, connName, le.LogDir), + LogExcerpt: reportLogExcerpt(redactForReport(runLog)), + Version: core.Version, + OS: reportOSName(), + SourceType: src, + TargetType: tgt, + SignatureID: sigID, + Skeleton: skel, + ExecID: le.ID, + Origin: origin, + PlatformHost: platformHost, + JobName: jobName, + ConnName: connName, + } + return d, nil +} + +func reportTitle(label, src, tgt string) string { + if label == "" { + label = "unknown_error" + } + return fmt.Sprintf("%s (%s→%s)", label, typeToken(dbio.Type(src)), typeToken(dbio.Type(tgt))) +} + +func reportDescription(redactedErr string) string { + redactedErr = strings.TrimSpace(redactedErr) + if redactedErr == "" { + return "(no error message captured)" + } + // Keep the debug message stack (caller frames + wrap messages), not a + // collapsed one-line summary. GitHub URL budget trims logs/config first. + return redactedErr +} + +func reportConfig(configPath, connName, logDir string) string { + if connName != "" { + return "" + } + snap := filepath.Join(logDir, "config.snapshot.yaml") + body := "" + if b, err := os.ReadFile(snap); err == nil { + body = strings.TrimSpace(redactForReport(string(b))) + } + // Fall back to the config path on disk (snapshots written before + // config.snapshot.yaml existed). + if body == "" && isConfigFilePath(configPath) { + if b, err := os.ReadFile(configPath); err == nil && int64(len(b)) <= 64*1024 { + body = strings.TrimSpace(redactForReport(string(b))) + } + } + if body != "" { + return body + } + return "(not captured)" +} + +func isConfigFilePath(path string) bool { + return g.In(strings.ToLower(filepath.Ext(path)), ".yaml", ".yml", ".json") +} + +func reportLogExcerpt(redacted string) string { + redacted = strings.TrimRight(redacted, "\n") + if redacted == "" { + return "(no log captured)" + } + if len(redacted) <= maxLogExcerptBytes { + return redacted + } + s := redacted[len(redacted)-maxLogExcerptBytes:] + if i := strings.IndexByte(s, '\n'); i >= 0 && i < 200 { + s = s[i+1:] + } + return "[...truncated...]\n" + s +} + +func reportOSName() string { + switch runtime.GOOS { + case "linux": + return "Linux" + case "darwin": + return "Mac" + case "windows": + return "Windows" + default: + return runtime.GOOS + } +} + +func readSnapshotFile(dir, name string) string { + b, err := os.ReadFile(filepath.Join(dir, name)) + if err != nil { + return "" + } + return string(b) +} + +// redactForReport scrubs secrets, then replaces only URLs and file paths. +// Identifiers, numbers and timestamps stay readable for debugging. +func redactForReport(s string) string { + s = env.ScrubLine(s) + lines := strings.Split(s, "\n") + for i, line := range lines { + line = reQuotedURL.ReplaceAllStringFunc(line, replaceURL) + line = reQuotedPath.ReplaceAllString(line, "") + line = reURL.ReplaceAllStringFunc(line, replaceURL) + line = reVersionBanner.ReplaceAllString(line, "") + line = replacePathKeepLead(reUnixPath, line) + line = replacePathKeepLead(reWinPath, line) + line = replacePathKeepLead(reHomePath, line) + lines[i] = line + } + return strings.Join(lines, "\n") +} + +// BodyMarkdown renders sections that match the GitHub issue template fields. +func (d ReportDraft) BodyMarkdown() string { + var b strings.Builder + b.WriteString("## Description\n\n") + if custom := strings.TrimSpace(d.CustomDescription); custom != "" { + b.WriteString(custom) + b.WriteString("\n\n") + } + b.WriteString("Error:\n\n```\n") + b.WriteString(strings.TrimSpace(d.Description)) + b.WriteString("\n```\n\n") + fmt.Fprintf(&b, "Exec ID: %s\n", d.ExecID) + fmt.Fprintf(&b, "Sling version: %s\n", d.Version) + fmt.Fprintf(&b, "OS: %s\n", d.OS) + if strings.EqualFold(d.Origin, "platform") { + fmt.Fprintf(&b, "Platform execution (host: %s, job: %s)\n", d.PlatformHost, d.JobName) + } + if d.ConnName != "" { + fmt.Fprintf(&b, "Connection: %s\n", d.ConnName) + } + if d.SourceType != "" { + fmt.Fprintf(&b, "Source: %s\n", typeToken(dbio.Type(d.SourceType))) + } + if d.TargetType != "" { + fmt.Fprintf(&b, "Target: %s\n", typeToken(dbio.Type(d.TargetType))) + } + if d.ConnName == "" { + b.WriteString("\n## Replication Configuration\n\n") + b.WriteString("```yaml\n") + b.WriteString(strings.TrimRight(d.Config, "\n")) + b.WriteString("\n```\n") + } + b.WriteString("\n## Log Output\n\n") + b.WriteString("```\n") + b.WriteString(strings.TrimRight(d.LogExcerpt, "\n")) + b.WriteString("\n```\n") + return b.String() +} + +// trimToBudget shrinks logs, then config, until build stays within max bytes. +// Never trims the description. +func (d ReportDraft) trimToBudget(max int, build func(logs, config string) string) string { + logs := d.LogExcerpt + config := d.Config + u := build(logs, config) + for len(u) > max && logsHasMore(logs) { + logs = dropFirstLine(logs) + u = build(logs, config) + } + if len(u) > max && len(logs) > 0 { + // One remaining line still too long: keep the tail. + keep := len(logs) / 2 + for keep > 32 && len(build(logs[len(logs)-keep:], config)) > max { + keep = keep / 2 + } + if keep < len(logs) { + logs = logs[len(logs)-keep:] + } + u = build(logs, config) + } + for len(u) > max && config != "" && config != "(not captured)" && config != "(truncated)" { + next := dropFirstLine(config) + if next == config || next == "" { + if len(config) > 64 { + config = config[len(config)/2:] + } else { + config = "(truncated)" + } + } else { + config = next + } + if strings.TrimSpace(config) == "" { + config = "(truncated)" + } + u = build(logs, config) + } + return u +} + +// GitHubIssueURL builds a prefilled new-issue URL. Issue forms (.yml) ignore +// query params except title, so the full report goes into body against the +// blank issue form. +func (d ReportDraft) GitHubIssueURL() string { + return d.trimToBudget(githubIssueURLMax, func(logs, config string) string { + trimmed := d + trimmed.LogExcerpt = logs + trimmed.Config = config + q := url.Values{} + q.Set("title", d.Title) + q.Set("body", trimmed.BodyMarkdown()) + return githubIssueBaseURL + "?" + q.Encode() + }) +} + +// ContactFormURL builds the contact-page URL with the full report body +// base64 encoded in the issue param. The form collects name/email and +// Turnstile. Never trims logs or config; oversized bodies go to a file +// instead (see deliverContactForm). +func (d ReportDraft) ContactFormURL() string { + return contactFormBaseURL + "?issue=" + base64.RawURLEncoding.EncodeToString([]byte(d.BodyMarkdown())) +} + +// deliverGitHubIssue opens a prefilled GitHub issue when the URL fits. +// When it still exceeds githubIssueURLMax after trimming, it falls back +// to the email contact form (or a temp file if that is also oversize). +func deliverGitHubIssue(d ReportDraft, interactive bool) error { + u := d.GitHubIssueURL() + if len(u) > githubIssueURLMax { + fmt.Fprintln(os.Stdout, "report too long for a GitHub issue URL; sending via email instead.") + return deliverContactForm(d, interactive) + } + fmt.Fprintln(os.Stdout, u) + if interactive { + if err := OpenBrowser(u); err != nil { + g.Warn("could not open browser: %s", err.Error()) + } + } + return nil +} + +// deliverContactForm opens the prefilled contact page when the URL fits. +// When it exceeds the Cloudflare request-line limit, it writes the full +// report to a temp file for manual email instead. +func deliverContactForm(d ReportDraft, interactive bool) error { + u := d.ContactFormURL() + if len(u) > contactFormURLMax { + return writeReportFile(d) + } + fmt.Fprintln(os.Stdout, u) + if interactive { + if err := OpenBrowser(u); err != nil { + g.Warn("could not open browser: %s", err.Error()) + } + fmt.Fprintln(os.Stdout, "complete the name/email fields in the form, then submit.") + } + return nil +} + +// writeReportFile saves the full report body for manual email as attachment. +func writeReportFile(d ReportDraft) error { + f, err := os.CreateTemp("", "sling-report-*.md") + if err != nil { + return g.Error("could not write report file: %s", err.Error()) + } + defer f.Close() + if _, err := f.WriteString(d.BodyMarkdown()); err != nil { + return g.Error("could not write report file: %s", err.Error()) + } + fmt.Fprintln(os.Stdout, "report too long for the web form.") + fmt.Fprintf(os.Stdout, "attach this file and email it to %s:\n %s\n", "support@slingdata.io", f.Name()) + return nil +} + +func logsHasMore(s string) bool { + return strings.Contains(s, "\n") +} + +func dropFirstLine(s string) string { + i := strings.IndexByte(s, '\n') + if i < 0 { + return "" + } + return s[i+1:] +} + +// OpenBrowser opens url in the default browser. Always also print the URL. +func OpenBrowser(rawURL string) error { + if browserEnv := strings.TrimSpace(os.Getenv("BROWSER")); browserEnv != "" { + return exec.Command(browserEnv, rawURL).Start() + } + var cmd string + var args []string + switch runtime.GOOS { + case "windows": + cmd = "rundll32" + args = []string{"url.dll,FileProtocolHandler", rawURL} + case "darwin": + cmd = "open" + args = []string{rawURL} + default: + cmd = "xdg-open" + args = []string{rawURL} + } + return exec.Command(cmd, args...).Start() +} + +func composeReportWithPlatformFallback(execID string, localErr error) (ReportDraft, error) { + pe, err := resolvePlatformFallback(execID, localErr) + if err != nil { + return ReportDraft{}, err + } + if _, err := materializePlatformExec(pe); err != nil { + return ReportDraft{}, err + } + return ComposeReport(execID) +} + +// ReportCmd is the `sling assist report` entry. +type ReportCmd struct { + ExecID string + Title string // optional override + Description string // optional custom context, shown above the error + GitHub bool + Email bool + Submit bool // skip the confirm prompt (for agents) +} + +// RunReport prints the redacted draft, then optionally sends it. +func RunReport(opts ReportCmd) error { + d, err := ComposeReport(opts.ExecID) + if err != nil { + d, err = composeReportWithPlatformFallback(opts.ExecID, err) + if err != nil { + return err + } + } + if opts.Title != "" { + d.Title = opts.Title + } + if opts.Description != "" { + d.CustomDescription = opts.Description + } + fmt.Fprintln(os.Stdout, d.Title) + fmt.Fprintln(os.Stdout, "") + body := d.BodyMarkdown() + fmt.Fprint(os.Stdout, body) + if !strings.HasSuffix(body, "\n") { + fmt.Fprintln(os.Stdout) + } + + if opts.GitHub && opts.Email { + return g.Error("use only one of --github or --email") + } + + route := "" + switch { + case opts.GitHub: + route = "github" + case opts.Email: + route = "email" + } + + if !env.IsInteractiveTerminal() { + if route == "github" { + return deliverGitHubIssue(d, false) + } + if route == "email" { + // Form flow: the user confirms by submitting the contact form. + return deliverContactForm(d, false) + } + return nil + } + + if route == "" { + picked, err := pickReportRoute() + if err != nil { + return err + } + if picked == "cancel" || picked == "" { + return nil + } + route = picked + } + + if !opts.Submit { + ok, err := confirmSendReport() + if err != nil { + return err + } + if !ok { + return nil + } + } + return sendReport(d, route) +} + +func sendReport(d ReportDraft, route string) error { + switch route { + case "github": + return deliverGitHubIssue(d, true) + case "email": + // The contact form's Turnstile check and name/email fields validate + // the submission. + return deliverContactForm(d, true) + default: + return g.Error("unknown report route %q", route) + } +} + +func pickReportRoute() (string, error) { + route := "cancel" + form := huh.NewForm( + huh.NewGroup( + huh.NewSelect[string](). + Title("How do you want to send this report?"). + Options( + huh.NewOption("GitHub issue (public, needs account)", "github"), + huh.NewOption("Email to support", "email"), + huh.NewOption("Cancel", "cancel"), + ). + Value(&route), + ), + ).WithTheme(huh.ThemeCharm()) + if err := form.Run(); err != nil { + return "", err + } + return route, nil +} + +func confirmSendReport() (bool, error) { + ok := false + form := huh.NewForm( + huh.NewGroup( + huh.NewConfirm(). + Title("Send this report? [y/N]"). + Affirmative("Yes"). + Negative("No"). + Value(&ok), + ), + ).WithTheme(huh.ThemeCharm()) + if err := form.Run(); err != nil { + return false, err + } + return ok, nil +} + +// PlatformExec is the platform view of one execution. +type PlatformExec struct { + ExecID string + Status string // success | error | running | ... + Type string // replication | pipeline | query | monitor + JobName string + FileName string + StartTime string // formatted, may be empty + EndTime string + ErrSummary string // first error line from the record, if present + HostLabel string + Rows string + Duration string + Object string + Version string +} + +// resolvePlatformFallback looks up a missed local exec on the Sling Platform. +var resolvePlatformFallback = func(execID string, localErr error) (*PlatformExec, error) { + if localErr == nil { + return nil, g.Error("see ~/.sling/assist/errors/") + } + return nil, g.Error(localErr, "see ~/.sling/assist/errors/") +} + +// materializePlatformExec writes a local snapshot from platform data. +var materializePlatformExec = func(pe *PlatformExec) (LocalExec, error) { + return LocalExec{}, g.Error("use the official release of sling-cli to materialize platform executions") +} + +func reportOriginFromMeta(meta map[string]any) (origin, host, job string) { + if len(meta) == 0 { + return "", "", "" + } + return cast.ToString(meta["origin"]), cast.ToString(meta["platform_host"]), cast.ToString(meta["job_name"]) +} diff --git a/core/sling/assist/report_test.go b/core/sling/assist/report_test.go new file mode 100644 index 000000000..bf7046ae9 --- /dev/null +++ b/core/sling/assist/report_test.go @@ -0,0 +1,358 @@ +package assist + +import ( + "encoding/base64" + "io" + "net/url" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/slingdata-io/sling-cli/core/dbio" +) + +func writeReportFixture(t *testing.T, id string) string { + t.Helper() + withTempHomeDir(t) + errMsg := "could not read /Users/alice/secret/file.csv from host" + WriteFailureSnapshot(FailureSnapshot{ + ExecID: id, + ErrMsg: errMsg, + ConfigPath: "/Users/alice/secret/repl.yaml", + SignMeta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbSnowflake}, + RunLog: strings.Join([]string{ + "opened https://db.internal.example.com/sync", + "path=/Users/alice/secret/file.csv", + "failed to copy rows", + }, "\n"), + }) + dir := findLocalExecDir(id) + if dir == "" { + t.Fatal("fixture dir missing") + } + return dir +} + +func TestComposeReportGoldenDraft(t *testing.T) { + writeReportFixture(t, "exec_report1") + d, err := ComposeReport("exec_report1") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(d.Title, " (postgres→snowflake)") { + t.Fatalf("title = %q", d.Title) + } + md := d.BodyMarkdown() + descAt := strings.Index(md, "## Description") + cfgAt := strings.Index(md, "## Replication Configuration") + logAt := strings.Index(md, "## Log Output") + if descAt < 0 || cfgAt < 0 || logAt < 0 || !(descAt < cfgAt && cfgAt < logAt) { + t.Fatalf("markdown section order:\n%s", md) + } + if strings.Contains(d.LogExcerpt, "/Users/") { + t.Fatalf("path leaked in LogExcerpt: %q", d.LogExcerpt) + } + if strings.Contains(d.LogExcerpt, "db.internal.example.com") { + t.Fatalf("hostname leaked in LogExcerpt: %q", d.LogExcerpt) + } + if !strings.Contains(d.LogExcerpt, "") && !strings.Contains(d.LogExcerpt, "") { + t.Fatalf("expected placeholders in LogExcerpt: %q", d.LogExcerpt) + } + if strings.Contains(d.Config, "/Users/") { + t.Fatalf("path leaked in Config: %q", d.Config) + } + if strings.Contains(md, d.SignatureID) { + t.Fatalf("signature must not appear in markdown body") + } +} + +func TestComposeReportKeepsDebugStack(t *testing.T) { + withTempHomeDir(t) + errMsg := strings.Join([]string{ + "~ could not connect", + "--- database.go:123 Connect ---", + "~ failed to ping", + "--- task_run.go:140 Execute ---", + "connection refused", + }, "\n") + WriteFailureSnapshot(FailureSnapshot{ + ExecID: "exec_stack1", + ErrMsg: errMsg, + SignMeta: SignMeta{SourceType: dbio.TypeDbPostgres, TargetType: dbio.TypeDbSnowflake}, + }) + d, err := ComposeReport("exec_stack1") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(d.Description, "--- database.go:123 Connect ---") { + t.Fatalf("debug stack stripped from Description:\n%s", d.Description) + } + if !strings.Contains(d.Description, "--- task_run.go:140 Execute ---") { + t.Fatalf("debug stack stripped from Description:\n%s", d.Description) + } + if strings.Contains(d.Description, "could not connect failed to ping") { + t.Fatalf("debug stack collapsed to one line:\n%s", d.Description) + } + md := d.BodyMarkdown() + if !strings.Contains(md, "--- task_run.go:140 Execute ---") { + t.Fatalf("debug stack stripped from body:\n%s", md) + } +} + +func TestComposeReportRedactsPathAndHost(t *testing.T) { + writeReportFixture(t, "exec_redact1") + d, err := ComposeReport("exec_redact1") + if err != nil { + t.Fatal(err) + } + if strings.Contains(d.LogExcerpt, "/Users/alice") { + t.Fatalf("raw path in excerpt: %q", d.LogExcerpt) + } + if strings.Contains(d.LogExcerpt, "db.internal.example.com") { + t.Fatalf("raw host in excerpt: %q", d.LogExcerpt) + } +} + +func TestComposeReportPrefixID(t *testing.T) { + writeReportFixture(t, "exec_prefix_abc") + if _, err := ComposeReport("exec_pre"); err != nil { + t.Fatal(err) + } +} + +func TestGitHubIssueURLBudget(t *testing.T) { + var b strings.Builder + for i := 0; i < 400; i++ { + b.WriteString("line ") + b.WriteString(strings.Repeat("x", 80)) + b.WriteByte('\n') + } + d := ReportDraft{ + Title: "boom (postgres→snowflake)", + Description: "kept description", + Config: "repl.yaml", + LogExcerpt: b.String(), + Version: "1.4.24", + OS: "Mac", + } + u := d.GitHubIssueURL() + if len(u) > githubIssueURLMax { + t.Fatalf("url len %d > %d", len(u), githubIssueURLMax) + } + parsed, err := url.Parse(u) + if err != nil { + t.Fatal(err) + } + q := parsed.Query() + if q.Get("template") != "" { + t.Fatalf("template param must be absent for body prefill, got %q", q.Get("template")) + } + body := q.Get("body") + if !strings.Contains(body, "kept description") { + t.Fatalf("description trimmed from body") + } + if !strings.Contains(body, "Sling version: 1.4.24") { + t.Fatalf("version missing from body") + } + if !strings.Contains(body, "OS: Mac") { + t.Fatalf("os missing from body") + } + if !strings.Contains(body, "```") { + t.Fatal("logs stripped entirely from body") + } +} + +func TestContactFormURLNoTrim(t *testing.T) { + d := ReportDraft{ + Title: "boom", + Description: "kept", + Config: strings.Repeat("col: value\n", 4000), + LogExcerpt: strings.Repeat("log line\n", 4000), + Version: "1.4.24", + OS: "Mac", + } + u := d.ContactFormURL() + if !strings.HasPrefix(u, contactFormBaseURL+"?issue=") { + t.Fatalf("bad prefix: %q", u[:60]) + } + // Over-limit bodies must stay complete; delivery falls back to a file. + if len(u) <= contactFormURLMax { + t.Fatalf("expected over-limit url, got %d", len(u)) + } + enc := strings.TrimPrefix(u, contactFormBaseURL+"?issue=") + decoded, err := base64.RawURLEncoding.DecodeString(enc) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(decoded), "log line") || !strings.Contains(string(decoded), "col: value") { + t.Fatal("body trimmed") + } +} + +func TestDeliverContactFormOversizedWritesFile(t *testing.T) { + d := ReportDraft{ + Title: "boom", + Description: "kept", + Config: strings.Repeat("col: value\n", 4000), + LogExcerpt: "log", + Version: "1.4.24", + OS: "Mac", + } + // Writes to stdout; failure surfaces as an error return. + if err := deliverContactForm(d, false); err != nil { + t.Fatal(err) + } +} + +func captureStdout(t *testing.T, fn func()) string { + t.Helper() + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + old := os.Stdout + os.Stdout = w + fn() + _ = w.Close() + os.Stdout = old + b, err := io.ReadAll(r) + if err != nil { + t.Fatal(err) + } + return string(b) +} + +func TestDeliverGitHubIssueOversizedFallsBackToEmail(t *testing.T) { + d := ReportDraft{ + Title: "boom", + Description: "kept", + CustomDescription: strings.Repeat("context ", 4000), + Config: "cfg", + LogExcerpt: "log", + Version: "1.4.24", + OS: "Mac", + } + if n := len(d.GitHubIssueURL()); n <= githubIssueURLMax { + t.Fatalf("fixture not over github budget: %d", n) + } + out := captureStdout(t, func() { + if err := deliverGitHubIssue(d, false); err != nil { + t.Fatal(err) + } + }) + if !strings.Contains(out, "sending via email instead") { + t.Fatalf("missing email fallback:\n%s", out) + } + if strings.Contains(out, githubIssueBaseURL) { + t.Fatalf("still printed a GitHub URL:\n%s", out) + } +} + +func TestDeliverGitHubIssueFitsStaysOnGitHub(t *testing.T) { + d := ReportDraft{ + Title: "boom", + Description: "kept", + Config: "cfg", + LogExcerpt: "log", + Version: "1.4.24", + OS: "Mac", + } + u := d.GitHubIssueURL() + if len(u) > githubIssueURLMax { + t.Fatalf("fixture over github budget: %d", len(u)) + } + out := captureStdout(t, func() { + if err := deliverGitHubIssue(d, false); err != nil { + t.Fatal(err) + } + }) + if !strings.Contains(out, githubIssueBaseURL) { + t.Fatalf("missing GitHub URL:\n%s", out) + } + if strings.Contains(out, "sending via email instead") { + t.Fatalf("should not fall back to email:\n%s", out) + } +} + +func TestGitHubIssueURLDropdownAndEscape(t *testing.T) { + d := ReportDraft{ + Title: "hash # and café", + Description: "line1\nline2 #frag", + Config: "a=b", + LogExcerpt: "ok", + OS: "FreeBSD", + Version: "dev", + } + u := d.GitHubIssueURL() + parsed, err := url.Parse(u) + if err != nil { + t.Fatal(err) + } + q := parsed.Query() + body := q.Get("body") + if !strings.Contains(body, "line1\nline2 #frag") { + t.Fatalf("description unescape = %q", body) + } + if !strings.Contains(body, "OS: FreeBSD") { + t.Fatalf("non-dropdown os should stay in body text, got %q", body) + } + if q.Get("title") != "hash # and café" { + t.Fatalf("title unescape = %q", q.Get("title")) + } +} + +func TestComposeReportConnTestHasNoConfig(t *testing.T) { + withTempHomeDir(t) + WriteFailureSnapshot(FailureSnapshot{ + ExecID: "exec_conn1", + ErrMsg: "could not connect", + ConnName: "MY_PG", + SignMeta: SignMeta{SourceType: dbio.TypeDbPostgres}, + }) + dir := findLocalExecDir("exec_conn1") + if dir == "" { + t.Fatal("fixture dir missing") + } + if _, err := os.Stat(filepath.Join(dir, "config.snapshot.yaml")); !os.IsNotExist(err) { + t.Fatal("conns test must not write config.snapshot.yaml") + } + d, err := ComposeReport("exec_conn1") + if err != nil { + t.Fatal(err) + } + if d.ConnName != "MY_PG" { + t.Fatalf("ConnName=%q", d.ConnName) + } + if d.Config != "" { + t.Fatalf("config text must be empty for conns test, got %q", d.Config) + } + md := d.BodyMarkdown() + if !strings.Contains(md, "Connection: MY_PG") { + t.Fatalf("missing connection line:\n%s", md) + } + if strings.Contains(md, "## Replication Configuration") { + t.Fatalf("conns test must not include config section:\n%s", md) + } + if strings.Contains(md, "Target:") { + t.Fatalf("conns test must not invent a target:\n%s", md) + } +} + +func TestHandleReportComposeRedacts(t *testing.T) { + writeReportFixture(t, "exec_mcp1") + d, err := ComposeReport("exec_mcp1") + if err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(filepath.Join(findLocalExecDir("exec_mcp1"), "stderr.log")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(raw), "/Users/alice") { + t.Fatal("fixture missing raw path") + } + if strings.Contains(d.LogExcerpt, "/Users/alice") { + t.Fatalf("compose leaked path: %q", d.LogExcerpt) + } +} diff --git a/core/sling/assist/session.go b/core/sling/assist/session.go new file mode 100644 index 000000000..fbc214f82 --- /dev/null +++ b/core/sling/assist/session.go @@ -0,0 +1,820 @@ +package assist + +import ( + "bufio" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "syscall" + "time" + + "github.com/charmbracelet/huh" + "github.com/flarco/g" + "github.com/google/uuid" + "github.com/slingdata-io/sling-cli/core" + "github.com/slingdata-io/sling-cli/core/env" + "golang.org/x/term" +) + +// assistOut is stdout for --out - / nested-launch / the open card. Tests swap it. +var assistOut io.Writer = os.Stdout + +const ( + modeAsk = "ask" + modeOpen = "open" +) + +// SessionOptions is the flags-only `sling assist` invocation. +type SessionOptions struct { + Ask string + Name string + Agent string + Model string + Print bool + OutputFile string + Headless bool + ExecID string // --id: investigate this failure + ResumeID string + ResumeSet bool // --resume present (empty id → picker already resolved) + NonInteractive map[string]string +} + +// NestedLaunch reports an already-running CLI agent (or non-TTY stdin). +// When true, print the prompt instead of spawning another agent. +func NestedLaunch() bool { + if os.Getenv("CLAUDECODE") != "" { + return true + } + if os.Getenv("CURSOR_TRACE_ID") != "" { + return true + } + if os.Getenv("OPENCODE") != "" || os.Getenv("OPENCODE_SESSION") != "" { + return true + } + return false +} + +// Session probes local state, renders the five-section prompt, then prints or launches. +func Session(opts SessionOptions) (string, error) { + if opts.ResumeID != "" { + return resumeSession(opts) + } + + if len(opts.NonInteractive) > 0 { + if v := opts.NonInteractive["Intention"]; v != "" && opts.Ask == "" { + opts.Ask = v + } + if v := opts.NonInteractive["intention"]; v != "" && opts.Ask == "" { + opts.Ask = v + } + } + opts.Ask = strings.TrimSpace(opts.Ask) + + if opts.Headless && opts.Ask == "" && opts.ExecID == "" { + return "", g.Error(`no ask given; pass it as an argument: sling assist ""`) + } + + // Failure details enter the prompt only when the ask targets the + // failure: the investigate pick, or the empty-ask fallback. + ctx := Probe(ProbeOptions{ + Ask: opts.Ask, + ExecID: opts.ExecID, + IncludeFailure: opts.Ask == "", + }) + if opts.ExecID != "" && ctx.TargetExec == nil { + _, localErr := ResolveLocalExec(opts.ExecID) + if localErr != nil { + pe, err := resolvePlatformFallback(opts.ExecID, localErr) + if err != nil { + return "", err + } + ctx.PlatformExec = pe + ctx.Route = "platform_failed_run" + } + } + mode := modeAsk + if shouldOpenScreen(opts) { + ask, investigate, ok := runOpenScreen(ctx) + if !ok { + return "", nil + } + opts.Ask = ask + ctx = Probe(ProbeOptions{Ask: opts.Ask, IncludeFailure: investigate}) + mode = modeOpen + } + + prompt, err := RenderPrompt(ctx) + if err != nil { + return "", err + } + + if opts.Print { + fmt.Fprint(assistOut, prompt) + return prompt, nil + } + if opts.OutputFile != "" { + if err := os.WriteFile(opts.OutputFile, []byte(prompt), 0o644); err != nil { + return "", g.Error(err, "write --out %s", opts.OutputFile) + } + return prompt, nil + } + if NestedLaunch() || (!ttyCheck(os.Stdin) && !opts.Headless) { + fmt.Fprint(assistOut, prompt) + return prompt, nil + } + + if err := EnsureAssistReady(); err != nil { + return "", err + } + prof, _, _ := LoadProfile() + resolvedAgent, agentErr := ResolveAgent(opts.Agent, prof) + if agentErr != nil { + return "", agentErr + } + + a := AnswersFile{ + Name: opts.Name, + Task: mode, + SlingVersion: core.Version, + Created: time.Now().UTC(), + Agent: resolvedAgent, + Cwd: mustGetwd(), + Answers: map[string]any{"intention": opts.Ask, "ask": opts.Ask}, + } + if a.Name == "" { + if s := slugify(opts.Ask); s != "" && s != "entry" { + a.Name = s + } else { + a.Name = "assist" + } + } + now := time.Now().UTC() + harnessID := newHarnessSessionID(resolvedAgent) + m := Meta{ + Task: mode, + Agent: resolvedAgent, + Model: opts.Model, + HarnessSessionID: harnessID, + LaunchedAt: &now, + } + id, err := SaveEntry(a, prompt, m) + if err != nil { + return prompt, err + } + if err := AutoTrim(); err != nil { + g.Debug("assist: auto-trim: %s", err.Error()) + } + if resolvedAgent == "" { + return id, nil + } + promptPath := filepath.Join(HistoryDir(), id, "prompt.md") + g.Info("submitting prompt to agent %s: %s", + env.CyanString(resolvedAgent), env.DarkGrayString(collapseHome(promptPath))) + snap := snapshotHarnessFiles(resolvedAgent) + err = LaunchAgent(LaunchOptions{ + Agent: resolvedAgent, + Prompt: prompt, + PromptPath: promptPath, + Model: opts.Model, + SessionID: harnessID, + }) + if harnessID == "" { + if hid := discoverHarnessSessionID(resolvedAgent, snap); hid != "" { + e, lerr := LoadEntry(id) + if lerr == nil { + e.Meta.HarnessSessionID = hid + if serr := e.saveMeta(); serr != nil { + g.Debug("assist: save harness session id: %s", serr.Error()) + } + } + } + } + if err != nil { + var ae *AgentExitError + if errors.As(err, &ae) { + return id, ae + } + return id, g.Error(err, "agent launch failed") + } + return id, nil +} + +func resumeSession(opts SessionOptions) (string, error) { + e, err := LoadEntry(opts.ResumeID) + if err != nil { + return "", g.Error("unknown session %q; run `sling assist --resume` to pick one", opts.ResumeID) + } + + promptPath := filepath.Join(e.Path, "prompt.md") + promptBytes, _ := os.ReadFile(promptPath) + prompt := string(promptBytes) + + if opts.Print { + fmt.Fprint(assistOut, prompt) + return prompt, nil + } + if opts.OutputFile != "" { + if err := os.WriteFile(opts.OutputFile, promptBytes, 0o644); err != nil { + return "", g.Error(err, "write --out %s", opts.OutputFile) + } + return prompt, nil + } + + if err := EnsureAssistReady(); err != nil { + return "", err + } + + agent := e.Answers.Agent + if e.Meta.Agent != "" { + agent = e.Meta.Agent + } + if opts.Agent != "" && opts.Agent != agent { + return "", g.Error("session %q was launched with agent %q; cannot resume with --agent %s", e.ID, agent, opts.Agent) + } + + hid := e.Meta.HarnessSessionID + if hid == "" { + hid = discoverHarnessSessionID(agent, nil) + if hid != "" { + e.Meta.HarnessSessionID = hid + if serr := e.saveMeta(); serr != nil { + g.Debug("assist: save harness session id: %s", serr.Error()) + } + } + } + if hid == "" { + return "", g.Error("session %q has no harness session id; cannot resume", e.ID) + } + + if err := LaunchResume(agent, hid, opts.Model); err != nil { + var ae *AgentExitError + if errors.As(err, &ae) { + return e.ID, ae + } + return e.ID, g.Error(err, "agent resume failed") + } + return e.ID, nil +} + +// assistIn is stdin for the open screen. Tests swap it. +var assistIn io.Reader = os.Stdin + +// ttyCheck is the TTY probe. Tests swap it. +var ttyCheck = isTTY + +func shouldOpenScreen(opts SessionOptions) bool { + if strings.TrimSpace(opts.Ask) != "" || opts.ExecID != "" { + return false + } + if opts.Print || opts.OutputFile != "" || opts.Headless || NestedLaunch() { + return false + } + return ttyCheck(os.Stdin) +} + +func runOpenScreen(p PromptContext) (ask string, investigate, ok bool) { + width := 80 + if w, _, err := term.GetSize(int(os.Stdout.Fd())); err == nil && w > 0 { + width = w + } + opts := p.suggestions() + if assistIn == os.Stdin { + // Real terminal: the huh form draws its own list, no banner. + return pickOpenAsk(opts) + } + fmt.Fprint(assistOut, renderOpenCard(p, width)) + return readOpenAsk(assistIn, assistOut, opts) +} + +// pickOpenAsk is the huh picker: a select over the suggestion rows, then a +// text area when the user picks the describe-it row. +func pickOpenAsk(opts []suggestion) (ask string, investigate, ok bool) { + if len(opts) == 0 { + return "", false, false + } + sel := make([]huh.Option[int], 0, len(opts)) + for i, s := range opts { + sel = append(sel, huh.NewOption(s.Label, i)) + } + pick := 0 + free := "" + isFree := func() bool { return pick >= 0 && pick < len(opts) && strings.TrimSpace(opts[pick].Ask) == "" } + + form := huh.NewForm( + huh.NewGroup( + // No Height: huh sizes the viewport to all options, so a + // wrapped row never scrolls the list. + huh.NewSelect[int](). + Title("What would you like to do?"). + Options(sel...). + Value(&pick), + ), + huh.NewGroup( + huh.NewText(). + Title("Describe what you want"). + Placeholder("e.g. backfill orders from postgres into snowflake"). + Lines(5). + CharLimit(2000). + Value(&free), + ).WithHideFunc(func() bool { return !isFree() }), + ).WithTheme(huh.ThemeCharm()) + + if err := form.Run(); err != nil { + if !errors.Is(err, huh.ErrUserAborted) { + g.Debug("assist: open picker: %s", err.Error()) + } + fmt.Fprintln(assistOut, `No ask given. Run: sling assist ""`) + return "", false, false + } + + if pick < 0 || pick >= len(opts) { + return "", false, false + } + s := opts[pick] + if strings.TrimSpace(s.Ask) != "" { + return s.Ask, s.Investigate, true + } + free = strings.TrimSpace(free) + if free == "" { + fmt.Fprintln(assistOut, `No ask given. Run: sling assist ""`) + return "", false, false + } + return free, false, true +} + +func renderOpenCard(p PromptContext, width int) string { + if width <= 0 { + width = 80 + } + var b strings.Builder + b.WriteString(openSummary(p)) + b.WriteByte('\n') + for i, s := range p.suggestions() { + fmt.Fprintf(&b, " %d. %s\n", i+1, s.Label) + } + b.WriteByte('\n') + return wrapToWidth(strings.TrimRight(b.String(), "\n")+"\n", width) +} + +func openSummary(p PromptContext) string { + n := userConnectionCount(p.Connections) + noun := "connections" + if n == 1 { + noun = "connection" + } + var line1 string + switch { + case p.HasProject && p.ProjectName != "": + line1 = fmt.Sprintf("On project %s · %d %s", p.ProjectName, n, noun) + case p.HasProject: + line1 = fmt.Sprintf("On this project · %d %s", n, noun) + default: + line1 = fmt.Sprintf("No project in this folder · %d %s", n, noun) + } + if run, ok := latestRun(p.RecentRuns); ok { + label := run.ID + if obj := run.displayObject(); obj != "" { + label += " " + obj + } + s := fmt.Sprintf("%s [%s]", label, run.Status) + if !run.When.IsZero() { + rt := relTime(run.When) + if rt == "just now" { + s += " just now" + } else { + s += " " + rt + " ago" + } + } + return line1 + "\n last run: " + s + "\n" + } + return line1 + "\n" +} + +func readOpenAsk(r io.Reader, w io.Writer, opts []suggestion) (ask string, investigate, ok bool) { + br := bufio.NewReader(r) + empty := 0 + fmt.Fprint(w, "> ") + for { + line, err := br.ReadString('\n') + if err != nil && strings.TrimSpace(line) == "" { + fmt.Fprintln(w, `No ask given. Run: sling assist ""`) + return "", false, false + } + line = strings.TrimSpace(line) + if line == "" { + empty++ + if empty >= 2 { + fmt.Fprintln(w, `No ask given. Run: sling assist ""`) + return "", false, false + } + fmt.Fprintln(w, `Type a number or describe what you want.`) + fmt.Fprint(w, "> ") + continue + } + empty = 0 + if n, convErr := strconv.Atoi(line); convErr == nil && n >= 1 && n <= len(opts) { + s := opts[n-1] + if strings.TrimSpace(s.Ask) != "" { + return s.Ask, s.Investigate, true + } + fmt.Fprintln(w, "Describe it:") + fmt.Fprint(w, "> ") + continue + } + return line, false, true + } +} + +// isTTY reports whether the file descriptor is connected to a terminal. +func isTTY(f *os.File) bool { + if f == nil { + return false + } + info, err := f.Stat() + if err != nil { + return false + } + return (info.Mode() & os.ModeCharDevice) != 0 +} + +// ResolveAgent picks the agent to launch, in order: +// 1. --agent override (must be a known CLI agent and detected on disk). +// 2. profile.Agent (when not "auto"). +// 3. profile.Agent == "auto" → first detected CLI agent on $PATH. +func ResolveAgent(override string, prof Profile) (string, error) { + if override != "" { + c := LookupClient(override) + if c == nil { + return "", g.Error("unknown agent %q", override) + } + if c.Kind() != KindCLIAgent { + return "", g.Error("agent %q is not a launchable CLI agent (it's an install target)", override) + } + if !c.Detect() { + return "", g.Error("agent %q not detected; run `sling assist setup` to set it up", override) + } + return override, nil + } + if prof.Agent != "" && prof.Agent != "auto" { + c := LookupClient(prof.Agent) + if c == nil { + return "", g.Error("profile names unknown agent %q; run `sling assist setup`", prof.Agent) + } + if c.Kind() != KindCLIAgent { + return "", g.Error("profile names non-launchable agent %q; run `sling assist setup` or use --agent", prof.Agent) + } + return prof.Agent, nil + } + for _, c := range CLIAgents() { + if c.Detect() && commandOnPath(agentBinary(c.Name())) { + return c.Name(), nil + } + } + return "", g.Error("no AI agent on $PATH; run `sling assist setup` or pass --agent") +} + +type agentLaunchPlan struct { + Args []string + UseStdin bool +} + +func agentBinary(agent string) string { + if agent == "cursor" { + return "cursor-agent" + } + return agent +} + +func assignsHarnessSessionID(agent string) bool { + return agent == "claude" || agent == "grok" +} + +func newHarnessSessionID(agent string) string { + if !assignsHarnessSessionID(agent) { + return "" + } + return uuid.NewString() +} + +func agentLaunchArgs(agent, promptPath, model, sessionID string) agentLaunchPlan { + var p agentLaunchPlan + switch agent { + case "codex": + p = agentLaunchPlan{Args: []string{"exec", "-"}, UseStdin: true} + case "gemini": + p = agentLaunchPlan{Args: []string{"-p", "-"}, UseStdin: true} + case "grok": + // -p/--prompt-file are single-turn. Seed the interactive session + // with a positional prompt that @-references the file. + p = agentLaunchPlan{ + Args: []string{fmt.Sprintf("Read and execute the task in @%s", promptPath)}, + UseStdin: false, + } + case "pi": + p = agentLaunchPlan{Args: []string{"-p"}, UseStdin: true} + case "opencode": + p = agentLaunchPlan{ + Args: []string{"run", "--file", promptPath, "Read and execute the attached task"}, + UseStdin: false, + } + case "claude": + p = agentLaunchPlan{ + Args: []string{fmt.Sprintf("Read and execute the task in @%s", promptPath)}, + UseStdin: false, + } + case "cursor": + p = agentLaunchPlan{Args: []string{promptPath}, UseStdin: false} + default: + p = agentLaunchPlan{UseStdin: true} + } + p.Args = withModelAndSession(agent, p.Args, model, sessionID) + return p +} + +func agentResumeArgs(agent, harnessID, model string) agentLaunchPlan { + var p agentLaunchPlan + switch agent { + case "claude": + p = agentLaunchPlan{Args: []string{"--resume", harnessID}} + case "grok": + p = agentLaunchPlan{Args: []string{"--resume", harnessID}} + case "codex": + p = agentLaunchPlan{Args: []string{"resume", harnessID}} + case "gemini": + p = agentLaunchPlan{Args: []string{"--resume", harnessID}} + case "cursor": + p = agentLaunchPlan{Args: []string{"--resume=" + harnessID}} + case "opencode": + p = agentLaunchPlan{Args: []string{"--session", harnessID}} + case "pi": + p = agentLaunchPlan{Args: []string{"--session", harnessID}} + default: + p = agentLaunchPlan{Args: []string{"--resume", harnessID}} + } + p.Args = withModelAndSession(agent, p.Args, model, "") + return p +} + +func withModelAndSession(agent string, args []string, model, sessionID string) []string { + flags := []string{} + switch agent { + case "claude": + if sessionID != "" { + flags = append(flags, "--session-id", sessionID) + } + if model != "" { + flags = append(flags, "--model", model) + } + return append(flags, args...) + case "grok": + if sessionID != "" { + flags = append(flags, "--session-id", sessionID) + } + if model != "" { + flags = append(flags, "--model", model) + } + return append(flags, args...) + case "codex": + if model == "" { + return args + } + if len(args) == 0 { + return []string{"-m", model} + } + out := make([]string, 0, len(args)+2) + out = append(out, args[0], "-m", model) + out = append(out, args[1:]...) + return out + default: + if model != "" { + flags = append(flags, "--model", model) + } + return append(flags, args...) + } +} + +// LaunchOptions is one agent exec (first run or resume). +type LaunchOptions struct { + Agent string + Prompt string + PromptPath string + Model string + SessionID string // pre-assigned harness id (claude, grok) +} + +// LaunchAgent execs the given CLI agent with the prompt. +func LaunchAgent(opts LaunchOptions) error { + plan := agentLaunchArgs(opts.Agent, opts.PromptPath, opts.Model, opts.SessionID) + return startAgent(opts.Agent, plan, opts.Prompt) +} + +// LaunchResume execs the harness resume command. No prompt is sent. +func LaunchResume(agent, harnessSessionID, model string) error { + if strings.TrimSpace(harnessSessionID) == "" { + return g.Error("missing harness session id") + } + plan := agentResumeArgs(agent, harnessSessionID, model) + return startAgent(agent, plan, "") +} + +func startAgent(agent string, plan agentLaunchPlan, prompt string) error { + binary, err := lookPath(agent) + if err != nil { + return err + } + + args := plan.Args + useStdin := plan.UseStdin + + procAttr := &os.ProcAttr{ + Files: []*os.File{os.Stdin, os.Stdout, os.Stderr}, + Env: os.Environ(), + } + if useStdin { + pr, pw, err := os.Pipe() + if err != nil { + return err + } + go func() { + defer pw.Close() + _, _ = pw.Write([]byte(prompt)) + }() + procAttr.Files = []*os.File{pr, os.Stdout, os.Stderr} + argv := append([]string{binary}, args...) + proc, err := os.StartProcess(binary, argv, procAttr) + if err != nil { + return err + } + state, err := proc.Wait() + pr.Close() + if err != nil { + return err + } + return finishAgentExit(state, agent) + } + + argv := append([]string{binary}, args...) + proc, err := os.StartProcess(binary, argv, procAttr) + if err != nil { + return err + } + state, err := proc.Wait() + if err != nil { + return err + } + return finishAgentExit(state, agent) +} + +// AgentExitError is returned when a launched CLI agent exits non-zero. +type AgentExitError struct { + ExitCode int + Agent string +} + +func (e *AgentExitError) Error() string { + if e == nil { + return "agent exited with error" + } + if e.Agent != "" { + return fmt.Sprintf("agent %q exited with code %d", e.Agent, e.ExitCode) + } + return fmt.Sprintf("agent exited with code %d", e.ExitCode) +} + +// ExitCodeOf returns (code, true) when err is or wraps an AgentExitError. +func ExitCodeOf(err error) (int, bool) { + var ae *AgentExitError + if errors.As(err, &ae) && ae != nil { + return ae.ExitCode, true + } + return 0, false +} + +func finishAgentExit(state *os.ProcessState, agent string) error { + if state.Success() { + return nil + } + code := 1 + if ws, ok := state.Sys().(syscall.WaitStatus); ok { + code = ws.ExitStatus() + } + return &AgentExitError{ExitCode: code, Agent: agent} +} + +func lookPath(name string) (string, error) { + bin := agentBinary(name) + if bin == "opencode" { + p, err := EnsureBinOpenCode() + if err != nil { + return "", g.Error(err, "agent %q not found on $PATH", name) + } + return p, nil + } + p, err := exec.LookPath(bin) + if err != nil { + return "", g.Error(err, "agent %q not found on $PATH", name) + } + return p, nil +} + +func harnessSessionRoot(agent string) string { + home := userHome() + switch agent { + case "claude": + return filepath.Join(home, ".claude", "projects") + case "codex": + return filepath.Join(home, ".codex", "sessions") + case "gemini": + return filepath.Join(home, ".gemini", "tmp") + case "cursor": + return filepath.Join(home, ".cursor") + case "opencode": + dataHome := os.Getenv("XDG_DATA_HOME") + if dataHome == "" { + dataHome = filepath.Join(home, ".local", "share") + } + return filepath.Join(dataHome, "opencode") + case "pi": + return filepath.Join(home, ".pi", "agent", "sessions") + case "grok": + return filepath.Join(home, ".grok", "sessions") + default: + return "" + } +} + +func snapshotHarnessFiles(agent string) map[string]time.Time { + root := harnessSessionRoot(agent) + out := map[string]time.Time{} + if root == "" { + return out + } + _ = filepath.Walk(root, func(p string, info os.FileInfo, err error) error { + if err != nil || info == nil || info.IsDir() { + return nil + } + rel, rerr := filepath.Rel(root, p) + if rerr != nil { + rel = p + } + out[rel] = info.ModTime() + return nil + }) + return out +} + +func discoverHarnessSessionID(agent string, before map[string]time.Time) string { + root := harnessSessionRoot(agent) + if root == "" { + return "" + } + if before == nil { + before = map[string]time.Time{} + } + var bestPath string + var bestTime time.Time + _ = filepath.Walk(root, func(p string, info os.FileInfo, err error) error { + if err != nil || info == nil || info.IsDir() { + return nil + } + rel, rerr := filepath.Rel(root, p) + if rerr != nil { + rel = p + } + prev, known := before[rel] + if known && !info.ModTime().After(prev) { + return nil + } + id := idFromSessionPath(p) + if id == "" { + return nil + } + if bestPath == "" || info.ModTime().After(bestTime) { + bestPath = p + bestTime = info.ModTime() + } + return nil + }) + if bestPath == "" { + return "" + } + return idFromSessionPath(bestPath) +} + +func idFromSessionPath(p string) string { + base := filepath.Base(p) + base = strings.TrimSuffix(base, filepath.Ext(base)) + if base == "" || strings.HasPrefix(base, ".") { + return "" + } + switch strings.ToLower(base) { + case "meta", "index", "config", "settings": + return "" + } + return base +} diff --git a/core/sling/assist/skills/agent-browser/CORE.md b/core/sling/assist/skills/agent-browser/CORE.md new file mode 100644 index 000000000..cf4433981 --- /dev/null +++ b/core/sling/assist/skills/agent-browser/CORE.md @@ -0,0 +1,516 @@ +--- +name: core +description: Core agent-browser usage guide. Read this before running any agent-browser commands. Covers the snapshot-and-ref workflow, navigating pages, interacting with elements (click, fill, type, select), extracting text and data, taking screenshots, managing tabs, handling forms and auth, waiting for content, running multiple browser sessions in parallel, and troubleshooting common failures. Use when the user asks to interact with a website, fill a form, click something, extract data, take a screenshot, log into a site, test a web app, or automate any browser task. +allowed-tools: Bash(agent-browser:*), Bash(npx agent-browser:*) +--- + +# agent-browser core + +Fast browser automation CLI for AI agents. Chrome/Chromium via CDP, no Playwright or Puppeteer dependency. Accessibility-tree snapshots with compact `@eN` refs let agents interact with pages in ~200-400 tokens instead of parsing raw HTML. + +Most normal web tasks (navigate, read, click, fill, extract, screenshot) are covered here. Load a specialized skill when the task falls outside browser web pages — see [When to load another skill](#when-to-load-another-skill). + +## The core loop + +```bash +agent-browser open # 1. Open a page +agent-browser snapshot -i # 2. See what's on it (interactive elements only) +agent-browser click @e3 # 3. Act on refs from the snapshot +agent-browser snapshot -i # 4. Re-snapshot after any page change +``` + +Refs (`@e1`, `@e2`, ...) are assigned fresh on every snapshot. They become **stale the moment the page changes** — after clicks that navigate, form submits, dynamic re-renders, dialog opens. Always re-snapshot before your next ref interaction. + +## Always use your own session + +Before your first command, set a named session for the whole task: + +```bash +export AGENT_BROWSER_SESSION="$(agent-browser session id --scope worktree --prefix task)" +``` + +The default (unnamed) session is a single shared browser: it is shared with every other agent on the machine and it persists across conversations, so working in it can hijack another agent's page mid-task or navigate away from something the human left open. Every example below assumes a named session is active. See [Run multiple browsers in parallel](#run-multiple-browsers-in-parallel) and `references/session-management.md`. + +## Quickstart + +```bash +# Install once +npm i -g agent-browser && agent-browser install + +# Linux hosts can install required browser libraries too +agent-browser install --with-deps + +# Take a screenshot of a page +agent-browser open https://example.com +agent-browser screenshot home.png +agent-browser close + +# Search, click a result, and capture it +agent-browser open https://duckduckgo.com +agent-browser snapshot -i # find the search box ref +agent-browser fill @e1 "agent-browser cli" +agent-browser press Enter +agent-browser wait --load networkidle +agent-browser snapshot -i # refs now reflect results +agent-browser click @e5 # click a result +agent-browser screenshot result.png +``` + +The browser stays running across commands so these feel like a single session. By default, an inactive daemon saves configured restore state, closes its headless browser, and exits after one hour; the next command starts it again. Without `--restore` or another restore key, shutdown discards transient browser state and open tabs. Dashboard mouse, keyboard, and touch input count as activity. Headed browsers, Safari and iOS WebDriver sessions, and user-attached browsers are exempt from the default; provider-owned cloud browsers are not. Use `--idle-timeout