diff --git a/README.md b/README.md index 39cd7dd6f..16eee31d1 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Omnigraph is engineered for the new workload introduced by long-horizon agents: | **Branching** | Agents propose reviewable changes instead of writing to main: a branch per agent or task, three-way merge, time travel across the whole graph. | | **Declared as code** | A `cluster.yaml` declares graphs, schemas, stored queries, embedding providers and policies. `cluster plan` previews, `cluster apply` converges, Terraform-style. | | **Security as code** | Cedar policy enforced **server-side on every mutation**, per graph or server-wide; bearer auth, server-resolved actors, audit tracking. | -| **Object-storage native, open format** | Scales at the lowest cost on local disk or any S3-compatible store: RustFS / MinIO, AWS S3 / R2 / GCS, Azure preview. Data lives in open, versioned [Lance](https://github.com/lance-format/lance) with native blob-as-data, and never leaves your store. | +| **Object-storage native, open format** | Scales at the lowest cost on local disk, AWS S3 or a self-hosted S3-compatible store such as RustFS, with Azure in preview. Data lives in open, versioned [Lance](https://github.com/lance-format/lance) with native blob-as-data, and never leaves your store. | ## What you can build @@ -137,9 +137,11 @@ policies: **2. Stand up your object store.** On-prem, run RustFS (or MinIO); Omnigraph writes [Lance](https://github.com/lance-format/lance) over the standard S3 API. -In the cloud, use S3 / R2 / GCS through `AWS_*`. Native `az://` roots are also -available as a qualification preview. Every Azure writer, including -`cluster apply` and the server, must use the checked-in admission wrapper; +In the cloud, use AWS S3 through `AWS_*`; other S3-compatible services must +support [conditional writes](docs/user/concepts/storage.md#supported-graph-uris). +Native `az://` roots are also available as a qualification preview. Every +Azure writer, including `cluster apply` and the server, must use the +checked-in admission wrapper; follow the [Azure deployment guide](docs/user/deployment.md#azure-blob-preview) instead of running the bare commands below. diff --git a/docs/user/concepts/storage.md b/docs/user/concepts/storage.md index cd2732775..8fb36464e 100644 --- a/docs/user/concepts/storage.md +++ b/docs/user/concepts/storage.md @@ -13,6 +13,15 @@ datasets directly. | `s3://bucket/prefix/graph.omni` | Amazon S3 or an S3-compatible service | | `az://container/prefix/graph.omni` | Azure Blob Storage qualification preview | +Other schemes, including `gs://`, are refused; there is no native Google Cloud +Storage support. Writers rely on conditional writes: the store must refuse to +create an object that already exists (`If-None-Match: *`) and refuse to replace +one whose ETag has changed (`If-Match`). Amazon S3 supports both, and CI tests +the `s3://` path against RustFS. Other S3-compatible services, such as +Cloudflare R2 or Google Cloud Storage through its S3-compatible API, are not +qualified. Confirm that a service enforces both conditions before you write to +it: one that ignores them can let two writers overwrite each other's commits. + An `http://` or `https://` URL addresses an OmniGraph server; it is not a graph storage URI. CLI commands use `--target NAME|ADDRESS` for both server and direct-storage destinations; the selected kind determines command availability. diff --git a/docs/user/deployment.md b/docs/user/deployment.md index e50a21d87..e06365c08 100644 --- a/docs/user/deployment.md +++ b/docs/user/deployment.md @@ -210,7 +210,8 @@ docker run --rm -p 8080:8080 \ ``` Terminate TLS at a load balancer or trusted reverse proxy. Keep bearer tokens -and storage credentials in the platform's secret store. +and storage credentials in the platform's secret store. For encryption at rest +and data sent to external services, see [data protection](operations/data-protection.md). ## Unix-socket access @@ -331,9 +332,8 @@ Configure tokens from environment or a mounted secret file: - `OMNIGRAPH_SERVER_BEARER_TOKENS_FILE` - `OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET` in the AWS-enabled build -Policy bundles come from the applied cluster. See -[HTTP server](operations/server.md) and -[Authorization and actors](operations/policy.md). +Policy bundles come from the applied cluster. See [HTTP server](operations/server.md) +and [Authorization and actors](operations/policy.md). ## Upgrade and backup diff --git a/docs/user/index.md b/docs/user/index.md index f5c22a35c..a00157d6b 100644 --- a/docs/user/index.md +++ b/docs/user/index.md @@ -46,6 +46,7 @@ deployment. - [Deployment](deployment.md) - [Storage failures](operations/storage-failures.md) - [Authorization and actors](operations/policy.md) +- [Data protection](operations/data-protection.md) - [Maintenance](operations/maintenance.md) - [Storage-format upgrades](operations/upgrade.md) diff --git a/docs/user/operations/data-protection.md b/docs/user/operations/data-protection.md new file mode 100644 index 000000000..d68e725f3 --- /dev/null +++ b/docs/user/operations/data-protection.md @@ -0,0 +1,44 @@ +# Data protection + +OmniGraph does not encrypt what it writes. It has no client-side or field-level +encryption: graph data, history, managed Blob bytes, schemas, stored queries and +policy bundles are stored in the cluster root as written. Protecting them at +rest and in transit is the job of the storage service and the network in front +of the server. + +## Encryption at rest + +Use the storage service's encryption: + +- **Amazon S3.** Default encryption on the bucket (SSE-S3 or SSE-KMS) applies + to every object OmniGraph writes, with no OmniGraph setting. With a + customer-managed KMS key, the credentials OmniGraph uses also need permission + to use that key. Per-request settings such as `AWS_SERVER_SIDE_ENCRYPTION` + are not needed, and CLI [targets](../cli/targets.md) refuse them. +- **Azure Blob Storage.** Azure Storage encrypts Blob data at rest by default. +- **Local filesystem.** Put the root on an encrypted volume. + +## Encryption in transit + +`omnigraph-server` serves plain HTTP. Terminate TLS in front of it, as +described under [Container](../deployment.md#container). Storage traffic uses +HTTPS unless you configure an `http://` endpoint for local development. + +## Data sent to external services + +The only external service OmniGraph sends text to is the embedding provider. A +text `nearest` query sends its query text, and the offline `omnigraph embed` +command sends the values it embeds. Stored graph data is never sent. See +[network egress](../search/embeddings.md#network-egress) for when a provider is +called and how a graph selects one. + +## Erasure + +Deleting data or dropping a property changes the current graph only. Earlier +commits keep the old values readable until [cleanup](maintenance.md#cleanup) +removes them, and the bytes can remain in storage files after that. To erase a +dropped property's values, follow the procedure in +[schema changes](../schema/index.md#schema-changes). + +This is not secure erasure. Object-store versioning, retention policies and +backups can keep copies that OmniGraph does not manage. diff --git a/docs/user/search/embeddings.md b/docs/user/search/embeddings.md index 88954373b..b0987ecbc 100644 --- a/docs/user/search/embeddings.md +++ b/docs/user/search/embeddings.md @@ -57,6 +57,28 @@ graphs: Inline API keys are rejected. `${ENV_VAR}` references are resolved when the server starts, not when the cluster configuration is planned or applied. +## Network egress + +OmniGraph calls the embedding provider only to turn text into a vector: + +- a `nearest` query whose argument is text sends that text; +- `omnigraph embed` sends each source value it embeds. + +Loads, mutations and `nearest` queries given a vector never call a provider; +`@embed` does not populate vectors during a load +([schema annotation](#schema-annotation)). + +A cluster-served graph uses the provider its `embedding_provider` names. A +served graph without one, and every direct or embedded use, resolves the +provider from the process environment as described above; with nothing set, +that is OpenRouter. When the provider's API key is missing, the call fails +before any request is sent. + +No setting turns providers off. `mock` makes no network calls, but it answers +text queries with deterministic vectors that are not comparable with real +ones. A text `nearest` on a property whose `@embed` records a `model` is +refused under `mock` unless `OMNIGRAPH_EMBED_MODEL` names that model. + ## Schema annotation Associate a vector with its source text: