From 07c7efec5e14b7993d55bf1e75babb97fb3b7865 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Wed, 13 May 2026 14:24:18 -0600 Subject: [PATCH 01/22] added first draft for targetsource implementation docs --- docs/content/docs/user-guide/targetsource.md | 152 ++++++++----------- 1 file changed, 62 insertions(+), 90 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource.md b/docs/content/docs/user-guide/targetsource.md index d1fee0c..a3a3eb1 100644 --- a/docs/content/docs/user-guide/targetsource.md +++ b/docs/content/docs/user-guide/targetsource.md @@ -10,15 +10,11 @@ The `TargetSource` resource enables dynamic discovery of network devices from ex ## Discovery Sources -TargetSource supports multiple discovery backends: +TargetSource supports the following discovery providers: | Source | Description | |--------|-------------| | `http` | Fetch targets from an HTTP endpoint | -| `consul` | Discover targets from Consul service registry | -| `configMap` | Read targets from a Kubernetes ConfigMap | -| `podSelector` | Create targets from Kubernetes Pods | -| `serviceSelector` | Create targets from Kubernetes Services | ## HTTP Discovery @@ -30,92 +26,55 @@ kind: TargetSource metadata: name: http-discovery spec: - http: - url: http://inventory-service:8080/targets - labels: + provider: + http: + url: http://inventory-service:8080/targets + targetProfile: default + targetLabels: source: inventory ``` -The HTTP endpoint should return a JSON array of target objects. - -## Consul Discovery - -Discover targets from Consul service registry: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: consul-discovery -spec: - consul: - url: http://consul:8500 - labels: - source: consul - datacenter: dc1 +The HTTP endpoint should return a JSON array of target objects. The following is an example for a valid JSON array: + +```json +[ + { + "address": "spine1:57400", + "name": "spine1", + "labels": { + "role": "spine" + } + }, + { + "address": "leaf1:57400", + "name": "leaf1", + "labels": { + "role": "leaf" + } + }, + { + "address": "leaf2:57400", + "name": "leaf2", + "labels": { + "role": "leaf" + } + } +] ``` -## ConfigMap Discovery +## TargetProfile Inheritance -Read targets from a Kubernetes ConfigMap: +Within the `TargetSource`, the default `TargetProfile` for all targets can be defined using `targetProfile`. Each target discovered inherits the defined value. -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: configmap-targets -spec: - configMap: network-devices - labels: - source: configmap -``` - -The ConfigMap should contain target definitions in a structured format. - -## Kubernetes Pod Discovery - -Create targets from Kubernetes Pods matching a label selector: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: pod-discovery -spec: - podSelector: - matchLabels: - app: network-simulator - gnmi: enabled - labels: - source: kubernetes - type: simulator -``` - -This is useful for: -- Containerized network simulators -- Virtual network functions (VNFs) -- Development/testing environments - -## Kubernetes Service Discovery +## Label Inheritance -Create targets from Kubernetes Services matching a label selector: +Each discovered target has a label defined to identify the owning `TargetSource`: +- `operator.gnmic.dev/targetsource: datacenter-a` -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: service-discovery -spec: - serviceSelector: - matchLabels: - protocol: gnmi - labels: - source: kubernetes -``` -## Label Inheritance +This label is needed to identify all targets owned by this resource and determine which devices get applied or removed. This label takes precedence over all other labels on the target. -Labels defined in the `TargetSource.spec.labels` field are applied to all discovered targets: +Labels defined in the `TargetSource.spec.targetLabels` field are applied to all discovered targets: ```yaml apiVersion: operator.gnmic.dev/v1alpha1 @@ -123,21 +82,33 @@ kind: TargetSource metadata: name: datacenter-a spec: - consul: - url: http://consul-dc-a:8500 - labels: + provider: + http: + url: http://datacenter-a:8080/targets + targetLabels: datacenter: dc-a environment: production - source: consul ``` All targets discovered from this source will have: - `datacenter: dc-a` - `environment: production` -- `source: consul` This enables using label selectors in Pipelines to select targets by their discovery source. +## Labels from Source of Truth + +Targets can also have labels defined by the external system. These get directly applied to the target with their original key/value pair. + +The gNMIc Operator has a reserved namespace for labels which alter the behavior of the target: +- `gnmic_operator_` + +Following are all supported operator-specific labels: + +| Label | Description | +|--------|-------------| +| `gnmic_operator_target_profile` | Overwrite the `TargetProfile` which is defined in the `TargetSource` | + ## Status The TargetSource status shows discovery state: @@ -155,7 +126,7 @@ status: | `targetsCount` | Number of targets discovered | | `lastSync` | Timestamp of last successful sync | -## Example: Multi-Source Discovery + ## Lifecycle @@ -219,12 +190,13 @@ spec: When a TargetSource discovers a new device: 1. A new `Target` resource is created -2. Labels from `spec.labels` are applied -3. Owner reference is set to the TargetSource +2. The `Profile` gets specified from `spec.targetProfile` +3. Labels from `spec.targetLabels` are applied +4. Owner reference is set to the TargetSource ### Target Updates -When a discovered device's properties change: +Discovered devices get reapplied each time the target gets discovered, overwriting any changes manually made: 1. The corresponding `Target` is updated 2. Clusters using that target are reconciled From 8bab96ce2719903fc42deed051220557ce764bd4 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Wed, 13 May 2026 15:46:47 -0600 Subject: [PATCH 02/22] aligning doc with existing pages --- docs/content/docs/user-guide/targetsource.md | 143 +++++++++++++------ 1 file changed, 96 insertions(+), 47 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource.md b/docs/content/docs/user-guide/targetsource.md index a3a3eb1..5895a79 100644 --- a/docs/content/docs/user-guide/targetsource.md +++ b/docs/content/docs/user-guide/targetsource.md @@ -8,53 +8,86 @@ description: > The `TargetSource` resource enables dynamic discovery of network devices from external sources. The operator automatically creates, updates, and deletes `Target` resources based on discovered devices. -## Discovery Sources - -TargetSource supports the following discovery providers: - -| Source | Description | -|--------|-------------| -| `http` | Fetch targets from an HTTP endpoint | - -## HTTP Discovery - -Discover targets from an HTTP endpoint that returns a JSON list of targets: +## Basic Configuration ```yaml apiVersion: operator.gnmic.dev/v1alpha1 kind: TargetSource metadata: - name: http-discovery + name: targetsource-1 spec: provider: - http: - url: http://inventory-service:8080/targets + # see Discovery Providers section targetProfile: default targetLabels: source: inventory ``` -The HTTP endpoint should return a JSON array of target objects. The following is an example for a valid JSON array: +## Spec Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `provider` | object | Yes | Provider-specific discovery configuration. Exactly one provider must be configured | +| `targetProfile` | string | Yes | Reference to `TargetProfile` applied to all targets | +| `targetLabels` | map[string]string | No | Labels added to all discovered targets | + + +## Discovery Providers + +`TargetSource` supports the following discovery providers: + +| Provider | Description | +|----------|-------------| +| `http` | Discover targets from an HTTP JSON endpoint | + +### HTTP Provider + +The HTTP provider discovers targets from an HTTP endpoint returning a JSON array of target definitions. + +```yaml +spec: + provider: + http: + url: http://inventory-service:8080/targets +``` + +#### HTTP Spec Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `url` | string | Yes | URL pointing to the inventory server | + +#### Response Format + +The endpoint must return a JSON array of objects with the following structure: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | Name of the generated `Target` resource | +| `address` | string | Yes | Device address in `host:port` format | +| `labels` | map[string]string | No | Labels added to the generated `Target` resource | + +Example response: ```json [ { - "address": "spine1:57400", "name": "spine1", + "address": "spine1:57400", "labels": { "role": "spine" } }, { - "address": "leaf1:57400", "name": "leaf1", + "address": "leaf1:57400", "labels": { "role": "leaf" } }, { - "address": "leaf2:57400", "name": "leaf2", + "address": "leaf2:57400", "labels": { "role": "leaf" } @@ -62,29 +95,32 @@ The HTTP endpoint should return a JSON array of target objects. The following is ] ``` -## TargetProfile Inheritance - -Within the `TargetSource`, the default `TargetProfile` for all targets can be defined using `targetProfile`. Each target discovered inherits the defined value. - ## Label Inheritance -Each discovered target has a label defined to identify the owning `TargetSource`: -- `operator.gnmic.dev/targetsource: datacenter-a` +Each generated `Target` receives an ownership label identifying the originating `TargetSource`: +```yaml +operator.gnmic.dev/targetsource: targetsource-1 +``` + +This label is automatically managed by the operator and is used to: +- Identify targets owned by a specific `TargetSource` +- Determine which targets should be updated or deleted during reconciliation +The `operator.gnmic.dev/targetsource` label is reserved and always takes precedence over any provider-supplied labels. -This label is needed to identify all targets owned by this resource and determine which devices get applied or removed. This label takes precedence over all other labels on the target. +### TargetSource Labels -Labels defined in the `TargetSource.spec.targetLabels` field are applied to all discovered targets: +Additional labels can be applied to all generated targets using `spec.targetLabels`: ```yaml apiVersion: operator.gnmic.dev/v1alpha1 kind: TargetSource metadata: - name: datacenter-a + name: targetsource-1 spec: provider: http: - url: http://datacenter-a:8080/targets + url: http://targetsource-1:8080/targets targetLabels: datacenter: dc-a environment: production @@ -94,20 +130,27 @@ All targets discovered from this source will have: - `datacenter: dc-a` - `environment: production` -This enables using label selectors in Pipelines to select targets by their discovery source. +This enables Pipelines to select targets using label selectors. -## Labels from Source of Truth +### Labels from Discovery Providers -Targets can also have labels defined by the external system. These get directly applied to the target with their original key/value pair. +Discovery providers may return additional labels for each target. These labels are applied directly to the generated `Target` resource. -The gNMIc Operator has a reserved namespace for labels which alter the behavior of the target: -- `gnmic_operator_` +The `gnmic_operator_` label prefix is reserved for operator-specific behavior. Labels using this prefix are interpreted by the operator and are not applied directly to the generated `Target` resource. -Following are all supported operator-specific labels: +Supported operator labels: | Label | Description | |--------|-------------| -| `gnmic_operator_target_profile` | Overwrite the `TargetProfile` which is defined in the `TargetSource` | +| `gnmic_operator_target_profile` | Overrides the `TargetProfile` configured in the `TargetSource` | + +### Label Precedence + +If the same label key is defined in multiple places, labels are applied in the following order (highest precedence first): + +1. `TargetSource` ownership label (`operator.gnmic.dev/targetsource`) +2. Labels from `TargetSource.spec.targetLabels` +3. Labels returned by the discovery provider ## Status @@ -188,27 +231,33 @@ spec: ### Target Creation -When a TargetSource discovers a new device: +When a `TargetSource` discovers a new device: + 1. A new `Target` resource is created -2. The `Profile` gets specified from `spec.targetProfile` +2. The `TargetProfile` referenced in `spec.targetProfile` is assigned 3. Labels from `spec.targetLabels` are applied -4. Owner reference is set to the TargetSource +4. The `TargetSource` is set as the owner reference ### Target Updates -Discovered devices get reapplied each time the target gets discovered, overwriting any changes manually made: -1. The corresponding `Target` is updated -2. Clusters using that target are reconciled +On each discovery cycle, existing `Target` resources are reconciled with the latest discovered state: + +1. The corresponding `Target` resource is updated and overwritten +2. Clusters consuming the target are reconciled automatically + +> Manual changes to `Target` resources managed by a `TargetSource` are overwritten on every reconciliation cycle. ### Target Deletion -When a device is no longer discovered: -1. The `Target` resource is deleted -2. Clusters stop collecting from that target +When a device is no longer returned by the discovery provider: + +1. The corresponding `Target` resource is deleted +2. Clusters automatically stop using the target ### TargetSource Deletion -When a TargetSource is deleted: -1. All Targets owned by it are deleted (via owner references) -2. Clusters are reconciled to remove those targets +When a `TargetSource` is deleted: + +1. All `Target` resources owned by it are deleted via owner references +2. Clusters are reconciled and remove the deleted targets From 2d1f5fb28a113511781e0c6623f676981b8a6db5 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Wed, 20 May 2026 19:31:17 -0600 Subject: [PATCH 03/22] updated targetSource guide based on new CRD --- docs/content/docs/user-guide/targetsource.md | 176 ++++++++++++++++++- 1 file changed, 170 insertions(+), 6 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource.md b/docs/content/docs/user-guide/targetsource.md index 5895a79..896b405 100644 --- a/docs/content/docs/user-guide/targetsource.md +++ b/docs/content/docs/user-guide/targetsource.md @@ -18,6 +18,7 @@ metadata: spec: provider: # see Discovery Providers section + targetPort: 57400 targetProfile: default targetLabels: source: inventory @@ -28,6 +29,7 @@ spec: | Field | Type | Required | Description | |-------|------|----------|-------------| | `provider` | object | Yes | Provider-specific discovery configuration. Exactly one provider must be configured | +| `targetPort` | int32 | No | Default port used if the discovered target does not provide a port. | | `targetProfile` | string | Yes | Reference to `TargetProfile` applied to all targets | | `targetLabels` | map[string]string | No | Labels added to all discovered targets | @@ -49,6 +51,7 @@ spec: provider: http: url: http://inventory-service:8080/targets + ``` #### HTTP Spec Fields @@ -56,16 +59,173 @@ spec: | Field | Type | Required | Description | |-------|------|----------|-------------| | `url` | string | Yes | URL pointing to the inventory server | +| `acceptPush` | bool | No | Enable webhook-based target updates. Defaults to `false`. | +| `authorization` | object | No | Credentials used to access the HTTP endpoint. See _Authorization_ section. | +| `pollInterval` | metav1.Duration | No | Polling interval used to fetch targets from the endpoint. Defaults to `30s`. | +| `timeout` | metav1.Duration | No | Timeout for HTTP requests. Defaults to `10s`. | +| `tls` | object | No | Client TLS configuration for HTTPS endpoints. See _TLS_ section. | +| `pagination` | object | No | Pagination configuration for parsing responses from the HTTP endpoint. See _Pagination_ section. | +| `responseMapping` | object | No | JSON path mapping definitions. See _Response Mapping_ section. | + +##### Authorization + +The HTTP provider supports authenticated requests to the inventory endpoint. + +Exactly one authorization method can be configured. + +###### Basic Authentication + +Credentials can either be defined inline or referenced from a Secret. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + authorization: + basic: + username: admin + password: secret +``` + +Using a Secret reference: + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + authorization: + basic: + credentialsSecretRef: + name: inventory-credentials + key: username +``` + +###### Token Authentication + +Static token authentication can be configured using either an inline token or a Secret reference. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + authorization: + token: + scheme: Bearer + token: eyJhbGciOi... +``` + +Using a Secret reference: + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + authorization: + token: + scheme: Bearer + tokenSecretRef: + name: inventory-token + key: token +``` + +##### TLS + +TLS settings can be configured for HTTPS endpoints. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + tls: + insecureSkipVerify: false +``` + +###### TLS Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `insecureSkipVerify` | bool | No | Skip verification of the server certificate. Defaults to `false`. | +| `caBundle` | []byte | No | Base64-encoded PEM CA bundle used to validate the server certificate. | +| `caBundleSecretRef` | object | No | Reference to a Secret containing a PEM CA bundle. | + +`caBundle` and `caBundleSecretRef` are mutually exclusive. + +##### Pagination + +Pagination can be configured for APIs returning paginated responses. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/devices + pagination: + itemsField: results + nextField: next +``` + +###### Pagination Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `itemsField` | string | No | Top-level JSON field containing the list of target objects. | +| `nextField` | string | No | Top-level JSON field containing the next page reference or pagination token. | + +The `nextField` value may either contain: +- A full URL for the next request +- A pagination token appended as a query parameter to the original URL + +##### Response Mapping + +By default, the HTTP response must follow the structure defined in the _Response Format_ section. + +`responseMapping` allows extracting target fields from arbitrary JSON structures using JSONPath expressions. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/devices + responseMapping: + name: "$.hostname" + ip: "$.management.ip" + port: "$.gnmi.port" + targetProfile: "$.profile" + labels: + role: "$.metadata.role" + site: "$.metadata.site" +``` + +###### Response Mapping Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | JSONPath expression extracting the target name. | +| `ip` | string | Yes | JSONPath expression extracting the target IP address or hostname. | +| `port` | string | No | JSONPath expression extracting the gNMI port. | +| `targetProfile` | string | No | JSONPath expression extracting the `TargetProfile`. | +| `labels` | map[string]string | No | JSONPath expressions extracting target labels. | + +Labels extracted through `responseMapping.labels` are merged with labels from `spec.targetLabels`. + +If the same label key exists in both locations, labels extracted through `responseMapping.labels` take precedence. #### Response Format -The endpoint must return a JSON array of objects with the following structure: +If `responseMapping` is not configured, the endpoint must return a JSON array of objects with the following structure: | Field | Type | Required | Description | |-------|------|----------|-------------| | `name` | string | Yes | Name of the generated `Target` resource | -| `address` | string | Yes | Device address in `host:port` format | +| `address` | string | Yes | Device address (FQDN or IP address) | +| `port` | int32 | No | Port used for gNMI connections. If omitted, `spec.targetPort` is used. | | `labels` | map[string]string | No | Labels added to the generated `Target` resource | +| `targetProfile` | string | No | Reference to a `TargetProfile`. If omitted, `spec.targetProfile` is used. | Example response: @@ -73,21 +233,25 @@ Example response: [ { "name": "spine1", - "address": "spine1:57400", + "address": "spine1", + "port": 57400, "labels": { "role": "spine" - } + }, + "targetProfile": "spine-profile" }, { "name": "leaf1", - "address": "leaf1:57400", + "address": "leaf1", + "port": 57400, "labels": { "role": "leaf" } }, { "name": "leaf2", - "address": "leaf2:57400", + "address": "leaf2", + "port": 57400, "labels": { "role": "leaf" } From abbbee4d33aa4dc5d7c7a5d6098c7bdaafb178fc Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Wed, 20 May 2026 20:38:47 -0600 Subject: [PATCH 04/22] renamed response field ip to address --- docs/content/docs/user-guide/targetsource.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/docs/user-guide/targetsource.md b/docs/content/docs/user-guide/targetsource.md index 896b405..9a699ff 100644 --- a/docs/content/docs/user-guide/targetsource.md +++ b/docs/content/docs/user-guide/targetsource.md @@ -193,7 +193,7 @@ spec: url: https://inventory.example.com/devices responseMapping: name: "$.hostname" - ip: "$.management.ip" + address: "$.management.ip" port: "$.gnmi.port" targetProfile: "$.profile" labels: From 21496fc079150a2d74784f65c72533bf70b751b9 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Wed, 20 May 2026 20:48:43 -0600 Subject: [PATCH 05/22] moved http into a subfile of TargetSource --- .../docs/user-guide/targetsource/_index.md | 211 ++++++++++++++++ .../{targetsource.md => targetsource/http.md} | 233 ++---------------- 2 files changed, 225 insertions(+), 219 deletions(-) create mode 100644 docs/content/docs/user-guide/targetsource/_index.md rename docs/content/docs/user-guide/{targetsource.md => targetsource/http.md} (50%) diff --git a/docs/content/docs/user-guide/targetsource/_index.md b/docs/content/docs/user-guide/targetsource/_index.md new file mode 100644 index 0000000..16861e7 --- /dev/null +++ b/docs/content/docs/user-guide/targetsource/_index.md @@ -0,0 +1,211 @@ +--- +title: "TargetSource" +linkTitle: "TargetSource" +weight: 4 +description: > + Dynamic target discovery from external sources +--- + +The `TargetSource` resource enables dynamic discovery of network devices from external sources. The operator automatically creates, updates, and deletes `Target` resources based on discovered devices. + +## Basic Configuration + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: targetsource-1 +spec: + provider: + # see Discovery Providers section + targetPort: 57400 + targetProfile: default + targetLabels: + source: inventory +``` + +## Spec Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `provider` | object | Yes | Provider-specific discovery configuration. Exactly one provider must be configured | +| `targetPort` | int32 | No | Default port used if the discovered target does not provide a port. | +| `targetProfile` | string | Yes | Reference to `TargetProfile` applied to all targets | +| `targetLabels` | map[string]string | No | Labels added to all discovered targets | + + +## Discovery Providers + +`TargetSource` supports the following discovery providers: + +| Provider | Description | +|----------|-------------| +| `http` | Discover targets from an HTTP JSON endpoint. [Configuration]({{< relref "http.md" >}}) | + + +## Label Inheritance + +Each generated `Target` receives an ownership label identifying the originating `TargetSource`: +```yaml +operator.gnmic.dev/targetsource: targetsource-1 +``` + +This label is automatically managed by the operator and is used to: +- Identify targets owned by a specific `TargetSource` +- Determine which targets should be updated or deleted during reconciliation + +The `operator.gnmic.dev/targetsource` label is reserved and always takes precedence over any provider-supplied labels. + +### TargetSource Labels + +Additional labels can be applied to all generated targets using `spec.targetLabels`: + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: targetsource-1 +spec: + provider: + http: + url: http://targetsource-1:8080/targets + targetLabels: + datacenter: dc-a + environment: production +``` + +All targets discovered from this source will have: +- `datacenter: dc-a` +- `environment: production` + +This enables Pipelines to select targets using label selectors. + +### Labels from Discovery Providers + +Discovery providers may return additional labels for each target. These labels are applied directly to the generated `Target` resource. + +The `gnmic_operator_` label prefix is reserved for operator-specific behavior. Labels using this prefix are interpreted by the operator and are not applied directly to the generated `Target` resource. + +Supported operator labels: + +| Label | Description | +|--------|-------------| +| `gnmic_operator_target_profile` | Overrides the `TargetProfile` configured in the `TargetSource` | + +### Label Precedence + +If the same label key is defined in multiple places, labels are applied in the following order (highest precedence first): + +1. `TargetSource` ownership label (`operator.gnmic.dev/targetsource`) +2. Labels from `TargetSource.spec.targetLabels` +3. Labels returned by the discovery provider + +## Status + +The TargetSource status shows discovery state: + +```yaml +status: + status: Synced + targetsCount: 42 + lastSync: "2024-01-15T10:30:00Z" +``` + +| Field | Description | +|-------|-------------| +| `status` | Current sync status (Synced, Error, Pending) | +| `targetsCount` | Number of targets discovered | +| `lastSync` | Timestamp of last successful sync | + + + +## Lifecycle + +### Target Creation + +When a `TargetSource` discovers a new device: + +1. A new `Target` resource is created +2. The `TargetProfile` referenced in `spec.targetProfile` is assigned +3. Labels from `spec.targetLabels` are applied +4. The `TargetSource` is set as the owner reference + +### Target Updates + +On each discovery cycle, existing `Target` resources are reconciled with the latest discovered state: + +1. The corresponding `Target` resource is updated and overwritten +2. Clusters consuming the target are reconciled automatically + +> Manual changes to `Target` resources managed by a `TargetSource` are overwritten on every reconciliation cycle. + +### Target Deletion + +When a device is no longer returned by the discovery provider: + +1. The corresponding `Target` resource is deleted +2. Clusters automatically stop using the target + +### TargetSource Deletion + +When a `TargetSource` is deleted: + +1. All `Target` resources owned by it are deleted via owner references +2. Clusters are reconciled and remove the deleted targets + diff --git a/docs/content/docs/user-guide/targetsource.md b/docs/content/docs/user-guide/targetsource/http.md similarity index 50% rename from docs/content/docs/user-guide/targetsource.md rename to docs/content/docs/user-guide/targetsource/http.md index 9a699ff..9fea5d3 100644 --- a/docs/content/docs/user-guide/targetsource.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -1,49 +1,11 @@ --- -title: "TargetSource" -linkTitle: "TargetSource" +title: "HTTP Provider" +linkTitle: "HTTP" weight: 4 description: > - Dynamic target discovery from external sources + HTTP TargetSource Discovery Provider --- -The `TargetSource` resource enables dynamic discovery of network devices from external sources. The operator automatically creates, updates, and deletes `Target` resources based on discovered devices. - -## Basic Configuration - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: targetsource-1 -spec: - provider: - # see Discovery Providers section - targetPort: 57400 - targetProfile: default - targetLabels: - source: inventory -``` - -## Spec Fields - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `provider` | object | Yes | Provider-specific discovery configuration. Exactly one provider must be configured | -| `targetPort` | int32 | No | Default port used if the discovered target does not provide a port. | -| `targetProfile` | string | Yes | Reference to `TargetProfile` applied to all targets | -| `targetLabels` | map[string]string | No | Labels added to all discovered targets | - - -## Discovery Providers - -`TargetSource` supports the following discovery providers: - -| Provider | Description | -|----------|-------------| -| `http` | Discover targets from an HTTP JSON endpoint | - -### HTTP Provider - The HTTP provider discovers targets from an HTTP endpoint returning a JSON array of target definitions. ```yaml @@ -54,7 +16,7 @@ spec: ``` -#### HTTP Spec Fields +## HTTP Spec Fields | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -67,13 +29,13 @@ spec: | `pagination` | object | No | Pagination configuration for parsing responses from the HTTP endpoint. See _Pagination_ section. | | `responseMapping` | object | No | JSON path mapping definitions. See _Response Mapping_ section. | -##### Authorization +### Authorization The HTTP provider supports authenticated requests to the inventory endpoint. Exactly one authorization method can be configured. -###### Basic Authentication +#### Basic Authentication Credentials can either be defined inline or referenced from a Secret. @@ -102,7 +64,7 @@ spec: key: username ``` -###### Token Authentication +#### Token Authentication Static token authentication can be configured using either an inline token or a Secret reference. @@ -132,7 +94,7 @@ spec: key: token ``` -##### TLS +### TLS TLS settings can be configured for HTTPS endpoints. @@ -145,7 +107,7 @@ spec: insecureSkipVerify: false ``` -###### TLS Fields +#### TLS Fields | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -155,7 +117,7 @@ spec: `caBundle` and `caBundleSecretRef` are mutually exclusive. -##### Pagination +### Pagination Pagination can be configured for APIs returning paginated responses. @@ -169,7 +131,7 @@ spec: nextField: next ``` -###### Pagination Fields +#### Pagination Fields | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -180,7 +142,7 @@ The `nextField` value may either contain: - A full URL for the next request - A pagination token appended as a query parameter to the original URL -##### Response Mapping +### Response Mapping By default, the HTTP response must follow the structure defined in the _Response Format_ section. @@ -201,7 +163,7 @@ spec: site: "$.metadata.site" ``` -###### Response Mapping Fields +#### Response Mapping Fields | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -215,7 +177,7 @@ Labels extracted through `responseMapping.labels` are merged with labels from `s If the same label key exists in both locations, labels extracted through `responseMapping.labels` take precedence. -#### Response Format +## Response Format If `responseMapping` is not configured, the endpoint must return a JSON array of objects with the following structure: @@ -258,170 +220,3 @@ Example response: } ] ``` - -## Label Inheritance - -Each generated `Target` receives an ownership label identifying the originating `TargetSource`: -```yaml -operator.gnmic.dev/targetsource: targetsource-1 -``` - -This label is automatically managed by the operator and is used to: -- Identify targets owned by a specific `TargetSource` -- Determine which targets should be updated or deleted during reconciliation - -The `operator.gnmic.dev/targetsource` label is reserved and always takes precedence over any provider-supplied labels. - -### TargetSource Labels - -Additional labels can be applied to all generated targets using `spec.targetLabels`: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: targetsource-1 -spec: - provider: - http: - url: http://targetsource-1:8080/targets - targetLabels: - datacenter: dc-a - environment: production -``` - -All targets discovered from this source will have: -- `datacenter: dc-a` -- `environment: production` - -This enables Pipelines to select targets using label selectors. - -### Labels from Discovery Providers - -Discovery providers may return additional labels for each target. These labels are applied directly to the generated `Target` resource. - -The `gnmic_operator_` label prefix is reserved for operator-specific behavior. Labels using this prefix are interpreted by the operator and are not applied directly to the generated `Target` resource. - -Supported operator labels: - -| Label | Description | -|--------|-------------| -| `gnmic_operator_target_profile` | Overrides the `TargetProfile` configured in the `TargetSource` | - -### Label Precedence - -If the same label key is defined in multiple places, labels are applied in the following order (highest precedence first): - -1. `TargetSource` ownership label (`operator.gnmic.dev/targetsource`) -2. Labels from `TargetSource.spec.targetLabels` -3. Labels returned by the discovery provider - -## Status - -The TargetSource status shows discovery state: - -```yaml -status: - status: Synced - targetsCount: 42 - lastSync: "2024-01-15T10:30:00Z" -``` - -| Field | Description | -|-------|-------------| -| `status` | Current sync status (Synced, Error, Pending) | -| `targetsCount` | Number of targets discovered | -| `lastSync` | Timestamp of last successful sync | - - - -## Lifecycle - -### Target Creation - -When a `TargetSource` discovers a new device: - -1. A new `Target` resource is created -2. The `TargetProfile` referenced in `spec.targetProfile` is assigned -3. Labels from `spec.targetLabels` are applied -4. The `TargetSource` is set as the owner reference - -### Target Updates - -On each discovery cycle, existing `Target` resources are reconciled with the latest discovered state: - -1. The corresponding `Target` resource is updated and overwritten -2. Clusters consuming the target are reconciled automatically - -> Manual changes to `Target` resources managed by a `TargetSource` are overwritten on every reconciliation cycle. - -### Target Deletion - -When a device is no longer returned by the discovery provider: - -1. The corresponding `Target` resource is deleted -2. Clusters automatically stop using the target - -### TargetSource Deletion - -When a `TargetSource` is deleted: - -1. All `Target` resources owned by it are deleted via owner references -2. Clusters are reconciled and remove the deleted targets - From db090b3e6c8f5ff622e0677c5fe2fd0b6e211431 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Wed, 20 May 2026 21:00:38 -0600 Subject: [PATCH 06/22] restructured http chapter hierarchy --- .../docs/user-guide/targetsource/http.md | 88 ++++++++++--------- 1 file changed, 45 insertions(+), 43 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index 9fea5d3..b6ff8e4 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -13,7 +13,6 @@ spec: provider: http: url: http://inventory-service:8080/targets - ``` ## HTTP Spec Fields @@ -23,19 +22,19 @@ spec: | `url` | string | Yes | URL pointing to the inventory server | | `acceptPush` | bool | No | Enable webhook-based target updates. Defaults to `false`. | | `authorization` | object | No | Credentials used to access the HTTP endpoint. See _Authorization_ section. | -| `pollInterval` | metav1.Duration | No | Polling interval used to fetch targets from the endpoint. Defaults to `30s`. | -| `timeout` | metav1.Duration | No | Timeout for HTTP requests. Defaults to `10s`. | +| `pollInterval` | duration | No | Polling interval used to fetch targets from the endpoint. Defaults to `30s`. | +| `timeout` | duration | No | Timeout for HTTP requests. Defaults to `10s`. | | `tls` | object | No | Client TLS configuration for HTTPS endpoints. See _TLS_ section. | | `pagination` | object | No | Pagination configuration for parsing responses from the HTTP endpoint. See _Pagination_ section. | -| `responseMapping` | object | No | JSON path mapping definitions. See _Response Mapping_ section. | +| `responseMapping` | object | No | JSONPath mapping definitions. See _Response Processing_ section. | -### Authorization +## Authorization The HTTP provider supports authenticated requests to the inventory endpoint. Exactly one authorization method can be configured. -#### Basic Authentication +### Basic Authentication Credentials can either be defined inline or referenced from a Secret. @@ -64,7 +63,7 @@ spec: key: username ``` -#### Token Authentication +### Token Authentication Static token authentication can be configured using either an inline token or a Secret reference. @@ -94,7 +93,7 @@ spec: key: token ``` -### TLS +## TLS TLS settings can be configured for HTTPS endpoints. @@ -107,7 +106,7 @@ spec: insecureSkipVerify: false ``` -#### TLS Fields +### TLS Fields | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -117,7 +116,7 @@ spec: `caBundle` and `caBundleSecretRef` are mutually exclusive. -### Pagination +## Pagination Pagination can be configured for APIs returning paginated responses. @@ -131,7 +130,7 @@ spec: nextField: next ``` -#### Pagination Fields +### Pagination Fields | Field | Type | Required | Description | |-------|------|----------|-------------| @@ -142,42 +141,16 @@ The `nextField` value may either contain: - A full URL for the next request - A pagination token appended as a query parameter to the original URL -### Response Mapping +## Response Processing -By default, the HTTP response must follow the structure defined in the _Response Format_ section. +The HTTP provider supports two methods for processing responses from the inventory endpoint: -`responseMapping` allows extracting target fields from arbitrary JSON structures using JSONPath expressions. +- **Default Response Format**: The endpoint returns a predefined JSON structure understood directly by the operator. +- **Response Mapping via JSONPath**: Arbitrary JSON structures can be mapped to target fields using JSONPath expressions. -```yaml -spec: - provider: - http: - url: https://inventory.example.com/devices - responseMapping: - name: "$.hostname" - address: "$.management.ip" - port: "$.gnmi.port" - targetProfile: "$.profile" - labels: - role: "$.metadata.role" - site: "$.metadata.site" -``` +If `responseMapping` is configured, the custom mappings are used. Otherwise, the default response format is expected. -#### Response Mapping Fields - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `name` | string | Yes | JSONPath expression extracting the target name. | -| `ip` | string | Yes | JSONPath expression extracting the target IP address or hostname. | -| `port` | string | No | JSONPath expression extracting the gNMI port. | -| `targetProfile` | string | No | JSONPath expression extracting the `TargetProfile`. | -| `labels` | map[string]string | No | JSONPath expressions extracting target labels. | - -Labels extracted through `responseMapping.labels` are merged with labels from `spec.targetLabels`. - -If the same label key exists in both locations, labels extracted through `responseMapping.labels` take precedence. - -## Response Format +### Default Response Format If `responseMapping` is not configured, the endpoint must return a JSON array of objects with the following structure: @@ -220,3 +193,32 @@ Example response: } ] ``` + +### Response Mapping via JSONPath + +`responseMapping` allows extracting target fields from arbitrary JSON structures using JSONPath expressions. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/devices + responseMapping: + name: "$.hostname" + address: "$.management.ip" + port: "$.gnmi.port" + targetProfile: "$.profile" + labels: + role: "$.metadata.role" + site: "$.metadata.site" +``` + +#### Response Mapping Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | JSONPath expression extracting the target name | +| `address` | string | Yes | JSONPath expression extracting the target IP address or hostname | +| `port` | string | No | JSONPath expression extracting the gNMI port | +| `targetProfile` | string | No | JSONPath expression extracting the `TargetProfile` | +| `labels` | map[string]string | No | JSONPath expressions extracting target labels | From 69d82bb884225763cb880ad2055a3f12f50326c2 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Wed, 20 May 2026 21:03:02 -0600 Subject: [PATCH 07/22] changed example router names --- docs/content/docs/user-guide/targetsource/http.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index b6ff8e4..b5ff3d0 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -168,7 +168,7 @@ Example response: [ { "name": "spine1", - "address": "spine1", + "address": "spine1.local", "port": 57400, "labels": { "role": "spine" @@ -177,7 +177,7 @@ Example response: }, { "name": "leaf1", - "address": "leaf1", + "address": "leaf1.local", "port": 57400, "labels": { "role": "leaf" @@ -185,7 +185,7 @@ Example response: }, { "name": "leaf2", - "address": "leaf2", + "address": "leaf2.local", "port": 57400, "labels": { "role": "leaf" From 7c3417113509097c939f40a5ca3f1e014846ce9c Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Fri, 29 May 2026 07:48:44 +0000 Subject: [PATCH 08/22] update http provider documentation --- docs/content/docs/reference/api.md | 99 +++++++++++-- .../docs/user-guide/targetsource/_index.md | 126 ++++++++-------- .../docs/user-guide/targetsource/http.md | 135 +++++++++--------- 3 files changed, 220 insertions(+), 140 deletions(-) diff --git a/docs/content/docs/reference/api.md b/docs/content/docs/reference/api.md index 1cfe4fb..8c6a41f 100644 --- a/docs/content/docs/reference/api.md +++ b/docs/content/docs/reference/api.md @@ -153,30 +153,111 @@ description: > | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| -| `http` | HTTPConfig | No | - | HTTP endpoint for target discovery | -| `consul` | ConsulConfig | No | - | Consul service discovery config | -| `configMap` | string | No | - | ConfigMap name containing targets | -| `podSelector` | LabelSelector | No | - | Select Kubernetes Pods as targets | -| `serviceSelector` | LabelSelector | No | - | Select Kubernetes Services as targets | -| `labels` | map[string]string | No | - | Labels to apply to discovered targets | +| `provider` | ProviderSpec | Yes | - | Provider-specific discovery configuration | +| `targetPort` | int32 | No | - | Default port used when the discovered target does not provide a port | +| `targetProfile` | string | Yes | - | Reference to `TargetProfile` applied to discovered targets | +| `targetLabels` | map[string]string | No | - | Labels added to all discovered targets | + +### ProviderSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `http` | HTTPConfig | No | HTTP provider configuration | ### HTTPConfig +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `url` | string | No | - | HTTP endpoint used to pull targets. Required unless push is enabled | +| `method` | string | No | GET | HTTP request method | +| `headers` | map[string]string | No | - | HTTP headers to include in requests | +| `body` | string | No | - | Request body for POST requests | +| `authorization` | AuthorizationSpec | No | - | Authentication configuration for the HTTP endpoint | +| `interval` | duration | No | 6h | Polling interval used to refresh targets | +| `timeout` | duration | No | 10s | Timeout for HTTP requests | +| `tls` | ClientTLSConfig | No | - | Client TLS configuration for HTTPS endpoints | +| `pagination` | PaginationSpec | No | - | Pagination settings for parsing responses | +| `mapping` | ResponseMappingSpec | No | - | Response mapping configuration for JSON responses | +| `push` | PushSpec | No | - | Push-based update configuration | + +### ClientTLSConfig + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `insecureSkipVerify` | bool | No | false | Skip verification of the server certificate | +| `caBundleRef` | ConfigMapKeySelector | No | - | Reference to a ConfigMap containing a PEM CA bundle | + +### AuthorizationSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `basic` | BasicAuthSpec | No | Basic authentication configuration | +| `token` | TokenAuthSpec | No | Token authentication configuration | + +### BasicAuthSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `credentialsSecretRef` | SecretKeySelector | Yes | Reference to a Secret containing username/password keys | + +### TokenAuthSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `scheme` | string | Yes | Token scheme, e.g. Bearer | +| `tokenSecretRef` | SecretKeySelector | Yes | Reference to a Secret containing the token | + +### PaginationSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `nextField` | string | No | JSON field containing the next page reference or pagination token | + +### ResponseMappingSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `targetsField` | string | No | CEL expression selecting the list of targets from the response | +| `name` | string | No | CEL expression for the target name | +| `address` | string | No | CEL expression for the target address | +| `port` | string | No | CEL expression for the target port | +| `labels` | string | No | CEL expression returning a map of labels | +| `targetProfile` | string | No | CEL expression for the target profile | + +### PushSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `enabled` | bool | No | Enable push updates | +| `auth` | PushAuthSpec | No | Push authentication configuration | + +### PushAuthSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `bearer` | PushBearerAuthSpec | No | Bearer token authentication configuration | +| `signature` | PushSignatureAuthSpec | No | Signature authentication configuration | + +### PushBearerAuthSpec + | Field | Type | Required | Description | |-------|------|----------|-------------| -| `url` | string | Yes | URL of the HTTP endpoint | +| `tokenSecretRef` | SecretKeySelector | Yes | Reference to a Secret containing the bearer token | -### ConsulConfig +### PushSignatureAuthSpec | Field | Type | Required | Description | |-------|------|----------|-------------| -| `url` | string | Yes | Consul server URL | +| `secretRef` | SecretKeySelector | Yes | Reference to a Secret used to verify request signatures | +| `header` | string | Yes | Header containing the signature | +| `algorithm` | string | No | Signature algorithm | ### TargetSourceStatus | Field | Type | Description | |-------|------|-------------| | `status` | string | Sync status (Synced, Error, Pending) | +| `observedGeneration` | int64 | Observed generation of the spec | | `targetsCount` | int32 | Number of discovered targets | | `lastSync` | Time | Last successful sync timestamp | diff --git a/docs/content/docs/user-guide/targetsource/_index.md b/docs/content/docs/user-guide/targetsource/_index.md index 16861e7..7e957ec 100644 --- a/docs/content/docs/user-guide/targetsource/_index.md +++ b/docs/content/docs/user-guide/targetsource/_index.md @@ -29,8 +29,8 @@ spec: | Field | Type | Required | Description | |-------|------|----------|-------------| | `provider` | object | Yes | Provider-specific discovery configuration. Exactly one provider must be configured | -| `targetPort` | int32 | No | Default port used if the discovered target does not provide a port. | -| `targetProfile` | string | Yes | Reference to `TargetProfile` applied to all targets | +| `targetPort` | int32 | No | Default port used when the discovered target does not provide a port | +| `targetProfile` | string | Yes | Reference to `TargetProfile` applied to all discovered targets | | `targetLabels` | map[string]string | No | Labels added to all discovered targets | @@ -40,9 +40,67 @@ spec: | Provider | Description | |----------|-------------| -| `http` | Discover targets from an HTTP JSON endpoint. [Configuration]({{< relref "http.md" >}}) | +| `http` | Discover targets from an HTTP JSON endpoint or receive webhook updates. [Configuration]({{< relref "http.md" >}}) | + + ## Label Inheritance Each generated `Target` receives an ownership label identifying the originating `TargetSource`: @@ -102,11 +160,12 @@ If the same label key is defined in multiple places, labels are applied in the f ## Status -The TargetSource status shows discovery state: +The `TargetSource` status shows discovery state: ```yaml status: status: Synced + observedGeneration: 1 targetsCount: 42 lastSync: "2024-01-15T10:30:00Z" ``` @@ -114,67 +173,10 @@ status: | Field | Description | |-------|-------------| | `status` | Current sync status (Synced, Error, Pending) | +| `observedGeneration` | Generation of the spec last processed by the controller | | `targetsCount` | Number of targets discovered | | `lastSync` | Timestamp of last successful sync | - - ## Lifecycle ### Target Creation diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index b5ff3d0..fd52094 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -6,7 +6,7 @@ description: > HTTP TargetSource Discovery Provider --- -The HTTP provider discovers targets from an HTTP endpoint returning a JSON array of target definitions. +The HTTP provider discovers targets from an HTTP endpoint returning JSON, or receives webhook-based updates when push mode is enabled. ```yaml spec: @@ -17,39 +17,43 @@ spec: ## HTTP Spec Fields -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `url` | string | Yes | URL pointing to the inventory server | -| `acceptPush` | bool | No | Enable webhook-based target updates. Defaults to `false`. | -| `authorization` | object | No | Credentials used to access the HTTP endpoint. See _Authorization_ section. | -| `pollInterval` | duration | No | Polling interval used to fetch targets from the endpoint. Defaults to `30s`. | -| `timeout` | duration | No | Timeout for HTTP requests. Defaults to `10s`. | -| `tls` | object | No | Client TLS configuration for HTTPS endpoints. See _TLS_ section. | -| `pagination` | object | No | Pagination configuration for parsing responses from the HTTP endpoint. See _Pagination_ section. | -| `responseMapping` | object | No | JSONPath mapping definitions. See _Response Processing_ section. | +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `url` | string | No | - | HTTP endpoint used to pull targets. Required unless `push.enabled` is enabled | +| `method` | string | No | GET | HTTP method used for requests | +| `headers` | map[string]string | No | - | HTTP headers to include in requests | +| `body` | string | No | - | Request body for POST requests | +| `authorization` | object | No | - | Authentication configuration for the HTTP endpoint | +| `interval` | duration | No | 6h | Polling interval used to refresh targets | +| `timeout` | duration | No | 10s | Timeout for HTTP requests | +| `tls` | object | No | - | Client TLS configuration for HTTPS endpoints | +| `pagination` | object | No | - | Pagination configuration for parsing HTTP responses | +| `mapping` | object | No | - | Response mapping configuration for JSON responses | +| `push` | object | No | - | Push-based update configuration | -## Authorization - -The HTTP provider supports authenticated requests to the inventory endpoint. - -Exactly one authorization method can be configured. - -### Basic Authentication +## Push Mode -Credentials can either be defined inline or referenced from a Secret. +The HTTP provider supports webhook-based target updates via `spec.provider.http.push`. ```yaml spec: provider: http: - url: https://inventory.example.com/targets - authorization: - basic: - username: admin - password: secret + push: + enabled: true ``` -Using a Secret reference: +When `push.enabled` is true, the operator accepts incoming webhook notifications and can update targets without polling a remote endpoint. The `url` field is optional when push mode is enabled, but can still be used for polling and fallback behavior. + +## Authorization + +The HTTP provider supports authenticated requests to the inventory endpoint. + +Exactly one authorization method can be configured. + +### Basic Authentication + +Credentials are referenced from a Secret. ```yaml spec: @@ -65,20 +69,7 @@ spec: ### Token Authentication -Static token authentication can be configured using either an inline token or a Secret reference. - -```yaml -spec: - provider: - http: - url: https://inventory.example.com/targets - authorization: - token: - scheme: Bearer - token: eyJhbGciOi... -``` - -Using a Secret reference: +Token authentication is configured using a Secret reference. ```yaml spec: @@ -104,17 +95,17 @@ spec: url: https://inventory.example.com/targets tls: insecureSkipVerify: false + caBundleRef: + name: inventory-ca + key: ca.crt ``` ### TLS Fields | Field | Type | Required | Description | |-------|------|----------|-------------| -| `insecureSkipVerify` | bool | No | Skip verification of the server certificate. Defaults to `false`. | -| `caBundle` | []byte | No | Base64-encoded PEM CA bundle used to validate the server certificate. | -| `caBundleSecretRef` | object | No | Reference to a Secret containing a PEM CA bundle. | - -`caBundle` and `caBundleSecretRef` are mutually exclusive. +| `insecureSkipVerify` | bool | No | Skip verification of the server certificate. Defaults to `false` | +| `caBundleRef` | object | No | Reference to a ConfigMap containing a PEM-encoded CA bundle | ## Pagination @@ -126,7 +117,6 @@ spec: http: url: https://inventory.example.com/devices pagination: - itemsField: results nextField: next ``` @@ -134,8 +124,7 @@ spec: | Field | Type | Required | Description | |-------|------|----------|-------------| -| `itemsField` | string | No | Top-level JSON field containing the list of target objects. | -| `nextField` | string | No | Top-level JSON field containing the next page reference or pagination token. | +| `nextField` | string | No | Top-level JSON field containing the next page reference or pagination token | The `nextField` value may either contain: - A full URL for the next request @@ -143,24 +132,24 @@ The `nextField` value may either contain: ## Response Processing -The HTTP provider supports two methods for processing responses from the inventory endpoint: +The HTTP provider supports two response processing modes: -- **Default Response Format**: The endpoint returns a predefined JSON structure understood directly by the operator. -- **Response Mapping via JSONPath**: Arbitrary JSON structures can be mapped to target fields using JSONPath expressions. +- **Default response format**: The endpoint returns a JSON array of target objects. +- **Response mapping**: Custom JSON structures are mapped to target fields using CEL expressions. -If `responseMapping` is configured, the custom mappings are used. Otherwise, the default response format is expected. +If `mapping` is configured, the custom mapping rules are used. Otherwise, the response itself must be a JSON array. ### Default Response Format -If `responseMapping` is not configured, the endpoint must return a JSON array of objects with the following structure: +If `mapping` is not configured, the endpoint must return a JSON array of objects with the following structure: | Field | Type | Required | Description | |-------|------|----------|-------------| | `name` | string | Yes | Name of the generated `Target` resource | | `address` | string | Yes | Device address (FQDN or IP address) | -| `port` | int32 | No | Port used for gNMI connections. If omitted, `spec.targetPort` is used. | +| `port` | int32 | No | Port used for gNMI connections. If omitted, `spec.targetPort` is used | | `labels` | map[string]string | No | Labels added to the generated `Target` resource | -| `targetProfile` | string | No | Reference to a `TargetProfile`. If omitted, `spec.targetProfile` is used. | +| `targetProfile` | string | No | Reference to a `TargetProfile`. If omitted, `spec.targetProfile` is used | Example response: @@ -194,31 +183,39 @@ Example response: ] ``` -### Response Mapping via JSONPath +### Response Mapping via CEL -`responseMapping` allows extracting target fields from arbitrary JSON structures using JSONPath expressions. +`mapping` allows extracting target fields from arbitrary JSON structures using CEL expressions. ```yaml spec: provider: http: url: https://inventory.example.com/devices - responseMapping: - name: "$.hostname" - address: "$.management.ip" - port: "$.gnmi.port" - targetProfile: "$.profile" + mapping: + targetsField: "self.results" + name: "item.hostname" + address: "item.management.ip" + port: "item.gnmi.port" + targetProfile: "item.profile" labels: - role: "$.metadata.role" - site: "$.metadata.site" + role: "item.metadata.role" + site: "item.metadata.site" ``` -#### Response Mapping Fields +#### Mapping Fields | Field | Type | Required | Description | |-------|------|----------|-------------| -| `name` | string | Yes | JSONPath expression extracting the target name | -| `address` | string | Yes | JSONPath expression extracting the target IP address or hostname | -| `port` | string | No | JSONPath expression extracting the gNMI port | -| `targetProfile` | string | No | JSONPath expression extracting the `TargetProfile` | -| `labels` | map[string]string | No | JSONPath expressions extracting target labels | +| `targetsField` | string | No | CEL expression selecting the target list from the response | +| `name` | string | No | CEL expression for the target name | +| `address` | string | No | CEL expression for the target address | +| `port` | string | No | CEL expression for the target port | +| `labels` | string | No | CEL expression returning a map of labels | +| `targetProfile` | string | No | CEL expression for the target profile | + +### CEL variables + +The mapping expressions support the following variables: +- `item`: the current target object +- `self`: the full JSON response From 28d8027f64c29a306f09d6a520780655a63c3157 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Fri, 29 May 2026 09:39:19 +0000 Subject: [PATCH 09/22] tune the http provider doc --- .../docs/user-guide/targetsource/http.md | 361 +++++++++++++++++- 1 file changed, 350 insertions(+), 11 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index fd52094..2e18d16 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -24,7 +24,7 @@ spec: | `headers` | map[string]string | No | - | HTTP headers to include in requests | | `body` | string | No | - | Request body for POST requests | | `authorization` | object | No | - | Authentication configuration for the HTTP endpoint | -| `interval` | duration | No | 6h | Polling interval used to refresh targets | +| `interval` | duration | No | 30m | Polling interval used to refresh targets | | `timeout` | duration | No | 10s | Timeout for HTTP requests | | `tls` | object | No | - | Client TLS configuration for HTTPS endpoints | | `pagination` | object | No | - | Pagination configuration for parsing HTTP responses | @@ -109,7 +109,7 @@ spec: ## Pagination -Pagination can be configured for APIs returning paginated responses. +Pagination enables the operator to retrieve all targets from APIs that return results in multiple pages. This is common with inventory systems that limit response sizes for performance reasons. ```yaml spec: @@ -124,12 +124,50 @@ spec: | Field | Type | Required | Description | |-------|------|----------|-------------| -| `nextField` | string | No | Top-level JSON field containing the next page reference or pagination token | +| `nextField` | string | No | Top-level JSON field name containing the pagination reference or token | The `nextField` value may either contain: - A full URL for the next request - A pagination token appended as a query parameter to the original URL +### How Pagination Works + +The operator handles two common pagination patterns used by network inventory systems: + +#### 1. Cursor-Based Pagination (Tokens) +When your inventory API returns a pagination token (e.g., `"next": "abc123xyz"`), the operator automatically appends it as a query parameter to construct the next request: + +``` +First request: GET https://inventory.example.com/devices +Response contains: "next": "page2token" +Next request: GET https://inventory.example.com/devices?next=page2token +``` + +Example response: +```json +{ + "devices": [...], + "next": "page2token" +} +``` + +#### 2. URL-Based Pagination +When your API returns a complete URL for the next page (e.g., `"next": "https://inventory.example.com/devices?offset=50"`), the operator uses it directly without modification: + +``` +First request: GET https://inventory.example.com/devices +Response contains: "next": "https://inventory.example.com/devices?offset=50" +Next request: GET https://inventory.example.com/devices?offset=50 +``` + +Example response: +```json +{ + "devices": [...], + "next": "https://inventory.example.com/devices?offset=50" +} +``` + ## Response Processing The HTTP provider supports two response processing modes: @@ -185,7 +223,7 @@ Example response: ### Response Mapping via CEL -`mapping` allows extracting target fields from arbitrary JSON structures using CEL expressions. +When your inventory API's JSON structure differs from the default format, use CEL (Common Expression Language) mapping to extract target fields. ```yaml spec: @@ -198,24 +236,325 @@ spec: address: "item.management.ip" port: "item.gnmi.port" targetProfile: "item.profile" - labels: - role: "item.metadata.role" - site: "item.metadata.site" + labels: "{'role': item.metadata.role, 'site': item.metadata.site}" +``` + +#### Understanding `targetsField` + +The `targetsField` expression tells the operator where to find the list of target objects in your API response. It's particularly important when your API wraps the target list in a data structure. + +**When to use `targetsField`:** +- Your API returns `{"results": [...]}` -> use `"self.results"` +- Your API returns `{"data": {"devices": [...]}}` -> use `"self.data.devices"` +- Your API returns a plain array `[...]` -> omit `targetsField` (default behavior) + +**Example scenarios:** + +*Custom API response example 1:* +```json +{ + "count": 42, + "next": "https://...", + "results": [ + {"id": 1, "name": "device1", "primary_ip": "10.0.0.1"}, + {"id": 2, "name": "device2", "primary_ip": "10.0.0.2"} + ] +} +``` +Usage: `targetsField: "self.results"` + +*Custom API response example 2:* +```json +{ + "status": "success", + "data": { + "timestamp": "2024-01-01T00:00:00Z", + "devices": [ + {"name": "router1", "mgmt_ip": "192.168.1.1"}, + {"name": "router2", "mgmt_ip": "192.168.1.2"} + ] + } +} ``` +Usage: `targetsField: "self.data.devices"` #### Mapping Fields | Field | Type | Required | Description | |-------|------|----------|-------------| -| `targetsField` | string | No | CEL expression selecting the target list from the response | +| `targetsField` | string | No | CEL expression selecting the target list from the response. If omitted, assumes response is a direct JSON array | | `name` | string | No | CEL expression for the target name | | `address` | string | No | CEL expression for the target address | | `port` | string | No | CEL expression for the target port | | `labels` | string | No | CEL expression returning a map of labels | | `targetProfile` | string | No | CEL expression for the target profile | -### CEL variables +#### CEL Variables The mapping expressions support the following variables: -- `item`: the current target object -- `self`: the full JSON response +- `item`: the current target object being processed +- `self`: the complete unprocessed response from the HTTP endpoint + +### Performance: CEL vs Direct Mapping + +Understanding the performance implications helps optimize your configurations: + +**Direct Mapping (No CEL)** - *Fastest* +- Used when your API response matches the default structure exactly +- No expression compilation or evaluation overhead +- Suitable for high-frequency polling (e.g., every minute) +- Example: API returns `[{"name": "...", "address": "..."}]` + +**CEL Mapping** - *Slight overhead* +- CEL expressions are compiled once at startup (not per request) +- Evaluation is performed per target object during each poll cycle +- At high scale (10,000+ targets), consider the `interval` between polls + +**Best practices:** +- Use direct mapping if your API already returns the correct structure +- For large result sets, increase the interval +- Combine CEL and direct mapping for efficiency (see hybrid mapping below) +- Use CEL extensions (see reference table below) to reduce complexity and improve readability + +### CEL Extensions + +The operator includes a set of standard CEL extensions from the official [CEL Go library](https://github.com/google/cel-go) to enable more advanced expressions. + +These [extensions](https://pkg.go.dev/github.com/google/cel-go/ext) expand CEL with additional capabilities commonly needed when transforming API responses: + +| Extension | Purpose | +|----------|----------| +| **Strings** | String manipulation such as splitting values, case conversion, and extracting parts of text (e.g. parsing hostnames or IPs) | +| **Math** | Numeric operations and comparisons (e.g. calculations, min/max, type conversions) | +| **Lists** | Working with arrays (e.g. indexing, filtering, joining values) | +| **Sets** | Set-style operations such as membership checks and comparisons | +| **Regex** | Pattern matching and validation using regular expressions | +| **Bindings** | Defining intermediate variables to simplify complex expressions | + +**Examples:** + +```yaml +mapping: + # Extract site from hostname + labels: | + { + 'site': item.name.split('-')[0] + } + + # Conditional profile + targetProfile: "item.type == 'edge' ? 'edge' : 'core'" + + # Pattern-based classification + labels: | + { + 'role': item.name.matches('^spine') ? 'spine' : 'leaf' + } +``` + +### Combining CEL and Direct Mapping (Hybrid Approach) + +You don't need to map all fields with CEL. The operator supports mixing CEL expressions and direct field lookups for maximum efficiency: + +| Scenario | Behavior | Use Case | +|----------|----------|----------| +| `name`, `address` use CEL; others omitted | Extracts mapped fields via CEL; looks for `port`, `labels`, `targetProfile` directly in item JSON | Simple API where only some fields need transformation | +| Only `labels` uses CEL | Other fields use direct mapping; labels constructed from CEL expression | API returns correct `name`, `address`, `port` but custom labels need extraction | +| Only `address` uses CEL | Direct mapping for other fields; only address requires transformation | Most fields match API exactly except address requires CIDR parsing or format conversion | +| All fields use CEL | Complete transformation via expressions | API structure completely different from expected format | + +This hybrid approach optimizes performance by only compiling and evaluating CEL where needed. + +**Example - Partial CEL mapping (only transform what needs transforming):** +```yaml +mapping: + # name: "item.name" -> OMITTED: already matches default "name" field + address: "item.primary_ip4 != null ? item.primary_ip4.split('/')[0] : item.primary_ip6.split('/')[0]" # CEL: parse CIDR + # port: OMITTED: already exists as "port" field in item + labels: | # CEL: construct labels from custom fields + { + 'site': item.site.name, + 'role': item.device_role.name + } + # targetProfile: OMITTED: already exists as "targetProfile" field in item +``` + +In this example, only `address` and `labels` use CEL expressions; `name`, `port`, and `targetProfile` use direct field lookups for efficiency. + +### Using YAML `|` for Complex CEL Expressions + +When writing more complex CEL expressions, it is recommended to use YAML’s pipe (`|`) literal block instead of inline strings. + +This is especially useful for expressions that span multiple lines or contain nested logic. + +#### Recommended pattern (labels example) + +```yaml +mapping: + labels: | + { + "site": item.site.name, + "rack": item.rack != null ? item.rack.name : "", + "role": item.role != null ? item.role : "unknown", + "tags": item.tags.size() > 0 ? ','.join(item.tags) : "" + } +``` + +**Why use `|` instead of quoted strings:** +- **Readability**: Multi-line expressions are easier to understand +- **Maintainability**: Complex CEL expressions don't require escaping +- **YAML best practice**: Literal blocks handle special characters naturally + +## Recommended Production Settings + +When deploying HTTP TargetSource providers in production networks, follow these guidelines to ensure reliable and efficient target discovery: + +### Polling Configuration +| Scenario | Setting | Rationale | +|----------|---------|-----------| +| **Small environment** (< 100 targets) | `interval: 5m` | Frequent updates without excessive load | +| **Medium environment** (100-500 targets) | `interval: 10m` | Balance between freshness and API load | +| **Large environment** (500-2000 targets) | `interval: 15m` | Reduce API polling overhead | +| **Very large environment** (2000+ targets) | `interval: 30m` | Minimize impact on inventory system | +| **High-frequency changes** | Use `push` mode with `interval` | Enables sub-minute updates via push while periodic polling ensures completeness and consistency | + +**Timeout Configuration:** +```yaml +timeout: 30s # Allows for network latency +``` + +If timeouts consistently occur, increase `interval` instead of timeout (don't poll faster) + +### Authentication & Security + +**Always use TLS in production:** +```yaml +tls: + insecureSkipVerify: false # Never skip verification in production + caBundleRef: + name: inventory-ca-bundle + key: ca.crt +``` + +**For authenticated APIs:** +- Store credentials in Kubernetes Secrets +- Rotate credentials periodically +- Use token-based auth when possible (simpler secret rotation) + +Example: +```yaml +authorization: + token: + scheme: Bearer + tokenSecretRef: + name: inventory-api-token + key: token +``` + +### Pagination & Large Result Sets + +**Configuration for APIs returning large result sets:** +```yaml +pagination: + nextField: next # Always configure pagination if your API supports it + +interval: 30m # Increase interval for large datasets (reduces cumulative API load) +timeout: 60s # Increase only if individual requests are slow or responses are large +``` + +Pagination splits large datasets into multiple smaller HTTP requests. This improves reliability and reduces the likelihood of timeouts compared to fetching a single large response. + +**Optimization strategies:** +- Request API filtering (if supported) to reduce result set size (e.g. ?limit=1000 or ?status=active) +- If the API does not support pagination or filtering increase the timeout +- Consider webhook push mode for frequently-changing inventories (if API supports it) + +### Mapping Optimization + +**Use hybrid CEL and direct mapping for performance:** +```yaml +# EFFICIENT - Only CEL-transform what needs it +mapping: + # These fields exist directly in API response -> no CEL needed + name: "item.hostname" # Direct: omit CEL, fallback to field lookup + # port: (OMITTED) # Direct: exists as "port" in item + + # Only these need transformation -> use CEL + address: "item.primary_ip.split('/')[0]" # CEL: parse CIDR + labels: | # CEL: construct from nested fields + {'site': item.site.name} +``` + +**Avoid unnecessary CEL complexity:** +```yaml +# GOOD - Simple expressions +mapping: + address: "item.management_ip" + port: "int(item.gnmi_port)" + +# AVOID - Nested ternary logic (hard to debug) +mapping: + name: "item.has_override ? item.override_name : (item.hostname != '' ? item.hostname : 'default-' + string(item.id))" +``` + +**CEL expression best practices:** +- Compile expressions once at startup (not per request), so complexity is paid only once +- Use `ext.Bindings` for repeated expressions to avoid redundant evaluation +- Test CEL expressions thoroughly; they're compiled but errors only appear during evaluation +- Keep expressions under 200 characters for maintainability + +### Example Production Configuration + +```yaml +apiVersion: gnmic.openconfig.net/v1alpha1 +kind: TargetSource +metadata: + name: production-inventory +spec: + provider: + http: + # Security + url: https://inventory.prod.example.com/api/dcim/devices/?limit=100 + tls: + insecureSkipVerify: false + caBundleRef: + name: netbox-ca + key: ca.crt + + # Authentication + authorization: + token: + scheme: Bearer + tokenSecretRef: + name: api-token + key: token + + # Timing + interval: 15m # Balanced update frequency + timeout: 30s # Allow for network latency + + # Pagination + pagination: + nextField: next + + # Mapping for fields + mapping: + targetsField: "self.results" + #name: "item.name" -> already handled with fallback direct mapping + address: "item.primary_ip4 != null ? item.primary_ip4.split('/')[0] : item.primary_ip6.split('/')[0]" + port: "item.custom_fields.gnmi_port" + labels: "{\n 'site': item.site.name,\n 'role': item.device_role.name,\n 'status': item.status.value\n }" + targetProfile: "item.custom_fields.gnmi_profile" + + # Global settings + targetPort: 9339 + targetProfile: default-profile +``` + +This configuration ensures: + +- Secure HTTPS communication with certificate validation +- API authentication with token-based credentials +- Balanced polling interval for stable environments +- Proper pagination handling for large device inventories +- Rich label extraction from custom fields +- Fallback to defaults when fields are missing From 929a4c2e1812c54724273b9c5f33278df9eab989 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Fri, 29 May 2026 09:42:54 +0000 Subject: [PATCH 10/22] update default value of timeout --- docs/content/docs/user-guide/targetsource/http.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index 2e18d16..9e569be 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -25,7 +25,7 @@ spec: | `body` | string | No | - | Request body for POST requests | | `authorization` | object | No | - | Authentication configuration for the HTTP endpoint | | `interval` | duration | No | 30m | Polling interval used to refresh targets | -| `timeout` | duration | No | 10s | Timeout for HTTP requests | +| `timeout` | duration | No | 30s | Timeout for HTTP requests | | `tls` | object | No | - | Client TLS configuration for HTTPS endpoints | | `pagination` | object | No | - | Pagination configuration for parsing HTTP responses | | `mapping` | object | No | - | Response mapping configuration for JSON responses | From c3fe3ca0abb415646893ddc14a1c0f9c65e16470 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Fri, 29 May 2026 09:49:39 +0000 Subject: [PATCH 11/22] fix CEL expression --- docs/content/docs/user-guide/targetsource/http.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index 9e569be..cafa24b 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -395,7 +395,7 @@ mapping: "site": item.site.name, "rack": item.rack != null ? item.rack.name : "", "role": item.role != null ? item.role : "unknown", - "tags": item.tags.size() > 0 ? ','.join(item.tags) : "" + "tags": item.tags.size() > 0 ? item.tags.join(',') : "" } ``` From 0389b9ff024bc940577d41c56b3959b602eb4038 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Fri, 29 May 2026 12:57:40 +0000 Subject: [PATCH 12/22] docs: update pull mode --- .../docs/user-guide/targetsource/http.md | 26 ++++++++++++++----- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index cafa24b..9789961 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -8,13 +8,6 @@ description: > The HTTP provider discovers targets from an HTTP endpoint returning JSON, or receives webhook-based updates when push mode is enabled. -```yaml -spec: - provider: - http: - url: http://inventory-service:8080/targets -``` - ## HTTP Spec Fields | Field | Type | Required | Default | Description | @@ -31,6 +24,25 @@ spec: | `mapping` | object | No | - | Response mapping configuration for JSON responses | | `push` | object | No | - | Push-based update configuration | +## Pull Mode + +The HTTP provider supports pull-based target discovery by periodically querying a remote HTTP endpoint that returns target data in JSON format. + +```yaml +spec: + provider: + http: + url: http://inventory-service:8080/targets +``` + +In pull mode, the operator sends HTTP requests to the configured url at a fixed interval and updates targets based on the response. The `push.enabled` field is optional when pull mode is enabled, but can still be used for accepting incoming webhook notifications. + +*How Pull Mode Works* +1. The operator sends an HTTP request to the configured url +2. The response is parsed (either directly or via mapping) +3. Targets are created, updated, or removed based on the returned data +4. This process repeats according to the configured interval + ## Push Mode The HTTP provider supports webhook-based target updates via `spec.provider.http.push`. From 31fd036e2afca76555ba603e0f05832aae811426 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Mon, 1 Jun 2026 09:48:17 +0000 Subject: [PATCH 13/22] update docs pagination and hybrid CEL mapping --- .../docs/user-guide/targetsource/http.md | 96 ++++++++++++------- 1 file changed, 63 insertions(+), 33 deletions(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index 9789961..df1cfd6 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -121,7 +121,7 @@ spec: ## Pagination -Pagination enables the operator to retrieve all targets from APIs that return results in multiple pages. This is common with inventory systems that limit response sizes for performance reasons. +Pagination enables the operator to retrieve complete result sets from APIs that return data in multiple pages. The operator automatically follows pagination until no further pages are available. ```yaml spec: @@ -129,14 +129,15 @@ spec: http: url: https://inventory.example.com/devices pagination: - nextField: next + nextField: "self.next" ``` ### Pagination Fields | Field | Type | Required | Description | |-------|------|----------|-------------| -| `nextField` | string | No | Top-level JSON field name containing the pagination reference or token | +| `nextField` | string | No | CEL expression used to extract the next page reference from the response | +| `requestParam` | string | No | Query parameter used when the extracted value is a token | The `nextField` value may either contain: - A full URL for the next request @@ -144,42 +145,64 @@ The `nextField` value may either contain: ### How Pagination Works -The operator handles two common pagination patterns used by network inventory systems: +The operator handles the following pagination patterns: + +#### 1. Link Header Pagination +If the API provides a Link response header with `rel="next"`, the operator will automatically follow it. -#### 1. Cursor-Based Pagination (Tokens) -When your inventory API returns a pagination token (e.g., `"next": "abc123xyz"`), the operator automatically appends it as a query parameter to construct the next request: +Example response header: +``` +Link: ; rel="next" +``` +Behavior: ``` -First request: GET https://inventory.example.com/devices -Response contains: "next": "page2token" -Next request: GET https://inventory.example.com/devices?next=page2token +Request 1: GET /devices?page=1 +Request 2: GET /devices?page=2 +Request 3: GET /devices?page=3 +... ``` +#### 2. URL-Based Pagination +If the response contains a full URL in the body (e.g. `"next": "https://..."`), it will be used directly. + Example response: ```json { "devices": [...], - "next": "page2token" + "next": "https://inventory.example.com/devices?offset=50" } ``` -#### 2. URL-Based Pagination -When your API returns a complete URL for the next page (e.g., `"next": "https://inventory.example.com/devices?offset=50"`), the operator uses it directly without modification: +#### 3. Token-Based Pagination +If the response contains a pagination token, the operator appends it as a query parameter. +Example: +```yaml +pagination: + nextField: "self.next_token" + requestParam: "page_token" ``` -First request: GET https://inventory.example.com/devices -Response contains: "next": "https://inventory.example.com/devices?offset=50" -Next request: GET https://inventory.example.com/devices?offset=50 + +Example: +``` +GET /devices +-> "next_token": "abc123" +GET /devices?page_token=abc123 ``` -Example response: -```json -{ - "devices": [...], - "next": "https://inventory.example.com/devices?offset=50" -} +#### CEL-Based Extraction +The nextField is evaluated as a CEL expression using: +- `self` -> entire JSON response + +Example: +```yaml +pagination: + nextField: "self['@odata.nextLink']" ``` +This allows extracting values from nested or special keys. + ## Response Processing The HTTP provider supports two response processing modes: @@ -379,15 +402,22 @@ This hybrid approach optimizes performance by only compiling and evaluating CEL **Example - Partial CEL mapping (only transform what needs transforming):** ```yaml mapping: - # name: "item.name" -> OMITTED: already matches default "name" field + # Use CEL only when you need to transform a field + name: "item.hostname" address: "item.primary_ip4 != null ? item.primary_ip4.split('/')[0] : item.primary_ip6.split('/')[0]" # CEL: parse CIDR - # port: OMITTED: already exists as "port" field in item - labels: | # CEL: construct labels from custom fields + + # Fields that already exist should be omitted + # Port already exists as "port" field in item + # port: item.port <- omit this + + # Use CEL for structured or derived values + labels: | { - 'site': item.site.name, - 'role': item.device_role.name + "site": item.site.name, + "role": item.device_role.name } - # targetProfile: OMITTED: already exists as "targetProfile" field in item + + # targetProfile can also be omitted if already present or not needed ``` In this example, only `address` and `labels` use CEL expressions; `name`, `port`, and `targetProfile` use direct field lookups for efficiency. @@ -407,7 +437,7 @@ mapping: "site": item.site.name, "rack": item.rack != null ? item.rack.name : "", "role": item.role != null ? item.role : "unknown", - "tags": item.tags.size() > 0 ? item.tags.join(',') : "" + "tags": item.tags.join(',') } ``` @@ -427,7 +457,7 @@ When deploying HTTP TargetSource providers in production networks, follow these | **Medium environment** (100-500 targets) | `interval: 10m` | Balance between freshness and API load | | **Large environment** (500-2000 targets) | `interval: 15m` | Reduce API polling overhead | | **Very large environment** (2000+ targets) | `interval: 30m` | Minimize impact on inventory system | -| **High-frequency changes** | Use `push` mode with `interval` | Enables sub-minute updates via push while periodic polling ensures completeness and consistency | +| **High-frequency changes** | Use `push` mode with `interval` | Enables updates via push while periodic polling ensures completeness and consistency | **Timeout Configuration:** ```yaml @@ -486,13 +516,13 @@ Pagination splits large datasets into multiple smaller HTTP requests. This impro ```yaml # EFFICIENT - Only CEL-transform what needs it mapping: - # These fields exist directly in API response -> no CEL needed - name: "item.hostname" # Direct: omit CEL, fallback to field lookup - # port: (OMITTED) # Direct: exists as "port" in item + # + name: "item.hostname" # CEL expression + # port: (OMITTED) # Direct: exists as "port" in item # Only these need transformation -> use CEL address: "item.primary_ip.split('/')[0]" # CEL: parse CIDR - labels: | # CEL: construct from nested fields + labels: | # CEL: construct from nested fields {'site': item.site.name} ``` From c12683d2811fdee774da299d4ff288cef4e863f3 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Mon, 1 Jun 2026 14:39:47 +0000 Subject: [PATCH 14/22] add Netbox example --- .../examples/NetBox/Export Template/_index.md | 338 ++++++++++++++++++ docs/content/docs/examples/NetBox/_index.md | 8 + 2 files changed, 346 insertions(+) create mode 100644 docs/content/docs/examples/NetBox/Export Template/_index.md create mode 100644 docs/content/docs/examples/NetBox/_index.md diff --git a/docs/content/docs/examples/NetBox/Export Template/_index.md b/docs/content/docs/examples/NetBox/Export Template/_index.md new file mode 100644 index 0000000..2691a13 --- /dev/null +++ b/docs/content/docs/examples/NetBox/Export Template/_index.md @@ -0,0 +1,338 @@ +--- +title: "NetBox (Export Template)" +linkTitle: "NetBox Export" +weight: 3 +description: > + Discover targets from NetBox using HTTP provider with export templates +--- + +This guide shows how to use **NetBox Export Templates** with the HTTP provider to discover and sync targets. + +Export Templates offer powerful filtering, transformation, and formatting directly in NetBox, reducing load on the operator and enabling complex discovery logic. + +## Overview + +An **Export Template** is a Jinja2 template defined in NetBox that: + +1. **Queries** NetBox's internal database (devices, interfaces, etc.) +2. **Filters** results based on custom criteria +3. **Transforms** data into your desired output format (JSON, YAML, CSV, etc.) +4. **Returns** the formatted output via a custom REST API endpoint + +When used with gNMIc's HTTP provider, the operator simply fetches the rendered template and parses the result — no additional transformation needed. + +--- + +## Prerequisites + +- A running Kubernetes cluster with gNMIc Operator installed +- A reachable NetBox instance with **permissions to create Export Templates** +- A NetBox API token +- `kubectl` access to your cluster +- Familiarity with Jinja2 templates + +--- + +## Step 1: Create a Secret for the NetBox API Token + +Create a Kubernetes Secret containing your NetBox API token. This keeps credentials secure and out of your TargetSource manifests. + +```bash +# Substitute YOUR_NETBOX_TOKEN with your actual token +kubectl create secret generic netbox-api-token \ + --from-literal=token=YOUR_NETBOX_TOKEN \ + -n your-namespace +``` + +Verify the Secret was created: + +```bash +kubectl get secret netbox-api-token -n your-namespace -o yaml +``` + +--- + +## Step 2: Create an Export Template in NetBox + +Log in to your NetBox instance and navigate to **Customization > Export Templates**. + +### Step 2a: Create a New Template + +Click **Add Export Template** and fill in the details: + +| Field | Value | Notes | +|-------|-------|-------| +| **Name** | `gNMIc Device Export` | Descriptive name for your template | +| **Content Type** | `dcim > device` | Export template applies to Device objects | +| **Template Code** | (see below) | Jinja2 template | +| **File Extension** | `json` | Output format | +| **Mime Type** | `application/json` | Correct MIME type for JSON | + +### Step 2b: Template Code Example + +#### Basic Template (All Devices) + +```jinja2 +{ + "targets": [ + {% for device in queryset %} + { + "name": "{{ device.name }}", + "address": "{{ device.primary_ip4.address.split('/')[0] }}:57400", + "labels": { + "site": "{{ device.site.name }}", + "role": "{{ device.device_role.name }}", + "region": "{{ device.site.region.name }}", + "type": "{{ device.device_type.model }}" + } + }{{ "," if not loop.last }} + {% endfor %} + ] +} +``` + +#### Advanced Template (Filtered by Status and Role) + +```jinja2 +{ + "targets": [ + {% for device in queryset.filter(status='active', device_role__name__in=['leaf', 'spine']) %} + { + "name": "{{ device.name }}", + "address": "{{ device.primary_ip4.address.split('/')[0] }}:57400", + "labels": { + "site": "{{ device.site.name }}", + "role": "{{ device.device_role.name }}", + "region": "{{ device.site.region.name }}", + "model": "{{ device.device_type.model }}", + "serial": "{{ device.serial }}", + "asset_tag": "{{ device.asset_tag }}" + } + }{{ "," if not loop.last }} + {% endfor %} + ] +} +``` + +**Key template elements:** + +- `queryset`: The filtered set of devices (all unless you add `.filter()`) +- `device.name`: Device hostname +- `device.primary_ip4.address.split('/')[0]`: Extract IP from CIDR (e.g., `192.0.2.1/24` to `192.0.2.1`) +- `device.site.name`, `device.device_role.name`: NetBox relationships (site, role, etc.) +- `loop.last`: Jinja2 loop variable to avoid trailing comma on last item + +### Step 2c: Save and Access the Template + +Once saved, NetBox exposes the template via: + +``` +http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc+Device+Export +``` + +Or fetch it directly: + +```bash +# Replace with your NetBox URL and template name +curl -H "Authorization: Token YOUR_NETBOX_TOKEN" \ + "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" +``` + +The response is a JSON array of targets ready for gNMIc. + +--- + +## Step 3: Create a TargetProfile + +Define how discovered targets should be configured: + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-devices + namespace: your-namespace +spec: + credentialsRef: device-credentials + timeout: 10s +``` + +--- + +## Step 4: Create a TargetSource Using Export Template + +Create a `TargetSource` that references your NetBox export template endpoint: + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-export-source + namespace: your-namespace +spec: + # Specify the HTTP provider + provider: + http: + # NetBox API endpoint with export template query + # Replace: netbox.example.com, template name (gNMIc+Device+Export), token + url: "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + + # Do not embed plaintext tokens in the TargetSource YAML. Instead, use a secret reference: + token: + scheme: Bearer + tokenSecretRef: + name: netbox-api-token + key: token + + + # Reference the TargetProfile + targetProfile: netbox-device + + # Optional: Apply labels to all discovered targets + targetLabels: + inventory: netbox + sync-source: export-template +``` + +--- + +## Step 5: Verify Target Discovery + +Once the `TargetSource` is deployed, verify that targets are being discovered: + +```bash +# List discovered targets +kubectl get targets -n your-namespace + +# Check TargetSource status and sync details +kubectl describe ts netbox-export-source -n your-namespace +``` + +Successful sync shows: + +- `status.status`: "success" +- `status.targetsCount`: number of devices +- `status.lastSync`: recent timestamp + +--- + +## Example: Complete Setup + +Here's a full example combining all components: + +```yaml +--- +# Secret for NetBox API token +apiVersion: v1 +kind: Secret +metadata: + name: netbox-api-token + namespace: gnmic +type: Opaque +data: + # base64-encoded token (echo -n "YOUR_TOKEN" | base64) + token: YOUR_BASE64_ENCODED_TOKEN + +--- +# TargetProfile for NetBox devices +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-device + namespace: gnmic +spec: + credentialsRef: device-credentials + timeout: 10s + +--- +# TargetSource using Export Template +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-export-source + namespace: gnmic +spec: + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: export-template + +--- +# Device Credentials (referenced by TargetProfile) +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: gnmic +type: Opaque +data: + username: Z25taWM= # base64: gnmic + password: Z25taaWNQYXNz= # base64: gnmicPass +``` + +--- + +## Advantages of Export Templates + +- **Powerful Filtering**: Filter devices by site, status, role, tags, etc. directly in NetBox +- **Reduced Operator Load**: NetBox handles data transformation; operator just fetches JSON +- **Reusability**: One template can serve multiple consumers +- **Maintainability**: Update discovery logic in NetBox without changing Kubernetes manifests +- **Performance**: Avoids REST API pagination for large inventories + +--- + +## Limitations & Considerations + +### 1. Reverse Proxy and URL Path Rewriting + +If NetBox is behind a reverse proxy with URL path rewriting: + +- **Issue**: The export template endpoint uses query parameters that may not survive proxy transformation. +- **Solution**: + - Ensure the proxy preserves query strings exactly. + - Test the export URL directly: + ```bash + curl -H "Authorization: Token YOUR_TOKEN" \ + "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + ``` + - If the proxy blocks or modifies parameters, consider using a direct NetBox endpoint without proxying. + +### 2. Large Inventory Rendering + +- Very large device counts can cause NetBox to take time rendering the template. +- **Solution**: + - Use `.filter()` in your template to limit results. + - Create separate export templates for different device groups (e.g., by site or role). + +### 3. Complex Jinja2 Logic + +- NetBox's Jinja2 sandbox restricts some Python functions for security. +- **Solution**: Keep templates simple and use NetBox's built-in filters and objects. Test in the NetBox UI before deploying. + +--- + +## Template Troubleshooting + +### Missing Data in Output + +- **Check**: Are all required fields populated in NetBox? (e.g., `primary_ip4` may be `None` if not set) +- **Solution**: Add conditional checks: + ```jinja2 + {% if device.primary_ip4 %} + "address": "{{ device.primary_ip4.address.split('/')[0] }}" + {% endif %} + ``` + +### Authorization Fails + +If you get a 403 error: + +- Verify the token is valid and not expired. +- Check token permissions in NetBox admin (User > API Tokens). +- Ensure the API token is enabled. + +--- diff --git a/docs/content/docs/examples/NetBox/_index.md b/docs/content/docs/examples/NetBox/_index.md new file mode 100644 index 0000000..4885d60 --- /dev/null +++ b/docs/content/docs/examples/NetBox/_index.md @@ -0,0 +1,8 @@ +--- +title: "NetBox" +linkTitle: "NetBox" +weight: 6 +# draft: true +description: > + Discover targets from NetBox using the HTTP provider +--- From b9ddb8b7b2723f4965f764667ab8f259daef89c0 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Mon, 1 Jun 2026 14:50:35 +0000 Subject: [PATCH 15/22] imporve export template documentation --- .../examples/NetBox/Export Template/_index.md | 39 ++++++++++++++++--- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/docs/content/docs/examples/NetBox/Export Template/_index.md b/docs/content/docs/examples/NetBox/Export Template/_index.md index 2691a13..23bf20e 100644 --- a/docs/content/docs/examples/NetBox/Export Template/_index.md +++ b/docs/content/docs/examples/NetBox/Export Template/_index.md @@ -33,12 +33,25 @@ When used with gNMIc's HTTP provider, the operator simply fetches the rendered t --- -## Step 1: Create a Secret for the NetBox API Token +## Step 1: Create a NetBox API Token -Create a Kubernetes Secret containing your NetBox API token. This keeps credentials secure and out of your TargetSource manifests. +### Step 1a: Create the API Token in NetBox + +Create a dedicated API token in NetBox for gNMIc Operator access. + +1. Log in to NetBox. +2. Go to **Admin > API Tokens**. +3. Click **Add** or **Add token**. +4. Enter a descriptive name such as `gNMIc Operator`. +6. Do not grant "Write enabled" +7. Copy the token value and store it safely; NetBox will not show it again. + +### Step 1b: Store the Token in a Kubernetes Secret + +Create a Kubernetes Secret containing the token so it is not embedded in manifests. ```bash -# Substitute YOUR_NETBOX_TOKEN with your actual token +# API Token Format: nbt_. kubectl create secret generic netbox-api-token \ --from-literal=token=YOUR_NETBOX_TOKEN \ -n your-namespace @@ -144,19 +157,35 @@ The response is a JSON array of targets ready for gNMIc. ## Step 3: Create a TargetProfile -Define how discovered targets should be configured: +Define how discovered targets should be configured. The `TargetProfile` points to a Secret containing device credentials, such as username/password or client certificates. + +Create a credentials Secret first, then reference it from the profile. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: your-namespace +type: Opaque +stringData: + username: gnmic + password: gnmicPass +``` ```yaml apiVersion: operator.gnmic.dev/v1alpha1 kind: TargetProfile metadata: - name: netbox-devices + name: netbox-device namespace: your-namespace spec: credentialsRef: device-credentials timeout: 10s ``` +For more TargetProfile options and credential handling, see the operator documentation for `TargetProfile`. + --- ## Step 4: Create a TargetSource Using Export Template From 9b7dc0b6b5155788e6bc4adcc44800964ebacdb1 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Mon, 1 Jun 2026 09:38:00 -0600 Subject: [PATCH 16/22] docs: made TargetProfile optional --- docs/content/docs/user-guide/targetsource/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/docs/user-guide/targetsource/_index.md b/docs/content/docs/user-guide/targetsource/_index.md index 7e957ec..10157b7 100644 --- a/docs/content/docs/user-guide/targetsource/_index.md +++ b/docs/content/docs/user-guide/targetsource/_index.md @@ -30,7 +30,7 @@ spec: |-------|------|----------|-------------| | `provider` | object | Yes | Provider-specific discovery configuration. Exactly one provider must be configured | | `targetPort` | int32 | No | Default port used when the discovered target does not provide a port | -| `targetProfile` | string | Yes | Reference to `TargetProfile` applied to all discovered targets | +| `targetProfile` | string | No | Reference to default `TargetProfile` applied to all discovered targets if no profile was discovered | | `targetLabels` | map[string]string | No | Labels added to all discovered targets | From b47901792e21496107c2ca7c13179f7c95dfb199 Mon Sep 17 00:00:00 2001 From: Valentino Diller Date: Mon, 1 Jun 2026 10:26:08 -0600 Subject: [PATCH 17/22] changed authorization to authentication --- docs/content/docs/reference/api.md | 4 ++-- docs/content/docs/user-guide/targetsource/http.md | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/content/docs/reference/api.md b/docs/content/docs/reference/api.md index 8c6a41f..d20cb3e 100644 --- a/docs/content/docs/reference/api.md +++ b/docs/content/docs/reference/api.md @@ -172,7 +172,7 @@ description: > | `method` | string | No | GET | HTTP request method | | `headers` | map[string]string | No | - | HTTP headers to include in requests | | `body` | string | No | - | Request body for POST requests | -| `authorization` | AuthorizationSpec | No | - | Authentication configuration for the HTTP endpoint | +| `authentication` | AuthenticationSpec | No | - | Authentication configuration for the HTTP endpoint | | `interval` | duration | No | 6h | Polling interval used to refresh targets | | `timeout` | duration | No | 10s | Timeout for HTTP requests | | `tls` | ClientTLSConfig | No | - | Client TLS configuration for HTTPS endpoints | @@ -187,7 +187,7 @@ description: > | `insecureSkipVerify` | bool | No | false | Skip verification of the server certificate | | `caBundleRef` | ConfigMapKeySelector | No | - | Reference to a ConfigMap containing a PEM CA bundle | -### AuthorizationSpec +### AuthenticationSpec | Field | Type | Required | Description | |-------|------|----------|-------------| diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index df1cfd6..2c93fce 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -16,7 +16,7 @@ The HTTP provider discovers targets from an HTTP endpoint returning JSON, or rec | `method` | string | No | GET | HTTP method used for requests | | `headers` | map[string]string | No | - | HTTP headers to include in requests | | `body` | string | No | - | Request body for POST requests | -| `authorization` | object | No | - | Authentication configuration for the HTTP endpoint | +| `authentication` | object | No | - | Authentication configuration for the HTTP endpoint | | `interval` | duration | No | 30m | Polling interval used to refresh targets | | `timeout` | duration | No | 30s | Timeout for HTTP requests | | `tls` | object | No | - | Client TLS configuration for HTTPS endpoints | @@ -57,11 +57,11 @@ spec: When `push.enabled` is true, the operator accepts incoming webhook notifications and can update targets without polling a remote endpoint. The `url` field is optional when push mode is enabled, but can still be used for polling and fallback behavior. -## Authorization +## Authentication The HTTP provider supports authenticated requests to the inventory endpoint. -Exactly one authorization method can be configured. +Exactly one authentication method can be configured. ### Basic Authentication @@ -72,7 +72,7 @@ spec: provider: http: url: https://inventory.example.com/targets - authorization: + authentication: basic: credentialsSecretRef: name: inventory-credentials @@ -88,7 +88,7 @@ spec: provider: http: url: https://inventory.example.com/targets - authorization: + authentication: token: scheme: Bearer tokenSecretRef: @@ -484,7 +484,7 @@ tls: Example: ```yaml -authorization: +authentication: token: scheme: Bearer tokenSecretRef: @@ -563,7 +563,7 @@ spec: key: ca.crt # Authentication - authorization: + authentication: token: scheme: Bearer tokenSecretRef: From 50866482b21335aa679c3471b466ef5d87d1e3c1 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Tue, 2 Jun 2026 08:51:57 +0000 Subject: [PATCH 18/22] add netbox rest api example --- .../docs/examples/NetBox/REST API/_index.md | 318 ++++++++++++++++++ 1 file changed, 318 insertions(+) create mode 100644 docs/content/docs/examples/NetBox/REST API/_index.md diff --git a/docs/content/docs/examples/NetBox/REST API/_index.md b/docs/content/docs/examples/NetBox/REST API/_index.md new file mode 100644 index 0000000..7998069 --- /dev/null +++ b/docs/content/docs/examples/NetBox/REST API/_index.md @@ -0,0 +1,318 @@ +--- +title: "NetBox (REST API)" +linkTitle: "NetBox REST" +weight: 2 +description: > + Discover targets from NetBox using the HTTP provider and NetBox REST API +--- + +This guide shows how to configure the HTTP provider to discover targets from NetBox using its REST API. + +The REST API approach is direct and straightforward — query NetBox's standard API endpoints to retrieve devices that match your criteria. + +## Prerequisites + +- A running Kubernetes cluster with gNMIc Operator installed +- `kubectl` access to your cluster +- A reachable NetBox instance (inside or outside the cluster) +- A NetBox API token + +## Overview + +The HTTP `TargetSource` loader performs these steps: + +1. **Fetch** JSON device data from a NetBox REST API endpoint (`/api/dcim/devices/`) +2. **Transform** each device record into a gNMIc target using CEL expressions +3. **Create** or **update** `Target` resources in Kubernetes with the extracted data + +--- + +## Step 1: Create a NetBox API Token and Store It Securely + +### Step 1a: Create the API Token in NetBox + +Create a dedicated API token in NetBox for gNMIc Operator access. + +1. Log in to NetBox. +2. Open your user profile or go to **User > API Tokens**. +3. Click **Add** or **Add token**. +4. Enter a descriptive name such as `gNMIc Operator`. +5. Grant the minimum permissions required for read-only device discovery. +6. Copy the token value and store it safely; NetBox will not show it again. + +### Step 1b: Store the Token in a Kubernetes Secret + +Create a Kubernetes Secret containing the token so it is not embedded in manifests. + +```bash +# Substitute YOUR_NETBOX_API_TOKEN with your actual token +# Bearer Token Format (v2): nbt_. +kubectl create secret generic netbox-api-token \ + --from-literal=token=YOUR_NETBOX_API_TOKEN \ + -n your-namespace +``` + +Verify the Secret was created: + +```bash +kubectl get secret netbox-api-token -n your-namespace -o yaml +``` + +--- + +## Step 2: Create a TargetProfile + +Define how discovered targets should be configured. The `TargetProfile` points to a Secret containing device credentials, such as username/password or client certificates. + +Create a credentials Secret first, then reference it from the profile. + +```yaml +# Replace YOUR_DEVICE_USERNAME and YOUR_DEVICE_PASSWORD with your corresponding default device username and password +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: your-namespace +type: Opaque +stringData: + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD +``` + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-device + namespace: your-namespace +spec: + credentialsRef: device-credentials + timeout: 10s +``` + +For more TargetProfile options and credential handling, see the operator documentation for `TargetProfile`. + +--- + +## Step 3: Create a TargetSource Using REST API + +The following `TargetSource` queries NetBox's REST API to discover devices: + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-rest-source + namespace: your-namespace +spec: + targetPort: 57400 + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: rest-api + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?limit=1000" + method: GET + interval: 5m + timeout: 30s + authorization: + token: + scheme: Bearer + tokenSecretRef: + name: netbox-api-token + key: token + mapping: + targetsField: "self.results" + address: "item.primary_ip4 != null ? item.primary_ip4.address.split('/')[0] : ''" + labels: | + { + "site": item.site.name, + "role": item.device_role.name, + "model": item.device_type.model, + "status": item.status.value + } +``` + + +The HTTP loader supports `targetsField` and individual CEL expressions for: + +- `name` +- `address` +- `port` +- `labels` +- `targetProfile` + +Use `self` for the full response and `item` for each candidate object. + + +--- + +## Step 4: Verify Target Discovery + +Once the `TargetSource` is deployed, check that targets are being discovered and synced: + +```bash +# List discovered targets +kubectl get targets -n your-namespace + +# Check TargetSource status +kubectl describe targetsource netbox-rest-source -n your-namespace +``` + +Look for: +- `status.status`: "success" (or similar) +- `status.targetsCount`: number of discovered devices +- `status.lastSync`: recent timestamp + +--- + +## Example: Complete Setup + +Here's a complete example combining all resources: + +```yaml +--- +# Secret for NetBox API token +apiVersion: v1 +kind: Secret +metadata: + name: netbox-api-token + namespace: gnmic +type: Opaque +data: + # base64-encoded token (echo -n "YOUR_TOKEN" | base64) + token: YOUR_BASE64_ENCODED_TOKEN + +--- +# Secret for Target Credential +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: your-namespace +type: Opaque +stringData: + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD + +--- +# TargetProfile +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-device + namespace: your-namespace +spec: + credentialsRef: device-credentials + timeout: 10s + +--- +# TargetSource with REST API +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-rest-source + namespace: gnmic +spec: + targetPort: 57400 + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: rest-api + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?limit=1000" + method: GET + interval: 5m + timeout: 30s + authorization: + token: + scheme: Bearer + tokenSecretRef: + name: netbox-api-token + key: token + mapping: + targetsField: "self.results" + address: "item.primary_ip4 != null ? item.primary_ip4.address.split('/')[0] : ''" + labels: | + { + "site": item.site.name, + "role": item.device_role.name, + "model": item.device_type.model, + "status": item.status.value + } +``` + +--- + +## Performance Considerations & Limitations + +### REST API Query Limits + +- **Query Size**: The example uses `limit=1000`. Adjust based on your NetBox instance's pagination settings and response size limits. +- **Response Timeout**: Large device lists can take time. Set appropriate timeouts in your `TargetSource`. + +### Reverse Proxy Considerations + +If NetBox is behind a reverse proxy: + +- **Base URL**: Ensure the reverse proxy correctly handles the `/api/dcim/devices/` path. +- **Authentication**: Some proxies may require additional headers; verify with your proxy and NetBox admin. +- **HTTPS**: If using HTTPS, ensure certificates are trusted by the operator or else use the `tls` setting. + +### Large Inventories + +For inventories with thousands of devices: + +- Consider using **Export Templates** (see [NetBox Export Templates]({{< relref "../Export Template" >}})) for better filtering and performance. +- Implement pagination or filtering in the REST API URL (e.g., `?site=us-west&status=active`). + +--- + +## Security Considerations + +### Token Storage + +- **Never** embed plaintext tokens in manifests or YAML files. +- Always store tokens in Kubernetes Secrets. +- Restrict RBAC permissions on the Secret to only necessary service accounts. + +### HTTPS and Certificates + +If connecting to NetBox via HTTPS: + +- Ensure cluster DNS resolves the hostname correctly. +- Mount CA certificates if using self-signed certificates. +- Verify the operator's HTTP client configuration for certificate validation. + +--- + +## Troubleshooting + +### Show TargetSource Errors + +```bash +kubectl describe targetsource netbox-rest-source -n your-namespace +``` + +### Targets Not Appearing + +- Check that the `TargetProfile` exists and is correctly referenced. +- Verify labels and addresses are being extracted correctly from the NetBox response. +- Review operator logs for parsing errors: + ```bash + kubectl logs -l app=gnmic-operator -n gnmic-operator-system + ``` + +### Rate Limiting or Timeouts + +Increase the sync interval in your `TargetSource` or adjust timeouts: + +```yaml +spec: + provider: + http: + interval: 1h + timeout: 1m +``` From fc6af9cf725484250c3e8aee3534952b5bf5ccad Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Tue, 2 Jun 2026 09:11:06 +0000 Subject: [PATCH 19/22] fix incorrect namespace --- docs/content/docs/examples/NetBox/REST API/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/content/docs/examples/NetBox/REST API/_index.md b/docs/content/docs/examples/NetBox/REST API/_index.md index 7998069..33d486f 100644 --- a/docs/content/docs/examples/NetBox/REST API/_index.md +++ b/docs/content/docs/examples/NetBox/REST API/_index.md @@ -178,7 +178,7 @@ apiVersion: v1 kind: Secret metadata: name: netbox-api-token - namespace: gnmic + namespace: your-namespace type: Opaque data: # base64-encoded token (echo -n "YOUR_TOKEN" | base64) @@ -213,7 +213,7 @@ apiVersion: operator.gnmic.dev/v1alpha1 kind: TargetSource metadata: name: netbox-rest-source - namespace: gnmic + namespace: your-namespace spec: targetPort: 57400 targetProfile: netbox-device From f0d63ffed106ce5c0124a24f3e880af959ce33fc Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Tue, 2 Jun 2026 09:11:17 +0000 Subject: [PATCH 20/22] update netbox export template guide --- .../examples/NetBox/Export Template/_index.md | 185 +++++++++--------- 1 file changed, 96 insertions(+), 89 deletions(-) diff --git a/docs/content/docs/examples/NetBox/Export Template/_index.md b/docs/content/docs/examples/NetBox/Export Template/_index.md index 23bf20e..ad6c35f 100644 --- a/docs/content/docs/examples/NetBox/Export Template/_index.md +++ b/docs/content/docs/examples/NetBox/Export Template/_index.md @@ -1,14 +1,14 @@ --- title: "NetBox (Export Template)" -linkTitle: "NetBox Export" +linkTitle: "NetBox Export Template" weight: 3 description: > - Discover targets from NetBox using HTTP provider with export templates + Discover targets from NetBox using HTTP provider with NetBox Export Template --- This guide shows how to use **NetBox Export Templates** with the HTTP provider to discover and sync targets. -Export Templates offer powerful filtering, transformation, and formatting directly in NetBox, reducing load on the operator and enabling complex discovery logic. +Export Templates offer powerful filtering, transformation, and formatting directly in NetBox, reducing the load on the operator. ## Overview @@ -19,41 +19,42 @@ An **Export Template** is a Jinja2 template defined in NetBox that: 3. **Transforms** data into your desired output format (JSON, YAML, CSV, etc.) 4. **Returns** the formatted output via a custom REST API endpoint -When used with gNMIc's HTTP provider, the operator simply fetches the rendered template and parses the result — no additional transformation needed. +When used with gNMIc's HTTP provider, the operator simply fetches the rendered template and parses the result — no additional gNMIc Operator transformation needed if done correctly. --- ## Prerequisites - A running Kubernetes cluster with gNMIc Operator installed -- A reachable NetBox instance with **permissions to create Export Templates** -- A NetBox API token - `kubectl` access to your cluster +- A reachable NetBox instance with permissions to create Export Templates +- A NetBox API token - Familiarity with Jinja2 templates --- -## Step 1: Create a NetBox API Token +## Step 1: Create a NetBox API Token and Store It Securely ### Step 1a: Create the API Token in NetBox Create a dedicated API token in NetBox for gNMIc Operator access. 1. Log in to NetBox. -2. Go to **Admin > API Tokens**. +2. Open your user profile or go to **User > API Tokens**. 3. Click **Add** or **Add token**. 4. Enter a descriptive name such as `gNMIc Operator`. -6. Do not grant "Write enabled" -7. Copy the token value and store it safely; NetBox will not show it again. +5. Grant the minimum permissions required for read-only device discovery. +6. Copy the token value and store it safely; NetBox will not show it again. ### Step 1b: Store the Token in a Kubernetes Secret Create a Kubernetes Secret containing the token so it is not embedded in manifests. ```bash -# API Token Format: nbt_. +# Substitute YOUR_NETBOX_API_TOKEN with your actual token +# Bearer Token Format (v2): nbt_. kubectl create secret generic netbox-api-token \ - --from-literal=token=YOUR_NETBOX_TOKEN \ + --from-literal=token=YOUR_NETBOX_API_TOKEN \ -n your-namespace ``` @@ -86,45 +87,41 @@ Click **Add Export Template** and fill in the details: #### Basic Template (All Devices) ```jinja2 -{ - "targets": [ - {% for device in queryset %} - { - "name": "{{ device.name }}", - "address": "{{ device.primary_ip4.address.split('/')[0] }}:57400", - "labels": { - "site": "{{ device.site.name }}", - "role": "{{ device.device_role.name }}", - "region": "{{ device.site.region.name }}", - "type": "{{ device.device_type.model }}" - } - }{{ "," if not loop.last }} - {% endfor %} - ] -} +[ + {% for device in queryset %} + { + "name": "{{ device.name }}", + "address": "{{ device.primary_ip4.address.split('/')[0] }}", + "labels": { + "site": "{{ device.site.name }}", + "role": "{{ device.device_role.name }}", + "region": "{{ device.site.region.name }}", + "type": "{{ device.device_type.model }}" + } + }{{ "," if not loop.last }} + {% endfor %} +] ``` #### Advanced Template (Filtered by Status and Role) ```jinja2 -{ - "targets": [ - {% for device in queryset.filter(status='active', device_role__name__in=['leaf', 'spine']) %} - { - "name": "{{ device.name }}", - "address": "{{ device.primary_ip4.address.split('/')[0] }}:57400", - "labels": { - "site": "{{ device.site.name }}", - "role": "{{ device.device_role.name }}", - "region": "{{ device.site.region.name }}", - "model": "{{ device.device_type.model }}", - "serial": "{{ device.serial }}", - "asset_tag": "{{ device.asset_tag }}" - } - }{{ "," if not loop.last }} - {% endfor %} - ] -} +[ + {% for device in queryset.filter(status='active', device_role__name__in=['leaf', 'spine']) %} + { + "name": "{{ device.name }}", + "address": "{{ device.primary_ip4.address.split('/')[0] }}", + "labels": { + "site": "{{ device.site.name }}", + "role": "{{ device.device_role.name }}", + "region": "{{ device.site.region.name }}", + "model": "{{ device.device_type.model }}", + "serial": "{{ device.serial }}", + "asset_tag": "{{ device.asset_tag }}" + } + }{{ "," if not loop.last }} + {% endfor %} +] ``` **Key template elements:** @@ -147,12 +144,16 @@ Or fetch it directly: ```bash # Replace with your NetBox URL and template name -curl -H "Authorization: Token YOUR_NETBOX_TOKEN" \ +# Substitute YOUR_NETBOX_API_TOKEN with your actual token +# Bearer Token Format (v2): nbt_. +curl -H "Authorization: Bearer YOUR_NETBOX_API_TOKEN" \ "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" ``` The response is a JSON array of targets ready for gNMIc. +> If you instead return a JSON object with a nested array, add a mapping section such as `targetsField: "self.targets"` to the TargetSource CR. + --- ## Step 3: Create a TargetProfile @@ -162,6 +163,7 @@ Define how discovered targets should be configured. The `TargetProfile` points t Create a credentials Secret first, then reference it from the profile. ```yaml +# Replace YOUR_DEVICE_USERNAME and YOUR_DEVICE_PASSWORD with your corresponding default device username and password apiVersion: v1 kind: Secret metadata: @@ -169,8 +171,8 @@ metadata: namespace: your-namespace type: Opaque stringData: - username: gnmic - password: gnmicPass + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD ``` ```yaml @@ -199,28 +201,23 @@ metadata: name: netbox-export-source namespace: your-namespace spec: - # Specify the HTTP provider + targetPort: 57400 + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: export-template provider: http: - # NetBox API endpoint with export template query - # Replace: netbox.example.com, template name (gNMIc+Device+Export), token url: "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" - - # Do not embed plaintext tokens in the TargetSource YAML. Instead, use a secret reference: - token: - scheme: Bearer + method: GET + interval: 30m + timeout: 30s + authorization: + token: + scheme: Token tokenSecretRef: name: netbox-api-token key: token - - - # Reference the TargetProfile - targetProfile: netbox-device - - # Optional: Apply labels to all discovered targets - targetLabels: - inventory: netbox - sync-source: export-template ``` --- @@ -234,12 +231,12 @@ Once the `TargetSource` is deployed, verify that targets are being discovered: kubectl get targets -n your-namespace # Check TargetSource status and sync details -kubectl describe ts netbox-export-source -n your-namespace +kubectl describe targetsource netbox-export-source -n your-namespace ``` Successful sync shows: -- `status.status`: "success" +- `status.status`: "success" (or similar) - `status.targetsCount`: number of devices - `status.lastSync`: recent timestamp @@ -256,50 +253,61 @@ apiVersion: v1 kind: Secret metadata: name: netbox-api-token - namespace: gnmic + namespace: your-namespace type: Opaque data: # base64-encoded token (echo -n "YOUR_TOKEN" | base64) token: YOUR_BASE64_ENCODED_TOKEN --- -# TargetProfile for NetBox devices +# Secret for Target Credential +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: your-namespace +type: Opaque +stringData: + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD + +--- +# TargetProfile apiVersion: operator.gnmic.dev/v1alpha1 kind: TargetProfile metadata: name: netbox-device - namespace: gnmic + namespace: your-namespace spec: credentialsRef: device-credentials timeout: 10s + --- # TargetSource using Export Template apiVersion: operator.gnmic.dev/v1alpha1 kind: TargetSource metadata: name: netbox-export-source - namespace: gnmic + namespace: your-namespace spec: - provider: - http: - url: "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + targetPort: 57400 targetProfile: netbox-device targetLabels: inventory: netbox sync-source: export-template - ---- -# Device Credentials (referenced by TargetProfile) -apiVersion: v1 -kind: Secret -metadata: - name: device-credentials - namespace: gnmic -type: Opaque -data: - username: Z25taWM= # base64: gnmic - password: Z25taaWNQYXNz= # base64: gnmicPass + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + method: GET + interval: 30m + timeout: 30s + authorization: + token: + scheme: Token + tokenSecretRef: + name: netbox-api-token + key: token ``` --- @@ -340,13 +348,13 @@ If NetBox is behind a reverse proxy with URL path rewriting: ### 3. Complex Jinja2 Logic - NetBox's Jinja2 sandbox restricts some Python functions for security. -- **Solution**: Keep templates simple and use NetBox's built-in filters and objects. Test in the NetBox UI before deploying. +- **Solution**: Keep templates simple and use NetBox's built-in filters and objects. Test URL with curl or similar before deploying. --- ## Template Troubleshooting -### Missing Data in Output +### Missing Targets in Kubernetes - **Check**: Are all required fields populated in NetBox? (e.g., `primary_ip4` may be `None` if not set) - **Solution**: Add conditional checks: @@ -361,7 +369,6 @@ If NetBox is behind a reverse proxy with URL path rewriting: If you get a 403 error: - Verify the token is valid and not expired. -- Check token permissions in NetBox admin (User > API Tokens). - Ensure the API token is enabled. --- From 8aa4ac842a424d5eeacf59cb5846e29631099948 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Tue, 2 Jun 2026 09:13:21 +0000 Subject: [PATCH 21/22] update authorization to authentication --- docs/content/docs/examples/NetBox/Export Template/_index.md | 4 ++-- docs/content/docs/examples/NetBox/REST API/_index.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/content/docs/examples/NetBox/Export Template/_index.md b/docs/content/docs/examples/NetBox/Export Template/_index.md index ad6c35f..cb360b0 100644 --- a/docs/content/docs/examples/NetBox/Export Template/_index.md +++ b/docs/content/docs/examples/NetBox/Export Template/_index.md @@ -212,7 +212,7 @@ spec: method: GET interval: 30m timeout: 30s - authorization: + authentication: token: scheme: Token tokenSecretRef: @@ -302,7 +302,7 @@ spec: method: GET interval: 30m timeout: 30s - authorization: + authentication: token: scheme: Token tokenSecretRef: diff --git a/docs/content/docs/examples/NetBox/REST API/_index.md b/docs/content/docs/examples/NetBox/REST API/_index.md index 33d486f..1ae58b9 100644 --- a/docs/content/docs/examples/NetBox/REST API/_index.md +++ b/docs/content/docs/examples/NetBox/REST API/_index.md @@ -116,7 +116,7 @@ spec: method: GET interval: 5m timeout: 30s - authorization: + authentication: token: scheme: Bearer tokenSecretRef: @@ -226,7 +226,7 @@ spec: method: GET interval: 5m timeout: 30s - authorization: + authentication: token: scheme: Bearer tokenSecretRef: From c3c58af562f74adf7ac2255661eaa08d18f041a0 Mon Sep 17 00:00:00 2001 From: Daniel Schatzmann Date: Tue, 2 Jun 2026 13:04:59 +0000 Subject: [PATCH 22/22] re-word credentialsSecretRef --- docs/content/docs/user-guide/targetsource/http.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/docs/user-guide/targetsource/http.md b/docs/content/docs/user-guide/targetsource/http.md index 2c93fce..96ef58c 100644 --- a/docs/content/docs/user-guide/targetsource/http.md +++ b/docs/content/docs/user-guide/targetsource/http.md @@ -74,7 +74,7 @@ spec: url: https://inventory.example.com/targets authentication: basic: - credentialsSecretRef: + credentialSecretRef: name: inventory-credentials key: username ```