Repository navigation
[ci] Add mock LLM for Lightspeed deployment testing - #4222
malingatembo wants to merge 13 commits into
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 35m 50s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 47m 27s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 13m 42s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 03m 01s |
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 19s |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
openshift-ci
Bot
added
do-not-merge/contains-merge-commits
and removed
needs-rebase
labels
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 14m 59s |
7261428 to
efd9055
Compare
openshift-ci
Bot
added
needs-rebase
and removed
do-not-merge/contains-merge-commits
labels
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
efd9055 to
908e89a
Compare
openshift-ci
Bot
added
do-not-merge/contains-merge-commits
and removed
needs-rebase
labels
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 14m 27s |
9f515ff to
6ad71b7
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 46s |
lpiwowar
left a comment
lpiwowar
left a comment
There was a problem hiding this comment.
Comments from the meeting:)
Deploy a mock provider so Lightspeed can be tested without external LLM credentials. Support ordinary and streamed chat completions required by the current provider adapter. Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
The upstream mock-provider deployment timed out while application images were still starting. The collected custom resource became Ready about 50 seconds after the playbook failed. Increase the default CR readiness retries from 30 to 120 while retaining the ten-second polling interval, Ready condition and bounded failure. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
The operator now requires models and defaultModel. Move the existing provider settings into one configured model and select it as the default so the deployment hook can create the CR against the current operator. Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Wait for mock readiness, reload changed mock code, and allow bounded application startup. Run the prepared Lightspeed test checkout with an explicit client identity and retain JUnit/logs before propagating failure. Remove private client token files and API tunnels on success or failure. Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Serve GET /v1/models with the configured provider Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Recognize the project terms used in the Lightspeed testing guide. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Execute the prepared test source in an AnsibleTest Pod with explicit client authentication and PVC report collection. Replace the controller test hook and support stage-specific storage and security settings. Local checks passed. CRC validation is deferred; combined CI validation is pending. Reusing completed test resources remains a known limitation. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Request real document retrieval before the fixed answer so the OKP tests can validate retrieved documentation. Support JSON and streaming tool calls, and return the answer after the tool result to avoid repeated retrieval. Related-Issue: OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Add an optional OGX image override to the deployment hook while retaining operator defaults when unset. Document the parameter and recognize OGX in documentation spelling checks. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Restore the Lightspeed access role and binding plus the test-operator ConfigMap and two Secrets accidentally dropped during conflict resolution. Preserve upstream callers and retain the mock retrieval, readiness handling and optional OGX image override. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
Remove custom source packaging and bootstrap execution in favour of direct AnsibleTest configuration in the consuming job. Restore the shared role, remove the associated PR-added tests, and keep test-account permissions in the Lightspeed installer. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
86a94ba to
a8dabab
Compare
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 4h 45m 33s |
lpiwowar
left a comment
lpiwowar
left a comment
There was a problem hiding this comment.
Just a couple of non-blocking comments from the call:)
| - name: Create ClusterRole for Lightspeed tests | ||
| kubernetes.core.k8s: | ||
| state: present | ||
| definition: | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRole | ||
| metadata: | ||
| name: "lightspeed-test-role" | ||
| rules: | ||
| - nonResourceURLs: ["/ls-access"] | ||
| verbs: ["get"] | ||
| register: _result | ||
| retries: 5 | ||
| delay: 10 | ||
| until: _result is not failed |
There was a problem hiding this comment.
nit (non-blocking): Thanks to this PR -> openstack-k8s-operators/lightspeed-operator#133 we no longer have to create the ClusterRole here probably. We can just use the one created by the operator. But for not it is good as-is. We can always polish later.
Separate test-operator access and dummy prerequisites from service deployment, use the configured test namespace directly, and remove redundant rescue diagnostics. Preserve task behaviour and the existing must-gather collection. Validation: production ansible-lint, syntax and documentation build; offline task/object parity, namespace and natural failure-propagation checks passed. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 36m 19s Warning: |
There was a problem hiding this comment.
FYI, you can add these changes into the lightspeed operator itself, you don't need to have them in the ci-framework.
This would have the advantage of having docs local to the lightspeed operator, and also have the changes automatically tested in the CI jobs that lightspeed relies on.
You can find some examples of this in the telemetry-operator repo.
We have a playbook in telemetry-operator/ci: https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/cloudkitty-pre_deploy-install_loki.yml
We define the hook in: https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/vars-cloudkitty-tempest.yml#L20C1-L22C17 , using some zuul vars to reference the location of the checked out repo/playbook.
Then this is pulled in via cifmw_extras: https://github.com/openstack-k8s-operators/telemetry-operator/blob/15c5aef9b929d06f0338a4cc2910c9fdbe490381/zuul.d/projects.yaml#L160
| - name: Create ClusterRole for Lightspeed tests | ||
| kubernetes.core.k8s: | ||
| state: present | ||
| definition: | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRole | ||
| metadata: | ||
| name: "lightspeed-test-role" | ||
| rules: | ||
| - nonResourceURLs: ["/ls-access"] | ||
| verbs: ["get"] | ||
| register: _result | ||
| retries: 5 | ||
| delay: 10 | ||
| until: _result is not failed | ||
|
|
||
| - name: Create ClusterRoleBinding for Lightspeed tests | ||
| kubernetes.core.k8s: | ||
| state: present | ||
| definition: | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRoleBinding | ||
| metadata: | ||
| name: "lightspeed-test-role-binding" | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: ClusterRole | ||
| name: "lightspeed-test-role" | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: "default" | ||
| namespace: "{{ openstack_lightspeed_namespace }}" | ||
| register: _result | ||
| retries: 5 | ||
| delay: 10 | ||
| until: _result is not failed |
There was a problem hiding this comment.
nit (non-blocking): This is probably not needed as mentioned here -> #4222 (comment)
| - name: Verify LLM credentials secret exists | ||
| kubernetes.core.k8s_info: | ||
| kind: Secret | ||
| name: "{{ cifmw_openstack_lightspeed_llm_credentials | default('openstack-lightspeed-apitoken') }}" | ||
| namespace: "{{ openstack_lightspeed_namespace }}" | ||
| stringData: | ||
| apitoken: "{{ cifmw_openstack_lightspeed_api_token }}" | ||
| when: cifmw_openstack_lightspeed_api_token is defined | ||
| no_log: true | ||
|
|
||
| - name: Verify LLM credentials secret exists | ||
| kubernetes.core.k8s_info: | ||
| kind: Secret | ||
| name: "{{ cifmw_openstack_lightspeed_llm_credentials | default('openstack-lightspeed-apitoken') }}" | ||
| namespace: "{{ openstack_lightspeed_namespace }}" | ||
| register: secret_check | ||
| failed_when: secret_check.resources | length == 0 | ||
| when: cifmw_openstack_lightspeed_api_token is not defined | ||
| no_log: true | ||
|
|
||
| - name: Download TLS CA certificate from URL | ||
| ansible.builtin.uri: | ||
| url: "{{ cifmw_openstack_lightspeed_ca_cert_url }}" | ||
| return_content: true | ||
| register: ca_cert_download | ||
| when: cifmw_openstack_lightspeed_ca_cert_url is defined | ||
|
|
||
| - name: Create TLS CA certificate bundle ConfigMap | ||
| kubernetes.core.k8s: | ||
| state: present | ||
| definition: | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| register: secret_check | ||
| failed_when: secret_check.resources | length == 0 | ||
| when: cifmw_openstack_lightspeed_api_token is not defined | ||
| no_log: true |
There was a problem hiding this comment.
nit (non-blocking): We do not verify the existence of the secrete. The previous task already guarantees that.
Use the operator-managed lightspeed-access ClusterRole for the test Pod's ServiceAccount. Remove duplicate roles and the unused application-account binding while retaining authorization in the configured test namespace. Give the replacement binding a distinct name to avoid changing an existing binding's immutable roleRef. Update the test-access documentation. Related-Issue: #OSPRH-33344 Signed-off-by: malingatembo <tembomalinga@gmail.com>
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 21m 22s Warning: |
[ci] Add mock LLM for Lightspeed deployment testing
Deploy a mock LLM so Lightspeed can be tested without external provider
credentials. The mock supports ordinary JSON and streamed chat completions.
Validated locally on CRC:
Syntax and formatting checks passed. Ansible lint did not complete.
Remaining before final review:
Query checks were manual; automated lightspeed-tests integration remains
separate work..
Add a controller hook that runs the prepared lightspeed-tests checkout after
Lightspeed reaches Ready. The hook supplies client authentication, retains
JUnit reports and runner logs, propagates test failures, and cleans up its
temporary token files and API tunnel.
Test-runner changes:
openstack-k8s-operators/lightspeed-tests#17
Combined upstream integration validation remains pending.
Related-Issue: OSPRH-33344