Skip to content

feat(eval): automated skill assertion & prompt regression test runner (cinch eval) (#19) - #43

Merged
00200200 merged 2 commits into
mainfrom
feat/eval-runner
Oct 6, 2026
Merged

00200200 merged 2 commits into
mainfrom
feat/eval-runner

Conversation

@00200200

@00200200 00200200 commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Closes #19

Summary of Changes

  • Implemented cinch eval CLI subcommand for running test suites against project skills.
  • Supported assertions:
    • contains: substring check
    • not_contains: negative substring check
    • matches: regex pattern check
    • not_matches: negative regex check
    • max_length: maximum length constraint
    • min_length: minimum length constraint
    • json_schema: valid JSON and schema structure validation
  • Generates standard JUnit XML test reports via --output-junit <path>.
  • Rich-formatted console output table reporting test passes/failures.
  • Full test coverage in tests/test_eval.py.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

👋 Thanks for opening this pull request, @00200200!

We're thrilled to have your contribution to cinch! Here is a quick checklist to help get your PR merged smoothly:

  • Linked Issue: Issue reference detected in PR body. ✅
  • Local Verification: Verify all checks pass locally using uv run pytest and uv run ruff check ..
  • Commit Hygiene: Ensure commit messages follow conventional commit style (feat:, fix:, docs:).

A maintainer will review your code shortly. Feel free to ping in comments if you have any questions! 🚀

@github-actions github-actions Bot added enhancement New feature or request tests Testing and quality assurance labels Oct 6, 2026
@00200200
00200200 merged commit 8b6fd54 into main Oct 6, 2026
7 checks passed
@00200200
00200200 deleted the feat/eval-runner branch October 6, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request tests Testing and quality assurance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(eval): Automated skill assertion & prompt regression test runner (cinch eval)

1 participant