Skip to content

test: add error message quality validation suite #155

@membphis

Description

@membphis

Background

qjson reports parse errors with byte offsets (qjson_format_error), but there's no systematic validation that error messages are:

  • Consistent across similar error types
  • Informative enough for debugging
  • Stable across versions (no accidental message changes)

V8 maintains error message consistency as part of its test suite.

Goal

Add a test suite that validates error message quality and stability.

Scope

Validation Dimensions

  1. Offset accuracy: Error offset points to the actual problematic byte
  2. Message clarity: Message describes what was expected vs. found
  3. Consistency: Similar errors produce similar message patterns
  4. Completeness: All error codes have corresponding message templates
  5. Stability: Message format doesn't change unintentionally

Test Cases

Input Expected Message Pattern
{ parse error at byte 1: unexpected end of input, expected ...
{"a":} parse error at byte 5: unexpected '}', expected value
[01] invalid number at byte 1: leading zeros not allowed
{"\xff"} invalid UTF-8 at byte 2: ...
[[[[...1024 deep max depth exceeded at byte N

Snapshot Testing

Consider snapshot-based testing: store expected messages in files, fail on drift.

Acceptance Criteria

  • Test file covering all qjson_err variants with expected message patterns
  • Each error code has at least one test case validating its message
  • Messages are validated for presence of: error type, byte offset, context hint
  • CI fails if message format changes unexpectedly (snapshot or pattern match)

Technical Notes

  • Build on existing tests/ffi_parse_error_offsets.rs
  • Use qjson_format_error FFI function for message generation
  • Consider adding line:column in addition to byte offset (future enhancement)

References

  • Current error formatting: src/error.rs, qjson_format_error in src/ffi.rs
  • Existing offset tests: tests/ffi_parse_error_offsets.rs

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions