fix: strip NUL bytes from document metadata before pgvector insert

[nangelovv]

What

Strip NUL bytes (\u0000) from document metadata before it reaches the
pgvector cmetadata JSONB column, so PDFs with a NUL in fields like producer
can be embedded instead of failing the whole batch insert.

Fixes #295.

Why

cmetadata is JSONB and PostgreSQL cannot store \u0000 in text/JSONB.
PDFs produced by Canon / Adobe PSL PostScript drivers (and some Google-Docs
exports) pad producer/creator/title with a trailing NUL. The unsanitized
metadata merge in _prepare_documents_sync lets that NUL flow into the insert,
which PostgreSQL rejects with:

psycopg2.errors.UntranslatableCharacter: unsupported Unicode escape sequence
DETAIL:  \u0000 cannot be converted to text.

The entire multi-row batch INSERT is rolled back, so the file is silently never
stored or searchable (and the embedding provider cost is already spent).

rag_api already strips NUL from document text via clean_text /
remove_null (the LibreChat Discussion #2243 fix). This PR extends the same
protection to metadata fields, which JSONB rejects identically.

Changes

• app/utils/document_loader.py: add clean_metadata() — recursively strips
  NUL from all string values (including nested dict/list), reusing
  remove_null.
• app/routes/document_routes.py: apply clean_metadata() at the metadata
  merge in _prepare_documents_sync (the single choke point both the async
  pipeline and batched-sync insert paths flow through). Applied
  unconditionally, not gated behind the PDF-only clean_content flag,
  since the JSONB NUL constraint holds for every loader and file type. The
  known-clean keys (file_id / user_id / digest) stay outside the
  sanitizer.
• tests/utils/test_document_loader.py: unit tests for producer NUL, nested
  dict/list values, and no over-stripping of non-string/clean values.

Testing

• Unit tests: the full tests/utils/test_document_loader.py suite passes
  (25/25) in the project's Python 3.10 Docker image.
• End-to-end against pgvector + Azure OpenAI embeddings: a real PDF with a NUL
  in producer now embeds successfully (status: true) and is searchable via
  POST /query.
• Negative control: with the change stashed, the same PDF reproduces the
  original UntranslatableCharacter failure; with the change applied it
  succeeds. A normal PDF still embeds with intact metadata (no over-stripping).

---

🤖 Generated with Claude Code