Read JSON and write XML as UTF-8 regardless of locale

[davidpavlovschi]

Summary

• Read CLI JSON input with utf-8-sig instead of the host locale, fixing Wrong encoding when runned via module #516 while accepting Windows-style UTF-8 BOMs.
• Report genuinely non-UTF-8 JSON through the CLI's normal error path instead of leaking a UnicodeDecodeError traceback.
• Write DocxTemplate.write_xml() output as UTF-8 instead of the host locale.
• Add a locale-forced regression script that covers Unicode round-tripping, invalid input, and XML output.

The write_xml() half has no linked issue; I found it while tracing the same locale-dependent text-I/O pattern for #516.

Verification

I independently ran the locale regression, all 37 script tests, Flake8, and Black on macOS with Python 3.9.6:

tests/utf8_locale.py: pass
script tests: 37 passed, 0 failed
flake8: 0
black --check: 8 files unchanged

The test forces Python out of UTF-8 mode under a C locale. It also verifies that invalid Latin-1 JSON raises the CLI's normal RuntimeError, and that write_xml() emits bytes that decode as UTF-8.

Current CI does not run these scripts because test.yml invokes tests/runtests.py from the repository root. PR #652 already addresses that runner problem, so this PR does not overlap it.

Related: withdrawn PR #640 proposed the CLI half in June and closed without comments or review. This change was developed independently and also covers BOM input, the CLI error path, and write_xml().

AI disclosure

Claude Opus 5 implemented and tested this change. OpenAI Codex independently reviewed the diff and reran the regression, full script suite, and linters. David authorized this automated contribution workflow and owns the submission. The commit keeps an explicit Co-Authored-By trailer for Claude.

[JackSpiece]

The production cases pass for me, including the forced non-UTF locale under Python 3.8, and all 37 scripts pass under Python 3.12. I found one supported-version problem in the new test harness.

From the repository root, PYTHONPATH=. python3.8 tests/utf8_locale.py fails before starting the child interpreter with FileNotFoundError: .../tests/tests. Python 3.8 keeps __file__ relative here. The main block changes into tests, then rerun_with_non_utf8_locale() recomputes abspath(__file__) from that new directory and adds a second tests component. The same command passes under Python 3.12 because __file__ is absolute there.

Please capture the absolute tests directory once before changing directories and reuse it for both os.chdir() and the child cwd, or otherwise avoid recomputing it after the directory change. Python 3.7 and 3.8 are both listed as supported, so the regression script should work from this normal invocation on those versions too.

[davidpavlovschi]

Thanks for the careful review, and for pinning down the exact mechanism — that made it quick to reproduce.

Fixed in 1be323e. The tests directory and the script name are now resolved once, at import time, into TESTS_DIR / SCRIPT_NAME, and both os.chdir() and the child cwd reuse them. Nothing recomputes abspath(__file__) after the directory change, so the doubled tests/tests path is structurally gone rather than papered over: the value is fixed before any chdir can affect it, on every version, whether __file__ starts out relative or absolute.

Chasing your invocation turned up a second instance of the same relative-path problem, one step further along. The child inherits PYTHONPATH and runs in the tests directory, so with PYTHONPATH=. the . resolved to tests/ and python -m docxtpl then failed with No module named docxtpl. The entries handed to the child are now made absolute against the startup directory. You would not have hit this with docxtpl installed in the environment, which I assume is why your run got as far as it did.

Verified locally on CPython 3.8.20, 3.9.25 and 3.12.13:

• PYTHONPATH=. python tests/utf8_locale.py from the repository root, your exact command — fails on 3.8 before the change with FileNotFoundError: .../tests/tests, passes after it
• PYTHONPATH=.. python utf8_locale.py from inside tests/
• the script by absolute path with an unrelated working directory

The full runtests.py suite, all 37 scripts, passes on 3.8 and 3.12. I also re-checked that the test still fails when either production fix is reverted on its own, so it is still doing its job — dropping encoding="utf-8" from write_xml() reproduces the UnicodeEncodeError as expected.

I do not have a 3.7 interpreter to hand, so I have not run it there and would rather not claim otherwise. The path no longer depends on the working directory at all, though, so the 3.7 behaviour of __file__ cannot reintroduce this failure.