Add chunking edge case tests#481
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
2269611 to
1a6c12b
Compare
|
recheck |
|
recheck |
1a6c12b to
7068373
Compare
|
Your branch is 1 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
|
Your branch is 1 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
Description
Adds focused test coverage for chunking edge cases documented in #430:
SentenceIteratorstart-position guardsChunkIteratorconstructor fallbacksTextChunkmissing-document errors, sanitized text, and lazy cachingbroken_sentencereset behavior after a sentence is split mid-wayFixes #430
Testing
How Has This Been Tested?
Checklist:
Code of conduct.
Contributing
page, and I either signed the Google
Individual CLA
or am covered by my company's
Corporate CLA.
issue(s) and we have agreed upon the general approach.
issue(s) that documentation elsewhere needs updating.
Google's Python Style Guide
and ran
pylintover the affected code.