Separate words at the invisible separator, U+2063

docs/design/decisions/T1-separators-not-joiners.md

@@ -4,7 +4,7 @@
 
 - An ignorable character is a separator, never removed with its neighbors joined as v1's squash_emoji and squash_bidi did: `A😀B` reads given 'A', family 'B' — 2026-07 PR #288.
 - The `strip_bidi=False` half of the opt-out carries no example, by decision; with it `John‏Smith` (a right-to-left mark between the words) reads given 'John‏Smith', the mark kept — 2026-10-20 #761.
-- The zero-width space (U+200B) and the byte-order mark (U+FEFF) are separators with no opt-out; the zero-width non-joiner and joiner (U+200C, U+200D), the word joiner (U+2060) and the soft hyphen (U+00AD) stay inside the word, and an invisible character the tokenizer does not handle is reported nowhere — 2026-10-24 #851.
+- The zero-width space (U+200B), the invisible separator (U+2063) and the byte-order mark (U+FEFF) are separators with no opt-out; the zero-width non-joiner and joiner (U+200C, U+200D), the word joiner (U+2060), the soft hyphen (U+00AD), the invisible math operators (U+2061, U+2062, U+2064) and the Mongolian vowel separator (U+180E) stay inside the word, and an invisible character the tokenizer does not handle is reported nowhere — 2026-10-24 #851; the invisible separator and the four exclusions beside it, 2026-11-02 #1018.
 
 Open: none in this file.
 
@@ -16,3 +16,5 @@ Open: none in this file.
   - Excluded: the zero-width non-joiner and joiner, which decide how the letters beside them join and stand inside Persian (Arabic-script), Devanagari, Malayalam and Kannada names in the corpora (thousands between letters in `nepal-candidates` alone); about 575 of them stand at a word's edge (a run of joiners counted at an edge when whitespace or the string's edge is beside it), where they join nothing (`Antonio ‌Negri`), and are left to the caller. The word joiner, whose meaning is no break. The soft hyphen, mostly inside a word: of the 66 names holding one, 52 are an accented í decoded with the wrong encoding (`NATALíCIO`, and `Pri­ncipe` without the `Ã`), 10 are typesetting hyphens at a syllable break (`Puig­demont`), and 4 end the name (`Ayman al-Zawahiri­`, read with the hyphen kept in the family); none is the parser's to repair.
   - Accepted (Derek, 2026-10-24, PR #874 review): the two do not count as a space where a quote or a maiden marker is recognized, which happens before words are divided and tests for whitespace, so a glued quote is no nickname (`'Bob' Smith` reads given `'Bob'`) and a marker glued to its name leaves the clause a nickname (`Jane (née​Jones) Smith` reads nickname `née Jones`) — the katakana middle dot's shape, pinned in `tests/v2/cases.py` as `marker_divided_by_a_nakaguro_keeps_the_clause_a_nickname`. `ParsedName.replace()` and the v1 setters split a value at whitespace only, as for the middle dot and emoji. A zero-width space after a hyphen, a line-break hint in typeset text, divides the name (`Mary-​Jane Smith` reads given `Mary-`, middle `Jane`, where master kept `Mary-​Jane`). No corpus holds a zero-width space or byte-order mark inside a delimited clause or after a hyphen, so widening the quote and marker tests was declined as machinery for an unwritten shape.
   - Declined: an ambiguity reporting an invisible character the tokenizer leaves in a word. An ambiguity records a reading the parse had to choose; this is a fact about the data, which the caller can find as easily as the parser can. Declined too: a cleanup recipe in the user docs — cleaning input is the caller's job, and the docs would own it.
+- 2026-11-02 (Derek), #1018 — THE INVISIBLE SEPARATOR JOINS THE ZERO-WIDTH SPACE: U+2063, INVISIBLE SEPARATOR, separates under every policy as U+200B and U+FEFF do, so `John⁣Smith` reads given 'John', family 'Smith' where it read given 'John⁣Smith'; found comparing the tokenizer with name_tamer's, which strips it beside U+200B, U+FEFF, the joiners and U+180E. It is the one character beside the #851 pair whose Unicode name and use say separator (in mathematical text it marks the comma between invisible indices), and every shape #851 settled for the zero-width space holds for it unchanged: it reads as whitespace beside the interpunct, does not count as a space where a quote or a maiden marker is recognized (`Jane (née⁣Jones) Smith` reads nickname 'née Jones'), and divides a hyphenated name after its hyphen. No real name holds it: none of the 13 local corpora's `raw.jsonl` holds U+2063, nor U+2061, U+2062, U+2064 or U+180E, and of the 16 committed JSONL files only `corpus_rules.jsonl` and `snapshot.jsonl` do, each holding U+2063 once, in rules/T1's example `John⁣Smith` (`grep -c` over the UTF-8 bytes, 2026-11-02), so the change moves no measured name and is robustness against text pasted from documents.
+  - Excluded: the invisible math operators U+2061 FUNCTION APPLICATION, U+2062 INVISIBLE TIMES and U+2064 INVISIBLE PLUS, which join their operands rather than separate words, and U+180E MONGOLIAN VOWEL SEPARATOR, which decides the shape of the Mongolian letters beside it, as the joiners #851 excluded decide how theirs join; name_tamer strips U+180E, and stripping it would change how a Mongolian name is written. Each stays inside the word, a row of `test_joiners_and_soft_hyphen_stay_inside_the_word`.

docs/design/rules/T1-separators-not-joiners.md

@@ -1,6 +1,6 @@
 # T1 — separators, not joiners
 
-**When:** a name holds whitespace or an ignorable character, one that carries no name content, such as an emoji, an invisible directionality control, a zero-width space or a byte-order mark, as in "John😀Smith".
+**When:** a name holds whitespace or an ignorable character, one that carries no name content, such as an emoji, an invisible directionality control, a zero-width space, an invisible separator or a byte-order mark, as in "John😀Smith".
 
 **Then:** the name splits at each of them, so an ignorable character separates its neighbors and never joins them.
 
@@ -10,11 +10,12 @@
 "John😀Smith"                →  family="Smith"
 "​王小明"                    →  family="王小明"
 "​Dr. John Smith"          →  title="Dr."
+"John⁣Smith"                →  family="Smith"
 ```
 
 **Except:**
 
-- **Where the caller opts to keep emoji or directionality controls**, the name does not split at the ones kept; a zero-width space and a byte-order mark have no such opt-out.
+- **Where the caller opts to keep emoji or directionality controls**, the name does not split at the ones kept; a zero-width space, an invisible separator and a byte-order mark have no such opt-out.
   ```
   "John😀Smith"  keep-emoji    →  given="John😀Smith"  · boundary
   ```
@@ -25,11 +26,11 @@
   ```
   "Antonio ‌Negri"  →  family="‌Negri"
   ```
-- A zero-width space or byte-order mark does not count as a space where a quote or a maiden marker is recognized, as the katakana middle dot does not (rules/T2): a marker glued to its name by one leaves the clause a nickname.
+- A zero-width space, invisible separator or byte-order mark does not count as a space where a quote or a maiden marker is recognized, as the katakana middle dot does not (rules/T2): a marker glued to its name by one leaves the clause a nickname.
   ```
   "Jane (née​Jones) Smith"  →  nickname="née Jones"
   ```
-- A zero-width space after a hyphen, a line-break hint in typeset text, divides the hyphenated name.
+- A zero-width space after a hyphen, a line-break hint in typeset text, divides the hyphenated name, as an invisible separator there does.
   ```
   "Mary-​Jane Smith"  →  given="Mary-"
   ```

docs/release_log.rst

@@ -42,6 +42,7 @@ Release Log
     - **Fix British post-nominals read as the family name.** ``parse("Jane Doe KC")`` gives family ``Doe``, suffix ``KC``, where 1.4.0 through 2.8.0 gave middle ``Doe``, family ``KC``, and ``parse("Jane Doe, CChem")`` no longer reverses the name around the credential. ``KC``, ``PC``, ``DL``, ``FRS``, ``LLB``, ``CChem``, ``FRAeS`` and 16 more join the vocabulary. ``KC`` is also a Nepali surname, so ``Ram KC`` keeps it as the family name, as do ``K.C.`` and ``Kc``, while ``Gita Bahadur KC`` reads it as the suffix, and ``PC`` and ``DL`` keep a family name the same way (``Anna DL``); ``FIET`` and ``FREng`` keep the surnames Fiet and Freng the same way, so a bare ``Jane Doe FREng`` still gives family ``FREng``. :issue:`965`
     - **Fix Policy(name_order=...) accepting a set.** ``Policy(name_order=frozenset(FAMILY_FIRST))`` raises ``TypeError``, where 2.0.0 through 2.8.0 accepted it as an order under some hash seeds and raised ``ValueError`` under others; pass the tuple. :issue:`754`
     - **Divide five more mainland Chinese surnames under locales.ZH.** ``付晓明`` under ``locales.ZH`` gives family ``付``, given ``晓明``, where 2.1.0 through 2.8.0 gave family ``付晓明``; ``代``, ``候``, ``芦`` (``蘆``) and ``农`` (``農``) divide too. A Japanese name opening with ``代`` or ``芦`` now splits after it (``芦川誠`` gives family ``芦``). :issue:`952`
+    - **Separate words at the invisible separator, U+2063, as at a zero-width space.** ``parse("John\u2063Smith")`` gives given ``John``, family ``Smith``, where 1.4.0 through 2.8.0 gave given ``John\u2063Smith``; the character is unseen in pasted text. The other invisible characters are unchanged. :issue:`1018`
     - **Fix case repair writing Machado as MacHado.** ``parse("JOHN MACHADO").capitalized()`` gives ``John Machado``, where 1.4.0 through 2.8.0 gave ``John MacHado``: a name opening Mac is title-cased unless the vocabulary records its casing, as it does for 42 surnames, nearly all Gaelic, most of whose bearers write the inner capital (``RONALD MACDONALD`` still gives ``Ronald MacDonald``). One most of its bearers write without it, or that the vocabulary does not list, loses it (``James Mackenzie``, ``Shirley Maclaine``). A strict choice of languages keeps all 42 with English, 37 with Scottish Gaelic and the 4 Irish ones with Irish; add your own spelling with ``Lexicon.capitalization_exceptions``. Mc is unchanged. :issue:`1014`
     - **Read Никитич and Саввич as patronymics under locales.RU.** ``parser_for(locales.RU).parse("Петров Иван Никитич")`` gives family ``Петров``, given ``Иван``, where 2.0.0 through 2.8.0 gave family ``Никитич``, given ``Петров``; ``Саввич`` and the Latin ``Nikitich`` and ``Savvich`` read the same way. :issue:`994`
 

nameparser/_pipeline/_tokenize.py

@@ -40,13 +40,16 @@
 # transcription marker; only the Chinese 间隔号 is (#298).
 _NAME_DOT_SEPARATORS = frozenset({"\u30FB", "\uFF65"})
 
-# rules/T1: "a zero-width space and a byte-order mark have no such
-# opt-out" -- the Except clause: they separate under every policy,
-# carrying nothing a caller would keep. Not here: U+200C and U+200D,
-# which decide how the letters around them join, the word joiner
-# U+2060, which means no break, and the soft hyphen U+00AD, which
-# marks a break inside one word (decisions/T1, #851).
-_INVISIBLE_SEPARATORS = frozenset({"\u200B", "\uFEFF"})
+# rules/T1: "a zero-width space, an invisible separator and a
+# byte-order mark have no such opt-out" -- the Except clause: they
+# separate under every policy, carrying nothing a caller would keep.
+# Not here: U+200C and U+200D, which decide how the letters around
+# them join, the word joiner U+2060, which means no break, the soft
+# hyphen U+00AD, which marks a break inside one word, the invisible
+# math operators U+2061, U+2062 and U+2064, and the Mongolian vowel
+# separator U+180E, which shapes the letters beside it
+# (decisions/T1, #851).
+_INVISIBLE_SEPARATORS = frozenset({"\u200B", "\u2063", "\uFEFF"})
 
 _INTERPUNCT = "\u00B7"
 # rules/T3: "the interpunct divides a name only between two
@@ -71,8 +74,8 @@ def _stripped(ch: str, policy: Policy) -> bool:
     character the strip policy removes must not occupy a flank
     position either, so a new strip class added here stays transparent
     to the interpunct guard by construction. The separators (whitespace,
-    the name dots, the zero-width space and the byte-order mark) are
-    not in this set and do occupy one."""
+    the name dots and _INVISIBLE_SEPARATORS) are not in this set and do
+    occupy one."""
     if policy.strip_bidi and _BIDI.match(ch):
         return True
     return bool(policy.strip_emoji and _is_emoji(ch))
@@ -106,7 +109,8 @@ def _ignorable(ch: str, state: ParseState) -> bool:
     # COMMA_CHARS (commas RECORD an offset; these must not -- only the
     # 间隔号 marks a transcription and records, #298, see
     # _NAME_DOT_SEPARATORS above). The only load-bearing constraint is
-    # "after the isascii fast path" (all four are non-ASCII).
+    # "after the isascii fast path" (every member of both sets is
+    # non-ASCII).
     if ch in _NAME_DOT_SEPARATORS or ch in _INVISIBLE_SEPARATORS:
         return True
     return _stripped(ch, state.policy)

tests/v2/pipeline/test_tokenize.py

@@ -95,12 +95,14 @@ def test_emoji_and_bidi_are_ignorable_by_policy() -> None:
 
 @pytest.mark.parametrize("policy", [Policy(), _KEEP_ALL],
                          ids=["default", "keep-emoji-and-bidi"])
-@pytest.mark.parametrize("ch", ["​", ""],
-                         ids=["zero-width-space", "byte-order-mark"])
-def test_zero_width_space_and_bom_separate_under_every_policy(
+@pytest.mark.parametrize("ch", ["​", "\u2063", ""],
+                         ids=["zero-width-space", "invisible-separator",
+                              "byte-order-mark"])
+def test_the_invisible_separators_separate_under_every_policy(
         ch: str, policy: Policy) -> None:
     # #851: no opt-out reaches them -- neither strip switch covers
-    # them, since neither is an emoji or a direction control
+    # them, since none is an emoji or a direction control; U+2063, the
+    # invisible separator, is the one whose Unicode name says what it is
     for text, words in [(f"John{ch}Smith", ["John", "Smith"]),
                         (f"{ch}Dr. John", ["Dr.", "John"]),
                         (f"John {ch}Smith{ch}", ["John", "Smith"])]:
@@ -111,13 +113,18 @@ def test_zero_width_space_and_bom_separate_under_every_policy(
         assert state.comma_offsets == ()
 
 
-@pytest.mark.parametrize("ch", ["‌", "‍", "⁠", "­"],
-                         ids=["zwnj", "zwj", "word-joiner", "soft-hyphen"])
+@pytest.mark.parametrize("ch", ["‌", "‍", "⁠", "­",
+                                "\u2061", "\u2062", "\u2064", "\u180e"],
+                         ids=["zwnj", "zwj", "word-joiner", "soft-hyphen",
+                              "function-application", "invisible-times",
+                              "invisible-plus", "mongolian-vowel-separator"])
 def test_joiners_and_soft_hyphen_stay_inside_the_word(ch: str) -> None:
     # #851's exclusions: ZWNJ and ZWJ decide how the letters beside
     # them join, the word joiner means no break, and the soft hyphen
     # marks a break inside one word, so splitting at any of them would
-    # cut a real word in two
+    # cut a real word in two. The invisible math operators beside
+    # U+2063 join operands rather than separate words, and the
+    # Mongolian vowel separator shapes the letters beside it
     assert [t.text for t in _tokenized(f"ab{ch}cd ef").tokens] == [
         f"ab{ch}cd", "ef"]
 

tests/v2/rules_budget.json

@@ -73,7 +73,7 @@
   "S11": {"words": 668, "then_words": 399, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 3, "accepted_words": 161},
   "S12": {"words": 777, "then_words": 372, "because_sentences": 2, "except": 3, "precedence": 4, "accepted": 2, "accepted_words": 93},
   "S13": {"words": 431, "then_words": 180, "because_sentences": 3, "except": 2, "precedence": 1, "accepted": 4, "accepted_words": 138},
-  "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83},
+  "T1": {"words": 99, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 91},
   "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0},
   "T3": {"words": 56, "then_words": 23, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0},
   "T4": {"words": 334, "then_words": 236, "because_sentences": 2, "except": 0, "precedence": 1, "accepted": 1, "accepted_words": 56},

tests/v2/test_render.py

@@ -164,6 +164,7 @@ def test_default_str_includes_nickname_and_maiden() -> None:
     # space, a delimiter, a dropped marker word
     ("John Smith,, Jr.", None, "John Smith Jr."),
     ("John\u200bSmith", None, "John Smith"),
+    ("John\u2063Smith", None, "John Smith"),
     ("John (Jack) Smith", None, 'John "Jack" Smith'),
     ("Jane Smith née Jones, PhD", None, "Jane Smith (Jones) PhD"),
     ("Jane (née , Jones) Smith", None, 'Jane "née, Jones" Smith'),

tools/differential/corpus_rules.jsonl

@@ -656,6 +656,7 @@
 "John von PhD Jones"
 "John й Smith"
 "John ｢Jack｣ Smith"
+"John⁣Smith"
 "John😀Smith"
 "Jon Dough and"
 "Jones Jr. née Smith"

tools/differential/snapshot.jsonl

@@ -2025,6 +2025,7 @@
 {"name": "John, Smith, Dr.", "title": "Dr.", "given": "Smith", "family": "John", "initials": "S. J."}
 {"name": "Johnson Mary", "order": "FAMILY_FIRST", "given": "Mary", "family": "Johnson", "initials": "M. J."}
 {"name": "Johnson, Mary", "given": "Mary", "family": "Johnson", "initials": "M. J."}
+{"name": "John⁣Smith", "given": "John", "family": "Smith", "initials": "J. S."}
 {"name": "John😀Smith", "given": "John", "family": "Smith", "initials": "J. S."}
 {"name": "Jon Andoni Urrestaratzu", "given": "Jon", "middle": "Andoni", "family": "Urrestaratzu", "initials": "J. A. U."}
 {"name": "Jon Dough and", "given": "Jon", "family": "Dough and", "initials": "J. D."}