Add bare metal Mac installation and run process to remove admin requirements by tarahmarie · Pull Request #109 · ARTFL-Project/text-pair

tarahmarie · 2026-08-28T21:35:00Z

This is a multithreaded bare metal Mac version of text-pair that lets someone who may not have admin privileges over their laptop have a decent chance at running text-pair. It repairs some reproducibility issues and does not touch the supported Linux/Docker path.

Before committing this, I tested it, another dev (@karmichand, who also added the multithreading capabilities) tested on bare metal Mac as well, and before the upstream merge, I tested this on before-and-after alignments produced from a given corpus; the two were byte-identical.

The aligner is deterministic and code-stable; differences between runs trace to explicit, documented configuration choices, with a sub-0.1% residue from code evolution.

Below are more details; I hand-wrote much of the code but used Claude to check and handle pro forma fixes.

What's included

macOS support (opt-in)

install_bare_metal_mac.sh: standalone installer for a native macOS setup (pyenv-managed Python 3.11, builds compareNgrams from source since the prebuilt binaries are Linux ELF, patches PhiloLogic's line_count.py. see note below). Documented in a new README section with prerequisites and known limitations.
run() now raises RLIMIT_NOFILE on Darwin for the duration of the run (macOS defaults to 256 open fds, which PhiloLogic's sort/merge exceeds on large corpora) and restores it afterwards. No-op on Linux.
generate_ngrams.py: ngram generation now fans out with a ThreadPoolExecutor instead of relying on text_preprocessing's internal multiprocess.Pool. Forking again immediately after the parse stage's pool tears down reliably deadlocks on modern macOS (bpo-33725); threads sidestep this, and throughput is equivalent.
read_global_config() checks ~/.text-pair/global_settings.ini after /etc/text-pair/, so a user-level install doesn't need root to write /etc. All existing /etc installs behave exactly as before; the three call sites in web_loader.py that re-read the file directly now go through this helper.

Portability/robustness fixes (benefit all platforms)

Directory setup/cleanup in generate_ngrams.py, __main__.py, and banality_finder.py no longer shells out with unquoted interpolated paths (os.system("rm -rf {path}") etc.). Paths containing spaces previously failed — or worse, rm -rf word-split. Now shutil/os.replace (atomic for the banality rewrites), with the two remaining shell pipelines (external sort, batch merge) shell-quoted and the merge NUL-delimited (find -print0 | sort -zV | xargs -0).
saveAlignmentConfig in compareNgrams listed two field names (banalNgrams, oneWayMatching) that no longer exist on matchingParams, so every run's alignment_config.ini recorded <invalid reflect.Value> for both. Removed; the remaining fields all exist.

Testing

Full pipeline runs on a 3,514-document English corpus (ELTeC-100) and a 2,847-document French corpus (ELTeC-fra, exercising the language = french preprocessing path), 8 workers, macOS (Apple Silicon).
Output determinism verified: two identical runs on the 3,514-document corpus produce byte-identical alignment sets (same spans, same passages, same banality flags).
The space-handling fixes were verified with a functional test on paths containing spaces; go build and bash -n pass; branch merges cleanly against current master.

Note on PhiloLogic

Full end-to-end support for output paths containing spaces still requires a fix in PhiloLogic itself: Loader.setup_db_dir runs rm -rf through the shell unquoted (directly below its own unused shellquote helper). The macOS installer patches PhiloLogic's line_count.py for a separate issue (the non-lz4 code path and BSD wc output format).

I am more than happy to file these against PhiloLogic if useful.

…fig paths

macOS wc outputs leading whitespace, causing .split() to fail on banality_finder.py line 60. Adds sed patch to install script matching the manual fix already applied to the local copy. Third of three upstream macOS compatibility fixes: 1. line_count.py rewrite (non-lz4 branch broken) 2. ulimit increase for large corpora 3. banality_finder.py .strip() before .split()

Mac bare metal

describe mac bare metal process and tag for thesis usage

… pre and sets configs correctly

Installtion and configuration updates. Install script installs needed…

Install.sh script rewrite for a bare metal mac

Install.sh scaffold and config setup

multithread update

text_setup.sh

generate() shelled out via os.system with the output path interpolated unquoted in eleven places, including rm -rf: any output path containing a space word-splits in the shell, failing at best and deleting the wrong path at worst (macOS paths like ~/Library/Mobile Documents contain spaces). Directory setup and cleanup now use shutil.rmtree/os.makedirs; the external-sort index pipeline stays in the shell for performance but shell-quotes every path occurrence. Verified with a functional test on a path containing spaces.

saveAlignmentConfig writes alignment_config.ini by looking up a hardcoded list of field names on matchingParams via reflection. banalNgrams and oneWayMatching no longer exist on the struct, so FieldByName returned the zero Value and every run's provenance record contained '<invalid reflect.Value>' for both. Removed the two stale names; the remaining seventeen all exist on the struct. go build passes.

my_config.ini shipped with a machine-specific source_file_path; it is now blank like the config template, with the path applied at run time. text_setup.sh hardcoded another machine's absolute path in its sed line; it now derives the repo root from the script's own location.

A 4 MB build artifact; it also embeds build-machine paths. The install flow builds it from source, and it now stays untracked via .gitignore. The local copy is untouched.

Upstream fixes

Completes what the generate_ngrams fix started: __main__.py shelled out with unquoted interpolated paths (three rm -rf sites, the full compareNgrams command line, and the batch-merge pipeline), and banality_finder.py chained rm && mv through the shell. Directory removal and moves now use shutil/os.replace (atomic); the compareNgrams command shell-quotes every path-valued argument; the batch-merge pipeline is NUL-delimited (find -print0 | sort -zV | xargs -0) so batch paths with spaces survive. Not covered, deliberately: web_loader.py (web-app deployment) and vector_space_alignment (experimental). Note that full end-to-end support for output paths containing spaces additionally requires a fix in PhiloLogic's Loader.setup_db_dir, which runs rm -rf through the shell unquoted (directly below its own unused shellquote helper); textpair's own code no longer mishandles such paths. Verified: py_compile passes; a 30-document French corpus runs the full pipeline (parse, ngrams, alignment, banality flagging, grouping) with identical results before and after.

…pstream my_config.ini and text_setup.sh are machine-specific run helpers that do not belong in the shared history (local copies untouched, now gitignored). lib/textpair.egg-info/entry_points.txt is generated build metadata; reverted to the upstream version so it carries no diff.

Untrack fork-local run config and setup script; align egg-info with u…

Extend path-space safety to the core alignment pipeline

…un() entry point Conflict resolutions: - config: upstream's sa_config.ini/vsa_config.ini split adopted wholesale (fork's language=en template default dropped to keep upstream defaults) - install.sh: upstream's version kept verbatim; the fork's bare-metal installer ships as the new opt-in install_bare_metal_mac.sh (seed file reference updated to config/sa_config.ini) - entry point: upstream's run() name kept; the fork's macOS RLIMIT_NOFILE raise/restore folded into it; the .xml.xml input-dir renamer dropped (staging-side workaround, mutates user files) - lib/pyproject.toml: entry point follows run()

Sync upstream

The README's macOS section predated the upstream merge: it pointed at the retired mac-bare-metal branch and the old combined install.sh, and described patches (cli_entry wrapper, banality_finder wc fix, psycopg2-binary swap) that now live in the source or upstream. Rewritten around the opt-in install_bare_metal_mac.sh with accurate prerequisites and run instructions; the fork-voice note is gone, with the thesis attribution kept in the macOS section. The installer drops its three now-obsolete self-patching steps (cli_entry append, banality wc sed, psycopg2 swap - upstream ships psycopg2-binary and the source carries the fixes) and gains an lz4 dependency check, since the batch-merge step uses lz4cat. The PhiloLogic line_count patch stays: that package is still broken on macOS.

Update macOS docs and installer for the post-merge layout

tarahmarie · 2026-08-28T21:45:00Z

this changes psycopg2 → psycopg2-binary in the main dependency list, which affects Linux installs too. Reason: bare-metal Macs usually lack pg_config, so the source build fails at install. If you'd rather keep psycopg2 for production installs, I can rework.

clovis

Thanks for the pull request! Just a few issues to resolve (noted in comments) but otherwise looks good, thanks!

clovis · 2026-08-31T15:05:54Z

Could you make the ThreadPoolExecutor code only run under Darwin, and leave the older code which uses multiprocessing on Linux? Something like:

if platform.system() == "Darwin": # init preprocessor with workers=workers # for local_metadata in preprocessor.process_texts(files, progress=False) ... else: # workers=1, and ThreadPoolExecutor

Hope that makes sense!

cool thought, good call. just a sec.

clovis · 2026-08-31T15:18:20Z

+
+
+def read_global_config() -> configparser.ConfigParser:
+    """Read global_settings.ini from the first location(s) that exist. Missing files are silently skipped."""


This isn't quite right I believe. The second location overrides the first, no? Which is fine btw. Just a nitpick on the comment.

It does and its a fair nit, This is dug killing run a muck

clovis · 2026-08-31T15:25:22Z

@@ -0,0 +1 @@
+3.11.15


why hardcode to 3.11? The original install.sh lets you choose your Python executable which gives you more flexibility. I'm also planning on actually having uv install python directly (Python 3.12 most likely) to avoid any issues with Python versions.

Its an issue with the way python is upgrading on different systems, Im game to try some alts.

yeah, the hardcode will break stuff

clovis · 2026-08-31T15:41:18Z

    "text_preprocessing @ git+https://github.com/ARTFL-Project/text-preprocessing@v1.1.2#egg=text_preprocessing",
    "fastapi==0.110.3",
-    "psycopg2",
+    "psycopg2-binary",


that's actually not something I'd want in production. Perhaps you could instead add an environment marker in this file to handle the Mac and Linux case? E.g.:

'psycopg2; sys_platform != "darwin"', 'psycopg2-binary; sys_platform == "darwin"',

Ah, ok. Let me check and see if that is just the name for the package, or i did that to follow another bug.

Nope - i made this change, adn I don't like it. I'll fix it.

clovis · 2026-08-31T15:45:32Z

I wonder if compareNgrams should just be compiled by default. Adding Go as a dependency is not a huge ask anyway.

I don't remember, i have to get back in and check.

That's because I made the edit.

tarahmarie and others added 30 commits February 19, 2026 20:12

Mac bare metal install: psycopg2-binary, async cli_entry wrapper, con…

fbe8859

…fig paths

Update install.sh for Mac bare metal installation

a241a5a

looks liek this works for more corpora

29768d3

Add .gitignore, remove corpus texts and temp files from tracking

3b9ee45

Add .gitignore: exclude corpus data, macOS artifacts, backup files

42a5c9e

Merge pull request #1 from tarahmarie/mac-bare-metal

71cbf53

Mac bare metal

describe mac bare metal process and tag for thesis usage

1cce403

Merge pull request #2 from tarahmarie/mac-bare-metal

d5565e0

describe mac bare metal process and tag for thesis usage

Installtion and configuration updates. Install script installs needed…

af2ce5b

… pre and sets configs correctly

Merge pull request #3 from tarahmarie/install-fix

7357d6f

Installtion and configuration updates. Install script installs needed…

Install.sh script rewrite for a bare metal mac

6ef0557

Merge pull request #4 from tarahmarie/install-fix

0a4e57a

Install.sh script rewrite for a bare metal mac

Install.sh scaffold and config setup

c423f08

Merge branch 'master' into local-install-fix2

0b93f90

Merge pull request #5 from tarahmarie/local-install-fix2

d43e76a

Install.sh scaffold and config setup

multithread update

870c8f3

Merge pull request #6 from tarahmarie/multithread

4d2238c

multithread update

text_setup.sh

82a30e0

Merge pull request #7 from tarahmarie/multithread

973d62a

text_setup.sh

Stop tracking the compiled compareNgrams binary

db58e82

A 4 MB build artifact; it also embeds build-machine paths. The install flow builds it from source, and it now stays untracked via .gitignore. The local copy is untouched.

Merge pull request #8 from tarahmarie/upstream-fixes

b28028b

Upstream fixes

Merge pull request #9 from tarahmarie/pr-cleanup

0a1cc41

Untrack fork-local run config and setup script; align egg-info with u…

Merge pull request #10 from tarahmarie/space-safety-core

7e48340

Extend path-space safety to the core alignment pipeline

tarahmarie and others added 4 commits August 28, 2026 20:59

Merge pull request #11 from tarahmarie/sync-upstream

1f16474

Sync upstream

Merge pull request #12 from tarahmarie/readme-macos

44045c0

Update macOS docs and installer for the post-merge layout

Update README.md

ad697c8

clovis reviewed Aug 31, 2026

View reviewed changes



		def read_global_config() -> configparser.ConfigParser:
		"""Read global_settings.ini from the first location(s) that exist. Missing files are silently skipped."""

		@@ -0,0 +1 @@
		3.11.15

Uh oh!

Conversation

tarahmarie commented Aug 28, 2026

What's included

Testing

Note on PhiloLogic

Uh oh!

tarahmarie commented Aug 28, 2026

Uh oh!

clovis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants