chore: add internal markdown link check by Geethapranay1 · Pull Request #21831 · apache/datafusion

Geethapranay1 · 2026-04-24T10:58:17Z

Which issue does this PR close?

Closes chore: Create CI action that validates links in md files #21747

Rationale for this change

datafusion did not have a CI check for broken links in markdown content, docs workflows build and deploy docs, and dev checks formatting and spelling, but none of them validate link targets.
This pr adds a dedicated link check for internal markdown links so broken references fail early in PRs.
I kept the scope internal-only to avoid flaky CI failures from external websites and rate limits.
Rust doc comments remain covered by the existing rustdoc CI job.

What changes are included in this PR?

Added a new Dev workflow job, Check Markdown Links, in dev.yml.
Added LYCHEE_VERSION pin in tool_versions.sh.
Added markdown_link_check.sh to run lychee on the selected markdown paths.
Added lychee.toml with internal-link policy and exclusions.
Added check markdown links to required status checks in .asf.yaml.
Updated contributor testing docs with the new local command and scope note.
Fixed internal markdown links that failed under the new check in:
- roadmap.md
- 49.0.0.md
- overview.md
- dataframe.md
- format_options.md

Are these changes tested?

Yes,

python3 ci/scripts/check_asf_yaml_status_checks.py passed.
bash -n ci/scripts/markdown_link_check.sh passed.
bash ci/scripts/markdown_link_check.sh passed with 0 errors.
cargo fmt --all --check passed.

OK: All 5 required_status_checks match existing GitHub Actions jobs.
🔍 12824 Total (in 0s) ✅ 490 OK 🚫 0 Errors 👻 12334 Excluded

Are there any user-facing changes?

No,

There is one contributor-facing CI change: PRs now fail when internal markdown links break in the checked markdown files.

Geethapranay1 · 2026-04-24T10:58:36Z

@comphead PTAL

comphead · 2026-04-24T15:26:53Z

+
+Run the internal markdown link check locally:
+
+```shell


this is very nice

for local we need to document mapfile is needed

and also mapfile is not available on macOS

and cargo install lychee

comphead

Thanks @Geethapranay1 for the PR

I tried to check it locally but with macOS I'm missing some bash commands. Would you help to attach to the PR how bad link would look like.

I'm assuming it should be very clear to the user what link needs to be fixed

Geethapranay1 · 2026-04-24T15:48:03Z

Thanks @comphead for the detailed review and for testing this on macOS.

I will:

update the local script to avoid macOS-missing bash commands (mapfile),
document local prerequisites including cargo install lychee,
add an example of how a broken link error looks in output so it is clear what to fix.

Geethapranay1 · 2026-04-24T16:43:19Z

@comphead PTAL, i have changed according to the review

comphead · 2026-04-24T20:09:53Z

+
+```text
+[docs/source/user-guide/cli/overview.md]:
+  [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct


oh, lychee doesn't refer to a specific line in the md file?

This is some sample file name, but when there is link broken it will not show line number, but exactly file name and link broken line, but not number.

comphead

Thanks @Geethapranay1 this pr makes a lot of sense

alamb · 2026-04-25T16:03:27Z

+      - name: Install lychee
+        run: |
+          source ci/scripts/utils/tool_versions.sh
+          cargo install lychee --locked --version "${LYCHEE_VERSION}"


Looks like it takes 3m (on a beefy machine) to install lychee

https://github.com/apache/datafusion/actions/runs/24900880747/job/72917950438?pr=21831

Maybe we can install a debug version or somehting to speed up the process

good point @alamb on the installation time. Instead of building a debug version from source, I can update the action to just download the pre-compiled lychee release binary for linux. That should bring the installation step down from 3 minutes to just a couple of seconds! I'll push an update for this shortly.

alamb · 2026-04-25T16:04:31Z

+  "datafusion/core/benches/tpch-csv",
+]
+
+exclude = [


why not check http and https links?

i already had talk with @comphead in the #21747 (comment) i excluded external links because they frequently cause flaky CI runs. Things like temporary network hiccups, third-party site downtime, or github API rate limits (429 Too Many Requests) can lead to false positives and unnecessarily block PRs.

anyway, if you prefer stricter validation and don't mind the occasional flakiness, I'm happy to remove these exclusions, we can always just ignore specific flaky domains as they pop up. Let me know how you'd like to proceed.

Geethapranay1 · 2026-04-25T17:09:43Z

@alamb PTAL, I updated the workflow to use taiki-e/install-action. It now downloads the pre-built binary instead of compiling it from source, so the installation drops from 3 minutes to a few seconds!

https://github.com/apache/datafusion/actions/runs/24936080575

chore: add internal markdown link check

038894f

github-actions Bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion labels Apr 24, 2026

comphead reviewed Apr 24, 2026

View reviewed changes

Merge branch 'main' into chore/internal-markdown-link-check

296c651

comphead reviewed Apr 24, 2026

View reviewed changes

added testing docs and removed mapfile

0333673

comphead reviewed Apr 24, 2026

View reviewed changes

comphead approved these changes Apr 24, 2026

View reviewed changes

alamb reviewed Apr 25, 2026

View reviewed changes

optimize lychee

31a303b

github-advanced-security AI found potential problems Apr 25, 2026

View reviewed changes

Comment thread .github/workflows/dev.yml Fixed

add permissions

6b04cf6

Conversation

Geethapranay1 commented Apr 24, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Geethapranay1 commented Apr 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Geethapranay1 commented Apr 24, 2026

Uh oh!

Geethapranay1 commented Apr 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Geethapranay1 Apr 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Geethapranay1 commented Apr 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Geethapranay1 Apr 25, 2026 •

edited

Loading

Geethapranay1 commented Apr 25, 2026 •

edited

Loading