Improve cudf-spark-jni build workflow

[bdice]

Description

Rename the Spark JNI integration workflow for the cudf-spark-jni repository and make its cuDF and cudf-spark-jni revisions configurable. When no cudf-spark-jni revision is supplied, select the branch from the checked-out cuDF revision's RAPIDS_BRANCH.

Use the shared sccache-dist setup instead of maintaining token exchange and distributed-cache setup inline.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[davidwendt]

Could we move this to nightly tests as part of this change perhaps?

[vyasr]

Could we move this to nightly tests as part of this change perhaps?

Part of our discussion with the Spark team in the most recent sync was the cudf team being more proactive in helping the Spark team address breakages from our PRs, so I don't think we want to lose the per-PR reporting that we get today.

[davidwendt]

Could we move this to nightly tests as part of this change perhaps?

Part of our discussion with the Spark team in the most recent sync was the cudf team being more proactive in helping the Spark team address breakages from our PRs, so I don't think we want to lose the per-PR reporting that we get today.

I don't think moving them to nightlies should affect pro-active-ness. I doubt we are fixing these issues faster than that anyway.

[vyasr]

I don't think moving them to nightlies should affect pro-active-ness. I doubt we are fixing these issues faster than that anyway.

Speaking for myself, I haven't merged a PR that breaks that job without having a corresponding spark-rapids-jni PR open in a long time. Other cudf devs were encouraged to do the same in the last Spark meeting.