Allow extern getFd() to get file descriptors from Java for Kiwix Android #852

Activity

1. kelson42

   assigned

   mgautierfr

   on Jan 30, 2024
2. kelson42

   added

   question

   on Jan 30, 2024
3. kelson42

   added this to the 9.2.0 milestone on Jan 30, 2024
4. kelson42 commented on Mar 9, 2024

   

   kelson42

   on Mar 9, 2024

   Contributor

   More actions

   @mgautierfr Any feedback here?
5. kelson42

   assigned

   kelson42

   and unassigned

   mgautierfr

   on Mar 23, 2024
6. kelson42 commented on Mar 23, 2024

   

   kelson42

   on Mar 23, 2024

   Contributor

   More actions

   We have an issue in Android directly accessing the fd via file path on Android 11 and above without MANAGE_EXTERNAL_PERMISSION(Which is special permission in Android and Play Store does not allow us to take this permission).

   Phrasing seems incorrect. Correct phrasing is: Issues on Android to fully (suggestions/ft search) use ZIM files when open via fd only (so without a - ZIM - file path). In such a case Xapian indexes can not be open (as they do require a dedicated fd, which can only be properly retrieved in "Java world" from the ZIM file path).

   Archive and SuggestionSearcher need the file path to work properly with the Xapian index as per conversation with @mgautierfr in kiwix/kiwix-android#3636. See kiwix/kiwix-android#3636 (comment) for more details.

   So to solve this issue can we open a new fd from the existing fd via fileInputStream API and use it for Xapian without affecting the existing one? Somehow this is possible at the libzim level. Or if you think of any better suggestion that can achieve this with existing fd would be welcome.

   I answer here under the control of @mgautierfr, but "yes" you can open many fd from the same file. Each fileInputStream will have its own fd, but you can open many fileInputStream for the same file and then you should have many fds.

   So. here I see only one solution: the libzim should be modified to allow an "external" getFD() method which would be able to call java code to get one. This is a feature request which implies modifications here, in kiwix/java-libkiwix and kiwix/kiwix-android.

   👍React with 👍1Reacted by MohitMaliFtechiz
7. kelson42

   assigned

   mgautierfr

   on Mar 23, 2024
8. kelson42

   changed the title [-]Can we open the `SuggestionSearcher` and `Archive` with existing FileDescriptor[/-] [+]Allow extern `getFd()` to get file descriptors from Java for Kiwix Android[/+] on Mar 23, 2024
9. kelson42

   modified the milestones: 9.2.0, 9.3.0 on Mar 23, 2024
10. mgautierfr commented on May 13, 2024

    

    mgautierfr

    on May 13, 2024

    Contributor

    More actions

    This comment kiwix/libkiwix#1015 (comment) is interesting and must not be forgotten, and here an answer:

    Opening an archive with a list of (file-path, offset, size) would be interesting/needed in itself but not enough.
    We may have system (android, flatpak,...) where we cannot have full access to filesystem. In those cases, file is opened by an external tool and we only get a fd.

    👍React with 👍1Reacted by ProgrammerWhoPrograms
11. veloman-yunkan commented on May 13, 2024

    

    veloman-yunkan

    on May 13, 2024

    Last edited by veloman-yunkan

    Collaborator

    More actions

    This comment kiwix/libkiwix#1015 (comment) is interesting and must not be forgotten, and here an answer:

    Opening an archive with a list of (file-path, offset, size) would be interesting/needed in itself but not enough. We may have system (android, flatpak,...) where we cannot have full access to filesystem. In those cases, file is opened by an external tool and we only get a fd.

    @mgautierfr As I noted in the said comment, we convert a file descriptor to a file path (of the form /dev/fd/XXX) and actually open the file by path. What I had proposed was to move that trick out of libzim - let the users do that conversion themselves (libzim may simply provide a utility) and the public API will only accept paths.
12. mgautierfr commented on May 13, 2024

    

    mgautierfr

    on May 13, 2024

    Contributor

    More actions

    The problem is that the trick may be not possible at all. Moving it to the user side move the problem outside of libzim but the root issue is still there: We cannot assume we can open by path (even /dev/fd/xxx)
13. kelson42

    modified the milestones: 9.3.0, 3.4.0 on Sep 14, 2024
14. soloturn commented on Aug 30, 2026

    

    soloturn

    on Aug 30, 2026

    More actions

    AI-assisted comment. Left by agent driven by @soloturn via GDD.

    @mgautierfr — status update on the PRs open against this.

    What's there

    • fix: open fd-backed archive parts via dup() instead of reopening by path #1119 (this repo, open): dup() the fd we already have instead of reopening by path. Directly answers your 2024-05-13 concern above ("we cannot assume we can open by path, even /dev/fd/xxx") - dup() never opens by path at all, so there's no path-assumption left to make. Covers FilePart/getDbFromAccessInfo(), i.e. the Xapian index case this issue opened with.
    • feat: expose the fd from ItemDataDirectAccessInfo across JNI kiwix/java-libkiwix#151 (draft, depends on fix: open fd-backed archive parts via dup() instead of reopening by path #1119): wires the same fd through JNI for Item::getDirectAccessInformation(), so large media items get the same dup() treatment, not just the Xapian db.
    • fix(reader): use the JNI-provided fd for direct-access reads instead of reopening by path kiwix/kiwix-android#5056 (draft, depends on the above): ZimFileReader uses the JNI-provided fd instead of reopening by path for direct-access reads.
    • fix(reader): stop reopening SAF fds by path just to check they're readable kiwix/kiwix-android#5057 (open, independent, not blocked): a second manifestation of the same root cause - a pre-flight "can I open this fd" check in FileUtils.kt was also reopening by path (citing kiwix-android#3636 in its own comment) before ever reaching libkiwix. Fixed the same way, doesn't depend on fix: open fd-backed archive parts via dup() instead of reopening by path #1119.

    What's still missing

    • fix: open fd-backed archive parts via dup() instead of reopening by path #1119 has no maintainer review and no real CI run yet - only CodeFactor + readthedocs checks show on the PR; no build/test workflow has fired. Might be a fork-PR approval gate.
    • Only verified on-device on arm64-v8a (real emulator repro, not this repo's own test suite - see the PR's verification section). x86/x86_64 - the architectures also named in the original kiwix-android#3636 report - not independently confirmed, since the Apple Silicon host used can't run an x86_64 Android emulator.
    • java-libkiwix#151 / kiwix-android#5056 not yet exercised end-to-end with a real SAF-picked large media item on an affected device, since they're chained behind fix: open fd-backed archive parts via dup() instead of reopening by path #1119 merging and shipping a release.
    • This issue's title is broader than the Xapian case fix: open fd-backed archive parts via dup() instead of reopening by path #1119 covers ("Allow extern getFd()... for Kiwix Android") - closing it for real probably wants the JNI+Android layer (Delete "subprojects" directory / GoogleTest source code from the git repo #151/#5056) landed too, not just the libzim-side fix.
15. kelson42

    unassigned

    mgautierfr

    on Aug 31, 2026
16. soloturn

    added 3 commits that reference this issue on Aug 31, 2026

    fix: address review comments on the fd-based-xapian-open PR

    ...

    7daa73b

    feat: add Item::getDirectAccessFd(), a new ABI-safe public API for th…

    ...

    2d6b49d

    feat: expose a direct-access fd for items across JNI

    ...

    3113814