[YQ-Wang]

Description

Add a seekable Python S3 reader using the existing v2 gRPC API and generated dragonfly-api bindings. It preserves whole-object task identity, validates and trims complete pieces, and obtains AWS credentials (including STS tokens) through boto3.

The example includes an optional PyAV video-window consumer, focused tests, and a path-scoped Python 3.11/3.12 CI workflow. PyAV is not required for byte-range reads. The README documents the trusted-socket and immutable-object assumptions, memory limits, and small-range scheduling behavior.

Related Issue

Related to #1821. This demonstrates native gRPC integration for callers that can be adapted; it does not resolve signed HTTP proxy compatibility or close that issue.

Motivation and Context

Python data loaders need selected object bytes in process, including seekable reads for large videos. The existing daemon API can serve this use case while keeping normal task/piece caching and P2P behavior. No daemon or protocol changes are needed.

Validation

Validated the example files in 517d43b8370a6eb571d8e3045959b7fa9865a249 against upstream d5bccb022:

• Python 3.11 and 3.12 in Linux Docker: all 13 tests passed, including real protobuf/gRPC streaming and video decoding. A clean Python 3.11 install without PyAV passed the 12 core tests and skipped the optional video test.
• Ruff syntax/import checks and formatting; git diff --check.
• Real integration with two official dfdaemon v1.5.5 containers, separate caches, manager/scheduler, and MinIO: signed piece GETs returned 206; changing Range after signing in the control request returned 403.
• Repeated ranges fetched 8 MiB from the remote peer without a source GET; overlapping ranges retained the same task ID and hit the daemon's local cache. Small reads used remote peers with ALWAYS and fell back to source with AUTO.
• Two MinIO STS sessions succeeded; invalid credentials on cold objects failed. URL-encoded keys and empty objects passed.
• Two one-minute windows at the two-hour mark of a synthetic three-hour indexed MP4 matched sequential local decoding. The reader transferred 9,923,151 of 39,283,279 object bytes in three RPCs; a fresh reader repeated the windows entirely from daemon cache with no new source GET.

The live tests used MinIO, not AWS S3. Cache hits can still issue metadata HEADs, and these results are functional checks, not throughput benchmarks. Local test deployments and generated logs are excluded from this PR.

AI Assistance

AI tools assisted implementation and testing. I reviewed the code and the testing process.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.50%. Comparing base (d5bccb0) to head (517d43b).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2042   +/-   ##
=======================================
  Coverage   64.50%   64.50%           
=======================================
  Files         101      101           
  Lines       30203    30203           
=======================================
  Hits        19482    19482           
  Misses      10721    10721           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pmady]

Nice example. Splitting core deps from the optional PyAV path and running CI both ways is a good pattern. I also like that the README states the trusted socket and immutable object assumptions up front, examples usually skip that.

One question, more for the maintainers than the author: is examples/ in the client repo where we want runnable integrations to live, or should this sit with the docs on d7y.io? I lean toward keeping it next to the client so CI keeps it honest, and I'm happy to add a cross link from the d7y.io docs once this lands. Worth a quick check before merge that the gRPC calls match the dragonfly-api version pinned in requirements.txt.