Skip to content

add ftp download method - #80

Open
sahiljhawar wants to merge 4 commits into
GFZ:mainfrom
sahiljhawar:ftp-download
Open

add ftp download method#80
sahiljhawar wants to merge 4 commits into
GFZ:mainfrom
sahiljhawar:ftp-download

Conversation

@sahiljhawar

Copy link
Copy Markdown
Contributor

Closes #79

Copilot AI lite review requested due to automatic review settings August 14, 2026 11:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an FTP-based download implementation to el_paso.download, extending the existing download() dispatcher to support pulling files from FTP directories using a glob pattern.

Changes:

  • Added "ftp" as a supported method option in download() and _download_single_step.
  • Implemented FTP connection and download helpers (_ftp_connect, _ftp_download) with directory listing + fnmatch-based selection.
  • Updated docstring to describe the new FTP behavior and anonymous-login fallback.
Suppressed comments (2)

el_paso/download.py:376

  • FTP URLs that specify a non-default port (e.g. ftp://example.com:2121/path) are currently ignored because only parsed.hostname is used. Pass parsed.port through so these URLs work as expected.
    try:
        ftp = _ftp_connect(parsed.hostname or "", authentication_info)
    except OSError as e:
        logger.warning(f"Error connecting to FTP server for {url}: {e}")
        return

el_paso/download.py:352

  • The new FTP download logic isn’t covered by tests, while other download methods have unit/system coverage (see tests/unittests/test_download.py). Add unit tests that mock ftplib.FTP (connect/login/nlst/retrbinary) to cover: pattern matching, port handling, skip_existing behavior, and .gz decompression path.
def _ftp_download(
    current_time: datetime,
    save_path: Path,
    download_url: str,
    file_name_stem: str,
    authentication_info: tuple[str, str],
    rename_file_name_stem: str | None,
    *,
    skip_existing: bool,
) -> None:

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread el_paso/download.py Outdated
Comment thread el_paso/download.py
Comment thread el_paso/download.py Outdated
Comment thread el_paso/download.py
sahiljhawar and others added 3 commits August 14, 2026 13:20
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@sahiljhawar

Copy link
Copy Markdown
Contributor Author

@DoctorRabbit55 The tests is not running maybe because the username and password is not in my forked repo secrets. It works "on my machine"

@DoctorRabbit55

Copy link
Copy Markdown
Collaborator

Hi Sahil! I assume the fnmatch call is there for the version? In that case, we solved this in the past using a regex expression (see the other download methods).

It would probably be easier to add a test for an FTP server, which does not require login information (e.g. https://kp.gfz.de/daten). Otherwise, the test does not run locally for other people.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add ftp method

3 participants