add ftp download method - #80
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an FTP-based download implementation to el_paso.download, extending the existing download() dispatcher to support pulling files from FTP directories using a glob pattern.
Changes:
- Added
"ftp"as a supportedmethodoption indownload()and_download_single_step. - Implemented FTP connection and download helpers (
_ftp_connect,_ftp_download) with directory listing + fnmatch-based selection. - Updated docstring to describe the new FTP behavior and anonymous-login fallback.
Suppressed comments (2)
el_paso/download.py:376
- FTP URLs that specify a non-default port (e.g. ftp://example.com:2121/path) are currently ignored because only parsed.hostname is used. Pass parsed.port through so these URLs work as expected.
try:
ftp = _ftp_connect(parsed.hostname or "", authentication_info)
except OSError as e:
logger.warning(f"Error connecting to FTP server for {url}: {e}")
return
el_paso/download.py:352
- The new FTP download logic isn’t covered by tests, while other download methods have unit/system coverage (see tests/unittests/test_download.py). Add unit tests that mock ftplib.FTP (connect/login/nlst/retrbinary) to cover: pattern matching, port handling, skip_existing behavior, and .gz decompression path.
def _ftp_download(
current_time: datetime,
save_path: Path,
download_url: str,
file_name_stem: str,
authentication_info: tuple[str, str],
rename_file_name_stem: str | None,
*,
skip_existing: bool,
) -> None:
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
@DoctorRabbit55 The tests is not running maybe because the username and password is not in my forked repo secrets. It works "on my machine" |
|
Hi Sahil! I assume the fnmatch call is there for the version? In that case, we solved this in the past using a regex expression (see the other download methods). It would probably be easier to add a test for an FTP server, which does not require login information (e.g. https://kp.gfz.de/daten). Otherwise, the test does not run locally for other people. |
Closes #79