Conversation
* Add setup script and update entrypoint for Airflow environment initialization * Updates documentation for creating .env file and removes setup script * Add an initialization DAG to run the initialization script every two hours and launch the main DAG * Add Dockerfiles and entrypoint scripts for Airflow and web GUI setup * Remove the 'heasarc' directory creation from the Dockerfiles and the web GUI entrypoint script to make the given folder be mounted as the 'heasarc' directory * Refactor file handling in DataPipeline to use shutil.move for better directory management * Add an interface to browse PDF files in DL0 directory * Update .gitignore to exclude all files in the given folder except explorer.js and index.html * Update .gitignore to exclude all files in the data directory except explorer.js and index.html * Update the initialization DAG to start immediately and improve task management * Remove unused timedelta import in cosipipe_cosipy.py * Update cosipipe_cosipy.py Removed commented line code. * Rename initialization DAG to 'cosipy_contactsimulator' for clarity * Update README.md to improve DAG build and testing instructions * Update UI text for clarity and consistency in explorer.js and index.html * Updates the instructions in README.md for building and running Docker on Mac and Linux, improving clarity and consistency. * Set the start date of the DAG 'cosipy_contactsimulator' to a specific time to avoid unexpected behavior
…n the airflow service, and updated the data path in the docker-compose
…itialize the pipeline data fetch
…nge user id of gamma
- Created DAG `cosiflow_alert_monitor` that periodically reads log file `data_pipeline.log` - Implemented `alert_manager` module with error parsing, YAML rules, deduplication and notification sending - Integration with Mailhog for local email sending testing - Added Airflow plugin to access Mailhog Web UI via "Develop tools" menu - Added support for SMTP environment variables via `.env`
… SMTP settings using MailHog
…tatements. Add new notify_email and removed central logging file
…structure - Changed data mount path in docker-compose.yaml to align with new directory structure. - Added Conda Terms of Service acceptance in Dockerfile for required channels. - Enhanced entrypoint script to export COSI directory structure environment variables and create necessary directories if not present. - Updated Python version comment in environment.yml for clarity.
…ad functionality - Introduced `heasarc_explorer_plugin` for browsing data files in a specified directory. - Implemented Flask routes for home, folder navigation, and file downloads. - Added a basic HTML template for the data explorer interface. - Created a view plugin for integration with Airflow's app builder. - Removed the obsolete `dl3_explorer_view_plugin` to streamline the codebase.
…lowing all file types - Updated `explorer_home` and `explorer_folder` functions to include `current_path` in the template context. - Modified file listing in `explorer_folder` to show all file types instead of just PDFs. - Added a visual element in `explorer.html` to display the current path for better user navigation.
… directory paths - Updated directory path definitions in `DataPipeline` to utilize environment variables for better flexibility. - Ensured the input directory is created if it doesn't exist and adjusted the inotify watch to monitor the base directory directly.
… COSI installation - Added `unzip` to the list of installed packages for the Airflow Docker environment. - Updated the COSI installation process to install `py7zr` and changed the git checkout to version `v0.3.x` for compatibility.
… and navigation improvements - Added tags for better organization in the `fail_task` DAG. - Enhanced `explorer.html` with additional CSS comments for clarity. - Improved navigation button descriptions and added comments for folder and file link functionalities.
- Introduced `cosipipe_tsmap__extpythonenv.py` for multi-task TS map computation with external Python environment. - Added `cosipipe_tsmap__singletask__extpythonenv.py` for optimized single-task execution of the TS map pipeline. - Created `cosipipe_tsmap_mulres.py` for multi-resolution TS map computation. - Implemented `cosipipe_tsmap.py` for standard TS map computation. - Developed supporting scripts for data preparation, binning, aggregation, and TS map computation. - Enhanced `tsmap_pipeline.py` to manage the entire TS map processing workflow, supporting both standard and multi-resolution modes. - Added detailed logging and error handling for improved pipeline robustness. - Included cleanup tasks for better resource management post-execution.
- Introduced `dag_parallel_test_1` and `dag_parallel_test_2` for parallel task execution. - Each DAG includes two BashOperator tasks that simulate a 60-second sleep. - Configured with a maximum of 2 active runs and a concurrency of 3 for testing parallelism.
…vices - Added a new volume mapping for the pipeline directory to the Airflow service configuration, enhancing the environment setup for pipeline execution.
- Introduced `cosipipe_lightcurve.py` DAG to automate the process of generating GRB light curves from newly arrived compressed folders. - Implemented a sequence of tasks including waiting for new archives, decompressing them, binning GRB sources and backgrounds, and plotting the light curve. - Created `cosipipe_lc_ops.py` with utility functions for archive decompression, input validation, and data binning, ensuring modularity and reusability. - Enhanced the pipeline's robustness with error handling and validation checks for required input files.
…vigation - Added a new route for file previewing, allowing users to view file contents directly in the interface. - Implemented security checks to ensure safe access to files and directories. - Enhanced the HTML template with a two-column layout for file listings and previews, improving user experience. - Included JavaScript functionality for single-click preview and double-click download actions on files. - Added error handling and user feedback for file access issues and loading states.
- Created a new example environment file to define essential Airflow environment variables. - Included settings for Airflow admin credentials, SMTP configuration, and COSI directory structure. - This file serves as a template for users to set up their local environment for the Airflow application.
- Modified the Dockerfile to set default values for user and group IDs as empty, enabling users to specify their own values during the build process. - This change enhances flexibility for user management within the Airflow environment.
- Introduced UID, GID, and DISPLAY variables to the .env.example file, allowing users to specify their own bootstrap ID settings for containerized environments. - This update enhances the configurability of the Airflow environment setup.
…apping - Added UID and GID environment variables for user-specific configurations in the Postgres, Airflow, and Mailhog services. - Enabled volume mapping for Postgres data to persist across container restarts. - Updated Airflow service to utilize an environment file for configuration, improving setup flexibility.
- Changed default values for MY_UID and MY_GID in the Dockerfile to specific integers (12050 and 10000, respectively). - This update ensures a consistent user and group configuration for the Airflow environment during the build process.
- Changed the SQLAlchemy connection string to specify the Postgres host and port explicitly, enhancing clarity and ensuring proper connectivity for the Airflow environment.
…h color-coded messages
… and consumer. Add also a UI to fetch arrived notices
…tion and improved UI comments
… features, including UI updates for better user interaction and auto-refresh functionality. Add service status display for GCN client and improve notice table layouts for better readability.
…and user experience
- Introduced a new YAML configuration file `performance_test.yaml` to define settings for performance testing, including Airflow service details, CSV output, logging options, cleanup rules, and DAG configurations. - Created a new shell script `run_performance_test.sh` to execute the performance test within a Docker container, utilizing the defined configuration file and handling necessary environment variables.
…reating virtual environments
… task dependency handling
…lient README for prototype details
…view names for clarity
…tency across the UI
- Updated README.md files for gcn-client and modules to clarify configuration and usage instructions. - Improved descriptions and organization in the CosiDAG developer guide, including constructor parameters and monitoring policies. - Enhanced the HEASARC Explorer plugin documentation for clarity on menu entries and features. - Revised Explore Notices plugin documentation to emphasize authentication requirements and menu structure. - Updated MailHog Link plugin documentation to reflect menu changes and authentication needs. - Improved Refresh DAGs List plugin documentation to clarify usage and authentication requirements. - Enhanced Reset Cosidag plugin documentation for better user guidance on managing processed states. - Added authentication requirements to relevant plugin views to ensure secure access.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR promotes the accumulated changes from
devtomain, consolidating 163 commits into a new COSIflow baseline.It evolves COSIflow into a modular Airflow-based environment for orchestrating COSI scientific pipelines, with an enhanced deployment stack, the COSIDAG framework, data and notice exploration tools, operational plugins, and performance-testing utilities.
Main changes
Airflow environment and deployment
hot_load_module.shfor installing, updating, and removing scientific modules without rebuilding the full stack..envfile.COSIDAG framework
COSIDAGAirflow DAG subclass for filesystem-driven scientific workflows.dag_run.conf.HEASARC Data Explorer and Airflow plugins
GCN notice workflow
Testing, monitoring, and documentation
v0.1.0,v0.1.1, andv0.2.0tags.Deployment notes
After merging:
Review the UID, GID, host address, and exposed ports in
env/docker-compose.yaml.Store database passwords and optional GCN Kafka credentials in
env/.env.Create the required persistent directories:
Rebuild and restart the stack:
cd env docker compose build docker compose up -dReal GCN publication remains disabled by default.