Supply source disaggregation, irrigation conveyance, and feature enhancements

.github/workflows/build.yml

@@ -16,28 +16,26 @@ jobs:
       strategy:
         matrix:
           os: [ubuntu-latest]
+          python-version: ['3.9', '3.11']
 
       env:
         OS: ${{ matrix.os }}
-        PYTHON: '3.9'
+        PYTHON: ${{ matrix.python-version }}
 
       steps:
 
         - uses: actions/checkout@v1
 
         - name: Set up Python
-          uses: actions/setup-python@master
+          uses: actions/setup-python@v5
           with:
-            python-version: 3.9
+            python-version: ${{ matrix.python-version }}
 
         - name: Install dependencies
           run: |
             python -m pip install --upgrade pip
-            pip install click-default-group
-            pip install .
+            pip install -e .[dev]
 
         - name: Test and generate coverage report on Linux
           run: |
-            pip install pytest
-            pip install pytest-cov
             pytest --cov=./ --cov-report=xml

.github/workflows/paper.yml

@@ -1,5 +1,13 @@
 ---
-on: [push]
+name: paper
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
 
 jobs:
   paper:
@@ -15,7 +23,7 @@ jobs:
           # This should be the path to the paper within your repo.
           paper-path: paper/paper.md
       - name: Upload
-        uses: actions/upload-artifact@v1
+        uses: actions/upload-artifact@v4
         with:
           name: paper
           # This is the output path where Pandoc will write the compiled

README.md

@@ -14,7 +14,7 @@ Details on methodology and advanced usage are available on the [User Guide](http
 
 Install `tethys` using pip:
 
-`pip install tethys-downscale`
+`pip install tethys-downscaling`
 
 ## Contributing to Tethys
 

docs/source/conf.py

@@ -26,7 +26,7 @@
 
 project = 'tethys'
 copyright = '2021, Battelle Memorial Institute'
-author = 'Chris R. Vernon'
+author = 'Isaac Thompson, Chris R. Vernon, Hassan Niazi'
 
 # The full version, including alpha/beta/rc tags
 release = version

docs/source/getting_started.rst

@@ -21,7 +21,7 @@ As a prerequisite, you'll need to have `Python <https://www.python.org/downloads
 
 **tethys** can be installed from GitHub using pip::
 
-  pip install tethys-downscale
+  pip install tethys-downscaling
 
 This will automatically install the dependencies. In order to avoid package version conflicts, consider using a virtual environment.
 

docs/source/user_guide.rst

@@ -102,22 +102,24 @@ Configuration File
 ------------------
 **tethys** uses a `YAML <https://yaml.org/spec/1.2.2/>`_ configuration file. The options in this file correspond to the arguments passed to the :ref:`Tethys class <tethys.model>`. Options not present in the config file will use the default. An overview is provided in the following table, with more details and examples below.
 
-======================== =========================================================
-Option                   Description
-======================== =========================================================
-:ref:`years`             list of years to be included spatial downscaling
-:ref:`resolution`        resolution in degrees for spatial downscaling
-:ref:`demand_type`       choice between "withdrawals" (default) or "consumption"
-:ref:`perform_temporal`  choice between "false" (default) or "true"
-:ref:`gcam_db`           relative path to a GCAM database
-:ref:`csv`               relative path to csv file containing inputs
-:ref:`output_file`       name of file to write outputs to
-:ref:`downscaling_rules` mapping from water demand sectors to proxy variables
-:ref:`proxy_files`       mapping of spatial proxy files to their years/variables
-:ref:`map_files`         list of files containing region maps
-:ref:`temporal_methods`  mapping of sector to temporal downscaling method
-:ref:`temporal_files`    files that will be accessible during temporal downscaling
-======================== =========================================================
+======================================= =============================================================================
+Option                                  Description
+======================================= =============================================================================
+:ref:`years`                            list of years to be included spatial downscaling
+:ref:`resolution`                       resolution in degrees for spatial downscaling
+:ref:`demand_type`                      choice between "withdrawals" (default) or "consumption"
+:ref:`perform_temporal`                 choice between "false" (default) or "true"
+:ref:`gcam_db`                          relative path to a GCAM database
+:ref:`csv`                              relative path to csv file containing inputs
+:ref:`output_file`                      name of file to write outputs to
+:ref:`downscaling_rules`                mapping from water demand sectors to proxy variables
+:ref:`proxy_files`                      mapping of spatial proxy files to their years/variables
+:ref:`map_files`                        list of files containing region maps
+:ref:`temporal_methods`                 mapping of sector to temporal downscaling method
+:ref:`temporal_files`                   files that will be accessible during temporal downscaling
+:ref:`source_disaggregation`            if true, share of surface water vs groundwater withdrawal will be written out
+:ref:`irrigation_conveyance_efficiency` if set, irrigation withdrawal will be divided by this number
+======================================= =============================================================================
 
 
 years
@@ -272,6 +274,23 @@ Mapping of files that will be accessible to temporal downscaling methods.
     CDD: data/temporal/CDD_monthly.nc
 
 
+source_disaggregation
+^^^^^^^^^^^^^^^^^^^^
+If true, share of surface water vs groundwater withdrawal will be written out to a file called gridded_runoff_shares.nc
+.. code-block:: yaml
+
+  source_disaggregation: true
+
+
+irrigation_conveyance_efficiency
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+If set, irrigation withdrawal will be divided by this number, representing GCAM's irrigation conveyance efficiency
+
+.. code-block:: yaml
+
+  irrigation_conveyance_efficiency: 0.829937
+
+
 Generalization
 --------------
 **tethys** was developed with consideration for GCAM's breakdown of water demand, but was designed to be as flexible as possible with support for user-specified downscaling configurations.

requirements.txt

@@ -1,14 +1,12 @@
 PyYAML>=6.0
-gcamreader>=1.2.5
+gcamreader>=1.4.0
 numpy>=1.22
 pandas>=1.2.4
 netCDF4>=1.6
 xarray>=2022.09.0
 rioxarray>=0.12.4
 tqdm>=4.66.2
 matplotlib>=3.8.3
-dask[complete]>=2022.12.1
-bokeh!=3.0.*,>=2.4.2
-geopandas>=0.14.3
-dask_geopandas>=0.3.1
-hvplot>=0.9.2
+dask>=2022.12.1
+distributed>=2022.12.1
+setuptools<81

setup.py

@@ -23,35 +23,37 @@ def get_requirements():
     url='https://github.com/JGCRI/tethys',
     packages=find_packages(),
     license='BSD-2-Clause',
-    author='Isaac Thompson, Chris Vernon',
+    author='Isaac Thompson, Chris Vernon, Hassan Niazi',
     python_requires='>=3.9, <4',
     include_package_data=True,
     install_requires=[
         'PyYAML>=6.0',
-        'gcamreader>=1.2.5',
+        'gcamreader==1.4.0',
         'numpy>=1.22',
         'pandas>=1.2.4',
         'netCDF4>=1.6',
-        'dask[complete]>=2022.12.1',
-        'dask_geopandas>=0.3.1',
+        'dask==2024.8.0',
+        'distributed==2024.8.0',
         'xarray>=2022.09.0',
         'rioxarray>=0.12.4',
         'tqdm>=4.66.2',
         'matplotlib>=3.8.3',
+        'setuptools<81',
         'bokeh!=3.0.*,>=2.4.2',
         'geopandas>=0.14.3',
+        'dask_geopandas>=0.3.1',
         'hvplot>=0.9.2',
     ],
     extras_require={
         'dev': [
-            'build~=0.5.1',
-            'nbsphinx~=0.8.6',
-            'setuptools~=57.0.0',
-            'sphinx~=4.0.2',
-            'sphinx-panels~=0.6.0',
-            'sphinx-rtd-theme~=0.5.2',
-            'sphinx-mathjax-offline~=0.0.1',
-            'twine>=3.4.1',
+            'build==0.10.0',
+            'nbsphinx==0.8.12',
+            'sphinx==4.5.0',
+            'sphinx-panels==0.6.0',
+            'sphinx-rtd-theme==0.5.2',
+            'sphinx-mathjax-offline==0.0.2',
+            'pytest>=8.3.0',
+            'pytest-cov>=5.0.0',
             'click-default-group>=1.2.4',
         ]
     }

tethys/__init__.py

@@ -3,4 +3,4 @@
 from .utils.spatial import InputViewer
 
 
-__version__ = "2.1"
+__version__ = "2.2.0"

tethys/datareader/easy_query.py

@@ -1,7 +1,7 @@
 import gcamreader
 
 
-def easy_query(variable, year_axis=None, **kwargs):
+def easy_query(variable, year_axis=None, replace_filters=False, **kwargs):
     """Build a query for a GCAM database
 
     kwargs act as filters on nodes, eg. "sector='Beef'" is converted to the xpath "[@type='sector' and @name='Beef']"
@@ -24,8 +24,11 @@ def easy_query(variable, year_axis=None, **kwargs):
     :return: gcamreader Query object
     """
 
-    filters = dict(sector=None)  # match nodes where @type='sector' by default
-    filters.update(kwargs)  # update the filters with user-provided kwargs
+    if replace_filters:
+        filters = kwargs
+    else:
+        filters = dict(sector=None)  # match nodes where @type='sector' by default
+        filters.update(kwargs)  # update the filters with user-provided kwargs
 
     # filters are separated by "//*" to match any descendant
     xpath = "*" + "//*".join(handle_filter(k, v) for k, v in filters.items()) + f"//{variable}/node()"

tethys/datareader/regional.py

@@ -41,6 +41,19 @@ def load_region_data(gcam_db, sectors, demand_type='withdrawals'):
     return df
 
 
+def extract_resource_name(x):
+    """Removes '_water withdrawals' or '_water consumption' from GCAM sector or resource name.
+
+    :param x: resource name string.
+    :return: cleaned resource name without suffix.
+    """
+    if x.endswith('_water withdrawals'):
+        return x.removesuffix('_water withdrawals')
+    if x.endswith('_water consumption'):
+        return x.removesuffix('_water consumption')
+    return x
+
+
 def extract_basin_name(x):
     """Maps 'water_td_irr_basin_C' to '_basin', and water_td_elec_C to ''"""
     if x.startswith('water_td_irr_'):