This repo contains scripts to import and/or pre-process the layers required for the Rural Water Security Plugin.

Install miniconda and create a virutal environment for python3.7

refer Conda Cheat Sheet

The Home Directory for different OS,

windows: C:\Users\USERNAME

mac: /Users/USERNAME

linux: /home/USERNAME

create directory named 'Code' and 'Data' in Home Directory

mkdir Code

clone this repository inside 'Code' dir

cd Code
git clone https://github.com/atreebangalore/ruralwaterLayers.git

rename 'ruralwaterLayers' into 'atree'

mv ruralwaterLayers atree

Install the required packages in the python environment.

pip install -r requirements.txt

in general, execute the scripts from the home directory,

python Code/atree/scripts/{dataLayer}/{script}.py [arguments]

arguments may vary based on the script.

Calculation of Gini coefficient of Evapotranspiration vales for districts of the chosen states.

python Code/atree/scripts/equity/1_getETValues.py [Year] [ST]
python Code/atree/scripts/equity/2_calcGiniValues.py [Year] [ST]

Equity has two script to be executed successively.

They both take two arguments,

Year - interger value for the water year. (YYYY)

ST - Two letter Abbreviated State Names seperated by comma.

python Code/atree/scripts/equity/1_getETValues.py 2018 KA,TN
python Code/atree/scripts/equity/2_calcGiniValues.py 2018 KA,TN

The corresponding output csv files will be saved in the path Code/atree/outputs/equity

Downloads monthly SEBOP ET images and generation of yearly SEBOP ET image by summation of the monthly images and extraction of ET pixel values for districts of chosen state.

Requires the shape file of SOI National Boundary to be located at {Home Dir}\Data\gis\soi_national_boundary.shp

getWaterYrSEBOP.py will call getMonthlySEBOP.py script to download monthly images.

python Code/atree/scripts/evapotranspiration/getWaterYrSEBOP.py [Year]

Year - integer value for the water year

python Code/atree/scripts/evapotranspiration/getWaterYrSEBOP.py 2019

Monthly SEBOP images will be downloaded to {Home Dir}/Data/et/sebop/

also in that same loaction annual year SEBOP ET image will be generated as wy{year}_india.tif.

Downloads ET zip data from USGS for specified month and year. Extract the zip file and clip it to the soi_national_boundary.shp extent.

To download SEBOP data, wget utility is required.

windows - place the wget.exe file in the location C:\Windows\System32

mac & linux - prebuilt into the OS.

python Code/atree/scripts/evapotranspiration/getMonthlySEBOP.py [Year] [Month]
python Code/atree/scripts/evapotranspiration/clipMonthlySEBOP.py [Year] [Month]

Year - Year for which data is to be downloaded

Month - two digit month for which data to be downloaded (eg: 04)

upload the ET SSEBop files from GCP bucket to the GEE Image Collection

python Code/atree/scripts/rainfall/IMDHistoricalGCP2GEE.py [year] [bucket] [user] [coll]

year - monthly images of the year to upload

bucket - Google Cloud Platform bucket name

user - Google Earth Engine Username

coll - Google Earth Engine Image Collection name

MEAN DAILY PERCENTAGE (p) OF ANNUAL DAYTIME HOURS FOR DIFFERENT LATITUDES calculated by FAO method

Requires a tmax or tmin tif file from IMDHistoricalTif2Daily output as a reference to generate the skeleton of p value tif file.

python Code\atree\scripts\evapotranspiration\fao\pAnnualDaytimeHrs.py [var_type] [filename]

var_type: tmax or tmin

filename: filename of output from IMDHistoricalTif2Daily.py for tmax or tmin

{Home Dir}\Data\et\fao\p_value\

Calculate Reference Crop ET as per FAO, This script requires p-value from pAnnualDaytimeHrs.py and mean Temperature from IMDHistoricalMeanTemp.py which in-turn requires max and min IMD Temperature Imagery.

python Code/atree/scripts/evapotranspiration/fao/refCropET.py [start_yr] [end_yr]

start_yr : Starting year

end_yr : Ending year

{Home Dir}/Data/et/fao/refCropET/{year}

python Code/atree/scripts/evapotranspiration/getDistrictValues.py [StateName] [Year]

StateName - state name as in {StateName}_district_boundary.shp

Year - integer value for the water year (year for which monthly images were downloaded).

Requires district shape file of chosen state to be loacated at {Home Dir}\Data\gis\{State Name}_district_boundary.shp

python Code/atree/scripts/evapotranspiration/getDistrictValues.py tamilnadu 2019

csv files for all the districts of chosen state will be generated at Code/data/evapotranspiration/SEBOP/yearly/stats

Downloads IMD gridded data for precipitation or min temp or max temp (IMDHistoricalGrid.py), converts the grd data into annual tif imagery (IMDHistoricalGrid2Tif.py) and splits the annual imagery bands into individual daily imagery (IMDHistoricalTif2Daily.py). Upload of daily imagery to GEE can be done using IMDHistoricalGCP2GEE.py script. Summation of Daily imagery into "yearly" or "monthly" or "water yr" (IMDHistoricalDailySum.py).

Calculate monthly Effective Precipitation (effPrecipitation.py) from the Daily rainfall images of IMDHistoricalTif2Daily.py

Calculate mean Temperature (IMDHistoricalMeanTemp.py) both daily and monthly from the Daily max and min Temp images of IMDHistoricalTif2Daily.py

python Code/atree/scripts/rainfall/IMDHistoricalGrid.py [type] [start_yr] [end_yr]
python Code/atree/scripts/rainfall/IMDHistoricalGrid2Tif.py [type] [start_yr] [end_yr]
python Code/atree/scripts/rainfall/IMDHistoricalTif2Daily.py [type] [start_yr] [end_yr]
python Code/atree/scripts/rainfall/IMDHistoricalGCP2GEE.py [year] [bucket] [user] [coll]
python Code/atree/scripts/rainfall/IMDHistoricalDailySum.py [type] [start_yr] [end_yr] [period]
python Code/atree/scripts/rainfall/effPrecipitation.py [start_yr] [end_yr]
python Code/atree/scripts/rainfall/IMDHistoricalMeanTemp.py [start_yr] [end_yr]

type - rain or tmin or tmax

start_yr - starting year (YYYY)

end_yr - ending year (YYYY)

year - daily images of the year to upload

bucket - Google Cloud Platform bucket name

user - Google Earth Engine Username

coll - Google Earth Engine Image Collection name

period - annual or monthly or wateryr (wateryr is from june 1st of start_yr to may 31st of end_yr)

python Code/atree/scripts/rainfall/IMDHistoricalGrid.py rain 2018 2019
python Code/atree/scripts/rainfall/IMDHistoricalGrid2Tif.py rain 2018 2019
python Code/atree/scripts/rainfall/IMDHistoricalTif2Daily.py rain 2018 2019
python Code/atree/scripts/rainfall/IMDHistoricalDailySum.py rain 2018 2019 wateryr
python Code/atree/scripts/rainfall/effPrecipitation.py 2018 2019
python Code/atree/scripts/rainfall/IMDHistoricalMeanTemp.py 2018 2019

IMDHistoricalGrid.py - Gridded data stored in {Home Dir}/Data/imd

IMDHistoricalGrid2Tif.py - {Home Dir}/Data/imd (respective folders for grd and tif)

IMDHistoricalTif2Daily.py - {Home Dir}/Data/imd/rain/tif/{year}/YYYYMMDD.tif

IMDHistoricalGCP2GEE.py - daily images uploaded to GEE Assets - Image Collection

IMDHistoricalDailySum.py - {Home Dir}/Data/imd/{type}/tif/period

effPrecipitation.py - {Home Dir}/Data/imd/{var_type}/tif/eff_precipitation/monthly

IMDHistoricalMeanTemp.py - {Home Dir}/Data/imd/tmean

python Code/atree/scripts/groundwater/levels/1_gw_preProcess.py [ST]
python Code/atree/scripts/groundwater/levels/2_gw_elevations.py [ST]
python Code/atree/scripts/groundwater/levels/3_gw_rechargedischarge.py [ST]

ST - two letter abbreviated State names seperated by comma.

python Code/atree/scripts/groundwater/levels/1_gw_preProcess.py KA,MH
python Code/atree/scripts/groundwater/levels/2_gw_elevations.py KA,MH
python Code/atree/scripts/groundwater/levels/3_gw_rechargedischarge.py KA,MH

csv and shape files at {Home Dir}/Code/atree/outputs/groundwater/levels/preprocessed/

From the JRC Global Surface Water dataset calculate the Volume of surface water (PreMonsoon and PostMonsoon) as Raster image.

python Code/atree/scripts/surfaceWater/JRCgetSW.py [year] [ST]

year - year for which the Volume calculation to be made

ST - Two letter abbreviated State Name.

python Code/atree/scripts/surfaceWater/JRCgetSW.py 2019 KA

PreMonsoon and PostMonsoon - two raster images are exported to GDrive.

this repo contains scripts to import and/or pre-process the layers required for the Rural Water Security Plugin
below are a few examples showing how to run these scripts yourself

CSV : data\groundwater\cgwb_stationwise_historical\CGWB_original.csv Columns: STATE, DISTRICT, LAT, LON, SITE_TYPE, WLCODE, May-96, Aug-96, Nov-96, Jan-97,... Nov-16, Jan-17
Dataset contains 28k+ well locations in India, with their historical water levels (BGL) in May, Aug, Nov, Jan.

i/p: takes original dataset as input
processing: preprocessing involves subsetting by State , removing nulls, duplicates and duplicate geometries, then saving to CSV, SHP (with data) and SHP (geoms only)
Typical usage (in terminal from root directory):
$ python layers/groundwater/levels/gw_preProcess.py [ST] # [ST] is two letter short code for state name
o/p: outputs/groundwater/csv/[ST]_metadata.log,
outputs/groundwater/csv/[ST]_processed.csv
outputs/groundwater/shapefiles/[ST]_processed.shp

i/p: takes preprocessed dataset as input
processing: reads in a pre-processed groundwater layer (SHP) and returns elevations, depth-to-water (above MSL) for each
Typical usage (in terminal from root directory):
$ python layers/groundwater/levels/gw_elevations.py [ST]
o/p: outputs/groundwater/csv/[ST]_metadata.log,
outputs/groundwater/csv/[ST]_processed_wElev.csv
outputs/groundwater/shapefiles/[ST]_processed_wElev.shp

i/p: takes dataset with elevations as input
processing: reads in a pre-processed groundwater layer with depth to water (above MSL) (CSV) and returns monsoon recharge, non-monsoon discharge for all years
Typical usage (in terminal from root directory):
$ python layers/groundwater/levels/gw_RechargeDischarge.py [ST]
o/p: outputs/groundwater/csv/[ST]_metadata.log,
outputs/groundwater/csv/[ST]_processed_wRD.csv
outputs/groundwater/shapefiles/[ST]_processed_wRD.shp

i/p: takes dataset with recharge-discharge as input
processing: reads in a pre-processed groundwater layer, with recharge-discharge (CSV) and returns tif
Typical usage (in terminal from root directory):
$ python layers/groundwater/levels/gw_Rasterize.py [ST]
o/p: outputs/groundwater/csv/[ST]_metadata.log,
outputs/groundwater/csv/[ST]_processed_wRD.csv
outputs/groundwater/shapefiles/[ST]_processed_wRD.shp

To be completed

Source: monthly ET , yearly ET requires command line tool 'wget' to be installed' download these as needed with wget -i layers/evapotranspiration/listoflinks.txt

use gdal batch script in layers/evapotranspiration/

Source: historical rainfall