We derive computed tomography (CT) of a time-varying volumetric scattering object, using a small number of moving cameras. We focus on passive tomography of dynamic clouds, as clouds have a major effect on the Earth's climate. State of the art scattering CT assumes a static object. Existing 4D CT methods rely on a linear image formation model and often on significant priors. In this paper, the angular and temporal sampling rates needed for a proper recovery are discussed. Spatiotemporal CT is achieved using gradient-based optimization, which accounts for the correlation time of the dynamic object content. We demonstrate this in physics-based simulations and on experimental real-world data.
This repository contains the official implementation of 4D Cloud Scattering Tomography, which is implemented ontop of Pyshdom3.0 package [1].
Our framework recovers 4D cloud microphysics fields, using a small number of moving cameras. We exploits the natural temporal evolution of clouds for weighing the gradients of the different time steps during the recovery process. For more details see our paper [2] and supplementary material [3].
Installation using using anaconda package management
Start a clean virtual environment
conda create -n pyshdom python=3
source activate pyshdom
Install required packages
conda install anaconda dill tensorflow tensorboard pillow joblib
Install pyshdom distribution with (either install or develop flag)
python setup.py develop
We used cloud field data that were generated by Eshkol Eytan. Download here the data and place it in 4D-Cloud-Scattering-Tomography/synthetic_cloud_fields/WIZ_Clouds/BOMEX2. For AirMSPI raw data please download the files from here and place in 4D-Cloud-Scattering-Tomography/AirMSPI/21views.
Scripts are devided into three categories: generate, render and optimize. For more information please read here.
To generate Mie scattering tables for cloud simulation and AirMSPI wavelengths
python scripts/generate_mie_tables.py \
--start_reff 1.0 --end_reff 25.0 --num_reff 50 --start_veff 0.01 --end_veff 0.2 --num veff 50 \
--radius_cutoff 65.0 --wavelength 0.66 0.865Prapare the data of Cloud1 simulations by running the following notebooks (set n_satellite 2 or 3)
4D_cloud_scattering_tomography/CVPR/Cloud_1_Rendering_Dynamic_Scene.ipynb
4D_cloud_scattering_tomography/CVPR/Cloud_1_Rendering_Dynamic_Scene_Multispectral.ipynb
4D_cloud_scattering_tomography/CVPR/Cloud_1_Rendering_Dynamic_Scene_Single_Platform.ipynb
and for Cloud2
4D_cloud_scattering_tomography/CVPR/Cloud_2_Rendering_Dynamic_Scene.ipynb
4D_cloud_scattering_tomography/CVPR/Cloud_2_Rendering_Dynamic_Scene_Single_Platform.ipynb
Results reproducing
python 4D_cloud_scattering_tomography/scripts/optimize_dynamic_extinction_lbfgs.py --input_dir PATH_TO_DATA --add_rayleigh --use_forward_cloud_velocity --use_forward_grid --init Homogeneous --extinction 1 --space_carve_agreement 0.5 --log LOG_DIR --radiance_threshold 0.03 --n_jobs 72 --maxiter 100 --reg_const 1 --use_cross_validation -1 --num_mediums -1 --sigma 20
For microphysics estimation (LWC and Reff) use
python 4D_cloud_scattering_tomography/scripts/optimize_dynamic_microphysics_lbfgs.py --input_dir PATH_TO_DATA --add_rayleigh --use_forward_cloud_velocity --use_forward_grid --const_veff --const_reff --one_dim_reff --init Homogeneous --n_jobs 72 --log LOG_DIR --maxiter 100 --use_cross_validation -1 --num_mediums -1 --reg_const 1 --sigma 20 --lwc 0.06 --reff 6
Set different values of sigma for 4D recovery. Manually choose space_carve_agreement and radiance_threshold values according to the generated data (single platform or 2 or 3 satellites). Static recovery can be obtained by setting sigma=0, num_mediums=1 and reg_const=0.
Data preparation and preprocessing: Download here the data and place it in AirMSPI/21views. Then process and generate the data with the following notebook
4D-Cloud-Scattering-Tomography/AirMSPI/develop_AirMSPI_load.ipynb
Results reproducing for 4D recovery
python4D-Cloud-Scattering-Tomography/scripts/optimize_dynamic_extinction_AirMSPI_lbfgs.py --input_dir 4D-Cloud-Scattering-Tomography/experiments/AirMSPI/dynamic_medium/monochromatic --add_rayleigh --init Homogeneous --space_carve_agreement 0.9 --radiance_threshold -0.03 --ext 1 --nx 80 --ny 80 --nz 80 --log LOG_DIR --n_jobs 72 --maxiter 200 --reg_const 1 --use_cross_validation -1 --num_mediums -1 --sigma 60
and for static 3D results
python4D-Cloud-Scattering-Tomography/scripts/optimize_dynamic_extinction_AirMSPI_lbfgs.py --input_dir 4D-Cloud-Scattering-Tomography/experiments/AirMSPI/dynamic_medium/monochromatic --add_rayleigh --init Homogeneous --space_carve_agreement 0.9 --radiance_threshold -0.03 --ext 1 --nx 80 --ny 80 --nz 80 --log LOG_DIR --n_jobs 72 --maxiter 200 --reg_const 0 --use_cross_validation -1 --num_mediums 1 --sigma 0
For cross validation comparison set use_cross_validation as the index of the left out image.
A video of the paper's results visualisation can be found here. The official results can be downloaded from this link.
If you make use of our work, please cite our paper:
@InProceedings{Ronen_2021_ICCV,
author = {Ronen, Roi and Schechner, Yoav Y. and Eytan, Eshkol},
title = {4D Cloud Scattering Tomography},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2021},
pages = {5520-5529}
}
Thanks to Eshkol Eytan for the cloud simulation data. If you use it please cite:
@article{eytan2021revisiting,
title={Revisiting adiabatic fraction estimations in cumulus clouds: high-resolution simulations with a passive tracer},
author={Eytan, Eshkol and Koren, Ilan and Altaratz, Orit and Pinsky, Mark and Khain, Alexander},
journal={Atmospheric Chemistry and Physics},
volume={21},
number={21},
pages={16203--16217},
year={2021},
publisher={Copernicus GmbH}
}
If you use this package in an academic publication please acknowledge the appropriate publications (see LICENSE file).