| 81f9ff0 | Zhang Yunjun | 14 July 2020, 20:14:24 UTC | wrap-up before new release + README: update version number in the badge + version: update fall back release version number and date | 14 July 2020, 21:00:36 UTC |
| bc650da | Zhang Yunjun | 14 July 2020, 20:12:20 UTC | bugfix in save_gdal.py by @4restwilliams | 14 July 2020, 21:00:36 UTC |
| f95d820 | Zhang Yunjun | 14 July 2020, 20:02:27 UTC | utils0: add lalo/inc_angle_ground2iono_shell_along_los() to facilitate the calculation of range delay caused by ionosphere. | 14 July 2020, 21:00:36 UTC |
| 433b4f8 | Zhang Yunjun | 13 July 2020, 00:45:32 UTC | bugfix in prep_aria and ifg_inv (#399) * bugfix in prep_aria and ifg_inv prep_aria: + add missing geometry module import for prep_aria.py --update option + update error msg for missing dataset ifgram_inversion: + add missing return box in ifgram_inversion_patch() | 13 July 2020, 00:45:32 UTC |
| 7389d97 | Zhang Yunjun | 11 July 2020, 06:03:53 UTC | sbApp: replace args string with list of args smallbaselineApp: + update sub-module calling method, by composing the input using list of arguments instead of using command line in string then split, to support directory path with whitespace in it. + import prep_aria sub-module, instead of calling in command line. Note that with this change, gdal will be required in mintpy for aria users. remove os.sysmte() calls with python function calls in the following scripts: + load_data + mask + prep_roipac + timeseries_rms + tropo_pyaps + tropo_pyaps3 + utils.utils + utils.utils1 + utils.writefile + test/test_smallbaselineApp tropo_pyaps3: bugfix for correct_timeseries() save_hdfeos5: simplify the code with dict.get() for default value roll back. | 11 July 2020, 21:51:19 UTC |
| 663d9f0 | Zhang Yunjun | 10 July 2020, 18:24:08 UTC | stack*: clean up style for global variables + stack: udpate indentation for long global varaibles + stack/stackDict: remove unused BOOL/INT/FLOAT/CPX_ZERO + stackDict: get rid of global variable dataType, and use numpy definition directly for simplicity and clarity | 11 July 2020, 21:51:19 UTC |
| 8139b37 | Zhang Yunjun | 05 July 2020, 23:44:48 UTC | sim.decor: add cal_coh4phase_pdf_bias() sim.decor: add cal_coh4phase_pdf_bias() + add calibrate_coherence4phase_pdf_bias() to calibrate the coherence for the bias introduced by the imperfect phase PDF equation from Tough et al. (1995), in order to better simulate the decorrelation noise from a given spatial coherence. + coherence2phase_variance(): add scatter argument to support PS, besides the DS. update docs: + docs/api/data_structure.md + docs/CONTRIBUTING.md for typos. + docs/README.md to switch citing and contributing + .github/ISSUE_TEMPLATE/bug_report.md + smallbaselineApp.cfg: suitable scenarios for bridging method | 11 July 2020, 21:51:19 UTC |
| 621f9f9 | Zhang Yunjun | 04 July 2020, 02:53:38 UTC | update contributing guidelines + remove contributors list on README.md as it's now clearly displayed on the front page by default. + add contributing short message from the disclaimer adapted from MetPy project + adapt a development process from numpy project, and merge it with the guidelines from GMT | 04 July 2020, 19:00:28 UTC |
| 0a1c147 | Zhang Yunjun | 02 July 2020, 01:43:14 UTC | dem_error.get_design_matrix4time_func(): get rid of inps as arguments + replace the inps namespace argument with date_list, poly_order, step_func_dates, for more independence, so that it can be called from other scripts. + simulation.simulation.timeseries2ifgram(): - more comments - add ax argument for more pythonic call | 03 July 2020, 03:06:35 UTC |
| 4420359 | Zhang Yunjun | 01 July 2020, 23:05:42 UTC | sbApp: move generate_ifgram_aux_file() to ref_point step from modify_network step, to be consistent with the tutorial. | 03 July 2020, 03:06:35 UTC |
| 98c6018 | Zhang Yunjun | 30 June 2020, 01:10:18 UTC | add mintpy.load.autoPath + add a new template option `mintpy.load.autoPath` to use automatically grab the default file paths pre-defined in mintpy.defaults.auto_path.AUTO_PATH_*. The default is turned off. + defaults/auto_path.py - remove the previous global variable autoPath and its usage in smallbaselineApp.py, load_data.py and select_network.py - use relative pre-defined path for all AUTO_PATH_{PROCESSOR} global variables, assuming the current work directory is within the mintpy folder. - add get_dem_file() to simplify the code. + smallbaselineApp.py and load_data.py: remove the previous triplet checking of SCRATCHDIR, PROJECT_NAME, etc. for a more simplified code style + update test/configs/FernandinaSenDT128.txt to use auto path. + docs/dir_structure.md: update metaFile for isce/stripmapStack WARNING: + this new change has not been tested against the default file structure of univ of miami for gamma and roipac processors, although it should still work + select_network.py is not tested neither. | 03 July 2020, 03:06:35 UTC |
| d332b98 | Zhang Yunjun | 28 June 2020, 06:22:03 UTC | tropo_pyaps3: support .unw file input + support correcting tropospheric delay for .unw file using ERA5 via correct_single_ifgram(), convenient for single interferogram analysis. + update latency info of ERA5T + add reference info for ERA5 | 03 July 2020, 03:06:35 UTC |
| b4e00aa | Talib Oliver | 02 July 2020, 18:02:09 UTC | UAVSAR (stripmapStack) support via YYMMDDTHHMMSS time format (#384) This PR add UAVSAR support via YYYYMMDDTHHMM time format to import isce/stripmapStack processed uavsar coregistered stack data. Detailed changes includes: * utils.ptime.py: + get_date_str_format() to identify the time string format, including: - YYYYMMDDTHHMMSS - YYYYMMDDTHHMM - YYMMDDTHHMM - YYYYMMDD - YYMMDD + add time info, besides date info, while calculating the temporal baseline, float years, float days, etc. * replace dt(*time.strptime) with dt.strptime() in all scripts to be able to grab second info from input date string, and remove the unused time module import * adapt utils.readfile and ifgram_inversion to read the new time format * utils.isce_utils: adjust to recognize uavsar data as isce stripmap mode using the "data" metadata filename. * prep_isce: remove unnecessary code for date12 from the initial prep_fringe.py PR. * objects.sensor: + add uavsar to global varialbe SENSOR_NAMES + add PLATFORM name uav + update standardSensorNames and get_unavco_mission_name() in alphebet order with UAV as the mission short name * obj/stack: + evaluate the precision requirement of datasets with very short temporal baselines for the design matrix data type to float32. The test result with both Sentinel-1 of 12 days re-visit time and UAVSAR of 10 mins re-visit time shows float32 is precise enough with max residual less than 0.002 mm. + support time string format by calling ptime.get_time_str_format(). * simulation: + use lower case for all labels + move the old sim_variable_timeseries() to sim_variable_timeseries_v1() + add sim_variable_timeseries() that takes num_date so that it can generate the exact displacement list regardless of re-visit time. * utils/network.select_pairs_sequential(): re-write to support any list of date, in str, datetime.datetime obj, etc. * docs: + adjust docs.api.module_hierarchy.md for the dependency change of objects.stack.py and utils.readfile.py + update docs/hdfeos5.md for UAV + ports.txt: add py37-gdbm Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> | 02 July 2020, 18:02:09 UTC |
| 51e3863 | Zhang Yunjun | 28 June 2020, 22:25:08 UTC | utils1.get_lookup_file(): revert work_dir arg to fix bug (#389) * utils1.get_lookup_file(): revert work_dir arg to fix bug revert the previouly introduced work_dir argument to fix the bug it introduced; and replace it by adding more directory candicates to achive the same functionality. | 28 June 2020, 22:25:08 UTC |
| c2bedb2 | Zhang Yunjun | 27 June 2020, 22:42:05 UTC | view: geo & radar support for --pts-yx/lalo/file options (#388) * view: geo & radar support for --pts-yx/lalo/file options view: geo & radar support for --pts-yx/lalo/file options + utils1.get_lookup_file(): add work_dir argument to be able to specify the mintpy working directory, instead of the default current one only. + objects.coord: search lookup file based on the file path/dir of the input file + utils.plot.read_pts2inps(): fix bug on yx/lalo conversion and format output yx/lalo data type. + utils.plot.add_point_argument(): put --pts-lalo/yx/file as mutually exclusive group for easy management; update help msg. tsview: fix messy terminal msg while clicking masked out nan pixels + For multiple files, do not plot the TS if the current point TS is all nan. + if the point TS from the first file is all nan: - do not print out dis range info - do not estimate slope - do not draw/update figure + use inps.ts_plot_func for simpler code. docs/README: update running mintpy section * Update test_smallbaselineApp.py | 27 June 2020, 22:42:05 UTC |
| 6c7c4b6 | Zhang Yunjun | 27 June 2020, 05:30:58 UTC | tsview: remove --pts-* options + remove --pts-* options from input argument, because it's not implemented + utils/plot.add_point_argument(): change nargs for --pts-yx/lalo from * to 2 for simplicity, so that they can only be used to plot one point; to more multiple points, use --pts-file. + more comments in the code | 27 June 2020, 05:55:00 UTC |
| 961929d | Zhang Yunjun | 26 June 2020, 18:09:02 UTC | obj/ramp: reduced memory usage + constrain design matrix G to be all float32, instead of the previous float64 + delete X after usage | 27 June 2020, 05:55:00 UTC |
| ab7088d | Falk Amelung | 27 June 2020, 00:11:58 UTC | defaults/auto_path: added reading of ISCE watermask (#386) | 27 June 2020, 00:11:58 UTC |
| 40f572a | Sara Mirzaee | 25 June 2020, 02:23:08 UTC | add support for stripmapStack in auto_path (#385) The proposed change is to support auto_path for stripmapStack as the products are in different paths than topsStack. No need to change the template file | 25 June 2020, 02:23:08 UTC |
| 363875c | stackTom | 21 June 2020, 00:09:50 UTC | remove insarmaps scripts (#381) | 21 June 2020, 00:09:50 UTC |
| 82ac7c9 | Zhang Yunjun | 09 June 2020, 21:55:52 UTC | readfile.get_slice_list(): do not check num_band for .int/unw file + readfile.get_slice_list(): do not check num_band for .int/unw file to support amp / pha / cpx reading, which is like multiple bands, even though it's one band actually. + more obvious warning msg for mintpy/simulation + multilook: remove interpolation example as it's not implemented. | 20 June 2020, 03:23:31 UTC |
| 17ec483 | Falk Amelung | 16 June 2020, 01:52:52 UTC | Update dask.md (#377) Add note on Fernandina example and known issues for HPC cluster. | 16 June 2020, 01:52:52 UTC |
| fccb370 | Forrest Williams | 09 June 2020, 02:19:02 UTC | Create save_gdal.py (#363) * Create save_gdal.py to allow the export of mintpy files to GDAL supported file formats, such as geotiff, ENVI, ISCE, etc., for analysis in external GIS programs. * utils/readfile.read_envi_hdr(): fix a potential bug while reading the map info UNIT. Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> | 09 June 2020, 02:19:02 UTC |
| 1713286 | Zhang Yunjun | 05 June 2020, 23:16:02 UTC | readfile.read_binary_file(): + support cpx output with datasetName='cpx' + constrain the band index to the max number of band for isce binary files plot.scale_data2disp_unit: separate field / power quantities for dB multilook: move the warning suppress to the script top fix typo in installation.md | 08 June 2020, 21:16:02 UTC |
| ce60a10 | Zhang Yunjun | 07 June 2020, 03:26:08 UTC | circleci/config: add version | 07 June 2020, 04:49:33 UTC |
| a8acf52 | Zhang Yunjun | 07 June 2020, 04:19:49 UTC | circleci/config.yml: add apt update (#371) + circleci/config.yml: - add `apt update` to fix the failing test after merging. - merge mintpy/pyaps download and dependencies installation into one run - add installation testing for mintpy via smallbaselineApp.py and pyaps via tropo_pyaps3.py - remove Kuju dataset from example datasets to save time (the corresponding roipac processor is obsolete) + replace `raise SystemExit()` with `sys.exit()` for non-error exit in smallbaselineApp.py and load_data.py; clean up SystemExit() code in other scripts. | 07 June 2020, 04:19:49 UTC |
| b2f49e9 | mehdizadehm | 06 June 2020, 21:08:43 UTC | circle CI: build_and_test workflow (#370) * Add build_and_test workflow to circle CI upon the creation of a pull request and commit to the default branch, with the following 2 jobs: - install mintpy and pyaps on normal Linux machine - test mintpy on the 3 example datasets using test_smallbaselineApp.py * Update PULL_REQUEST_TEMPLATE.md | 06 June 2020, 21:08:43 UTC |
| b74768b | ranneylxr | 06 June 2020, 01:44:53 UTC | tropo_pyaps3: support Sentinel-1 SAFE filename as a list text file (#367) + support text file with Sentinel-1 SAFE filenames (generated by stackSentinel.py from ISCE/topsStack) for the --date-list option, so that GAM data downloading can start before the tropospheric delay correction, to save time. + add --verbose option to printout detailed message from pyaps3. Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> | 06 June 2020, 01:44:53 UTC |
| 7067cf1 | Zhang Yunjun | 05 June 2020, 07:52:50 UTC | turn OFF parallel computing by default + set mintpy.compute.cluster = auto to no by default + move template files from /test to /test/configs | 05 June 2020, 20:27:16 UTC |
| f1d5ccf | Zhang Yunjun | 04 June 2020, 21:09:44 UTC | install: use conda init bash 1. use conda init bash to replace the PYTHON3DIR and PATH setup for conda. 2. remove obsolete notes on dask and vim | 05 June 2020, 20:27:16 UTC |
| 4c6cd00 | Falk Amelung | 05 June 2020, 18:15:33 UTC | remove /tab in mintpy.yaml (#365) Remove the curious /tab causes, which causes problems when running process_rsmas.py (at an import mintpy... command). Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> | 05 June 2020, 18:15:33 UTC |
| f1a763b | Zhang Yunjun | 01 June 2020, 20:36:02 UTC | utils.utils0.enu2los(): bugfix for determining azimuth angle ifg_inv.calc_weight(): simple printout msg Update README.md | 03 June 2020, 22:09:14 UTC |
| 30c067b | Zhang Yunjun | 02 June 2020, 23:06:59 UTC | ifg_inv: calibrate memory usage (#361) * ifg_inv: calibrate memory usage defaults/smallbaselineApp*.cfg: change default memory size from 4GB to 2GB. ifgram_inversion: + add calc_temporal_coherence() to calculate temp_coh chunk-by-chunk to save memory usage, otherwise, this part will give a memory surge + add calc_weight() to convert coherence to weight chunk-by-chunk to save memory usage + turn OFF parallel if numWorker is 1 update docs/dask.md | 02 June 2020, 23:06:59 UTC |
| 34478e7 | Joshua Zahner | 01 June 2020, 19:40:41 UTC | DaskCluster object as a general dask wrapper for mintpy (#357) * objects/cluster.py: created DaskCluster object for generalization of parallel routines - add open() for cluster initiation and scaling - add submit_job() for submitting dask workers to the client scheduler - add collect_result() to recompile subbox results - add run() to wrap around submit_job() and collect_result() to simplify the API. - add close() for closing cluster/client connections and moving out/err files. - add format_config_name() from previous version of cluster.py - add format_num_worker() from ifgram_inversion.py to be more independent - move split_box2sub_boxes() from ifgram_inversion.py - move move_dask_stdout_stderr_files() from utils.utils.py * ifgram_inversion.py: - removed --walltime command-line option and the `mintpy.compute.walltime` template option, as it's included in the YAML file already - add block-by-block writing for all output files, including temporal coherence and number of inverted obs - move weight calculation before obs reading to reduce memory usage due to the internal float64 format while calculating weight. - remove obsolete write_aux2hdf5_file() - add more del in ifgram_inversion to cut memory usage - change default value of weightFunc from 'no' to 'var' to be consistent with smallbaselineApp.py * utils/writefile: add write_hdf5_block() to support block-by-block writing and remote its duplicated version in objects/stack.timeseries() * utils/utils1.py: fix bug for numWorker = all translation in check_template_auto_value() * mintpy/defaults/mintpy.dask: removed pegasus specific configuration details from mintpy.yaml * docs/dask.md: - update doc - update runtime figure for local cluster * docs/module_hierarchy.md: add objects/cluster Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> Co-authored-by: Falk Amelung <falkamelung@users.noreply.github.com> | 01 June 2020, 19:40:41 UTC |
| 906bb14 | ranneylxr | 01 June 2020, 10:07:14 UTC | using x0 + width/y0 + length instead of x1/y1 | 01 June 2020, 16:34:31 UTC |
| cf18f7e | Zhang Yunjun | 31 May 2020, 17:50:58 UTC | Revert "plot_smallbaselineApp.sh: running view.py commands in parallel (#345)" This reverts commit ae4fda1d3afd4978d284b577e7ffc99b43b10754. | 31 May 2020, 18:37:23 UTC |
| 83c0221 | yunjunz | 25 May 2020, 22:38:20 UTC | load_data: fix bug when running with custom template file without offset dataset multilook: use BIL band interleave for binary file with 2 bands tropo_pyaps3: more verbose message when downloading failed save_kmz: support DATE1_DATE2 format input (for time-series file only). utils/isce_utils.extract_geometry_metadata(): extract CENTER_INCIDENCE_ANGLE metadata from los.rdr file utils/utils0.enu2los: add return in the comment | 30 May 2020, 03:03:45 UTC |
| 230fc1c | Joshua Zahner | 25 May 2020, 21:23:08 UTC | passing dict object as function arguments to avoid parallel_*_patch() (#356) * removed need for parallel_ifgram_inversion_patch() for dask by passing a dictionary of function arguments directly to ifgram_inversion_patch * Updated ifgram_inversion_patch to return box to conform to dask requirements * Removed paralell_ifgram_inversion_patch * roll back to the explicit argument of ifg_inv_patch() Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> | 25 May 2020, 21:23:08 UTC |
| 78048fc | Zhang Yunjun | 25 May 2020, 18:29:29 UTC | add mintpy.compute.* options + rename all dask and memory related options with prefix mintpy.compute., to prepare for parallel support for multiple steps. + turn ON parallel with local cluster as default, because 1) it works on laptop, local cluster and HPC cluster, therefore pretty generic; 2) in all the 3 scenarios, parallel is faster than non-parallel. In macOS case, it reduces the time for Fernandina example from 16 mins to 9 mins. + remove the redundant --parallel option, and use --cluster no to turn OFF parallel instead, following Falk's suggestion. | 25 May 2020, 18:43:39 UTC |
| f56ecf1 | Zhang Yunjun | 25 May 2020, 17:42:02 UTC | ifg_inv_patch: simplify the input argument + simplify the input arguments of ifgram_inversion.ifgram_inversion_patch() to take only box and inps. The detailed description of inps content is left for bookkeeping. + move the weight funcations from ifgram_inversion() to simulation/decorrelation.py because this weight is essentially describing the variance of decorrelation noise. + more line breaks in dask.md | 25 May 2020, 18:43:39 UTC |
| d1b9f28 | Zhang Yunjun | 24 May 2020, 00:44:47 UTC | obj/conncomp: use native skimage to remove small objects to replace the previous custom recipe. | 24 May 2020, 00:56:07 UTC |
| bdb6ce6 | Zhang Yunjun | 23 May 2020, 23:52:29 UTC | Update google_earth.md for save_kmz_timeseries.py | 24 May 2020, 00:56:07 UTC |
| d3a1bfc | Zhang Yunjun | 20 May 2020, 04:04:44 UTC | numNonzeroIntClosure: continue w/o break if no triplets found If no triplets found on ifgramStack file, print out warning message instead of raise error and break the process, since this file is independent from the rest steps. Add more print out message. + view.py: simplify the subplot title to support more generic h5 files + a little bit of clean up of dask.md | 20 May 2020, 19:03:16 UTC |
| 550e13d | Joshua Zahner | 20 May 2020, 03:45:09 UTC | Block by block Writing with Dask (#351) * Reorganized ifgram_inversion to support block-by-block writing in parallel following #347. * Adjust split_box2sub_boxes() to support box input that is not started from zero, and to facilitate proper matrix reconstruction after each dask worker finished. * Add more printout message and comments for block-by-block algorithm * Use `memorySize` instead of chunkSize in command line input and in template file. And change the default value to 4GB. * Simplified data package to `parallel_ifgram_inversion_patch` * remove obsolete split_ifgram_file() * support for macports user - docs/ports.txt: add py37-distributed for macports user - objects/cluster: move the dask_jobqueue import into the get_cluster() since jobqueue is not available on macport. Rename check_config_name() to format_config_name() and check_walltime_format() to format_walltime() following Josh's suggestion. + move "numWorker = all" translation from utils to ifgram_inversion.py so that the command line could also use this syntax. Add num worker checking for values on the command line as well, otherwise, it will printout ERROR message. + remove --fast option since it's redundant and almost never used. + utils.move_dask_std*(): do not create directory if no .o/.e file found, which is the case for local cluster. + change the hardwired default value for numWorker to 4 since local is now the most frequently used one Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> | 20 May 2020, 03:45:09 UTC |
| aa0b564 | Joshua Zahner | 19 May 2020, 22:51:33 UTC | Dask LocalCluster Documentation Update (#349) * Updated dask documentation file with information about LocalCluster implementation * Run several datasets with different size and different number of cores to evaluate the performance of dask local cluster. Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> | 19 May 2020, 22:51:33 UTC |
| cac0fd4 | Zhang Yunjun | 15 May 2020, 00:59:41 UTC | readfile: support generic defined h5 with 3D matrix + more grouped help message for geocode.py + more help msg for asc_desc2horz_vert.py | 15 May 2020, 04:21:18 UTC |
| ae4fda1 | Falk Amelung | 12 May 2020, 21:37:25 UTC | plot_smallbaselineApp.sh: running view.py commands in parallel (#345) Add an '&' to all view.py commands and a 'wait' after the last one, so that they all run in parallel. On stampede this reduces the runtime from 2 minutes to 10 seconds. On systems that have less cores than commands it is supposed to run as much commands in parallel as cores are available. This worked for me on linux (16 core machine, pegasus login node) and macOS (6 cores). | 12 May 2020, 21:37:25 UTC |
| 5fdc589 | Zhang Yunjun | 11 May 2020, 04:41:30 UTC | utils1: bugfix for translating numWorker = all + use dict.get() to set default value if option/key is missing in the template file, which return an error in smallbaselineApp.py --dostep mode. + unwrap_error_phase_closure(): adjust plot_* to allow notebook figure display + adjust template comments fort shorter width. | 12 May 2020, 05:23:38 UTC |
| 597735b | Zhang Yunjun | 09 May 2020, 05:20:27 UTC | clean up insarmaps/dev scripts add `mintpy/insarmaps` folder and move the following script into it: + add_attribute_insarmaps.py + hdfeos5_2insarmaps.py + hdfeos5_2json_mbtiles.py + inarmaps_query.py + json_mbtiles2insarmaps.py adjust add_attribute_insarmaps module import add `mintpy/legacy/gui` folder and move all the gui related script into it. move the following two scripts into dev as they are not mintpy directly related. + process_isce_stack.py + split_jobs.py | 10 May 2020, 19:43:41 UTC |
| d1b814b | Joshua Zahner | 10 May 2020, 04:54:49 UTC | add Dask local cluster type (#339) * Added support for dask local cluster, allocating same number of workers as cpu cores. * Commented new LocalCluster code * Added logic for changing default number of workers based on cluster type, as well as allocating all available compute resources with special value * Updated default config values and documentation | 10 May 2020, 04:54:49 UTC |
| 61b8bda | Zhang Yunjun | 09 May 2020, 04:59:32 UTC | replace xml.etree with defusedxml for ElementTree use `defusedxml.ElementTree` module to replace the vulnerable `xml.etree.ElementTree` module for XML file parsing. This was used in utils.readfile.read_isce_xml() and dem_gsi.py, both of them are now either be replaced or unused. add defusedxml to dependency docs | 09 May 2020, 22:57:37 UTC |
| 1396502 | Zhang Yunjun | 09 May 2020, 00:46:56 UTC | fix xml warning from codacy | 09 May 2020, 01:39:24 UTC |
| 66f53bf | Zhang Yunjun | 08 May 2020, 17:59:00 UTC | move tropo_gacos & 2to2_ts to legacy + move tropo_gacos.py to legacy because its python2 syntax is not compatiable with current python3 setting and this script is not maintained. + move 2to3_ts to legacy becuase it's not used (not I am aware of) and not maintained. | 09 May 2020, 01:39:24 UTC |
| 559d630 | yunjunz | 08 May 2020, 05:44:35 UTC | prep_fringe: change default output directory "mintpy" inside "fringe" working directory add more example commands after prep_fringe.py bugfix in view.py for --lalo-label option | 09 May 2020, 01:39:24 UTC |
| cba20d0 | Zhang Yunjun | 08 May 2020, 05:37:31 UTC | ref_point: change default to update data value stead of writing few files | 09 May 2020, 01:39:24 UTC |
| 2520586 | Zhang Yunjun | 08 May 2020, 04:50:02 UTC | isce_utils: move isce import from the top into each function so that isce is not imported unless it has to. remove unnecessary gdal import | 09 May 2020, 01:39:24 UTC |
| 897960c | Zhang Yunjun | 08 May 2020, 04:16:27 UTC | refactoring prep_fringe.py move layout_hdf5() from prep_aria.py to utils/writefile.py for easy sharing among different scripts. utils/isce_utils: + read_baseline_timeseries(): support changing reference date with `ref_date` + extract_geometry_metadata(): add fbase_list and fext_list to more flexible searching capability + extract_geometry_metadata(): add `box` argument for accurate metadata extraction prep_fringe: + add read_vrt_info() to support --geom-dir option so that it can be used to grab both the geometry directory and the bounding box + move metadata prepare code into a new function prepare_metadata(); use pythonic function call to replace the os.system() call; update metadata based on the input bounding box + remove layout_hdf5() as now it can be called from utils.writefile + move all time-series related code into prepare_timeseries() + for geometry and temporal coherence, since their structure is simple and consistent among different datasets, use simple writefile.write() to replace layout_hdf5() and write_*(). + add more exmaple cmd usage following the attachment in https://github.com/insarlab/MintPy/pull/327 | 09 May 2020, 01:39:24 UTC |
| 94ef04a | Zhang Yunjun | 07 May 2020, 20:23:14 UTC | README: add mintpy module usage github page: replace button for wiki with the one for readthedocs | 09 May 2020, 01:39:24 UTC |
| 11c9a38 | ehavazli | 07 May 2020, 04:48:24 UTC | ts2vel: check bootstrap-count (#337) check bootstrap-count number as it should be larger than 1. If so, turn the bootstrap OFF and continue. | 07 May 2020, 04:48:24 UTC |
| 44e18d6 | ehavazli | 07 May 2020, 01:48:08 UTC | move bootstrapping to timeseries2velocity (#335) * add bootstrapping to timeseries2velocity: + add run_velocity_estimation() to include bootstrapping, together with new --bootstrap and --bootstrap-count options for the estimation of mean velocity and its standard deviation. + add estiamte_velocity() for pure mathematic velocity estimation, so that 1) it can be added later to include more complex temporal model and 2) the future added models can also apply to the bootstrapping. + add reference and update help message in the cmd parser. * support bootstrapping in the routine workflow by adding two new template options in the default template file and updated timeseries2velocity accordingly. + mintpy.velocity.bootstrap + mintpy.velocity.bootstrapCount * removed old boot_strap.py Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> | 07 May 2020, 01:48:08 UTC |
| 815f7e3 | Zhang Yunjun | 06 May 2020, 18:20:19 UTC | geo input dset: use center inc angle / range distance if missing in input template (#334) * geo input dset: use center inc angle / range distance if missing in input template + objects/stackDict: for dataset in geo-coordinates, use the INCIDENCE_ANGLE and SLANT_RANGE_DISTANCE from metadata to create the constant 2D matrix to write to geometry files if those two datasets are missing. + utils/utils0: add incidence_angle2slant_range_distance() based on the law of sines + prep_snap: - add INCIDENCE_ANGLE and SLANT_RANGE_DISTANCE metadata - rename utm_dim_to_rsc() to extract_snap_metadata() + utils/isce_utils: suppress matplotlib DEBUG msg + prep_aria: use wildcard in the example input | 06 May 2020, 18:20:19 UTC |
| 8f5a87f | Zhang Yunjun | 03 May 2020, 06:20:50 UTC | save_qgis: add bbox options + call ut.get_lat_lon() to get the pixel-wised lat/lon, instead of reading h5 directly, to fix the bug when input dataset is geocoded already and does not contain lat/lon h5 dataset, such as ARIA products. + add --bbox and --geo-bbox option to output subset into shape file, together with read_bounding_box() + use ptime.progressBar() to replace the gdal.TermProgress_nocb() for more information + move the funcationality code from main() to write_shape_file() + utils/utils0.get_lat_lon(): add geom_file optional argument | 03 May 2020, 06:34:38 UTC |
| 964fe29 | Zhang Yunjun | 02 May 2020, 22:17:47 UTC | readfile: add read_gdal_vrt() + add read_gdal_vrt() to grab metadata from gdal support dataset. + more generic metadata file searching in read_attribut() including .vrt files + update issue templates | 03 May 2020, 06:34:38 UTC |
| 2eb55e6 | Zhang Yunjun | 30 April 2020, 04:33:39 UTC | view: fix wrong auto col_num when row_num==1 | 03 May 2020, 06:34:38 UTC |
| 9f3d143 | Zhang Yunjun | 01 May 2020, 06:01:50 UTC | dem_gsi: use gdal2isce_xml.py directly without specifying dir to source | 01 May 2020, 19:11:21 UTC |
| fd2f0f8 | Zhang Yunjun | 01 May 2020, 04:39:01 UTC | add data_structure.md + create docs/api/data_structure.md for the HDF5 file structure used in the code and add link to readthedocs.io + obj*/stack*: remove duplicated file structure definitions in the code | 01 May 2020, 19:11:21 UTC |
| a9f9f39 | Zhang Yunjun | 01 May 2020, 04:33:19 UTC | ds name: iono to ionoPhase | 01 May 2020, 19:11:21 UTC |
| d83977e | Zhang Yunjun | 01 May 2020, 02:13:48 UTC | add utils/isce_utils.py + move all ISCE related utility functions from prep_isce.py to a new sub-module utils/isce_utils.py, for more importable usage. + utils/isce_utils: add the following new functions for multilook number calculation to facilitate ISCE stack processing - get_full_resolution() - resolution2multilook_number() - multilook_number2resolution() + adjust the module import in prep_isce.py and prep_fringe.py | 01 May 2020, 19:11:21 UTC |
| e4238c9 | Zhang Yunjun | 30 April 2020, 04:33:39 UTC | dem_gsi: use $DEMDB/GSI_DEHM10m as default source dir + prep_isce.extract_isce_metadata(): move default output rsc file out of the funtion for more flexibility + utils/readfile.read_binary(): support cpx output | 01 May 2020, 19:11:21 UTC |
| 4c8e9fd | Zhang Yunjun | 28 April 2020, 02:29:46 UTC | Update fallback version number/date in version.py for the new release | 28 April 2020, 02:59:38 UTC |
| 40e64af | Zhang Yunjun | 28 April 2020, 02:26:07 UTC | add QGIS doc from wiki to readthedocs | 28 April 2020, 02:59:38 UTC |
| 2f28021 | Zhang Yunjun | 23 April 2020, 05:08:14 UTC | install: recommend conda env for installation * install: + remove max version constraint in conda_env.yml for dask + use tools instead of python for example directory name + update links for miniconda and vim * README: + update badge version number to 1.2.2 + add link to FRInGE | 28 April 2020, 02:59:38 UTC |
| 90ea2f1 | sssangha | 26 April 2020, 04:07:30 UTC | bugfix in prep_aria (#329) + fix bugs addressed in ARIA-tools issue 147 (aria-tools/ARIA-tools#147). Specifically, prep_aria crashes when one IFG passed (relevant for unit tests). | 26 April 2020, 04:07:30 UTC |
| a0aee9a | Forrest Williams | 26 April 2020, 01:42:47 UTC | prep_fringe: ingest FRInGE outputs into MintPy (#327) * Update prep_isce.py so that it can prepare metadata for non-multilooked files generated by the sentinelStack SLC workflow. if *.rdr geometry files are not found, *.rdr.full geometry files are used instead; if DATE12 contains only one date, the master date from the input metadata file (IW*.xml) is used as the first date * Create prep_fringe.py based on prep_aria.py to convert the output of FRInGE to timeseries.h5 and temporalCoherence.h5 files and creates a geometryRadar.h5 file clipped to the boundary box used by FRInGE. More work could be done, but this is workable. * Explicitly set first time series entry * convert time-series unit from phase to meter and add UNIT metadata Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> | 26 April 2020, 01:42:47 UTC |
| 8d74c24 | ehavazli | 26 April 2020, 01:32:31 UTC | add scikit-learn to the dependency for bootstrapping (#328) * Time bug fix from ARIA products * add new dependency for boot_strap.py - scikit-laern added to dependency * Update ports.txt * revert the time format fix Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> | 26 April 2020, 01:32:31 UTC |
| 5945b9c | Zhang Yunjun | 20 April 2020, 00:22:18 UTC | fix link for example dset in README unw_err_phase_closure: matplotlib backend setting update | 20 April 2020, 23:05:49 UTC |
| 789e844 | Zhang Yunjun | 19 April 2020, 22:25:57 UTC | add numNonzeroIntClosure to routine workflow in quick_overview step + add `quick_overview` to routine workflow: - rename step `stack_interferograms` to `quick_overview` and move it from after correct_unwrap_error to before correct_unwrap_error - call unwrap_error_phase_closure.main() to calculate number of triplets with non-zero integer ambiguity of closure phase to show the distribution of phase unwrapping errors + add import unwrap_error_* in workflow/__init__.py because 1) they are now used multiple times; 2) installing skimage and cvxopt seems to work fine + update integer ambiguity functions in unwrap_error_phase_closure: - output default filename as "numNonzeroIntClosure.h5" - plot the distribution and histogram of the integer ambiguity same as the paper, but: 1) use RdBu_r colormap 2) plot reference point - enable the update mode - add template text from quick_overview step as well + view.py and utils/plot.py: use %(default)s in options help message | 20 April 2020, 00:04:13 UTC |
| 6a5c051 | Zhang Yunjun | 19 April 2020, 05:08:00 UTC | add reference link for scientific colormap | 20 April 2020, 00:04:13 UTC |
| 5588d58 | Zhang Yunjun | 19 April 2020, 03:45:08 UTC | support wildcard dset input in view.py + support input wildcard pattern for dataset input in view.py, for a more powerful search to plot a subset of images + support masking using the connected components from ifgramStack, works only for one interferogram because it read the date12 from the first dataset slice, and will not make sense if one use it while plotting mulitple pairs + rename the following functions to be more intuitive: - check_input_file_info --> read_input_file_info - check_dataset_input --> search_dataset_input + rename --no-glob option to --nosearch option. + update example usage. | 20 April 2020, 00:04:13 UTC |
| c96cdca | Zhang Yunjun | 16 April 2020, 23:32:34 UTC | calc # of non-zero closure phase unwrap_error_phase_closure.py + add --action option to calculate the number of non-zero integer ambiguity of closure phase only + add file writing to calc_num_nonzero_closure_phase() + add hist_num_nonzero_closure_phase() for plotting Fig. 3e in Yunjun et al. (2019). smallbaselineApp.py: update the usage of unwrap_error_phase_closure.py smallbaselineApp.cfg: more detailed section info in the paper for further reading | 20 April 2020, 00:04:13 UTC |
| 992d3c1 | ehavazli | 16 April 2020, 04:12:31 UTC | Bug fix in tsview.py for the coord_unit (#322) * Bug fix: - Output filename option works now * tsview.py bug fix: - Fixed coordiante unit bug reading in tsview.py * Update tsview.py in case Y_UNIT does not exists Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> | 16 April 2020, 04:12:31 UTC |
| b90bba4 | Zhang Yunjun | 15 April 2020, 19:02:02 UTC | add more links to example dir structure (#321) * add more links to example dir structure update ARIA template option as wildcard for geom files Update requirements.txt for the max version of dask align output msg | 15 April 2020, 19:02:02 UTC |
| 866abd1 | ehavazli | 14 April 2020, 20:16:30 UTC | Wildcard option to prep_aria.py and bug fix in spatial_filter.py (#320) * Tool to run boot strapping method to estimate velocity and uncertainty from MintPy time series file - input is time series file in MintPy format - output is bootstrap estimation of velocity and uncertainty in MintPy velocity file format * fixed url bug in GPS data download step * add defo area example figure * Option added: - incidence angle and azimuth angle files can be given as wildcards in the template - updated example to reflect the new option * Bug fix: - Output filename option works now * Update prep_aria.py * Update spatial_filter.py Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> | 14 April 2020, 20:16:30 UTC |
| a78b8d7 | Zhang Yunjun | 12 April 2020, 06:05:36 UTC | support `aria` processor in smallbaselineApp.py (#319) * support `aria` processor in smallbaselineApp.py + smallbaselineApp.py: support 'mintpy.load.processor = aria', this will call prep_aria.py instead of load_data.py to load the data. In this way, users do not need to run prep_aria.py manually, making the ARIA products the same experience with other products. + doc: add example ARIA data/directory structure with example template setup. + prep_aria.py: - add -t/--template option to input template file, together with read_template2inps() - add --update option to be able to skip re-writing if outputs exist and contains all info, together with run_or_skip() - replace -w with -o option to be more consistent with load_data.py - raise an error when gdal.Open() vrt file failed, which happens when the folder is moved and the file path in vrt file becomes invalid. This is fine as long as mintpy h5 files already exist. - add help message for i) example template and ii) updated usage + add docs/examples/input_files/SanFranSenDT42.txt and update the README.md file acoordingly. + more comments of mintpy.load.*File in the default template file. | 12 April 2020, 06:05:36 UTC |
| cca2bb8 | Zhang Yunjun | 03 April 2020, 03:25:38 UTC | bugfix for --scalebar-pad in view.py view: + fix broken option for --scalebar-pad introduced while replacing basemap with cartopy + use eval() to replace PROJECTION_NAME2OBJ to specify the map projection from cartopy for more generic supports with more projections. + update prep_slice() to use cartopy too utils/utils0: add highest_power_of_2() to find the highest power of 2 that is <= the input number more comments in: + objects/resample.py + tropo_pyaps3.py + defaults/smallbaselineApp.cfg with guidelines on reference point test/test_smallbaselineApp: add custom colormap for final velocit pop out | 09 April 2020, 01:08:43 UTC |
| e742fbc | BB | 08 April 2020, 23:53:21 UTC | bug fix for correct data in unwrapping corrections (#317) * bugfix for correct data in unwrapping corrections prior to fixing, mintpy would default to using corrected dataset in timeseries.h5 if unwrapping corrections were turned off after running them, this would not be reflected this obtains the name of the dataset from the input file | 08 April 2020, 23:53:21 UTC |
| 01ad0b2 | Joshua Zahner | 07 April 2020, 18:18:55 UTC | support dask configuration by config_name (#315) * Added support for specifying a dask configuration by config_name * Changed auto value of clusterConfig to 'no' to match other config styles * Implemented simple fallback logic for clusterConfig where the default configuation name is used if a user specified name is not found in the dask.yaml file * Updated dask.yaml with base SLURM configuration variables * Updated installation note with new information about dask * Update installation.md * Update smallbaselineApp.cfg * Update smallbaselineApp_auto.cfg * Update smallbaselineApp.cfg * fig_inv: more --config input checking into cmd_line_parse() * Fixed typo in inps * Removed cast of inps.config to lower in all cases, as config names do not have to be all lowercase * Removed erroneous print statements * Create dask.md * Rename dask.yaml to mintpy.yaml * Update installation.md * Removed case dependency when defining cluster type * Added function to reformat walltime parameter to proper format based on cluster style * Removed extra check for inps.config default value * Update installation.md * Update smallbaselineApp.cfg * Update smallbaselineApp_auto.cfg * Revert "Removed extra check for inps.config default value" This reverts commit 12058a5dcd2c7e00ace0db8e56c718bd1a6d0f4b. * Update cluster.py * Update ifgram_inversion.py * Update mkdocs.yml * Update cluster.py * Update ifgram_inversion.py * add cluster.check_config_name() + move the config_name checking from ifgram_inversion.cmd_line_parse() to objects.cluster.check_config_name() + rename format_walltime_for_cluster() to check_walltime_format() * Updated cluster.py * Removed and tested check in cluster object Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> Co-authored-by: Zhang Yunjun <yunjunz@users.noreply.github.com> | 07 April 2020, 18:18:55 UTC |
| 46da031 | bb | 06 April 2020, 19:57:27 UTC | correct optional argument name | 06 April 2020, 20:09:18 UTC |
| 3cd0b68 | Ovec8hkin | 04 April 2020, 22:47:04 UTC | Updated configuration files with mintpy.networkInversion.cluster option | 04 April 2020, 22:56:51 UTC |
| 5c0deab | Ovec8hkin | 04 April 2020, 22:09:42 UTC | Updated read_template2inps to read cluster parameter from template file | 04 April 2020, 22:56:51 UTC |
| 0b87d4b | Zhang Yunjun | 27 March 2020, 19:01:26 UTC | view: fix bug for referencing on the fly for unwrapPhase* dset + move data & figure link to the end of paper info. | 31 March 2020, 22:12:43 UTC |
| 390b716 | Zhang Yunjun | 27 March 2020, 01:23:15 UTC | bug fix in utils/plot.read_mask() (#312) utils/plot: * auto_adjust_xaxis_date(): support buffer_year=None -> keep the original x-axis range limit * bug fixed in read_mask() with more info regarding data size difference view: * add --title4sen for Sentinel-1. This option display the subplot title for S1A/B acquisition in blue/orange color, and show the IPF number. *check invalid input files | 27 March 2020, 01:23:15 UTC |
| 04392eb | Zhang Yunjun | 25 March 2020, 03:46:22 UTC | diff: support file2 without REF_DATE + check number of file2 inputed, it should be only one for timeseries and ifgramStack + support file2 without REF_DATE + use read_attribute() to get metadata for file1/2 because `timeseries` object will assign a default REF_DATE value even if it's not exsited. This interferences with the handling of REF_DATE as None for file2. + use read() to get data, just to be consistent with metadata handling, no specific reason. tropo_pyaps3.py + print out info on dry/wet/comb delay + return error if input file not exist. | 25 March 2020, 04:44:13 UTC |
| a608a4c | Zhang Yunjun | 23 March 2020, 23:11:02 UTC | tropo_pyaps3: save abs delay by default + save the absolute tropospheric delay by default, without referencing in space and time. - This works because diff.py can handle the different reference in space and time between two input files and this is used in both tropo_pyaps3.py and smallbaselineApp.py - This is more generic, suitable for both phase and offset time-series, the later does not have spatial reference. + more smart way for default output file name: delay file in the same dir as geom file; corrected ts file in the same dir as the input ts file + remove --ref-yx option to simplify the code + remove reference related metadata from delay h5 file. + comment out other models as they are currently not supported in pyaps3, to avoid confusion. + re-organized create_parser() for improved readability + clean up cmd_line_parse(): - check required options --file OR (--date-list and --hour) - ignore invalid h5 file inputs - grab default output files + split check_inputs() into read_inps2date_time() and get_grib_info() + more straight forward main() + more comments objects/stack.timeseris.write2hdf5(): fix a bug when output file directory does not exist. | 25 March 2020, 04:44:13 UTC |
| cac5c00 | Zhang Yunjun | 23 March 2020, 06:18:15 UTC | bug fix in ifgram_inversion.py | 23 March 2020, 06:30:21 UTC |
| e7f42ee | Zhang Yunjun | 23 March 2020, 05:45:40 UTC | refactor cmd_line_parse() and startup() in smallbaselineApp.py + add read_inps2run_steps() to simplify cmd_line_parse() + add get_the_latest_default_template_file() to simplify TimeSeriesAnalysis.startup() + update citation note in README.md + update ifgram_inversion.py following codacy suggestion | 23 March 2020, 06:12:51 UTC |
| 7eb384b | Zhang Yunjun | 22 March 2020, 18:33:50 UTC | writefile.write(): more generic writer without timeseries.write2hdf5() + remove the call of timeseries.write2hdf5() in order to: - merge the writing of time-series file with all the other HDF5 files. - more generic writing to time-series file, i.e. dataset that are not previsouly defined in original time-series object + clean up the info extraction from reference file + checking the required dataset for timeseries and ifgramStack file, which is 'date'. | 23 March 2020, 06:12:51 UTC |
| bd2cf8e | Zhang Yunjun | 22 March 2020, 00:47:14 UTC | tropo_pyaps3: use %(default)s in argparse help msg | 23 March 2020, 06:12:51 UTC |
| 8d0057c | Zhang Yunjun | 21 March 2020, 03:33:09 UTC | plot_smallbaline*: check dataset in h5 file before plotting + call `h5ls` command to check if the dataset exsits in the HDF5 file before calling view.py to plot, for a cleaner message output in plot_smallbaselineApp.sh. + plot phase-unwrapping error correction result if the dataset is available + fix a potetial bug in utils1.run_deramp() if the input mask file dataset name is not 'mask'. | 23 March 2020, 06:12:51 UTC |
| 3e807ec | Zhang Yunjun | 16 March 2020, 06:25:28 UTC | Update tropo_pyaps3.py and README.md | 23 March 2020, 06:12:51 UTC |
