library reorganization

[lbaumo]

I went through the library. I will put a first pass at some changes I think we should make to the global structure in this issue.

1. calibration.py should perform local and global metacal calibration
2. let's move the metacal class out of basic.py into a separate metacal.py and get rid of basic.py completely
3. add an ellipticity.py that switches between e1, e2, and e_t, e_x, as well as between different ellipticity conventions, this will help with handling different input catalogs
4. stats.py will handle all of the stats functions in basic.py
5. should the metacal class interact directly with catalogs? i feel like some of this should go into io.py
6. util.py should be split into binning functions and vos functions
7. survey.py has to do with the footprint, should we combine this with galaxy.py? at any rate, galaxy.py must be renamed. also, the conversion between fwhm and sigma belongs in stats
8. it makes sense to group all cosmology things in one place, but i think there needs to be a further separation of the theory and data functionality.
9. galaxy cluster functions needs to be together
10. catalog.py again is a mix of stats and io, though it does perform a number of initial sanity checks. maybe we should rename it to cat_quick_check or something along those lines?

comments welcome- I'm sure I didn't consider every use case, so there are probably adaptations of the above that will make more sense.

[martinkilbinger]

For point 1.:
At the moment only global calibration is working. Local calibration was implemented by Emma some time ago, this part of the code would need to be revised.

[martinkilbinger]

Since we don't have unit tests yet, I recommend doing an end-to-end run on W3 before a PR with clean-up changes to the code. This will give us some confidence in the changes. I am also happy to do this.

For comparison, previous runs are in candide:/home/mkilbing/astro/Runs/shapepipe/CFIS/v1/W3.

• sp_output (v1 run, sp_validation version 0.3.4)
• sp_output_new (new run on branch ‘fixes’; need to merge with master)

Comparison can consist in checking plots and, more importantly, the output diagnostics text file plots/stats_file.txt.

Required input files:

• params.py
• tiles_W3.txt
• final_cat.npy
• output/run_sp_MsPl/mccd_merge_starcat_runner/output/full_starcat-0000000.fits

Compile and run validation:

• cd /path/to/sp_validation
• conda env create -f environment.yml
• python setup.py install
• cd notebooks
• python config_convert.py
• cd /path/to/W3
• ipython /path/to/sp_validation/notbooks/validation.py

[martinkilbinger]

I can start working on survey.py. Adding functions from galaxy.py except those which should go into stats.py.

[martinkilbinger]

Some functions in the psf_leakage notebook can be replaced by using the shear_psf_leakage class.