Code cleanup: Improve CLI

[ayan-b]

• equal_matrices (tests added)
• make_metadata (tests added)
• merge_xena (tests added)
• gdc_check_new
• gdc2xena

Also set-up initial parser
Related to #16.

See also https://github.com/yunhailuo/xena-GDC-ETL/pull/17#issuecomment-475110147

[ayan-b]

Current argparser setup:
xge xena-eql df1 df2
Let me know if this is good to go, I will then send a commit updating the docs.

[yunhailuo]

Can you remind me about the overall structure/design of the CLI?

[ayan-b]

@yunhailuo, it is discussed here: https://github.com/yunhailuo/xena-GDC-ETL/issues/16#issue-421366023. Our target is to have a single CLI for the whole package instead of for individual modules.

[yunhailuo]

Trying to test this PR but find a potential problem with 4c11c57. Somehow having xena_gdc_etl/scripts/__init__.py prevent me from installing using pip install ./ under a simple conda python 2.7 environment. Part of the error log:

  running build_scripts
  creating build/scripts-2.7
  copying and adjusting xena_gdc_etl/scripts/make_metadata.py -> build/scripts-2.7
  copying and adjusting xena_gdc_etl/scripts/gdc2xena.py -> build/scripts-2.7
  copying and adjusting xena_gdc_etl/scripts/gdc_check_new.py -> build/scripts-2.7
  copying xena_gdc_etl/scripts/equal_matrices.py -> build/scripts-2.7
  copying and adjusting xena_gdc_etl/scripts/TARGET-CCSK_phenotype_ETL.py -> build/scripts-2.7
  copying and adjusting xena_gdc_etl/scripts/panTCGA.py -> build/scripts-2.7
  warning: build_scripts: xena_gdc_etl/scripts/__init__.py is an empty file (skipping)
  
  copying and adjusting xena_gdc_etl/scripts/merge_xena.py -> build/scripts-2.7
  copying xena_gdc_etl/scripts/join_xena.sh -> build/scripts-2.7
  copying xena_gdc_etl/scripts/union_xena.sh -> build/scripts-2.7
  changing mode of build/scripts-2.7/make_metadata.py from 644 to 755
  changing mode of build/scripts-2.7/gdc2xena.py from 644 to 755
  changing mode of build/scripts-2.7/gdc_check_new.py from 644 to 755
  changing mode of build/scripts-2.7/TARGET-CCSK_phenotype_ETL.py from 644 to 755
  changing mode of build/scripts-2.7/panTCGA.py from 644 to 755
  error: [Errno 2] No such file or directory: 'build/scripts-2.7/__init__.py'
  
  ----------------------------------------
  Failed building wheel for xena-gdc-etl
  Running setup.py clean for xena-gdc-etl
Failed to build xena-gdc-etl

I know travis tests all pass. I'm wondering:

1. Do we test the install of the repo in travis?
2. Do you have any problems pip install?

[ayan-b]

During development, I use python setup.py develop in order to install the package. So, I have never faced the issue, thanks for pointing out.

1. Do we test the install of the repo in travis?

No, I will add a commit addressing this.
[UPD: Done in 3dfd603]

2. Do you have any problems pip install?

Yes, it turns out that the problem lies in the name of directory scripts. If we change the directory name to something else (say xena_scripts), pip install works as expected. Let me know if you have a suitable name for the directory in your mind.

[yunhailuo]

If we change the directory name to something else (say xena_scripts), pip install works as expected.

I think this is defined in setup.py. They are scripts, not modules. So it shouldn't have__init__.py.

However, I do know we are going to have command line and get rid of scripts for good eventually.

Let me know when the PR is ready.

[ayan-b]

Ok, I will get rid of __init__py. However this creates a problem, it seems pytest can only discover and test modules, not individual scripts.

[ayan-b]

@yunhailuo This PR is ready except for gdc2xena module. It is a bit large and requires proper test cases before moving into utils.

[ayan-b]

@yunhailuo, please review.

[yunhailuo]

Looks good. Thank you!

[yunhailuo]

If possible, it will be better to have commits group together. Or separate features into different PRs. @ayan-b