Remove columns having list datatype

[ayan-b]

Closes #64.

[yunhailuo]

Commented in slack. Remove it before making dataframe: https://github.com/ucscXena/xena-GDC-ETL/blob/master/xena_gdc_etl/gdc.py#L431-L434. Checking for "[" is hacky and not right.

[yunhailuo]

As a matter of fact, there could be an easier solution depending on the data:

>>> from pandas.io.json import json_normalize
>>> data = [{'state': 'Florida',
...          'shortname': 'FL',
...          'info': {
...               'governor': 'Rick Scott'
...          },
...          'counties': [{'name': 'Dade', 'population': 12345},
...                      {'name': 'Broward', 'population': 40000},
...                      {'name': 'Palm Beach', 'population': 60000}]},
...         {'state': 'Ohio',
...          'shortname': 'OH',
...          'info': {
...               'governor': 'John Kasich'
...          },
...          'counties': [{'name': 'Summit', 'population': 1234},
...                       {'name': 'Cuyahoga', 'population': 1337}]}]
>>> result = json_normalize(data)
>>> result
                                            counties info.governor shortname    state
0  [{'name': 'Dade', 'population': 12345}, {'name...    Rick Scott        FL  Florida
1  [{'name': 'Summit', 'population': 1234}, {'nam...   John Kasich        OH     Ohio
>>> result.applymap(type)
         counties  info.governor      shortname          state
0  <class 'list'>  <class 'str'>  <class 'str'>  <class 'str'>
1  <class 'list'>  <class 'str'>  <class 'str'>  <class 'str'>

You would skip reduce_json_array since it's bad in messing up the data. You do need to process the JSON properly in regarding to samples. In the end, you can drop the column having list values.

[yunhailuo]

reduce_json_array will try to expand length 1 array: https://github.com/ucscXena/xena-GDC-ETL/blob/master/xena_gdc_etl/utils.py#L175 This might be a bad behavior. A field would be removed as long as it's datatype is list unless we decide differently. It shouldn't be used here. Honestly, reduce_json_array is probably a bad function in the first place.

[yunhailuo]

Those fields can be dropped in JSON and don't have to wait until here. Dropping something as early as possible would save extra computation.