predmaint-distributed.q - Tensorflow deep learning model for Cox regression - distributed training using Horovod(https://eng.uber.com/horovod)
datasciencebowl2017.q - Data science bowl 2017 on kaggle - https://www.kaggle.com/c/data-science-bowl-2017 - medical images, convolutional NN(3D).
statoil.q - Statoil competition at Kaggle - https://www.kaggle.com/c/statoil-iceberg-classifier-challenge/ - satellite images, convolutional NN(2D).
predictivemaintenance.q - Australian Water Corp. data analysis for predictive maintenance of their water system - https://ministryofdata.org.au/mod-2017-hackathon-problems/wc-pr11-predictive-pumps-pipes-maintenance/ - In progress, building RUL and label columns from unlabelled data.
2sigma_regression.q - Two Sigma Financial Modelling challenge - https://www.kaggle.com/c/two-sigma-financial-modeling - Linear regression.
pm.q - LSTM for predictive maintenance.
np:.p.import`numpy - returns an embedPy object with pointers to all methods inside numpy
npar:.p.import[`numpy;`:array;* or < or >] - returns a embedPy, q or foreign(python) object with the array() method as callable,qcallable, or pycallable -
The callable, pycallable or qcallable is implicit on usage of the [] in the above call, it automatically happens.
np:.p.import `numpy;
npar:.p.callable np`array; / or qcallable or pycallable
or
npar:np[`:array;*];
np:.p.import`numpy;
np[`:array;<;(1000 1000 1000;1000 1000 1000)]
drf:.p.import[`dicom;`:read_file;*] - .p.callable return of method read_file inside dicom
drf <filename> - string file name
pd:.p.import `pandas
pd[`:DataFrame;*][] - calls default constructor and instantiates
or
.p.import[`pandas;`:DataFrame;*][] - calls default constructor and instantiates
df:.p.import[`pandas;`:DataFrame;*] - get class , then instantiate later by df[<appropriate object>]; / instantiation happens here with value passed
q)np:.p.import`numpy
q)v:np[`:arange;*;12]
q)v`
0 1 2 3 4 5 6 7 8 9 10 11
q)v[`:mean;<][]
5.5
q)rs:v[`:reshape;<]
q)rs[3;4]
0 1 2 3
4 5 6 7
8 9 10 11
q)rs[2;6]
0 1 2 3 4 5
6 7 8 9 10 11
q)np[`:arange;*;12][`reshape;*;3;4]`
0 1 2 3
4 5 6 7
8 9 10 11
q)np[`:arange;*;12][`reshape;*;3;4][`T]`
0 1 2
3 4 5
6 7 8
9 10 11
q)stdout:.p.callable(.p.import[`:sys]`:stdout.write)
q)stdout"hello\n";
hello
q)stderr:.p.import[`sys;`:stderr.write;*]
q)stderr"goodbye\n";
goodbye
cph:.p.import `sksurv.linear_model.coxph;
cox:.p.import[`sksurv;`:linear_model;`:coxph;`:CoxPHSurvivalAnalysis;*]
q)print cox
<class 'sksurv.linear_model.coxph.CoxPHSurvivalAnalysis'>
q).p.import `sksurv.nonparametric; {[c;r;x;a]embedPy[c;r;x;a]}[0;0;foreign]enlist
q)km:.p.import[`sksurv;`:nonparametric ;`:kaplan_meier_estimator;*]
q)print km
<function kaplan_meier_estimator at 0x7f775b7d5ae8>
Not allowed - pyplot is a module, so must be loaded as below , since the [] tries to make the innermost import a .callable.
q)plt:.p.import[`matplotlib;`:pyplot;*] -
AttributeError: module 'matplotlib' has no attribute 'pyplot'
plt:.p.import `matplotlib.pyplot;
One would try to instantiate a directly, like so:
mms:.p.import[`sklearn;`:preprocessing;`:MinMaxScaler;*][], expecting an object of class MinMaxScaler to be instantiated.
But this call fails, with the error message : "AttributeError: module 'sklearn' has no attribute 'preprocessing'"
So to use MinMaxScaler, do the following :
pre:.p.import `sklearn.preprocessing;
mms:pre[`:MinMaxScaler;*][]; / now that higher-level module is imported into shared memory already.
Then, mms safely contains : MinMaxScaler(copy=True, feature_range=(0, 1))
If a method returns a foreign and one wants to access the returned object's methods, .p.wrap the foreign and then access methods. In order to bring a python object and access its methods inside q, use an embedPy object, via either .p.import, .p.get or .p.eval, or .p.wrap. An embedPy object is effectively the representation of a python object inside q, with behaviour just like in the python world. Once an embedPy object is created, one can either access attributes, or call methods and get q objects.