After spending most of our "research" time not on AI innovation but on dealing with numerous and ever changing AI engines, their API, and the whole software and hardware stack, we decided to take an alternative approach.

We are developing CK-TensorFlow which is an open-source suite of convenient wrappers and workflows powered by Collective Knowledge with unified JSON API for simple and customized installation/recompilation, usage, evaluation and multi-objective optimisation of various TensorFlow implementations (CPU,CUDA,OpenCL) across diverse platforms from mobile devices and IoT to supercomputers and TPU cloud.

See cKnowledge.org/ai, shared optimization statistics and online demo of CK AI API with self-optimizing DNN for more details.

We need to have easily customizable TensorFlow builds via JSON API to be able to plug it to our framework for collaborative benchmarking and optimization of realistic workloads and models (such as deep learning) across diverse inputs and hardware provided by volunteers (see cKnowledge.org/ai, live repo and papers 1, 2, 3 for more details).

We can now build TensorFlow library and run classification via CK for various Android and Linux platforms. You can even participate in collaborative evaluation and optimization of TF using your Android device (mobile phone, tablet, etc) via this engaging Android app. You can see and use all public results in the Collective Knowledge repository.

• cTuning Foundation
• dividiti

• BSD (3 clause)

• Python 2.7+ or 3.3+
• Collective Knowledge Framework
• Java 8 JDK (though can be automatically installed via CK)
• CUDA/cuDNN if you have CUDA-enabled GPU
• Android NDK if you want to compile and run TF for Android devices

• Python 2.x:

$ sudo apt-get install python-dev python-pip python-setuptools python-opencv git
$ sudo pip install --upgrade pip
$ sudo pip install protobuf easydict joblib image numpy scipy

• Python 3.x:

$ sudo apt-get install python3-dev python3-pip python3-setuptools
$ sudo pip3 install --upgrade pip
$ sudo pip3 install protobuf easydict joblib image wheel numpy scipy

$ pip install --upgrade pip
$ pip install protobuf easydict joblib image numpy scipy

You can find more details about customized TensorFlow builds via CK for Android, Linux, Windows, Raspberry Pi, odroid, etc here.

You can install all basic dependencies and CK as following

$ sudo pip install ck
$ ck pull repo:ck-tensorflow

$ ck install package:lib-tensorflow-1.1.0-cpu
$ ck install package:lib-tensorflow-1.1.0-cuda

 $ ck run program:tensorflow (--env.BATCH_SIZE=10) (--env.NUM_BATCHES=5)

Select one of the test_cpu and test_cuda commands; select an available version of TensorFlow, if prompted (more than one choice); select an available benchmark, if prompted (more than one choice), and select TensorFlow model.

$ ck run program:tensorflow --cmd_key=classify

Note, that you will be asked to select a jpeg image from available CK data sets. We added standard demo images (cat.jpg, catgrey.jpg, fish-bike.jpg, computer_mouse.jpg) to the 'ctuning-datasets-min' repository. You can list them via

 $ ck pull repo:ctuning-datasets-min
 $ ck search dataset --tags=dnn

It is now possible to participate in crowd-benchmarking of TensorFlow (early prototype):

$ ck crowdbench tensorflow --user={your email or ID to acknowledge contributions} (--env.BATCH_SIZE=128 --env.NUM_BATCHES=100)

You can see continuously aggregated results in the public Collective Knowledge repository under 'crowd-benchmark TensorFlow library' scenario.

Note, that this is an on-going, heavily evolving and long-term project to enable collaborative and systematic benchmarking and tuning of realistic workloads across diverse hardware (ARM TechCon'16 talk and demo, DATE'16, CPC'15). We also plan to add crowd-benchmarking and crowd-tuning of Caffe, TensorFlow and other DNN frameworks to our Android application soon - please, stay tuned!

It is now possible to take advantage of our universal multi-objective CK autotuner to optimize TensorFlow. As a first simple example, we added batch size tuning via CK. You can invoke it as following:

$ ck autotune tensorflow

All results will be recorded in the local CK repository and you will be given command lines to plot graphs or replay experiments such as:

$ ck plot graph:{experiment UID}
$ ck replay experiment:{experiment UID} --point={specific optimization point}

We are now working to extend above autotuner and crowdsource optimization of the whole SW/HW/model/data set stack (paper 1, paper 2).

We would like to thank the community for their interest and feedback about this collaborative AI optimization approach powered by CK at ARM TechCon'16 and the Embedded Vision Summit'17 - so please stay tuned ;) !

CK allows us to unify AI interfaces while collaboratively optimizing underneath engines. For example, we added similar support to install, use and evaluate Caffe/Caffe2 via CK:

$ ck pull repo:ck-caffe2
$ ck pull repo --url=https://github.com/dividiti/ck-caffe

$ ck install package:lib-caffe-bvlc-master-cpu-universal --env.CAFFE_BUILD_PYTHON=ON
$ ck install package:lib-caffe2-master-eigen-cpu-universal --env.CAFFE_BUILD_PYTHON=ON

$ ck run program:caffe --cmd_key=classify
$ ck run program:caffe2 --cmd_key=classify

$ ck crowdbench caffe --env.BATCH_SIZE=5
$ ck crowdbench caffe2 --env.BATCH_SIZE=5 --user=i_want_to_ack_my_contribution

$ ck autotune caffe
$ ck autotune caffe2

We provided an option in all our AI crowd-tuning tools to let the community report and share mispredictions (images, correct label and wrong misprediction) to gradually and collaboratively build realistic data/training sets:

• Public repository (see "mispredictions and unexpected behavior)
• Misclassified images via CK-based AI web-service

• Simple demo to classify images with continuous optimization of DNN engines underneath, sharing of mispredictions and creation of a community training set; and to predict compiler optimizations based on program features.

CK-Caffe, CK-Caffe2, CK-Tensorflow are part of an ambitious long-term and community-driven project to enable collaborative and systematic optimization of realistic workloads across diverse hardware in terms of performance, energy usage, accuracy, reliability, hardware price and other costs (ARM TechCon'16 talk and demo, DATE'16, CPC'15).

We are working with the community to unify and crowdsource performance analysis and tuning of various DNN frameworks (or any realistic workload) using Collective Knowledge Technology:

• Android app for DNN crowd-benchmarking and crowd-tuning
• CK-TensorFlow
• CK-Caffe
• CK-Caffe2

We continue gradually exposing various design and optimization choices including full parameterization of existing models.

We use crowd-benchmarking and crowd-tuning of such realistic workloads across diverse hardware for open academic and industrial R&D challenges - join this community effort!

@inproceedings{ck-date16,
    title = {{Collective Knowledge}: towards {R\&D} sustainability},
    author = {Fursin, Grigori and Lokhmotov, Anton and Plowman, Ed},
    booktitle = {Proceedings of the Conference on Design, Automation and Test in Europe (DATE'16)},
    year = {2016},
    month = {March},
    url = {https://www.researchgate.net/publication/304010295_Collective_Knowledge_Towards_RD_Sustainability}
}

@inproceedings{cm:29db2248aba45e59:cd11e3a188574d80,
    url = {https://arxiv.org/abs/1506.06256},
    title = {{Collective Mind, Part II: Towards Performance- and Cost-Aware Software Engineering as a Natural Science.}},
    author = {Fursin, Grigori and Memon, Abdul and Guillon, Christophe and Lokhmotov, Anton},
    booktitle = {{18th International Workshop on Compilers for Parallel Computing (CPC'15)}},
    publisher = {ArXiv},
    year = {2015},
    month = January,
    pdf = {https://arxiv.org/pdf/1506.06256v1}
}

• All references with BibTex related to CK concept

• 2017: We received CGO test of time award for our CGO'07 paper which later motivated creation of Collective Knowledge
• 2015: ARM and the cTuning foundation use CK to accelerate computer engineering: HiPEAC Info'45 page 17, ARM TechCon'16 presentation and demo, public CK repo

Get in touch with CK-AI developers here. Also feel free to engage with our community via this mailing list:

• https://groups.google.com/group/collective-knowledge