MITuna is a distributed tuning infrastructure that provides pre-compiled kernels
for MIOpen customers through automated Jenkins pipelines and SLURM scalable
architecture. MITuna also provides a scalable task management infrastructure
ready to integrate with external libaries. The Example library provides a sample on
how to achieve this.
- Install python3.9
.. code-block::
apt-get update && apt-get install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.9 python3.9-dev python3.9-venv
- Install pip for python3.9
.. code-block::
wget https://bootstrap.pypa.io/get-pip.py -o get-pip.py python3.9 get-pip.py rm get-pip.py
- Install MySQL server
.. code-block::
apt-get install mysql-server mysqld --initialize grep 'temporary password' /var/log/mysql/error.log
- Enable the service
.. code-block::
systemctl start mysql mysql -u root -p mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'root-password'; mysql> CREATE DATABASE <database_name>;
- Install ipmitool
.. code-block::
apt-get install ipmitool
- Setup passwordless ssh between all machines, for example:
.. code-block::
ssh-keygen -t rsa ssh-copy-id @ ssh @
For the tuning cycle, every machine needs to be able to access every other machine through passwordless ssh.
-
Install RabbitMQ
-
Install Redis:
.. code-block::
sudo apt-get install redis
.. note:: Check /etc/redis/redis.conf and allow for remote connections*.
- Clone the repo using
.. code-block::
git clone cd MITuna
- Create a virtual environment, and activate it (by sourcing its
activatescript)
.. code-block::
python3.9 -m venv myvenv source myvenv/bin/activate
- Install the required dependencies:
.. code-block::
python3.9 -m pip install -r requirements.txt
The above assumes that MITuna lives in the home directory and the virtual environment was created using the command indicated above.
- Add the following environment variables to a local file and then source the file:
.. code-block::
export TUNA_DB_USER_NAME=root export TUNA_DB_USER_PASSWORD= export TUNA_DB_HOSTNAME=localhost export TUNA_DB_NAME=<database_name> export TUNA_CELERY_JOB_BATCH_SIZE= #rabbitMQ export TUNA_CELERY_BROKER_HOST=localhost export TUNA_CELERY_BROKER_USER= export TUNA_CELERY_BROKER_PWD= export TUNA_CELERY_BROKER_PORT=5672 #default #redis export TUNA_CELERY_BACKEND_HOST=localhost export TUNA_CELERY_BACKEND_PORT=6379 #default #ipmi export gateway_ip=<gateway_ip> export gateway_port=<gateway_port> export gateway_user=<gateway_user>
All machines used in the tuning process must have ssh-keys enabled. MITuna needs to have all-to-all machine communication available and passwords must not be required at run-time.
- Run the setup scripts:
.. code-block::
python3.9 setup.py develop
- The root tuna folder needs to be appended to the PYTHONAPTH:
.. code-block::
export PYTHONPATH=/<path_to_MITuna>/:$PYTHONPATH
- To create the database run the following script:
.. code-block::
./tuna/miopen/db/build_schema.py
The installation and setup are now complete. To start a tuning cycle, please follow the steps documented in TuningCycle
For use cases requiring logs to be stored, searched and analyzed, Tuna integrates with Logstash and Elastic Search.
This can be done through exporting the logstash destination details to enable the logs storing service.
.. code-block::
export TUNA_LOGSTASH_STATUS=true // this will turn logstash exporting on export TUNA_LOGSTASH_HOST= export TUNA_LOGSTASH_PORT=
MITuna uses yapf for code formatting:
.. code-block::
cd MITuna/ yapf -i --style='{based_on_style: google, indent_width: 2}' --recursive tuna/ tests/ alembic/
In order for a PR to be accepted the following pylint command needs to result in 10/10 analysis:
.. code-block::
cd MITuna/tuna pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' .py miopen/.py example/.py rocmlir/.py utils/.py miopen/celery_tuning/ miopen/utils/.py cd tuna && find miopen/scripts/ -type f -name '.py' | xargs pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' cd tuna && find miopen/driver/ -type f -name '.py' | xargs pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' cd tuna && find miopen/worker/ -type f -name '.py' | xargs pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' cd tuna && pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' miopen/subcmd/import_configs.py cd tuna && pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' miopen/subcmd/import_db.py cd tuna && pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' miopen/subcmd/export_db.py cd tuna && pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' miopen/subcmd/merge_db.py cd tuna && pylint -f parseable --max-args=8 --ignore-imports=no --indent-string=' ' miopen/subcmd/update_golden.py mypy tuna/miopen/utils/config_type.py mypy tuna/connection.py --ignore-missing-imports mypy tuna/abort.py --ignore-missing-imports mypy tuna/miopen/utils/analyze_parse_db.py --ignore-missing-imports mypy tuna/miopen/scripts/build_driver_cmd.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/scripts/corrupt_configs.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/subcmd/import_configs.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/subcmd/load_job.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/subcmd/export_db.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/subcmd/update_golden.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/parse_miopen_args.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/driver/convolution.py --ignore-missing-imports --follow-imports=skip mypy tuna/yaml_parser.py --ignore-missing-imports --follow-imports=skip mypy tuna/flask_example.py --ignore-missing-imports --follow-imports=skip mypy tuna/go_fish.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/driver/batchnorm.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/worker/fin_class.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/worker/fin_eval.py --ignore-missing-imports --follow-imports=skip mypy tuna/miopen/worker/fin_utils.py --ignore-missing-imports --follow-imports=skip mypy tuna/utils/db_utility.py --ignore-missing-imports --follow-imports=skip mypy tuna/worker_interface.py --ignore-missing-imports --follow-imports=skip mypy tuna/grafana_dict.py --ignore-missing-imports --follow-imports=skip mypy tuna/mituna_interface.py --ignore-missing-imports --follow-imports=skip mypy tuna/libraries.py mypy tuna/lib_utils.py --ignore-missing-imports --follow-imports=skip mypy tuna/machine_management_interface.py --ignore-missing-imports --follow-imports=skip yamllint tuna/miopen/yaml_files/.yaml yamllint tuna/example/.yaml mypy tuna/miopen/driver/base.py --ignore-missing-imports --follow-imports=skip mypy tuna/machine.py --ignore-missing-imports --follow-imports=skip mypy tuna/db/session_mixin.py --ignore-missing-imports --follow-imports=skip mypy tuna/db/tuna_tables.py --ignore-missing-imports --follow-imports=skip mypy tuna/parse_args.py --ignore-missing-imports --follow-imports=skip mypy tuna/worker_interface.py --ignore-missing-imports --follow-imports=skip mypy tuna/tables_interface.py --ignore-missing-imports --follow-imports=skip mypy tuna/sql.py --ignore-missing-imports mypy tuna/example/example_lib.py --ignore-missing-imports --follow-imports=skip mypy tuna/example/example_tables.py --ignore-missing-imports --follow-imports=skip mypy tuna/dbBase/sql_alchemy.py --ignore-missing-imports --follow-imports=skip mypy tuna/dbBase/base_class.py --ignore-missing-imports mypy tuna/example/session.py --ignore-missing-imports --follow-imports=skip mypy tuna/example/tables.py --ignore-missing-imports --follow-imports=skip mypy tuna/example/load_job.py --ignore-missing-imports --follow-imports=skip mypy tuna/example/example_worker.py --ignore-missing-imports --follow-imports=skip mypy tuna/rocmlir/import_configs.py --ignore-missing-imports --follow-imports=skip mypy tuna/rocmlir/load_job.py --ignore-missing-imports --follow-imports=skip mypy tuna/rocmlir/rocmlir_lib.py --ignore-missing-imports --follow-imports=skip mypy tuna/rocmlir/rocmlir_tables.py --ignore-missing-imports --follow-imports=skip mypy tuna/rocmlir/rocmlir_worker.py --ignore-missing-imports --follow-imports=skip
MITuna provides general documentation in MITuna/tuna. This folder also contains pointers in
index.rst to project specific documentation such as MIOpen.
Once new documentation is added, it can be generated with:
.. code-block::
cd MITuna/doc
make html
The new generated html files can be found in the MITuna/doc/_build folder.