CB-Tumblebug (CB-TB for short) is a system for managing multi-cloud infrastructure consisting of resources from multiple cloud service providers. (Cloud-Barista)

• CB-Tumblebug Overview

• CB-Tumblebug Features

• CB-Tumblebug Architecture

• Hot usecase of CB-Tumblebug

  • Deploy a Multi-Cloud Infra with GPUs and Enjoy muiltple LLMs in parallel (YouTube)
  • LLM-related scripts

CB-TB is not v1.0 yet.
We welcome any new suggestions, issues, opinions, and contributors!
Please note that the functionalities of Cloud-Barista are not stable and secure yet.
Be careful if you plan to use the current release in production.
If you have any difficulties in using Cloud-Barista, please let us know.
(Open an issue or join the Cloud-Barista Slack)

As an open-source project initiated by Korean members, 
we would like to promote participation of Korean contributors during the initial stage of this project. 
So, the CB-TB repo will accept the use of the Korean language in its early stages.
However, we hope this project flourishes regardless of the contributor's country eventually.
So, the maintainers recommend using English at least for the titles of Issues, Pull Requests, and Commits, 
while the CB-TB repo accommodates local languages in their contents.

1. Prerequisites
2. How to Run
3. How to Use
4. How to Build
5. How to Contribute

• Linux (recommend: Ubuntu 22.04)
• Docker and Docker Compose
• Golang (recommend: v1.21.6) to build the source

Open source packages used in this project

• Dependencies
• SBOM

• Clone the CB-Tumblebug repository:

  git clone https://github.com/cloud-barista/cb-tumblebug.git $HOME/go/src/github.com/cloud-barista/cb-tumblebug
  cd ~/go/src/github.com/cloud-barista/cb-tumblebug

  Optionally, you can register aliases for the CB-Tumblebug directory to simplify navigation:

  echo "alias cdtb='cd $HOME/go/src/github.com/cloud-barista/cb-tumblebug'" >> ~/.bashrc
  echo "alias cdtbsrc='cd $HOME/go/src/github.com/cloud-barista/cb-tumblebug/src'" >> ~/.bashrc
  echo "alias cdtbtest='cd $HOME/go/src/github.com/cloud-barista/cb-tumblebug/src/testclient/scripts'" >> ~/.bashrc
  source ~/.bashrc

• Check Docker Compose Installation:

  Ensure that Docker Engine and Docker Compose are installed on your system. If not, you can use the following script to install them (note: this script is not intended for production environments):

  cd ~/go/src/github.com/cloud-barista/cb-tumblebug
  ./scripts/installDocker.sh

• Start All Components Using Docker Compose:

  To run all components, use the following command:

  cd ~/go/src/github.com/cloud-barista/cb-tumblebug
  sudo docker compose up

  This command will start all components as defined in the preconfigured docker-compose.yaml file. For configuration customization, please refer to the guide.

  The following components will be started:

  • ETCD: CB-Tumblebug KeyValue DB
  • CB-Spider: a Cloud API controller
  • CB-MapUI: a simple Map-based GUI web server
  • CB-Tumblebug: the system with API server

  

  After running the command, you should see output similar to the following:

  Now, the CB-Tumblebug API server is accessible at: http:https://localhost:1323/tumblebug/api Additionally, CB-MapUI is accessible at: http:https://localhost:1324

  Note: Before using CB-Tumblebug, you need to initialize it.

To provisioning multi-cloud infrastructures with CB-TB, it is necessary to register the connection information (credentials) for clouds, as well as commonly used images and specifications.

• Create credentials.yaml file and input your cloud credentials

  • Overview

    • credentials.yaml is a file that includes multiple credentials to use API of Clouds supported by CB-TB (AWS, GCP, AZURE, ALIBABA, etc.)
    • It should be located in the ~/.cloud-barista/ directory and securely managed.
    • Refer to the template.credentials.yaml for the template

  • Create credentials.yaml the file

    Automatically generate the credentials.yaml file in the ~/.cloud-barista/ directory using the CB-TB script

    cd ~/go/src/github.com/cloud-barista/cb-tumblebug
    ./init/genCredential.sh

  • Input credential data

    Put credential data to ~/.cloud-barista/credentials.yaml (Reference: How to obtain a credential for each CSP)

    ### Cloud credentials for credential holders (default: admin)
    credentialholder:
      admin:
        alibaba:
          # ClientId(ClientId): client ID of the EIAM application
          # Example: app_mkv7rgt4d7i4u7zqtzev2mxxxx
          ClientId:
          # ClientSecret(ClientSecret): client secret of the EIAM application
          # Example: CSEHDcHcrUKHw1CuxkJEHPveWRXBGqVqRsxxxx
          ClientSecret:
        aws:
          # ClientId(aws_access_key_id)
          # ex: AKIASSSSSSSSSSS56DJH
          ClientId:
          # ClientSecret(aws_secret_access_key)
          # ex: jrcy9y0Psejjfeosifj3/yxYcgadklwihjdljMIQ0
          ClientSecret:
        ...
    

• Encrypt credentials.yaml into credentials.yaml.enc

  To protect sensitive information, credentials.yaml is not used directly. Instead, it must be encrypted using encCredential.sh. The encrypted file credentials.yaml.enc is then used by init.py. This approach ensures that sensitive credentials are not stored in plain text.

  • Encrypting Credentials

    init/encCredential.sh

    

  If you need to update your credentials, decrypt the encrypted file using decCredential.sh, make the necessary changes to credentials.yaml, and then re-encrypt it.

  • Decrypting Credentials

    init/decCredential.sh

    

• (INIT) Register all multi-cloud connection information and common resources

  • How to register

    Refer to README.md for init.py, and execute the init.py script. (enter 'y' for confirmation prompts)

    cd ~/go/src/github.com/cloud-barista/cb-tumblebug
    ./init/init.sh

    • The credentials in ~/.cloud-barista/credentials.yaml.enc (encrypted file from the credentials.yaml) will be automatically registered (all CSP and region information recorded in cloudinfo.yaml will be automatically registered in the system)
      • Note: You can check the latest regions and zones of CSP using update-cloudinfo.py and review the file for updates. (contributions to updates are welcome)
    • Common images and specifications recorded in the cloudimage.csv and cloudspec.csv files in the assets directory will be automatically registered.

• Shutting down CB-TB and related components

  • Stop all containers by ctrl + c or type the command sudo docker compose stop / sudo docker compose down (When a shutdown event occurs to CB-TB, the system will be shutting down gracefully: API requests that can be processed within 10 seconds will be completed)

    

  • In case of cleanup is needed due to internal system errors

    • Check and delete resources created through CB-TB
    • Delete CB-TB & CB-Spider metadata using the provided script

      cd ~/go/src/github.com/cloud-barista/cb-tumblebug
      ./scripts/cleanDB.sh

• Upgrading the CB-TB & CB-Spider versions

  The following cleanup steps are unnecessary if you clearly understand the impact of the upgrade

  • Check and delete resources created through CB-TB
  • Delete CB-TB & CB-Spider metadata

    cd ~/go/src/github.com/cloud-barista/cb-tumblebug
    ./scripts/cleanDB.sh

  • Restart with the upgraded version

1. Using CB-TB MapUI (recommended)
2. Using CB-TB REST API (recommended)
3. Using CB-TB Test Scripts

• With CB-MapUI, you can create, view, and control Mutli-Cloud infra.
  • CB-MapUI is a project to visualize the deployment of MCIS in a map GUI.
  • Run the CB-MapUI container using the CB-TB script

    cd ~/go/src/github.com/cloud-barista/cb-tumblebug
    ./scripts/runMapUI.sh

  • Access via web browser at http:https://{HostIP}:1324

• Access to REST API dashboard

  • http:https://[IP]:1323/tumblebug/api

    • Upsteam online API document:

  • REST API AUTH

    CB-TB API is encoded with basic access authentication by default. (not fully secured yet!)

    You need to encode the Username and Password entered during server startup in Base64 and include it in the API header.

• A guide to quickly create a Multi-Cloud Infra

• Using individual APIs

  • Create resources required for VM provisioning by using MCIR(multi-cloud infrastructure resources) management APIs
    • Create VM spec object
    • Create VM image object
    • Create network object
    • Create security group object
    • Create access key object
  • Create, view, control, execute remote commands, shut down, and delete MCIS using the MCIS(multi-cloud infrastructure service) management APIs
    • Create MCIS
    • MCIS remote command
    • View and control MCIS
    • Terminate and Delete MCIS
  • CB-TB optimal and dynamic provisioning
    • CB-TB optimal and dynamic provisioning

src/testclient/scripts/ provides Bash shell-based scripts that simplify and automate the MCIS (MC-Infra) provisioning procedures, which require complex steps.

• Deploy Xonotic game servers on MCIS

• Install Weave Scope cluster on MCIS

• Install Jitsi video conferencing on MCIS

• Automatically configure Ansible execution environment on MCIS

• Setup required tools

  • Install: git, gcc, make

    sudo apt update
    sudo apt install make gcc git

  • Install: Golang

    • Check https://golang.org/dl/ and setup Go

      • Download

        wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz;
        sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz

      • Setup environment

        echo 'export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin' >> ~/.bashrc
        echo 'export GOPATH=$HOME/go' >> ~/.bashrc

        source ~/.bashrc
        echo $GOPATH
        go env
        go version

• Run Docker Compose with the build option

  To build the current CB-Tumblebug source code into a container image and run it along with the other containers, use the following command:

  cd ~/go/src/github.com/cloud-barista/cb-tumblebug
  sudo docker compose up --build

  This command will automatically build the CB-Tumblebug from the local source code and start it within a Docker container, along with any other necessary services as defined in the docker-compose.yml file.

• Build the Golang source code using the Makefile

  cd ~/go/src/github.com/cloud-barista/cb-tumblebug/src
  make

  All dependencies will be downloaded automatically by Go.

  The initial build will take some time, but subsequent builds will be faster by the Go build cache.

  Note To update the Swagger API documentation, run make swag

  • API documentation file will be generated at cb-tumblebug/src/api/rest/docs/swagger.yaml
  • API documentation can be viewed in a web browser at http:https://localhost:1323/tumblebug/api (provided when CB-TB is running)
  • Detailed information on how to update the API

• Set environment variables required to run CB-TB (in another tab)

  • Check and configure the contents of cb-tumblebug/conf/setup.env (CB-TB environment variables, modify as needed)
    • Apply the environment variables to the system

      cd ~/go/src/github.com/cloud-barista/cb-tumblebug
      source conf/setup.env

    • (Optional) Automatically set the TB_SELF_ENDPOINT environment variable (an externally accessible address) using a script if needed
      • This is necessary if you want to access and control the Swagger API Dashboard from outside when CB-TB is running

      cd ~/go/src/github.com/cloud-barista/cb-tumblebug
      source ./scripts/setPublicIP.sh

• Execute the built cb-tumblebug binary by using make run

  cd ~/go/src/github.com/cloud-barista/cb-tumblebug/src
  make run

CB-TB welcomes improvements from both new and experienced contributors!

Check out CONTRIBUTING.

Thanks goes to these wonderful people (emoji key):

Seokho Son 🚧 🤔 💻 👀	Jihoon Seo 🚧 🤔 💻 👀	Yunkon Kim 🤔 💻 👀 🚧	jmleefree 💻 👀	ByoungSeob Kim 🤔	Sooyoung Kim 🐛 🤔	KANG DONG JAE 🤔
Youngwoo-Jung 🤔	Sean Oh 🤔	MZC-CSC 🐛 🤔	Eunsang 📓	hyokyungk 📓	pjini 📓	sunmi 📓
sglim 📖 💻	jangh-lee 📖 💻	이도훈 📖 💻	Park Beomsu 💻	Hassan Alsamahi 💻	Taegeon An 💻	INHYO 💻
Modney 📖 💻	Seongbin Bernie Cho 💻 📖	Gibaek Nam 💻	Abidin Durdu 💻	soyeon Park 💻	Jayita Pramanik 📖	Mukul Kolpe 📖
EmmanuelMarianMat 💻	Carlos Felix 💻	Stuart Gilbert 💻	Ketan Deshmukh 💻	Tríona Barrow 💻	BamButz 💻	dogfootman 📓
Okhee Lee 📓	joowon 📓	Sanghong Kim 💻	Rohit Rajput 💻	Arshad 💻	Jongwoo Han 💻	Yoo Jae-Sung 📓
Minhyeok LEE 👀