Skip to content

This is fully Kubernetes-native testbed environment. Please contribute for add additional OSS (Vitess, NATS, etc) or microservices.

License

Notifications You must be signed in to change notification settings

niuton/kubernetes-native-testbed

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Kubernetes-native testbed for the future

This is fully Kubernetes-native testbed environment. Please contribute for add additional OSS (Vitess, NATS, etc) or microservices.

Kubernetes-native testbed

Note: This repository is still alpha release, and we focus on:

  • BETA-1 at JUNE 2020 (Kubernetes Community Day Tokyo)
  • BETA-2 at SEP 2020 (CloudNative Days Tokyo 2020, Kubernetes Forum Tokyo)
  • GA at KubeCon + CNCon NA 2020

For Cloud Providers, we are looking for a sponsor to provide cloud environment. If you can help, please email us ([email protected]).

Authors

ToC

Microservices

microservice datastore
cart TiKV
comment MongoDB
delivery-status Cassandra, NATS
order TiDB
payment-info PostgreSQL
point YugabyteDB, Kafka, Memcached(MCRouter)
product MySQL
rate Redis(Centinel)
search Elasticsearch
user MySQL
admin -

Kubernetes-native testbed architecture

CI/CD

You can try to commit to your own repo for CI/CD.

Kubernetes-native testbed CI/CD

OSS

Name Genre Version CNCF Project URL
Vitess Relational Database v0.0.0 planetscale/vitess-operator
MySQL Relational Database v0.0.0 presslabs/mysql-operator
MySQL Relational Database v0.0.0 oracle/mysql-operator
YugabyteDB Relational Database v0.0.0 rook-yugabytedb
PostgreSQL Relational Database v0.0.0 zalando/postgres-operator
TiDB Relational Database v0.0.0 pingcap/tidb-operator
TiKV Key Value Store v0.0.0 pingcap/tidb-operator
Memcached Key Value Store v0.0.0 geerlingguy/mcrouter-operator
Redis Key Value Store v0.0.0 spotahome/redis-operator
Apache Cassandra NoSQL (RDB) v0.0.0 Orange-OpenSource/casskop
MongoDB NoSQL (Document DB) v0.0.0 kubedb/operator
NATS Message Queue v0.0.0 nats-io/nats-operator
Apache Kafka Message Queue v0.0.0 ○(*1) strimzi/strimzi-kafka-operator
MinIO Object Storage v0.0.0 minio/minio-operator
Ceph Block Storage v0.0.0 rook-ceph
Rook Block Storage v0.0.0 rook/rook
Nginx Ingress Controller v0.0.0 kubernetes/ingress-nginx
Envoy L7 LoadBalancer v0.0.0 projectcontour/contour
Harbor Container Registry v0.0.0 goharbor/harbor , goharbor/harbor-operator
Kaniko Container Build Tool v0.0.0 GoogleContainerTools/kaniko
Tekton Triggers CI v0.0.0 tektoncd/triggers
Tekton Pipelines CI v0.0.0 tektoncd/pipeline
ArgoCD CD v0.0.0 argoproj/argo-cd
Prometheus Monitoring v0.0.0 coreos/prometheus-operator
Weave Scope Monitoring v0.0.0 weaveworks/scope
Kubernetes Dashboard Monitoring v2.0.0 kubernetes/dashboard
Loki Logging v0.0.0 grafana/loki
Telepresence Local Development v0.0.0 telepresenceio/telepresence
Kustomize Manifest Management v0.0.0 kubernetes-sigs/kustomize
conftest Manifest Management v0.0.0 instrumenta/conftest
gRPC Microservice Interconnect v0.0.0 grpc/grpc-go
gRPC-web Microservice Interconnect v0.0.0 grpc/grpc-web
Elasticsearch/Kibana EFKStack v0.0.0 elastic/cloud-on-k8s
Clair Security v0.0.0 quay/clair
(TBR): OPA Gatekeeper Security v0.0.0 open-policy-agent/gatekeeper
(TBR): Knative Serverless v0.0.0 knative/serving, knative/eventing

*1: strictly speaking, strimzi is just a member of CNCF project, not Apache Kafka.

How to use

Pre-requirements

  • kubectl ~1.17.x
  • You need over 40 cores+ compute resources

Fork repo to your org, and clone it

from https://github.com/kubernetes-native-testbed/kubernetes-native-testbed

Setup cloud settings

This testbed use your cloud settings, so please check your settings.

# at GCP
$ gcloud config list
[compute]
region = asia-northeast1

[core]
account = [email protected]
project = GCP_PROJECT

Set environment variable

PLEASE CREATE A NEW DEDICATED GITHUB USER FOR ONLY THIS TESTBED REPO. NOW TOKEN IS WEAK

export TB_GITHUB_USER=XXXXX
export TB_GITHUB_TOKEN=XXXXX

Install Kubectl plugin "Krew"

Please install Krew which is kubectl plugins manager.

# for macOS / Linux

(
  set -x; cd "$(mktemp -d)" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew.{tar.gz,yaml}" &&
  tar zxvf krew.tar.gz &&
  KREW=./krew-"$(uname | tr '[:upper:]' '[:lower:]')_amd64" &&
  "$KREW" install --manifest=krew.yaml --archive=krew.tar.gz &&
  "$KREW" update
)

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

For other operation system, please refer to official installation document.

Initialize

Following script do:

  • Download client tools
  • Allocate static IP address
  • Replace placeholder in manifests on this repo, with above IP address
./tools/init.sh

Add webhook settings for forked repo

from https://github.com/__TB_GITHUB_ORG_NAME__/kubernetes-native-testbed/settings/hooks/new

* Payload URL: https://tekton.__LOADBALANCER_IP_ADDRESS__.nip.io/event-listener
	* replace to your allocated address
* Content type: application/json
* Secret: sample-github-webhook-secret
  * if you want to change, please edit manifests/infra/instances/ci.yaml
* Enable SSL verification: [*]
* Just the push event: [*]
* Active: [*]

Deploy applications and so on

Following script do:

  • Create cluster
  • Deploy applications to Kubernetes
./tools/start.sh

Shutdown

Following script do:

  • Delete "Service" resource which use allocated IP address
  • Destroy cluster
./tools/shutdown.sh

Finalize

Following script do:

  • Deallocate IP Address
./tools/finalize.sh

Endpoint

  • Infrastructure
Kind Endpoint Username Password
Image Registry https://harbor.__LOADBALANCER_IP_ADDRESS__.nip.io admin admin
CI https://tekton.__LOADBALANCER_IP_ADDRESS__.nip.io NO NO
CD https://argocd.__LOADBALANCER_IP_ADDRESS__.nip.io NO NO
Object Storage https://minio.__LOADBALANCER_IP_ADDRESS__.nip.io minio miniosecret
Block/Shared Storage https://rook.__LOADBALANCER_IP_ADDRESS__.nip.io/ admin 'kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}"
Dashboard https://dashboard.__LOADBALANCER_IP_ADDRESS__.nip.io/ NO NO
Monitoring https://grafana.__LOADBALANCER_IP_ADDRESS__.nip.io/ admin admin
Monitoring https://prometheus.__LOADBALANCER_IP_ADDRESS__.nip.io/ NO NO
Monitoring https://weave-scope.__LOADBALANCER_IP_ADDRESS__.nip.io NO NO
Alerting https://alertmanager.__LOADBALANCER_IP_ADDRESS__.nip.io/ NO NO
Search - ES https://testbed.__LOADBALANCER_IP_ADDRESS__.nip.io/search/es elastic 'kubectl -n search get secret search-db-es-elastic-user -o jsonpath="{.data.elastic}"
Search - Kibana https://testbed.__LOADBALANCER_IP_ADDRESS__.nip.io/search/kibana elastic 'kubectl -n search get secret search-db-es-elastic-user -o jsonpath="{.data.elastic}"
Point - DB https://yugabyte.__LOADBALANCER_IP_ADDRESS__.nip.io/point/master/ NO NO
Point - DB https://yugabyte.__LOADBALANCER_IP_ADDRESS__.nip.io/point/tserver/ NO NO
  • Microservice
Kind Endpoint Username Password
Admin https://testbed.__LOADBALANCER_IP_ADDRESS__.nip.io/admin/ NO NO

Directory structure

  • manifests/
    • Kubernetes manifests
    • infra/: system or infrastructure manifests
    • cicd/: CI/CD pipeline settings
      • ci-manifests/: tekton pipelines manifests
      • cd-manifests/: argocd pipelines manifests
  • microservices/
    • microservice application source code
  • development/
    • development manifests (skaffold, kustomization patch file)
  • tools/
    • tools and scripts

Local development

# initializing
source ./development/initialize.sh

# start local development for admin
./development/local-development.sh admin

# access service with  whole system or only local check
https://testbed.__LOADBALANCER_IP_ADDRESS__.nip.io/admin/index.html
  or
https://localhost:8080/

For contributors only

About

This is fully Kubernetes-native testbed environment. Please contribute for add additional OSS (Vitess, NATS, etc) or microservices.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • JavaScript 77.5%
  • Go 15.3%
  • HTML 3.5%
  • Shell 1.6%
  • Dockerfile 1.1%
  • Makefile 0.5%
  • Other 0.5%