Diagrams-as-Code using the awesome D2 language, MermaidJS, Python diagrams and Graphviz.
Diagrams shown below are automatically (re)generated by GitHub Actions CI/CD 😎
I read an article that said:
the ability to create meaningful diagrams is the pinnacle of communication skills as an engineer
- Documentation
- Diagrams
- This Repo's Creation & GitHub Actions CI/CD to auto-(re)generate diagrams from code changes
- GitHub Flow with Jira ticket integration
- Git - why you shouldn't use long-lived feature branches
- AWS Web Traffic Classic
- Azure Active Directory Single Sign-On
- Jenkins CI/CD on Kubernetes
- ArgoCD - GitOps for Kubernetes
- GCP Cloudflare Web Architecture GKE
- GCP Malware Scanner with ClamAV
- Kubernetes Deployment with Horizontal Pod Autoscaler and Ingress
- Kubernetes Stateful Architecture with persistent volumes
- Kubernetes Service External Traffic Policy
- Kubernetes on Premise
- Traefik Kubernetes Ingress on GKE
- Kong API Gateway on Kubernetes (AWS EKS)
- OpenTSDB on Kubernetes and HBase
- MySQL Replica Architecture
- Kafka Flink Elasticsearch
- Elasticsearch Queries
- Cassandra Queries
- Devs Test in Production
- Git - Environment Branches
- LucidChart - GCP Architecture
- Web Basics
- Network - Layer 2 - Local - ARP
- Network - Layer 3 - Remote - IP
- Samples Revamped
- Build from Source
- Templates
- See Also
Gist from Knowledge-Base repo full of links to Diagrams technologies and Icon Sets.
They say a picture is worth a thousand words...
Open README.md to enlarge:
Prefix Git branches with Jira ticket numbers in Jira's AA-NNN
format for GitHub Pull Requests to automatically appear in Jira tickets (see this doc):
%% https://mermaid.js.org/syntax/gitgraph.html#gitgraph-specific-configuration-options
%% https://htmlcolorcodes.com/
%%{ init: {
"logLevel": "debug",
"theme": "dark",
"themeVariables": {
"git0": "#839192",
"git1": "#2874A6",
"gitInv0": "#FFFFFF",
"gitBranchLabel0": "#FFFFFF",
"commitLabelColor": "#FFFFFF"
}
}
}%%
gitGraph
commit
commit id: "branch"
branch AA-NNN-my-feature-branch
checkout AA-NNN-my-feature-branch
commit id: "add code"
commit id: "refine code"
checkout main
merge AA-NNN-my-feature-branch id: "merge PR" type: HIGHLIGHT tag: "2023.15 release"
commit
commit
* Environment Branches may be one of the few exceptions but requires workflow discipline.
See Also: 100+ scripts for Git and the major Git repo providers like GitHub, GitLab, Bitbucket, Azure DevOps in my DevOps-Bash-tools repo.
%% https://mermaid.js.org/syntax/gitgraph.html#gitgraph-specific-configuration-options
%% https://htmlcolorcodes.com/
%%{ init: {
"logLevel": "debug",
"theme": "dark",
"gitGraph": {
"mainBranchName": "master"
},
"themeVariables": {
"git0": "#839192",
"git1": "#C0392B ",
"git2": "#2E86C1",
"gitInv0": "#FFFFFF",
"gitBranchLabel0": "#FFFFFF",
"commitLabelColor": "#FFFFFF"
}
}
}%%
gitGraph
commit id: "commit 1"
commit id: "branch"
branch long-lived-branch
checkout long-lived-branch
commit id: "50 clever commits"
checkout master
commit id: "commit 2"
checkout long-lived-branch
commit id: "too clever"
checkout master
commit id: "commit 3"
checkout long-lived-branch
commit id: "too long"
checkout master
commit id: "commit 4"
checkout long-lived-branch
commit id: "try to merge back"
checkout master
merge long-lived-branch id: "Merge Conflict!!" type: REVERSE
checkout long-lived-branch
commit id: "trying to fix"
commit id: "still trying to fix"
commit id: "struggling to fix"
commit id: "ask Hari for help"
branch fixes-branch-to-send-to-naughty-colleague
checkout fixes-branch-to-send-to-naughty-colleague
commit id: "fix 1"
commit id: "fix 2"
commit id: "fix 3"
commit id: "could have been working on better things!"
checkout long-lived-branch
merge fixes-branch-to-send-to-naughty-colleague id: "merge fixes" type: HIGHLIGHT
commit id: "more commits"
commit id: "because this branch only had 105 commits already"
checkout master
merge long-lived-branch id: "Finallly Merged!" type: HIGHLIGHT
commit id: "Please never do that again"
I've administered Azure Active Directory at a couple of companies and integrated a variety of applications including GitHub Enterprise Cloud, AWS IAM Identity Center (formerly AWS SSO), Jenkins, ArgoCD, Keycloak, Hubspot etc using the typical OIDC or SAML integration mechanisms.
azure_ad_aws_github_keycloak.d2:
A production Jenkins on Kubernetes I built for a client with auto-spawning agents for horizontal scaling and integration with Docker, SonarQube, Clair, Grype and Trivy for code & container scanning.
- GitHub repo: HariSekhon/Kubernetes-configs
- GitHub repo: HariSekhon/Jenkins
- Advanced Jenkinsfile
- Groovy Shared Library with the code & container scanning functions
- clair.groovy
- grype.groovy
- trivy.groovy, trivyFS.groovy, trivyImages.groovy
- gcrDockerAuth.groovy, garDockerAuth.groovy
- and others in vars/, and don't forget about the epic Jenkinsfile
- https://github.com/HariSekhon/Jenkins
- https://github.com/HariSekhon/Kubernetes-configs
screenshot:
A production internet customer facing website and apps replatform to Google Kubernetes Engine I did for an internet startup client using:
- GitHub repo: HariSekhon/Terraform
- GitHub repo: HariSekhon/Kubernetes-configs
- External DNS automatic DNS record creation in Cloudflare for any Kubernetes ingresses
- External Secrets pulling into Kubernetes from GCP Secret Manager
There are Cloudflare API scripts in the HariSekhon/DevOps-Bash-tools repo.
gcp_cloudflare_web_architecture_gke.py:
A variation using Kubernetes and Cloud Functions of this GCP malware scanner solution architecture:
- GitHub repo: HariSekhon/Kubernetes-configs
kubernetes_deployment_hpa_ingress.py:
- GitHub repo: HariSekhon/Kubernetes-configs
kubernetes_stateful_architecture.py:
- GitHub repo: HariSekhon/Kubernetes-configs
kubernetes_external_traffic_policy.d2:
- GitHub repo: Kubernetes-configs
- GitHub repo: HAProxy-configs
Traditionally:
with MetalLB:
- GitHub repo: HariSekhon/Kubernetes-configs
Is it just me or do MetaLB think they're Starfleet? (compare their logos)
kubernetes_on_premise_metallb.d2:
A Traefik deployment I did for a client.
- GitHub repo: HariSekhon/Kubernetes-configs
kubernetes_traefik_ingress_gke.py:
kubernetes_traefik_ingress_gke.d2:
A Kong API Gateway deployment I did for a client.
- GitHub repo: HariSekhon/Kubernetes-configs
kubernetes_kong_api_gateway_eks.py:
A high scale production OpenTSDB replatform I did to Kubernetes for a client, ingesting 9 billion data points per day and serving 3 million queries per day.
I also had to do advanced performance tuning of their production HBase cluster which was suffering from frequent outages at this scale due to being set up by a non-SME on the wrong hardware (I had to make do with the existing hardware of course).
This was the second client I did in-depth performance tuning of HBase for - I've published a selection of useful HBase tools - see hbase_*.py
and opentsdb_*.py
in HariSekhon/DevOps-Python-tools.
mysql_replica_architecture.d2:
Iirc I created and stuck this meme pic of The Most Interesting Man in the World on the wall of my tech dept back in 2011 while leading the infra team of an internet Ad Tech company doing several production releases a day. We literally did test in production using a small fraction of live internet traffic via canary deployments.
At least they don't only test in Production!
Another internet facing client refused to use tagging because they didn't want to have to think up version or release numbers for their website releases.
Not everybody likes environment branches, but they worked in production for over 2 years and they are easy to use.
Also, contrary to some naysayers it's quite easy to diff environment branches as everything should be in Git, so you can get a very quick and easy difference between your environments in a single git diff
command. It's also easy to automate backporting hotfixes to lower environments:
- GitHub repo: HariSekhon/Jenkins
%%{ init: {
"logLevel": "debug",
"theme": "dark",
"gitGraph": {
"mainBranchName": "dev"
},
"themeVariables": {
"git0": "red",
"git1": "blue ",
"git2": "green",
"gitInv0": "#FFFFFF",
"gitBranchLabel0": "#FFFFFF",
"commitLabelColor": "#FFFFFF"
}
}
}%%
gitGraph
branch staging
branch production
checkout dev
commit id: "commit 1"
checkout staging
commit id: "QA fix 1 "
checkout production
commit id: "hotfix commit"
checkout dev
commit id: "commit 2"
checkout staging
merge dev id: "fast-forward merge" tag: "CI/CD + QA Tests"
checkout production
merge staging id: "fast-forward merge " tag: "v2023.1 Release (CI/CD)"
checkout dev
commit id: "commit 3"
checkout staging
commit id: "QA fix 2 "
#checkout production
#commit id: "commit 3 "
checkout dev
commit id: "commit 4"
checkout staging
merge dev id: "fast-forward merge 2" tag: "CI/CD + QA Tests"
checkout production
merge staging id: "fast-forward merge 2 " tag: "v2023.2 Release (CI/CD)"
checkout dev
commit id: "commit 5"
checkout staging
commit id: "QA fix 3 "
#checkout production
#commit id: "commit 5 "
checkout dev
commit id: "commit 6"
checkout staging
merge dev id: "fast-forward merge 3" tag: "CI/CD + QA Tests"
checkout production
merge staging id: "fast-forward merge 3 " tag: "v2023.3 Release (CI/CD)"
Note: I did eventually move this client to tagged releases using YYYY.NN
release format, just incrementing NN
which is a no brainer (githubNextRelease.groovy). It turns out the developers had eventually started using releases in Jira labelled as YYYY.NN
to track which tickets were going into which production deployment, so when I pushed for this, it made sense to them finally as not being too great an inconvenience! It's also easy to automate by creating GitHub Releases in Jenkins (githubCreateRelease.groovy).
A sample architecture I did for a client for us to talk through, which was similar to what they had in mind (I won the gig).
This is the only diagram not as code (here for historical interest). I would embed the interactive live diagram but GitHub markdown doesn't allow HTML iframes so this is the png.
When you're trying to explain to your kids how the internet works...
These are reworked from Python diagrams and Cloudgram examples.
aws_load_balanced_web_farm.py:
aws_clustered_web_services.py:
advanced_web_services_open_source.py:
aws_serverless_image_processing.py:
Install D2, Graphviz, Python3 and 'diagrams' pip module:
git clone https://github.com/HariSekhon/Diagrams-as-Code diagrams
cd diagrams
make install
Create all the .png
and .svg
diagrams in the images/
dir:
make
Generate only the D2 svg diagrams:
make d2
Generate only the Python png diagrams:
make py
Create any single D2 diagram by running the d2 script file:
./jenkins_kubernetes_docker.d2
Create any single Python diagram and have it open automatically by running the python script file:
./gcp_cloudflare_web_architecture_gke.py
The templates/diagram.d2 and templates/diagram.py show the basics of each language.
They are a good starting point for creating your own diagrams, and come pre-loaded with many useful icons, links to docs and links to icon sets.
For tonnes of great free tech programs and scripts, see also:
-
DevOps Bash Tools - 1000+ DevOps Bash Scripts, Advanced
.bashrc
,.vimrc
,.screenrc
,.tmux.conf
,.gitconfig
, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies -
DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython
-
SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery
-
Jenkins - Advanced Jenkinsfile & Jenkins Groovy Shared Library
-
GitHub-Actions - GitHub Actions master template & GitHub Actions Shared Workflows library
-
Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.
-
Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments
-
Terraform - Terraform templates for AWS / GCP / Azure / GitHub management
-
The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)
-
Nagios Plugin Kafka - Kafka API pub/sub Nagios Plugin written in Scala with Kerberos support
-
DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...
-
HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.
-
Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3
-
HashiCorp Packer templates - Linux automated bare-metal installs and portable virtual machines OVA format appliances using HashiCorp Packer, Redhat Kickstart, Debian Preseed and Ubuntu AutoInstaller / Cloud-Init
-
Knowledge-Base - IT Knowledge Base from 20 years in DevOps, Linux, Cloud, Big Data, AWS, GCP etc.
Pre-built Docker images are available for those repos (which include this one as a submodule) and the "docker available" icon above links to an uber image which contains all my github repos pre-built. There are Centos, Alpine, Debian and Ubuntu versions of this uber Docker image containing all repos.