If you haven't seen Monty Python and The Holy Grail recently rewatch it.
Engineering Topic: Data and Test
A generic CICD pipeline:
By measuring test efficacy we can improve engineering workflows and value delivery to customers.
Ad-hoc testing, code and documentation.
Configuration and environment data
Documentation lives in version control too.
Ideally there is metadata embedded in the code which can be used for feature labeling.
Test data
Test data
Test data
Customer data
Each cycle produces data. The feedback loop needs to be fast and tightly coupled.
This requires teams to work directly with customers on both escalations and feature development.
As regression suites grow, data informs us of which tests actually need to run.
Configuration data is test data. Infrastructure and environment data is test data.
By measuring test efficacy we can improve engineering workflows and value delivery to customers.
How do you extract meaningful information from test data?
Capturing structured test data enables meaningful information to be extracted from test results.
Once test efficacy data is available, it can be used to inform test process improvements.
Approprate measurements and labels must be automatically associated with test data.
An extensible test framework combined with static analysis can achieve this.
Eventually, this process can be fully automated as well, with AI/ML informing which tests provide the best value.
The sensitivity to releasing broken product must be weighed carefully against efficiency.
However this requires a lot of data to really automate.
I'm passionate about test. I'm also passionate about using data to make test better.
This aligns with optimizing the ability of delivery teams to deliver value.
Making customers happy can be a fun and engaging engineering challenge.
However, not all organizations allow customers to engage with engineering directly.
Organizations must demand processes to support this, to support customers.
Data can help reduce the friction in the value delivery stream, but data isn't enough.
By measuring test efficacy we can improve engineering workflows and value delivery to customers.
References:
Continuous Delivery by Jez Humble and David Farley
https://testing.googleblog.com/2018/09/efficacy-presubmit.html
Hypergrowth by David Cancel