Releases: awslabs/deequ
Releases · awslabs/deequ
2.0.7
What's Changed
Upgrades
New Features
- New type of MetricsRepository by @VenkataKarthikP:
- Using Spark tables as the data source in #518
- Row Level Result Treatment Options by @eycho-am:
- Anomaly Detection Changes by @zeotuan:
- Add Daily Season with Hourly Interval to HoltWinter in #546
- New analyzers:
- RatioOfSums by @scott-gunn in #552
- Column Count Analyzer and Check by @mentekid in #555
Maintenance/Fixes
- Fix Breeze dependency conflict in Anomaly Detection Spark 3.4+ by @zeotuan in #545
- Data Sync / DatasetMatch changes by @VenkataKarthikP:
- Row level results fixes:
- Add analyzerOption to add filteredRowOutcome for isPrimaryKey Check by @eycho-am in #537
- Fix bug in MinLength and MaxLength when NullBehavior.EmptyString by @eycho-am in #538
- [Min/Max] Apply filtered row behavior at the row level evaluation by @rdsharma26 in #543
- [MinLength/MaxLength] Apply filtered row behavior at the row level evaluation by @rdsharma26 in #547
- Fix for satisfies row level results bug by @rdsharma26 in #553
New Contributors
- @VenkataKarthikP made their first contribution in #518
- @scott-gunn made their first contribution in #552
Full Changelog: 2.0.6...2.0.7
Contributors
mentekid, VenkataKarthikP, and 5 other contributors
Assets 2
2.0.6
What's Changed
- NEW: Exact Quantile Check
- Creation of Exact Quantile Check by @jmilis2000 in #512
- Data Synchronization/Matching fixes
- Delegate to Spark for checking existence of columns in the given dataframes by @rdsharma26 in #515
- Verify that non key columns exist in each dataset by @rdsharma26 in #517
- Addition of tests
- Test that exceptions within a check's constraints do not affect other… by @tylermcdaniel0 in #516
New Contributors
- @jmilis2000 made their first contribution in #512
- @tylermcdaniel0 made their first contribution in #516
Full Changelog: 2.0.5...2.0.6
Contributors
jmilis2000, rdsharma26, and tylermcdaniel0
Assets 2
2.0.5
What's Changed
- Spark 3.4 Update
- NEW: Custom SQL analyzer
- Analyzer Improvements
New Contributors
Full Changelog: 2.0.4...2.0.5
Contributors
mentekid, jhchee, and eycho-am
Assets 2
2.0.4
What's Changed
- Row-Level Results:
- MinLength by @eycho-am in #465
- Uniqueness by @eycho-am in #471
- ColumnValues by @zixianzh1 in #476
- ReferentialIntegrity by @rdsharma26 in #466
- [Experimental] DataSynchronization by @rdsharma26 in #473
- Referential Integrity:
- Updated Referential Integrity to support multiple columns by @rdsharma26 in #463
- Constraints and Condition Changes:
- Add population stability index (PSI) to distance methods by @bevhanno in #480
- Fix chi-square test conditions by @bevhanno in #482
- Missing Column Precondition for Compliance Check - issue fix 467 by @samarth-c1 in #478
- Addition of HasMax/HasMin/HasStandardDeviation/HasMean constraint suggestions by @rdsharma26 in #489
- Alternative aggregate functions to calculate histogram values. by @akalotkin in #475
New Contributors
- @zixianzh1 made their first contribution in #476
- @samarth-c1 made their first contribution in #478
- @akalotkin made their first contribution in #475
Full Changelog: 2.0.3...2.0.4
Contributors
bevhanno, akalotkin, and 4 other contributors
Assets 2
2.0.3
What's Changed
- Adding chi-square distance method for categorical variables by @bevhanno in #444
- [WIP] Row Level Results by @mentekid in #451
- [Experimental] Addition of dataset comparison utilities by @rdsharma26 in #449
New Contributors
- @rdsharma26 made their first contribution in #447
- @bevhanno made their first contribution in #444
- @mentekid made their first contribution in #451
Full Changelog: 2.0.2...2.0.3
Contributors
bevhanno, mentekid, and rdsharma26
Assets 2
2.0.2
2.0.1
2.0.0
Add Spark 3.1 compatibility.
Note: this version is no longer compatible with Spark <=3.0. Use previous versions and branch legacy-spark-3.0 instead.
Fix build setup to make artefact importable with maven/sbt
This release updates the build setup (i.e. the pom.xml and the publishing process) so that the artefacts published to maven can now be imported using maven or sbt. There are four branches associated with this new release:
- for spark 2.2: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.2
- for spark 2.3: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.3
- for spark 2.4: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.4
- for spark 2.5: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.5
1.1.0
Changes to the build setup to support Spark 2.2.x to 2.4.x and 3.0.x. There now is one maven release available per Spark version:
- spark-3.0-scala-2.12
- spark-2.4-scala-2.11
- spark-2.3-scala-2.11
- spark-2.2-scala-2.11