False positives and negatives: random strings ignored, alphabet flagged

[jcaswell-gd]

🐛 Bug Report

Tartufo flags the ASCII alphabet as high entropy and ignores randomly-generated strings of equal length.

To Reproduce

Please run this script or equivalent commands with tartufo v3.3.1

#!/usr/bin/env bash

cd /tmp
mkdir scan-this
cd scan-this
git init --initial-branch=main

# 26-character random ASCII string
echo '8d(VnIcY&/.U?:[lg6tDad5cg0' > secrets.txt
git add secrets.txt
git commit -m "Initial commit"

# Scan passes
tartufo scan-local-repo .

# 26-character random alnum string
echo 'Yfqt8oplkER6VtxtCH1XO7PZ8Y' >> secrets.txt
git add secrets.txt
git commit -m "This is not my password"

# Scan passes
tartufo scan-local-repo .

# 26-character "FIPS-181 compliant" string generated by Mac Keychain Access
echo 'nafowobovajiskakykyrikuitd' >> secrets.txt
git add secrets.txt
git commit -m "Nothing to see here"

# Scan passes
tartufo scan-local-repo .

echo "abcdefghijklmnopqrstuvwxyz" > alpha.txt
git add alpha.txt
git commit -m "Too many secrets"

# Scan fails, flagging alphabet
tartufo scan-local-repo .

Output:

% ./tartufo-false.sh 
Initialized empty Git repository in /private/tmp/scan-this/.git/
[main (root-commit) 4ee6eec] Initial commit
 1 file changed, 1 insertion(+)
 create mode 100644 secrets.txt
Time: 2022-12-06T10:52:06.489760
All clear. No secrets detected.
[main ab012c0] This is not my password
 1 file changed, 1 insertion(+)
Time: 2022-12-06T10:52:06.788933
All clear. No secrets detected.
[main be1d91f] Nothing to see here
 1 file changed, 1 insertion(+)
Time: 2022-12-06T10:52:07.081072
All clear. No secrets detected.
[main 6614526] Too many secrets
 1 file changed, 1 insertion(+)
 create mode 100644 alpha.txt
~~~~~~~~~~~~~~~~~~~~~
Reason: High Entropy
Filepath: alpha.txt
Signature: 036642242db40553ab31943aea7d694796bda82ae4d1f93fc5196b0a485ac3af
Commit time: 2022-12-06 10:52:07
Commit message: Too many secrets

Commit hash: 6614526736065ee21dfd904c64dcebdd4138022d
Branch: main
diff --git a/alpha.txt b/alpha.txt
new file mode 100644
index 0000000..b0883f3
--- /dev/null
+++ b/alpha.txt
@@ -0,0 +1 @@
+abcdefghijklmnopqrstuvwxyz

~~~~~~~~~~~~~~~~~~~~~

Expected Behavior

The randomly-generated strings should be flagged; the alphabet should be ignored.

We have also had false positives with longer non-random alphanumeric strings, e.g. "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789": https://github.com/gdcorp-mobile/Leka-iOS/actions/runs/3569972591/jobs/6000862649.

Code Example

Please see the script above

Environment

% python --version
Python 3.9.14
% tartufo --version
tartufo, version 3.3.1
% git --version
git version 2.38.0
% sw_vers
ProductName:	macOS
ProductVersion:	12.6
BuildVersion:	21G115