Improved testing scripts and testing README

MacSymbolicatorTests/README.md

@@ -0,0 +1,29 @@
+# Testing
+
+For validating the behavior of MacSymbolicator, we need to follow these steps:
+
+1. Create the binaries and the dSYMs of the test project
+ - Run `MacSymbolicatorTests/create_binaries_and_dsyms.sh`
+ - This script will build the test apps, then move the binaries and dSYMs to their respective folders.
+ - This script will also create a `Payload.zip` that we will use later
+2. On a second computer, create the crash files, the sample files, and the spindumps
+ - Transfer `Payload.zip` to your second computer and extract it
+ - Run `create_crashes_samples_spindumps.sh` from the extracted directory (requires sudo to sample/spindump processes)
+ - This script will create a `Result.zip` that we will transfer back to the first computer
+3. Back on the main computer, extract `Result.zip` into `MacSymbolicatorTests/Resources`
+4. Create the "expect output" of symbolication
+ - Run `MacSymbolicatorTests/create_expected_output.sh`
+ - This script will build `MacSymbolicatorCLI` and use it to symbolicate the reports from the second computer
+ - This script will save the symbolicated output files suffixed with `_symbolicated`
+5. Manually check that the expected output is symbolicated correctly
+6. Run the tests in Xcode, which will symbolicate the reports and compare them to the expected output
+
+
+### Q&A
+
+- Why is it done this way?
+ - So that we don't have include all the binaries, the dSYMs, the samples, and spindumps which are pretty big for git, pollute the history, but also might contain information about my computer that I don't want to share.
+- For step 2, why need a second computer?
+ - If you use the same computer that created the crashing binary (and dSYM), when macOS creates a crash report for an app it will also automatically symbolicate it. My guess is that dSYMs are indexed by Xcode as soon as they're created.
+- For step 5, doesn't manually checking the output defeat the point of automated testing?
+ - Yes, but at least the expected output is created automatically which saves a LOT of time.

MacSymbolicatorTests/create_binaries_and_dsyms.sh

@@ -1,7 +1,18 @@
 #!/usr/bin/env bash
 cd TestProject
 rm -rf build
-xcodebuild clean archive -configuration Release -alltargets
+xcodebuild clean
+
+if ! command -v xcbeautify &> /dev/null
+ xcodebuild archive -configuration Release -target CrashingTest -target CrashingInAnotherTargetTest
+ xcodebuild archive -configuration Release -target SingleThreadHangingTest -target MultiThreadHangingTest -target MultiTargetHangingTest
+ xcodebuild archive -configuration Release -target iOSCrashingTest
+else
+ xcodebuild archive -configuration Release -target CrashingTest -target CrashingInAnotherTargetTest | xcbeautify
+ xcodebuild archive -configuration Release -target SingleThreadHangingTest -target MultiThreadHangingTest -target MultiTargetHangingTest | xcbeautify
+ xcodebuild archive -configuration Release -target iOSCrashingTest | xcbeautify
+fi
+
 cd ..
 
 rm -rf Resources/dSYMs/
@@ -15,10 +26,13 @@ mkdir -p Binaries
 mv /tmp/TestProject.dst/usr/local/bin/* Binaries/
 mv /tmp/TestProject.dst/Applications/* Binaries/
 
-rm -rf build
+rm -rf TestProject/build
+rm -rf TestProject/DerivedData
+
+./create_payload.sh
 
 echo "Done!"
-echo "Use the binaries in Binaries/ to create your crash logs & samples on macOS."
+echo "Use the script and binaries in Payload.zip to automatically create your crash logs, samples and spindumps on macOS."
 echo "For iOS, install the .app on your device using Xcode."
 echo "The dSYMs are in Resources/dSYMs/"
 echo

MacSymbolicatorTests/create_crashes_samples_spindumps.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+mkdir Crashes
+mkdir Samples
+mkdir Spindumps
+
+rm ~/Library/Logs/DiagnosticReports/CrashingTest*
+./CrashingTest
+sleep 3 # give the system some time to create the crash report
+cp ~/Library/Logs/DiagnosticReports/CrashingTest* Crashes/
+mv Crashes/CrashingTest* Crashes/single-target-crash.ips
+
+rm ~/Library/Logs/DiagnosticReports/CrashingInAnotherTargetTest*
+./CrashingInAnotherTargetTest
+sleep 3 # give the system some time to create the crash report
+cp ~/Library/Logs/DiagnosticReports/CrashingInAnotherTargetTest* Crashes/
+mv Crashes/CrashingInAnotherTargetTest* Crashes/multi-target-crash.ips
+
+./SingleThreadHangingTest&
+pid=$!
+sleep 1 # give the process some time to start up and hang
+sudo sample $pid -f ./Samples/singlethread-sample.txt
+sudo spindump $pid -o ./Spindumps/singlethread-spindump.txt
+kill -9 $pid
+
+./MultiThreadHangingTest&
+pid=$!
+sleep 1 # give the process some time to start up and hang
+sudo sample $pid -f ./Samples/multithread-sample.txt
+sudo spindump $pid -o ./Spindumps/multithread-spindump.txt
+kill -9 $pid
+
+./MultiTargetHangingTest&
+pid=$!
+sleep 1 # give the process some time to start up and hang
+sudo sample $pid -f ./Samples/multitarget-sample.txt
+sudo spindump $pid -o ./Spindumps/multitarget-spindump.txt
+kill -9 $pid
+
+# Create a zip file of the results to send back
+zip -1 -r Result.zip Crashes Samples Spindumps

MacSymbolicatorTests/symbolicate_tests.sh → MacSymbolicatorTests/create_expected_output.sh

@@ -3,7 +3,13 @@
 cd ..
 rm -rf build
 rm -rf DerivedData
-xcodebuild clean build -configuration Release -scheme MacSymbolicatorCLI
+
+if ! command -v xcbeautify &> /dev/null
+ xcodebuild clean build -configuration Release -scheme MacSymbolicatorCLI
+else
+ xcodebuild clean build -configuration Release -scheme MacSymbolicatorCLI | xcbeautify
+fi
+
 cd MacSymbolicatorTests
 
 cli="../DerivedData/MacSymbolicator/Build/Products/Release/MacSymbolicatorCLI"
@@ -27,11 +33,15 @@ $cli "./Resources/Crashes/multi-target-crash.crash" "./Resources/dSYMs/CrashingI
 $cli "./Resources/Crashes/ios-crash.ips" "./Resources/dSYMs/iOSCrashingTest.app.dSYM" -o "./Resources/Crashes/ios-crash_symbolicated.ips"
 $cli "./Resources/Crashes/ios-crash.crash" "./Resources/dSYMs/iOSCrashingTest.app.dSYM" -o "./Resources/Crashes/ios-crash_symbolicated.crash"
 
-# Single-thread sample
+# Samples
 $cli "./Resources/Samples/singlethread-sample.txt" "./Resources/dSYMs/SingleThreadHangingTest.dSYM" -o "./Resources/Samples/singlethread-sample_symbolicated.txt"
-
-# Multi-thread sample
 $cli "./Resources/Samples/multithread-sample.txt" "./Resources/dSYMs/MultiThreadHangingTest.dSYM" -o "./Resources/Samples/multithread-sample_symbolicated.txt"
+$cli "./Resources/Samples/multitarget-sample.txt" "./Resources/dSYMs/MultiTargetHangingTest.dSYM" "./Resources/dSYMs/AnotherTarget.framework.dSYM" -o "./Resources/Samples/multitarget-sample_symbolicated.txt"
+
+# Spindumps
+# $cli "./Resources/Spindumps/singlethread-spindump.txt" "./Resources/dSYMs/SingleThreadHangingTest.dSYM" -o "./Resources/Spindumps/singlethread-spindump_symbolicated.txt"
+# $cli "./Resources/Spindumps/multithread-spindump.txt" "./Resources/dSYMs/MultiThreadHangingTest.dSYM" -o "./Resources/Spindumps/multithread-spindump_symbolicated.txt"
+# $cli "./Resources/Spindumps/multitarget-spindump.txt" "./Resources/dSYMs/MultiTargetHangingTest.dSYM" "./Resources/dSYMs/AnotherTarget.framework.dSYM" -o "./Resources/Spindumps/multitarget-spindump_symbolicated.txt"
 
 echo "Done creating the symbolicated output."
 echo "Make sure to check the output files manually before using them as 'expected' output in tests."

MacSymbolicatorTests/create_payload.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+# Prepare the zip to send to a second computer
+# (becauses crashes and samples will be automatically symbolicated if the dSYMs exist in the system)
+rm -rf Payload
+mkdir Payload
+cp -r Binaries/* Payload/
+cp create_crashes_samples_spindumps.sh Payload/
+zip -1 -r Payload.zip Payload
+rm -rf Payload