Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ACI-2512] Per-file DerivedData cache #40

Merged
merged 135 commits into from
Aug 12, 2024
Merged
Changes from 1 commit
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
3e6289d
save xcode deriveddata initial version
imrekel Jun 12, 2024
ff8d31a
restoreXcodeDerivedData + refactors
imrekel Jun 12, 2024
e0e6f04
param fix
imrekel Jun 12, 2024
47c7eed
fix save params
imrekel Jun 12, 2024
8f3d635
add tar compression
imrekel Jun 13, 2024
4aa43be
archive decompression
imrekel Jun 13, 2024
ee1e069
logging improvements
imrekel Jun 13, 2024
f32432f
ensure directories exist
imrekel Jun 13, 2024
18252d9
create dir fix
imrekel Jun 13, 2024
04fe721
updated log messages
imrekel Jun 13, 2024
2dbeefa
linter
imrekel Jun 13, 2024
386a262
test fix
imrekel Jun 13, 2024
5a02276
Merge branch 'main' of github.com:bitrise-io/bitrise-build-cache-cli …
imrekel Jun 13, 2024
eda5dff
goreleaser
imrekel Jun 13, 2024
adeecf0
goreleaser dep updated
imrekel Jun 13, 2024
f8b1f5a
goreleaser
imrekel Jun 13, 2024
94c9d7b
goreleaser
imrekel Jun 13, 2024
d6bde9a
release wf back to macos
imrekel Jun 13, 2024
42b7b9b
debug mode info log
imrekel Jun 14, 2024
0cbbaa5
more metadata logging
imrekel Jun 14, 2024
7564044
fix keys
imrekel Jun 14, 2024
ad11d92
single cache archive
imrekel Jun 18, 2024
8974575
move extraction before mtime restoration
imrekel Jun 18, 2024
39b0d56
key setting logging
imrekel Jun 19, 2024
2861d99
added some more logging
imrekel Jun 20, 2024
8c0b6a7
add RPC buildtool header
imrekel Jun 20, 2024
fe42dca
use [email protected]
imrekel Jun 21, 2024
5ff5217
add app slug to key
imrekel Jun 24, 2024
8607ef7
save test fix
imrekel Jun 24, 2024
a70c065
feat: Wording improvements
zsolt-marta-bitrise Jun 28, 2024
e6204d9
refactor: Extract file walking
zsolt-marta-bitrise Jun 28, 2024
65020f4
feat: Log nonexistent files
zsolt-marta-bitrise Jun 28, 2024
13636b3
feat: Automatically add port
zsolt-marta-bitrise Jun 28, 2024
84e54ea
fix: Send org ID when using PAT
zsolt-marta-bitrise Jul 1, 2024
02802c3
fix: Pass auth config to client
zsolt-marta-bitrise Jul 1, 2024
be7c583
Merge branch 'main' into dd-cache
zsolt-marta-bitrise Jul 1, 2024
f4e54f6
fix: Lintfix
zsolt-marta-bitrise Jul 1, 2024
f88cd61
extend cache archive metada and log data on restore
imrekel Jul 2, 2024
e84b41a
fix: Log parsed url
zsolt-marta-bitrise Jul 2, 2024
8cd387c
fix: Fix secure url parsing
zsolt-marta-bitrise Jul 2, 2024
6b0e570
fix: Return if insecure
zsolt-marta-bitrise Jul 2, 2024
21f208e
git commit hash alternative source; skip logging createdAt when not set
imrekel Jul 2, 2024
3c158f8
Merge branch 'dd-cache' of https://github.com/imrekel/bitrise-build-c…
imrekel Jul 2, 2024
e8f43e1
delete xcode deriveddata
imrekel Jul 3, 2024
2bdc566
added step analytics
imrekel Jul 5, 2024
8fe32f9
rename tracker field - update dependency
imrekel Jul 5, 2024
a4e82e7
Merge branch 'main' of github.com:bitrise-io/bitrise-build-cache-cli …
imrekel Jul 5, 2024
ea79200
go mod tidy
imrekel Jul 5, 2024
de2b296
fix: Fix restore step ID
zsolt-marta-bitrise Jul 8, 2024
0624e39
fix: Call GetCapabilities
zsolt-marta-bitrise Jul 12, 2024
a2d87f1
fix: Fix proto packages
zsolt-marta-bitrise Jul 12, 2024
a97c22d
fix: Redirect xcode to GCP
zsolt-marta-bitrise Jul 12, 2024
f47659d
file level upload
imrekel Jul 17, 2024
2958ef9
fix allocation
imrekel Jul 17, 2024
00cdde2
map fix
imrekel Jul 17, 2024
ffd1060
slice fix
imrekel Jul 17, 2024
3e12ba4
update logging
imrekel Jul 17, 2024
781e8c6
more timestamps
imrekel Jul 18, 2024
7a970dd
don't upload blobs twice
imrekel Jul 18, 2024
d080d7f
parallel upload
imrekel Jul 18, 2024
5053570
thread safety
imrekel Jul 18, 2024
354f582
more thread safety
imrekel Jul 18, 2024
320f1df
parallel 3
imrekel Jul 18, 2024
134d494
parallel 1
imrekel Jul 18, 2024
9d8fed0
parallel 20
imrekel Jul 18, 2024
f9e5696
5 parallel upload
imrekel Jul 18, 2024
928f86b
20 threads
imrekel Jul 18, 2024
3db468b
remove getcapabilities for each blob
imrekel Jul 18, 2024
2d2cc35
getcapabilities fix
imrekel Jul 18, 2024
3b54b32
skip DD symlinks
imrekel Jul 19, 2024
5edea77
per file download
imrekel Jul 19, 2024
e2ab534
parallel downloads
imrekel Jul 19, 2024
488c11d
limit parallelism
imrekel Jul 19, 2024
4a22442
also save file mode
imrekel Jul 20, 2024
c1d84a5
restore directories
imrekel Jul 22, 2024
eeb39c1
don't create folders twice
imrekel Jul 22, 2024
b2c8583
restore dirs after files
imrekel Jul 22, 2024
0a39c0f
fix: Fix error message
zsolt-marta-bitrise Jul 23, 2024
595e074
Merge branch 'main' of github.com:bitrise-io/bitrise-build-cache-cli …
imrekel Jul 23, 2024
7268bd3
Merge branch 'dd-cache-file-level' of github.com:bitrise-io/bitrise-b…
imrekel Jul 23, 2024
3536954
restore input directories + file mode
imrekel Jul 23, 2024
1fd6e4f
log largest DD file
imrekel Jul 23, 2024
da4a3c9
feat: Save / restore input file attributes
zsolt-marta-bitrise Jul 23, 2024
f4ac8db
fix: Fix message
zsolt-marta-bitrise Jul 23, 2024
ad14f0a
feat: Save and restore xcode files
zsolt-marta-bitrise Jul 23, 2024
28b5fdc
fix: Remove default for xcode cache path
zsolt-marta-bitrise Jul 23, 2024
421cffa
feat: Log largest download file size
zsolt-marta-bitrise Jul 23, 2024
bb1e955
refactors; metadata update
imrekel Jul 24, 2024
e4bb2a5
log project file count
imrekel Jul 24, 2024
4d6c957
rename -> UploadFileToBuildCache
imrekel Jul 26, 2024
dd92d41
2-phase metadata storage
imrekel Jul 26, 2024
dc48bfc
fix stream size
imrekel Jul 26, 2024
1e548e4
stream fix
imrekel Jul 26, 2024
d091052
log formatting
imrekel Jul 26, 2024
cce5b4b
more logging
imrekel Jul 26, 2024
d346827
debug
imrekel Jul 26, 2024
e5642f1
fix stream upload
imrekel Jul 26, 2024
8796c09
remove tar commands; linter and other fixes
imrekel Jul 29, 2024
be10aa5
feat: Dont retry if not found
zsolt-marta-bitrise Jul 29, 2024
4f1f7c7
linter, test fix
imrekel Jul 29, 2024
91242c9
Merge branch 'dd-cache-file-level' of github.com:bitrise-io/bitrise-b…
imrekel Jul 29, 2024
1754083
feat: Only log cache not found as info
zsolt-marta-bitrise Jul 29, 2024
acfde81
Merge branch 'dd-cache-file-level' of github.com:bitrise-io/bitrise-b…
imrekel Jul 29, 2024
bfea88d
linter
imrekel Jul 29, 2024
17d16bb
linter
imrekel Jul 29, 2024
30fd89b
feat: Try falling back to metadata key without branch
zsolt-marta-bitrise Jul 29, 2024
ee2a359
refactor: DI kv client, call GetCapabilities once
zsolt-marta-bitrise Jul 29, 2024
ff1d6d2
upload stat
imrekel Jul 30, 2024
4cf2c70
Merge branch 'dd-cache-file-level' of github.com:bitrise-io/bitrise-b…
imrekel Jul 30, 2024
8493299
feat: save metadata checksum for fallback key too
zsolt-marta-bitrise Jul 30, 2024
f8e4408
save tracking
imrekel Jul 30, 2024
f252192
Merge branch 'dd-cache-file-level' of github.com:bitrise-io/bitrise-b…
imrekel Jul 30, 2024
269b515
restore analytics
imrekel Jul 30, 2024
ef47257
add restore metada key type
imrekel Jul 30, 2024
292b360
add metadata size to download; add cli version to analytics events
imrekel Jul 30, 2024
2d3c58f
rename to cache_key_type
imrekel Jul 30, 2024
13a8292
track largest_file_size
imrekel Jul 30, 2024
e742d96
feat: Remove override to GCP
zsolt-marta-bitrise Jul 31, 2024
ecb4d71
fix dd downloaded event
imrekel Jul 31, 2024
9711c63
Merge branch 'dd-cache-file-level' of github.com:bitrise-io/bitrise-b…
imrekel Jul 31, 2024
7ef2fbb
mutex for upload stats
imrekel Jul 31, 2024
5a30c71
concurrent metadata collection
imrekel Aug 2, 2024
1427942
feat: Silence usage
zsolt-marta-bitrise Aug 2, 2024
fbf3999
feat: Implement actual delete cache
zsolt-marta-bitrise Aug 5, 2024
22ae8df
feat: Save metadata last
zsolt-marta-bitrise Aug 6, 2024
775ff0b
fix: Fix download stats
zsolt-marta-bitrise Aug 6, 2024
bd5809c
fix: Debug log stats
zsolt-marta-bitrise Aug 6, 2024
298c48b
feat: Delete DD if files are missing
zsolt-marta-bitrise Aug 6, 2024
d347be2
feat: Log DD clearing
zsolt-marta-bitrise Aug 6, 2024
57c17b4
fix: Treat missing as error
zsolt-marta-bitrise Aug 6, 2024
423173e
fix: Save stats if upload fails
zsolt-marta-bitrise Aug 6, 2024
1c5686b
refactor: Simplify files in xcode package
zsolt-marta-bitrise Aug 6, 2024
0f0cbe3
refactor: Move FileGroupInfo methods to file_group_info.go
zsolt-marta-bitrise Aug 6, 2024
47a2549
refactor: Use command's context
zsolt-marta-bitrise Aug 6, 2024
e149c89
fix: Lintfix
zsolt-marta-bitrise Aug 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test fix
  • Loading branch information
imrekel committed Jun 13, 2024
commit 386a2625a16332a991ba439d00e665bb7860ca78
4 changes: 2 additions & 2 deletions internal/xcode/metadata_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,12 @@ func Test_SaveMetadata(t *testing.T) {
wantErr: "failed to calculate file infos: missing rootDir",
},
{
name: "missing outputFile",
name: "missing fileName",
args: args{
rootDir: testRootDir,
outputFile: "",
},
wantErr: "missing output file",
wantErr: "missing output fileName",
},
{
name: "ok",
Expand Down