Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ACI-2512] File-level DerivedData cache #40

Draft
wants to merge 74 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
3e6289d
save xcode deriveddata initial version
imrekel Jun 12, 2024
ff8d31a
restoreXcodeDerivedData + refactors
imrekel Jun 12, 2024
e0e6f04
param fix
imrekel Jun 12, 2024
47c7eed
fix save params
imrekel Jun 12, 2024
8f3d635
add tar compression
imrekel Jun 13, 2024
4aa43be
archive decompression
imrekel Jun 13, 2024
ee1e069
logging improvements
imrekel Jun 13, 2024
f32432f
ensure directories exist
imrekel Jun 13, 2024
18252d9
create dir fix
imrekel Jun 13, 2024
04fe721
updated log messages
imrekel Jun 13, 2024
2dbeefa
linter
imrekel Jun 13, 2024
386a262
test fix
imrekel Jun 13, 2024
5a02276
Merge branch 'main' of github.com:bitrise-io/bitrise-build-cache-cli …
imrekel Jun 13, 2024
eda5dff
goreleaser
imrekel Jun 13, 2024
adeecf0
goreleaser dep updated
imrekel Jun 13, 2024
f8b1f5a
goreleaser
imrekel Jun 13, 2024
94c9d7b
goreleaser
imrekel Jun 13, 2024
d6bde9a
release wf back to macos
imrekel Jun 13, 2024
42b7b9b
debug mode info log
imrekel Jun 14, 2024
0cbbaa5
more metadata logging
imrekel Jun 14, 2024
7564044
fix keys
imrekel Jun 14, 2024
ad11d92
single cache archive
imrekel Jun 18, 2024
8974575
move extraction before mtime restoration
imrekel Jun 18, 2024
39b0d56
key setting logging
imrekel Jun 19, 2024
2861d99
added some more logging
imrekel Jun 20, 2024
8c0b6a7
add RPC buildtool header
imrekel Jun 20, 2024
fe42dca
use [email protected]
imrekel Jun 21, 2024
5ff5217
add app slug to key
imrekel Jun 24, 2024
8607ef7
save test fix
imrekel Jun 24, 2024
a70c065
feat: Wording improvements
zsolt-marta-bitrise Jun 28, 2024
e6204d9
refactor: Extract file walking
zsolt-marta-bitrise Jun 28, 2024
65020f4
feat: Log nonexistent files
zsolt-marta-bitrise Jun 28, 2024
13636b3
feat: Automatically add port
zsolt-marta-bitrise Jun 28, 2024
84e54ea
fix: Send org ID when using PAT
zsolt-marta-bitrise Jul 1, 2024
02802c3
fix: Pass auth config to client
zsolt-marta-bitrise Jul 1, 2024
be7c583
Merge branch 'main' into dd-cache
zsolt-marta-bitrise Jul 1, 2024
f4e54f6
fix: Lintfix
zsolt-marta-bitrise Jul 1, 2024
f88cd61
extend cache archive metada and log data on restore
imrekel Jul 2, 2024
e84b41a
fix: Log parsed url
zsolt-marta-bitrise Jul 2, 2024
8cd387c
fix: Fix secure url parsing
zsolt-marta-bitrise Jul 2, 2024
6b0e570
fix: Return if insecure
zsolt-marta-bitrise Jul 2, 2024
21f208e
git commit hash alternative source; skip logging createdAt when not set
imrekel Jul 2, 2024
3c158f8
Merge branch 'dd-cache' of https://github.com/imrekel/bitrise-build-c…
imrekel Jul 2, 2024
e8f43e1
delete xcode deriveddata
imrekel Jul 3, 2024
2bdc566
added step analytics
imrekel Jul 5, 2024
8fe32f9
rename tracker field - update dependency
imrekel Jul 5, 2024
a4e82e7
Merge branch 'main' of github.com:bitrise-io/bitrise-build-cache-cli …
imrekel Jul 5, 2024
ea79200
go mod tidy
imrekel Jul 5, 2024
de2b296
fix: Fix restore step ID
zsolt-marta-bitrise Jul 8, 2024
0624e39
fix: Call GetCapabilities
zsolt-marta-bitrise Jul 12, 2024
a2d87f1
fix: Fix proto packages
zsolt-marta-bitrise Jul 12, 2024
a97c22d
fix: Redirect xcode to GCP
zsolt-marta-bitrise Jul 12, 2024
f47659d
file level upload
imrekel Jul 17, 2024
2958ef9
fix allocation
imrekel Jul 17, 2024
00cdde2
map fix
imrekel Jul 17, 2024
ffd1060
slice fix
imrekel Jul 17, 2024
3e12ba4
update logging
imrekel Jul 17, 2024
781e8c6
more timestamps
imrekel Jul 18, 2024
7a970dd
don't upload blobs twice
imrekel Jul 18, 2024
d080d7f
parallel upload
imrekel Jul 18, 2024
5053570
thread safety
imrekel Jul 18, 2024
354f582
more thread safety
imrekel Jul 18, 2024
320f1df
parallel 3
imrekel Jul 18, 2024
134d494
parallel 1
imrekel Jul 18, 2024
9d8fed0
parallel 20
imrekel Jul 18, 2024
f9e5696
5 parallel upload
imrekel Jul 18, 2024
928f86b
20 threads
imrekel Jul 18, 2024
3db468b
remove getcapabilities for each blob
imrekel Jul 18, 2024
2d2cc35
getcapabilities fix
imrekel Jul 18, 2024
3b54b32
skip DD symlinks
imrekel Jul 19, 2024
5edea77
per file download
imrekel Jul 19, 2024
e2ab534
parallel downloads
imrekel Jul 19, 2024
488c11d
limit parallelism
imrekel Jul 19, 2024
4a22442
also save file mode
imrekel Jul 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove getcapabilities for each blob
  • Loading branch information
imrekel committed Jul 18, 2024
commit 3db468b0c4f6d897b2d0aa3420b6784073eb059d
14 changes: 9 additions & 5 deletions internal/build_cache/kv/methods.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,15 @@ type FileDigest struct {
SizeInBytes int64
}

func (c *Client) GetCapabilities(ctx context.Context) error {
_, err := c.capabilitiesClient.GetCapabilities(ctx, &remoteexecution.GetCapabilitiesRequest{})
if err != nil {
return fmt.Errorf("get capabilities: %w", err)
}

return nil
}

func (c *Client) Put(ctx context.Context, params PutParams) (io.WriteCloser, error) {
md := metadata.Pairs(
"authorization", fmt.Sprintf("bearer %s", c.authConfig.AuthToken),
Expand All @@ -36,11 +45,6 @@ func (c *Client) Put(ctx context.Context, params PutParams) (io.WriteCloser, err
}
ctx = metadata.NewOutgoingContext(ctx, md)

_, err := c.capabilitiesClient.GetCapabilities(ctx, &remoteexecution.GetCapabilitiesRequest{})
if err != nil {
return nil, fmt.Errorf("get capabilities: %w", err)
}

stream, err := c.bitriseKVClient.Put(ctx)
if err != nil {
return nil, fmt.Errorf("initiate put: %w", err)
Expand Down
5 changes: 5 additions & 0 deletions internal/xcode/batch_upload.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ func UploadDerivedDataFilesToBuildCache(dd DerivedData, cacheURL string, authCon
return fmt.Errorf("new KV client: %w", err)
}

err = kvClient.GetCapabilities(ctx)
if err != nil {
return fmt.Errorf("failed to get capabilities: %w", err)
}

missingBlobs, err := findMissingBlobs(ctx, dd, kvClient, logger)
if err != nil {
return fmt.Errorf("failed to check for missing blobs: %w", err)
Expand Down
5 changes: 5 additions & 0 deletions internal/xcode/upload.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ func UploadToBuildCache(filePath, key, cacheURL string, authConfig common.CacheA
return fmt.Errorf("new kv client: %w", err), false
}

err = kvClient.GetCapabilities(ctx)
if err != nil {
return err, false
}

fileSize, err := uploadFile(ctx, kvClient, filePath, key, checksum, logger)
logger.Infof("(i) Uploaded: %s", humanize.Bytes(uint64(fileSize)))

Expand Down