i#6685: Add core-sharded-on-disk analysis support #6680
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add core-sharded-on-disk analysis support Adds a new filetype OFFLINE_FILE_TYPE_CORE_SHARDED set by record_filter when core-sharded. Has the scheduler read ahead to the filetype in every input, and arrange to make it available at init time to output streams. The analyzer looks for the filetype and sets the shard type as core-sharded if set. This is different from a dynamic-scheduling core-sharded as we do not need a 1:1 shard:workers mapping. Allows header records in reader_t::process_input_entry(), which are now expected with core-sharded-on-disk traces. Adds core-sharded reader support for seeing only portions of some threads, where we have to relax some tid and pid checks. Checks in 6 core-sharded-on-disk x64 threadsig traces for use in a test of basic_counts analyzing them. That analysis fails without all of these framework changes. The changes to the record filter to support generating those traces in the first place will come in separately. Issue: #6635
… left here in part 3
…ing in scheduler init for online. Use output ords to avoid ord discrepancies from new read-ahead for parallel analysis mode.
…d for the filetype: + Breaks unit tests which have no filetype records => stopping readahead at pagesize marker. + Breaks replay-as-traced b/c input record ord is too far => disabling read-ahead for as-traced. + Breaks record filter legacy null check b/c legacy trace has no filetype and it reads the timestamp which means the stream's last-timestamp is now > stop-timestamp and so the resulting output trace has the filter-end marker 1st thing! => Made get_*_timestamp use output until output sees 1st timestamp. + Skip tests fail b/c timestamp has 1 instead of 120 ord -- coming from the change above to use the output ord until the 1st instr! => using last_timestamp_ > 0 instead. If we had metadata w/ the filetype instead of reading ahead and messing up the input ords that would be great, but we don't.
…re-on-disk-analysis
…to handle nondet core ordinal order
…_on_disk failure" This reverts commit b213ab3.
…k test w/ #workers<#shards
|
This has been a real nightmare to get all the modes and tests working: there is some fragility in the infrastructure for sure. Finally have everything passing except the invariant checker pthreads test on the aarch64-sve runner: but I cannot reproduce that failure anywhere else including aarch64 machines I have access to; the failure itself is one I saw a number of commits back on this same test locally and fixed so it is weird it shows up still on this one machine. I don't know that tmate works on self-hosted runners? |
This reverts commit 93f9e6b.
…re-on-disk-analysis
derekbruening
changed the title
abhinav92003
approved these changes
Feb 28, 2024
+ Restore basic_counts thread exit check + Add quite a few comments clarifying and explaining + Adjust interface docs to remove internal details
…re-on-disk-analysis
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add core-sharded-on-disk analysis support:
Adds a new filetype OFFLINE_FILE_TYPE_CORE_SHARDED set by record_filter when core-sharded.
Has the scheduler read ahead to the filetype in every input, and arrange to make it available at init time to output streams.
The analyzer looks for the filetype and sets the shard type as core-sharded if set. This is different from a dynamic-scheduling core-sharded as we do not need a 1:1 shard:workers mapping.
Allows header records in reader_t::process_input_entry(), which are now expected with core-sharded-on-disk traces.
Adds core-sharded reader support for seeing only portions of some threads, where we have to relax some tid and pid checks.
Checks in 6 core-sharded-on-disk x64 threadsig traces for use in a test of basic_counts analyzing them. That analysis fails without all of these framework changes.
The changes to the record filter to support generating those traces in the first place will come in separately.
Issue: #6685, #6635
Fixes #6685