Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: introduce the interface of RemoteJobScheduler #4181

Merged
Merged
Changes from 1 commit
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
0d5b790
refactor: add Compactor trait
zyy17 Jun 2, 2024
a91c5d6
chore: add compact() in Compactor trait and expose compaction module
zyy17 Jun 2, 2024
37b0464
refactor: add CompactionRequest and open_compaction_region
zyy17 Jun 2, 2024
014bc22
refactor: export the compaction api
zyy17 Jun 3, 2024
f9d1e66
refactor: add DefaultCompactor::new_from_request
zyy17 Jun 3, 2024
3eb0dab
refactor: no need to pass mito_config in open_compaction_region()
zyy17 Jun 3, 2024
8cd55a2
refactor: CompactionRequest -> &CompactionRequest
zyy17 Jun 3, 2024
891426d
fix: typo
zyy17 Jun 3, 2024
8ff9c6f
docs: add docs for public apis
zyy17 Jun 4, 2024
acabc07
refactor: remove 'Picker' from Compactor
zyy17 Jun 4, 2024
de8496c
chore: add logs
zyy17 Jun 4, 2024
6bee93e
chore: change pub attribute for Picker
zyy17 Jun 4, 2024
a63c9f5
refactor: remove do_merge_ssts()
zyy17 Jun 4, 2024
3a360c0
refactor: update comments
zyy17 Jun 4, 2024
35feb02
refactor: use CompactionRegion argument in Picker
zyy17 Jun 5, 2024
55033f9
chore: make compaction module public and remove unnessary clone
zyy17 Jun 6, 2024
15c102c
refactor: move build_compaction_task() in CompactionScheduler{}
zyy17 Jun 6, 2024
8ad6fff
chore: use in open_compaction_region() and add some comments for pub…
zyy17 Jun 6, 2024
26ae2a9
refactor: add 'manifest_dir()' in store-api
zyy17 Jun 6, 2024
9a05b6a
refactor: move the default implementation to DefaultCompactor
zyy17 Jun 6, 2024
130fb90
refactor: remove Options from MergeOutput
zyy17 Jun 6, 2024
e141e91
chore: minor modification
zyy17 Jun 6, 2024
e3e6f12
fix: clippy errors
zyy17 Jun 6, 2024
93f5a74
fix: unit test errors
zyy17 Jun 6, 2024
17bf852
refactor: remove 'manifest_dir()' from store-api crate(already have o…
zyy17 Jun 6, 2024
48f9398
refactor: use 'region_dir' in CompactionRequest
zyy17 Jun 6, 2024
2f6d9ef
refactor: refine naming
zyy17 Jun 6, 2024
861ea41
chore: sync main branch
zyy17 Jun 7, 2024
d26c64a
refactor: refine naming
zyy17 Jun 7, 2024
4968186
refactor: remove clone()
zyy17 Jun 7, 2024
574fd9f
chore: add comments
zyy17 Jun 7, 2024
77c085c
refactor: add PickerOutput field in CompactorRequest
zyy17 Jun 7, 2024
f8d88a2
chore: sync main branch
zyy17 Jun 16, 2024
f92c729
feat: introduce RemoteJobScheduler
zyy17 Jun 7, 2024
c0c7b6d
feat: add RemoteJobScheudler in schedule_compaction_request()
zyy17 Jun 7, 2024
84a8ff7
refactor: use Option type for senders field of CompactionFinished
zyy17 Jun 8, 2024
f421d08
refactor: modify CompactionJob
zyy17 Jun 8, 2024
6f24668
refactor: schedule remote compaction job by options
zyy17 Jun 9, 2024
cbe2115
refactor: remove unused Options
zyy17 Jun 9, 2024
66ee079
build: remove unused log
zyy17 Jun 9, 2024
3c183ff
refactor: fallback to local compaction if the remote compaction failed
zyy17 Jun 9, 2024
90a4794
fix: clippy errors
zyy17 Jun 17, 2024
c67b9ad
chore: sync main branch
zyy17 Jun 17, 2024
0f5c4d3
refactor: add plugins in mito2
zyy17 Jun 17, 2024
6ffdae2
refactor: add from_u64() for JobId
zyy17 Jun 18, 2024
06fc1e4
refactor: make schedule module public
zyy17 Jun 18, 2024
01b20b2
refactor: add error for RemoteJobScheduler
zyy17 Jun 18, 2024
b740c41
refactor: add Notifier
zyy17 Jun 18, 2024
4404973
refactor: use Arc for Notifier
zyy17 Jun 18, 2024
25bfdf9
chore: sync main branch
zyy17 Jun 19, 2024
1d6efa4
refactor: add 'remote_compaction' in compaction options
zyy17 Jun 19, 2024
07459d3
fix: clippy errors
zyy17 Jun 19, 2024
d08b56e
fix: unrecognized table option
zyy17 Jun 19, 2024
840e642
refactor: add 'start_time' in CompactionJob
zyy17 Jun 22, 2024
fc0e927
refactor: modify error type of RemoteJobScheduler
zyy17 Jun 22, 2024
2731e76
Merge branch 'main' into feat/add-experimental-remote-job-scheduler
zyy17 Jun 24, 2024
1a8a742
chore: revert changes for request
zyy17 Jun 26, 2024
bf55576
refactor: code refactor by review comment
zyy17 Jun 26, 2024
617f8f4
refactor: use string type for JobId
zyy17 Jun 26, 2024
f220543
Merge branch 'main' into feat/add-experimental-remote-job-scheduler
zyy17 Jun 26, 2024
2f92206
refactor: add 'waiters' field in DefaultNotifier
zyy17 Jun 26, 2024
fbef517
fix: build error
zyy17 Jun 26, 2024
613edc8
refactor: take coderabbit's review comment
zyy17 Jun 27, 2024
4e3ccdf
refactor: use uuid::Uuid as JobId
zyy17 Jun 27, 2024
d299592
refactor: return waiters when schedule failed and add on_failure for …
zyy17 Jun 27, 2024
ca1c8fa
refactor: move waiters from notifier to Job
zyy17 Jun 27, 2024
5246a22
refactor: use ObjectStoreManagerRef in open_compaction_region()
zyy17 Jun 28, 2024
a9b2d5e
Merge branch 'main' into feat/add-experimental-remote-job-scheduler
zyy17 Jul 1, 2024
6800383
refactor: implement for JobId and adds related unit tests
zyy17 Jul 1, 2024
6502caa
fix: run unit tests failed
zyy17 Jul 1, 2024
305ee27
Merge branch 'main' into feat/add-experimental-remote-job-scheduler
zyy17 Jul 1, 2024
0f9335d
refactor: add RemoteJobSchedulerError
zyy17 Jul 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: move the default implementation to DefaultCompactor
  • Loading branch information
zyy17 committed Jun 6, 2024
commit 9a05b6a28c326aba52989bf7d4dcab6b42c3fe30
34 changes: 26 additions & 8 deletions src/mito2/src/compaction/compactor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,32 @@ impl MergeOutput {
#[async_trait::async_trait]
pub trait Compactor: Send + Sync + 'static {
/// Merge SST files for a region.
async fn merge_ssts(
&self,
compaction_region: &CompactionRegion,
picker_output: PickerOutput,
) -> Result<MergeOutput>;

/// Update the manifest after merging SST files.
async fn update_manifest(
&self,
compaction_region: &CompactionRegion,
merge_output: MergeOutput,
) -> Result<()>;

/// Execute compaction for a region.
async fn compact(
&self,
compaction_region: &CompactionRegion,
compact_request_options: compact_request::Options,
) -> Result<()>;
}

/// DefaultCompactor is the default implementation of Compactor.
pub struct DefaultCompactor;

#[async_trait::async_trait]
impl Compactor for DefaultCompactor {
async fn merge_ssts(
&self,
compaction_region: &CompactionRegion,
Expand Down Expand Up @@ -324,7 +350,6 @@ pub trait Compactor: Send + Sync + 'static {
})
}

/// Update the manifest after merging SST files.
async fn update_manifest(
&self,
compaction_region: &CompactionRegion,
Expand Down Expand Up @@ -372,7 +397,6 @@ pub trait Compactor: Send + Sync + 'static {
Ok(())
}

/// Execute compaction for a region.
async fn compact(
&self,
compaction_region: &CompactionRegion,
Expand Down Expand Up @@ -407,9 +431,3 @@ pub trait Compactor: Send + Sync + 'static {
self.update_manifest(compaction_region, merge_output).await
}
}

/// DefaultCompactor is the default implementation of Compactor.
pub struct DefaultCompactor;

#[async_trait::async_trait]
impl Compactor for DefaultCompactor {}