Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Publish the list of partition ids on TABLE_PARTITION_UPDATE event #449

Merged
merged 2 commits into from
Jul 29, 2021
Merged

Publish the list of partition ids on TABLE_PARTITION_UPDATE event #449

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d62650
Publish the list of partition ids on TABLE_PARTITION_UPDATE event
insyncoss Jul 19, 2021
a9858f0
Publish the list of partition ids on TABLE_PARTITION_UPDATE event
insyncoss Jul 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Publish the list of partition ids on TABLE_PARTITION_UPDATE event
  • Loading branch information
@insyncoss
insyncoss committed Jul 29, 2021
commit a9858f0ed4cdfe4589e7259d38b72d74e69209b2
3 changes: 2 additions & 1 deletion ...test/groovy/com/netflix/metacat/common/dto/notifications/sns/SNSMessageFactorySpec.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,8 @@ class SNSMessageFactorySpec extends Specification {
UUID.randomUUID().toString(),
10,
15,
""
"",
Arrays.asList(UUID.randomUUID().toString())
)
)
) | UpdateTablePartitionsMessage.class
Expand Down
18 changes: 11 additions & 7 deletions .../java/com/netflix/metacat/main/services/notifications/sns/SNSNotificationServiceUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
import java.text.SimpleDateFormat;
import java.time.Instant;
import java.util.Arrays;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
Expand Down Expand Up @@ -60,7 +61,7 @@ public final class SNSNotificationServiceUtil {
//ISO basic date format: 20180101
private static final Pattern TIMESTAMP_FORMAT = Pattern.compile("^(?<time>\\d{10})(?:\\d{3})?(?:\\.\\d+)?$");
private static final Pattern ISO_BASIC = Pattern.compile("^\\d{8}$");
private static final int PARTS_UPDATED_LIST_MAX_SIZE = 1000;
private static final int PARTITIONS_UPDATED_LIST_MAX_SIZE = 1000;
private UserMetadataService userMetadataService;
private final DateFormat simpleDateFormatRegional = new SimpleDateFormat("yyyyMMdd");
private final DateFormat simpleDateFormatUTC = new SimpleDateFormat("yyyyMMdd");
Expand Down Expand Up @@ -179,17 +180,20 @@ private static List<String> getSortedDeletionPartitionKeys(final List<PartitionD
}

/**
* get partition name list from list of partitionDtos
* get partition name list from list of partitionDtos. The returned list is capped at
* the first PARTITIONS_UPDATED_LIST_MAX_SIZE elements, if there are more than that number of elements
* in the input then the return is empty which serves as a signal that complete list cannot be included
*
* @param partitionDtos partition DTOs
* @return descending order deletion column
* @return list of partition ids from the input list
*/
@VisibleForTesting
protected static List<String> getPartitionNameListFromDtos(final List<PartitionDto> partitionDtos) {
if (partitionDtos.size() > PARTITIONS_UPDATED_LIST_MAX_SIZE) {
// empty list signals
return Collections.emptyList();
}
return partitionDtos.stream()
.limit(PARTS_UPDATED_LIST_MAX_SIZE)
.map(PartitionDto::getName)
.map(QualifiedName::getPartitionName)
.map(dto -> dto.getName().getPartitionName())
.collect(Collectors.toList());
}

Expand Down