-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-2311] Update compatibility matrix Aggregators section for Metrics #248
Conversation
Refer to this link for build results (access rights to CI server needed): Jenkins built the site at commit id 9ea5300 with Jekyll and staged it here. Happy reviewing. Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again. |
Looks good re: What Spark runner supports. The terms logical and physical may confuse users here, especially since the API refers to these by methods |
Refer to this link for build results (access rights to CI server needed): Jenkins built the site at commit id 6625d21 with Jekyll and staged it here. Happy reviewing. Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again. |
src/_data/capability-matrix.yml
Outdated
values: | ||
- class: model | ||
l1: 'Partially' | ||
l2: user-provided metrics | ||
l3: Allow transforms to aggregate simple metrics across bundles in a <tt>DoFn</tt>. Semantically equivalent to using an additional output, but support partial results as the transform executes. Will likely want to augment <tt>Aggregators</tt> to be more useful for processing unbounded data by making them windowed. | ||
l3: Allow transforms to gather simple metrics across bundles in a <tt>PTransform</tt>. Provide a mechanism to obtain both committed and attempted metrics. Semantically equivalent to using an additional output, but support partial results as the transform executes. Will likely want to augment <tt>Metrics</tt> to be more useful for processing unbounded data by making them windowed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Committed metrics are semantically equivalent to using an additional output, right? Not attempted.
src/_data/capability-matrix.yml
Outdated
l2: may miscount in streaming mode | ||
l3: Current model is fully supported in batch mode. In streaming mode, <tt>Aggregators</tt> may under or overcount when bundles are retried. | ||
l2: Gauge metric type is not supported. | ||
l3: Current model is fully supported for Counter and Distribution metrics. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Metrics do not work in streaming mode?
Addressed Dan's comments. |
Refer to this link for build results (access rights to CI server needed): Jenkins built the site at commit id e00ffd5 with Jekyll and staged it here. Happy reviewing. Note that any previous site has been deleted. This staged site will be automatically deleted after its TTL expires. Push any commit to the pull request branch or re-trigger the build to get it staged again. |
src/_data/capability-matrix.yml
Outdated
- class: dataflow | ||
l1: 'Partially' | ||
l2: Gauge metric type is not supported. | ||
l3: Current model is fully supported for Counter and Distribution metrics. | ||
l2: Batch supports committed Counters and Distributions. No streamming support. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo in streaming
.
- class: dataflow | ||
l1: 'Partially' | ||
l2: may miscount in streaming mode | ||
l3: Current model is fully supported in batch mode. In streaming mode, <tt>Aggregators</tt> may under or overcount when bundles are retried. | ||
l2: Batch supports committed Counters and Distributions. No streamming support. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add link to JIRA, with some text like "coming soon [BEAM-XXX]"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you prefer, we can merge and make the minor fixes after – up to you.
r: @dhalperi @aviemzur @JingsongLi
Feel free to suggest rephrasing of the descriptions. I'll incorporate it.