[exporter | receiver/googlecloudpubsub] Initial implementation #8391
Conversation
|
Sorry, had no time (it's my thing to do on the weekend) to update the previous PR so it was autoclosed. @fbogsany @dashpole @MovieStoreGuy Same PR, but main merged in. And the last is fixes on remarks on the previous PR. |
| } | ||
|
|
||
| func (config *WatermarkConfig) validate() error { | ||
| if config.AllowedDrift == 0 { |
There was a problem hiding this comment.
config defaults should be set in createDefaultConfig(), rather than validate.
| switch config.Compression { | ||
| case "gzip": | ||
| return gZip, nil | ||
| case "": |
There was a problem hiding this comment.
nit: it might be nice to have an explicit "uncompressed" value, and default to that in createDefaultConfig(). That way you can separate defaulting vs parsing of config.
| return earliest, nil | ||
| case "current": | ||
| return current, nil | ||
| case "": |
There was a problem hiding this comment.
nit: (similar to above) can we default to "current" in createDefaultConfig(), so we don't need to handle this as a separate case or do defaulting here?
Co-authored-by: David Ashpole <[email protected]>
|
@dashpole I have added you're commit + rebase (to avoid merge conflict). I will add your improvements on the defaults to the list of things todo. Could you ping someone to merge (I don't have the authority to merge). Thanks. |
jpkrohling
changed the title
…telemetry#8391) * Add Google Pubsub as an OTLP exporter/receiver * Fix Pull Request comment on the GoogleCloudPububExporter * Fix Pull Request comment on the GoogleCloudPububReceiver * GoogleCloudPubsub linting * Changed the behaviour of allow_drift and make config fields private * Added documentation on the compression config * Update CHANGELOG.md Co-authored-by: David Ashpole <[email protected]> Co-authored-by: David Ashpole <[email protected]>
Description:
Add the implementation for Google Pubsub exporter and receiver for OTLP messages. The export is able to send OTEP messages to a Google Pubsub topic. The receiver and exporter are pushed in one PR as they work in pairs. A typical scenario is having an ingestion cluster (gRPC) pushing messages to Pubsub and then having receivers read again from Pubsub, having Pubsub as a buffer. That's why it doesn't make sense to have them reviewed separately.
This PR replaces the following PR:
#3879
#7172
The implementation has drifted and also includes several enhancements, it doesn't make sense to put it over the other PR.
The implementation is currently in use on production at Collibra.
Link to tracking Issue:
#1802
Testing:
exporter 93.1% test coverage
receiver 83.4% test coverage
Documentation:
Up-to-date README