Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/mongodb conditional updates #430

Merged
merged 15 commits into from
Nov 15, 2018
Merged

Conversation

amarziali
Copy link
Contributor

Several improvements here:

  • MongoDB: Allow to use custom condition for upserts
  • MongoDB: Fix potentials data loss (also reported by MongoDB and Solr Service sinks may retain some records under some conditions #409)
  • Framework: introduce datatime field type
  • Framework: extend the EL to use accessors (e.g. maps and nested types)
  • Serializers: Use Avro 1.8.2
  • Serializers: Allow ExtendedJsonSerializer to interpret date/time/datetime from an avro schema
  • Processors: ExpandMapField processor to flatten map field to the record root
  • HDFS: Allow hdfs burner to gather raw data and interpret as a json by joining dataframe with record type and datetime in order to have expected information needed for partitioning partitioning

@amarziali amarziali requested a review from oalam November 9, 2018 15:24
@oalam oalam merged commit 071cd2d into develop Nov 15, 2018
@oalam oalam deleted the feature/mongodb-conditional-updates branch November 15, 2018 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants