Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexing for model fields #221

Merged
merged 10 commits into from
Jan 21, 2020
Merged

Indexing for model fields #221

merged 10 commits into from
Jan 21, 2020

Conversation

carsonfarmer
Copy link
Member

@carsonfarmer carsonfarmer commented Jan 16, 2020

Building indexes currently does not affect any other tests, and adds a unique constraint on the ID field by default (so this is partial support for Unique). Any other fields can be specified using arbitrary 'paths' (e.g., Book.Title). Currently only JSON mode is supported. Structs could be added later, but this seems good enough for a POC and early tests. This work stores the indexes 'above' any model key paths in the underlying stores, so as to avoid showing up in queries etc. Finally, it requires an EventCodec to call an IndexFunc within the Reduce method as discussed.
Still left to do:

  • Use in Queries
  • Full support for Unique
  • Tests
  • Benchmarking

@carsonfarmer carsonfarmer marked this pull request as ready for review January 21, 2020 19:02
@carsonfarmer carsonfarmer changed the title [WIP] indexing for model fields Indexing for model fields Jan 21, 2020
@carsonfarmer carsonfarmer requested review from sanderpick, andrewxhill and jsign and removed request for andrewxhill January 21, 2020 19:02
@carsonfarmer carsonfarmer self-assigned this Jan 21, 2020
@carsonfarmer carsonfarmer added this to the Sprint 28 milestone Jan 21, 2020
Copy link
Member Author

@carsonfarmer carsonfarmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments for you guys. Be gentle!

jsonpatcher/jsonpatcher.go Show resolved Hide resolved
jsonpatcher/jsonpatcher.go Show resolved Hide resolved
store/README.md Show resolved Hide resolved
store/bench_test.go Show resolved Hide resolved
store/encode.go Show resolved Hide resolved
store/model.go Show resolved Hide resolved
store/query_json.go Outdated Show resolved Hide resolved
store/query_jsonmode_test.go Show resolved Hide resolved
store/store.go Show resolved Hide resolved
store/store.go Outdated Show resolved Hide resolved
Copy link
Contributor

@jsign jsign left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Heavy lifting.

I can see multiple points of improvement, but with some design changes that not sure may worth it. Also, It seems that we have the right extensibility points at different parts of the code.

Some memory benchs sounds good since some extra fetches of old state might hurt a bit in some cases (delete)... but feels that more for an optimization stage.

I think @sanderpick and you have a more clear need of why this was needed now, so maybe I'm missing a better analysis of the original motivation. I think was the unique index part, not sure about being slow for an existing use-case?

api/client/client.go Show resolved Hide resolved
api/client/client.go Outdated Show resolved Hide resolved
store/model.go Show resolved Hide resolved
store/query_json.go Outdated Show resolved Hide resolved
Copy link
Member

@sanderpick sanderpick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice @carsonfarmer 💪 !

Some high-level comments added. @jsign, correct, the immediate use case is to enable something like, give me the user with a given email. So, the primary focus is the unique field bit.

api/client/client.go Outdated Show resolved Hide resolved
api/client/client.go Show resolved Hide resolved
api/client/client_test.go Show resolved Hide resolved
store/model.go Show resolved Hide resolved
store/model.go Show resolved Hide resolved
Copy link
Member Author

@carsonfarmer carsonfarmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey gang, just one or two points of clarification and then I'll push two fixes before merging.

@carsonfarmer
Copy link
Member Author

Ok, I think all nits etc have been addressed. Ready for merge.

@carsonfarmer carsonfarmer added documentation Improvements or additions to documentation enhancement New feature or request labels Jan 21, 2020
@@ -96,7 +96,7 @@ func (c *Client) NewStore(ctx context.Context) (string, error) {
}

// RegisterSchema registers a new model shecma
func (c *Client) RegisterSchema(ctx context.Context, storeID, name, schema string, indexes []*store.IndexConfig) error {
func (c *Client) RegisterSchema(ctx context.Context, storeID, name, schema string, indexes ...*store.IndexConfig) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@carsonfarmer carsonfarmer merged commit a69c030 into master Jan 21, 2020
@carsonfarmer carsonfarmer deleted the carson/indexing branch January 21, 2020 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants