Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: MongoDB Adapter #8

Merged
merged 46 commits into from
May 17, 2021
Merged

Feat: MongoDB Adapter #8

merged 46 commits into from
May 17, 2021

Conversation

kodumbeats
Copy link
Contributor

This PR adds a MongoDB database adapter

@kodumbeats
Copy link
Contributor Author

@eldadfux I could use your advice going forward:

When fetching documents, Mongo client SDK returns everything in a MongoDB\Model\BSONDocument object:
https://docs.mongodb.com/php-library/v1.2/reference/bson/#phpclass.MongoDB\Model\BSONDocument

What'd be the best approach to parse these objects and get back the PHP array? I've found I can go one level deep with iterator_to_array, but the array contains BSON models as well (MongoDB\Model\BSONArray and MongoDB\BSON\ObjectId) - looks like this:

array(10) {
  ["_id"]=>
  object(MongoDB\BSON\ObjectId)#167 (1) {
    ["oid"]=>
    string(24) "60945e59acf8b169f83ce597"
  }
  ["+id"]=>
  string(9) "documents"
  ["+read"]=>
  object(MongoDB\Model\BSONArray)#164 (1) {
    ["storage":"ArrayObject":private]=>
    array(1) {
      [0]=>
      string(1) "*"
    }
  }
  ["+write"]=>
  object(MongoDB\Model\BSONArray)#169 (1) {
    ["storage":"ArrayObject":private]=>
    array(1) {
      [0]=>
      string(1) "*"
    }
  }
  ["name"]=>
  string(9) "documents"
  ["attributes"]=>
  string(2) "[]"
  ["indexes"]=>
  string(2) "[]"
  ["attributesInQueue"]=>
  string(2) "[]"
  ["indexesInQueue"]=>
  string(2) "[]"
  ["+collection"]=>
  string(11) "collections"
}

@TorstenDittmann
Copy link
Contributor

@eldadfux I could use your advice going forward:

When fetching documents, Mongo client SDK returns everything in a MongoDB\Model\BSONDocument object:
https://docs.mongodb.com/php-library/v1.2/reference/bson/#phpclass.MongoDB\Model\BSONDocument

What'd be the best approach to parse these objects and get back the PHP array? I've found I can go one level deep with iterator_to_array, but the array contains BSON models as well (MongoDB\Model\BSONArray and MongoDB\BSON\ObjectId) - looks like this:

array(10) {
  ["_id"]=>
  object(MongoDB\BSON\ObjectId)#167 (1) {
    ["oid"]=>
    string(24) "60945e59acf8b169f83ce597"
  }
  ["+id"]=>
  string(9) "documents"
  ["+read"]=>
  object(MongoDB\Model\BSONArray)#164 (1) {
    ["storage":"ArrayObject":private]=>
    array(1) {
      [0]=>
      string(1) "*"
    }
  }
  ["+write"]=>
  object(MongoDB\Model\BSONArray)#169 (1) {
    ["storage":"ArrayObject":private]=>
    array(1) {
      [0]=>
      string(1) "*"
    }
  }
  ["name"]=>
  string(9) "documents"
  ["attributes"]=>
  string(2) "[]"
  ["indexes"]=>
  string(2) "[]"
  ["attributesInQueue"]=>
  string(2) "[]"
  ["indexesInQueue"]=>
  string(2) "[]"
  ["+collection"]=>
  string(11) "collections"
}

https://stackoverflow.com/a/61972048

Did you try this?

@kodumbeats kodumbeats marked this pull request as ready for review May 11, 2021 20:23
src/Database/Adapter/MongoDB.php Outdated Show resolved Hide resolved
src/Database/Database.php Show resolved Hide resolved
@kodumbeats kodumbeats changed the base branch from v0 to main May 13, 2021 13:19
src/Database/Adapter/MongoDB.php Show resolved Hide resolved
src/Database/Adapter/MongoDB.php Outdated Show resolved Hide resolved
$options = [];

// set max limit
$options['limit'] = $max;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to make sure this limits the number of documents the DB scans and not just the number of results returned (which is always 1 anyway).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure on this one:
According to the mongo shell docs:

Field Type Description
limit integer Optional. The maximum number of documents to count.

But for the PHP lib implementation of the same method:

Field Type Description
limit integer Optional. The maximum number of matching documents to return.

I think all implementations just limit the number of documents passed to the next step in the pipeline:
https://docs.mongodb.com/manual/reference/operator/aggregation/limit/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can raise an issue on the Mongo SDK repo and ask. I feel like this is just a doc mistake

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, limit was the wrong setting. After more research, I have good news and bad news 😐

Good news: I found the exact setting we need for our desired count() and find() methods: maxScan:

maxScan Optional. Maximum number of documents or index keys to scan when executing the query.

Bad news: it was deprecated in Mongo 4.0 in favor of time-based limits:

MongoDB deprecates the option maxScan for the find command and the mongo shell helper cursor.maxScan(). Instead, use maxTimeMS option or the helper cursor.maxTimeMS().

https://docs.mongodb.com/manual/release-notes/4.0-compatibility/#maxscan-option

Since the behavior was specifically deprecated by the Mongo team, I'm not sure we'll be able to hack our way around it.

src/Database/Adapter/MongoDB.php Show resolved Hide resolved
src/Database/Adapter/MongoDB.php Show resolved Hide resolved
@eldadfux eldadfux merged commit b0c4e4f into utopia-php:main May 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants