Avoid populating _schema if the collection is empty in MongoDB connector #21236

build2create · 2024-03-25T13:35:36Z

Description

This PR is to fix the [issue][1]. Whenever a empty collection was created an entry was populated in default collection _schema. Further upon populating the collection with documents the fields in _schema remained empty. So a workaround there is a check introduced to avoid populating _schema if collection is empty.

Fixes #20972

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

cla-bot · 2024-03-25T13:35:39Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Mithilesh Wachasunder.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

ebyhr · 2024-03-25T13:37:35Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoSession.java

@@ -878,7 +878,7 @@ private List<Document> guessTableFields(String schemaName, String tableName)
 {
 MongoDatabase db = client.getDatabase(schemaName);
 Document doc = db.getCollection(tableName).find().first();
- if (doc == null) {
+ if (doc == null || doc.isEmpty()) {


Please add a test to TestMongoConnectorTest. We can use MongoClient for preparing an empty collection in MongoDB server.

Sure will do.

cla-bot · 2024-04-01T06:48:15Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Mithilesh Wachasunder.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[Update] Added the UTs

cla-bot · 2024-04-01T06:50:21Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Mithilesh Wachasunder.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

ebyhr · 2024-04-01T06:51:12Z

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java

+ database.getCollection(tableName).insertOne(new Document(ImmutableMap.of("k", "v")));
+ MaterializedResult result = getQueryRunner().execute(getSession(), "SELECT * FROM mongodb.test." + tableName);


We should add another SELECT statement before inserting a document. That's the original scenario we want to fix.

Also, this test doesn't reach doc.isEmpty() as far as I confirmed.

ebyhr · 2024-04-01T06:55:40Z

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java

+ @Test
+ public void testEmptyCollection()
+ {
+ String tableName = "test_empty";


Please append suffix with randomNameSuffix method.

ebyhr · 2024-04-01T06:55:43Z

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java

+ database.createCollection(tableName);
+ database.getCollection(tableName).insertOne(new Document(ImmutableMap.of("k", "v")));
+ MaterializedResult result = getQueryRunner().execute(getSession(), "SELECT * FROM mongodb.test." + tableName);
+ assertThat(result.getRowCount()).isEqualTo(1);


Please drop the table.

ebyhr · 2024-04-04T00:32:36Z

@cla-bot check

cla-bot · 2024-04-04T00:32:38Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Mithilesh Wachasunder.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2024-04-04T00:32:39Z

The cla-bot has been summoned, and re-checked this pull request!

github-actions · 2024-05-10T17:40:15Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

mosabua · 2024-05-10T17:57:33Z

👋 @build2create - are you still interested in progressing this PR? If yes, could you address the comments from @ebyhr and fix any build failures?

github-actions · 2024-06-04T17:03:45Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

build2create · 2024-06-10T18:01:53Z

are you still interested in progressing this PR? If yes, could you address the comments from @ebyhr and fix any build failures?

Yes sorry its been a while will resume on this. Based on the offline chat with @ebyhr seems like a different behaviour with Mongo Compass and Java Client so the test results are different.

ebyhr · 2024-06-18T04:02:59Z

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java

+ String tableName = "test_empty";
+ MongoDatabase database = client.getDatabase("test");
+ database.createCollection(tableName);
+ database.getCollection(tableName).insertOne(new Document(ImmutableMap.of("k", "v")));


#20972 issue can be reproduced once you replace the data with new Document().

github-actions bot added the mongodb MongoDB connector label Mar 25, 2024

build2create mentioned this pull request Mar 25, 2024

Don't populate _schema if the collection is empty in MongoDB connector #20972

Open

build2create requested a review from ebyhr March 25, 2024 13:36

ebyhr reviewed Mar 25, 2024

View reviewed changes

ebyhr assigned build2create Mar 29, 2024

github-actions bot added release-notes docs tests:hive iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector bigquery BigQuery connector labels Apr 1, 2024

build2create marked this pull request as draft April 1, 2024 06:50

Mithilesh Wachasunder added 2 commits April 1, 2024 15:50

Avoid populating schema if the collection is empty in MongoDB connector

eb1425a

Avoid populating schema if the collection is empty in MongoDB connector

190641d

[Update] Added the UTs

ebyhr force-pushed the mongodb_fix branch from e73f67d to 190641d Compare April 1, 2024 06:50

ebyhr reviewed Apr 1, 2024

View reviewed changes

build2create marked this pull request as ready for review April 1, 2024 06:51

ebyhr reviewed Apr 1, 2024

View reviewed changes

findepi removed the tests:hive label Apr 18, 2024

github-actions bot added the stale label May 10, 2024

github-actions bot removed the stale label May 13, 2024

github-actions bot added the stale label Jun 4, 2024

github-actions bot removed the stale label Jun 11, 2024

ebyhr reviewed Jun 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid populating _schema if the collection is empty in MongoDB connector #21236

Avoid populating _schema if the collection is empty in MongoDB connector #21236

build2create commented Mar 25, 2024 •

edited by ebyhr

Loading

cla-bot bot commented Mar 25, 2024

ebyhr Mar 25, 2024 •

edited

Loading

build2create Mar 25, 2024

cla-bot bot commented Apr 1, 2024

cla-bot bot commented Apr 1, 2024

ebyhr Apr 1, 2024 •

edited

Loading

ebyhr Apr 1, 2024 •

edited

Loading

ebyhr Apr 1, 2024

ebyhr commented Apr 4, 2024

cla-bot bot commented Apr 4, 2024

cla-bot bot commented Apr 4, 2024

github-actions bot commented May 10, 2024

mosabua commented May 10, 2024

github-actions bot commented Jun 4, 2024

build2create commented Jun 10, 2024

ebyhr Jun 18, 2024

		database.getCollection(tableName).insertOne(new Document(ImmutableMap.of("k", "v")));
		MaterializedResult result = getQueryRunner().execute(getSession(), "SELECT * FROM mongodb.test." + tableName);

Avoid populating _schema if the collection is empty in MongoDB connector #21236

Are you sure you want to change the base?

Avoid populating _schema if the collection is empty in MongoDB connector #21236

Conversation

build2create commented Mar 25, 2024 • edited by ebyhr Loading

Description

Release notes

cla-bot bot commented Mar 25, 2024

ebyhr Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

build2create Mar 25, 2024

Choose a reason for hiding this comment

cla-bot bot commented Apr 1, 2024

cla-bot bot commented Apr 1, 2024

ebyhr Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

ebyhr Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

ebyhr Apr 1, 2024

Choose a reason for hiding this comment

ebyhr commented Apr 4, 2024

cla-bot bot commented Apr 4, 2024

cla-bot bot commented Apr 4, 2024

github-actions bot commented May 10, 2024

mosabua commented May 10, 2024

github-actions bot commented Jun 4, 2024

build2create commented Jun 10, 2024

ebyhr Jun 18, 2024

Choose a reason for hiding this comment

build2create commented Mar 25, 2024 •

edited by ebyhr

Loading

ebyhr Mar 25, 2024 •

edited

Loading

ebyhr Apr 1, 2024 •

edited

Loading

ebyhr Apr 1, 2024 •

edited

Loading