[Failing Test]: BigtableIOWriteTest::test_bigtable_write is about 50% flaky

[tvalentyn]

What happened?

From #30867 (comment) and below, it appears that BigTable client initialization sometimes gets stuck and holding GIL indefinitely in:

Traceback for thread 100 (python) [Has the GIL] (most recent call last):
    (Python) File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
        self._bootstrap_inner()
    (Python) File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
        self.run()
    (Python) File "/usr/local/lib/python3.8/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
    (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
        work_item.run()
    (Python) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
    (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/batcher.py", line 385, in _flush_rows
        response = self.table.mutate_rows(rows_to_flush)
    (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 724, in mutate_rows
        self.name,
    (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/table.py", line 160, in name
        table_client = self._instance._client.table_data_client
    (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 332, in table_data_client
        transport = self._create_gapic_client_channel(
    (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable/client.py", line 285, in _create_gapic_client_channel
        channel = grpc_transport.create_channel(
    (Python) File "/usr/local/lib/python3.8/site-packages/google/cloud/bigtable_v2/services/bigtable/transports/grpc.py", line 217, in create_channel
        return grpc_helpers.create_channel(
    (Python) File "/usr/local/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 386, in create_channel
        return grpc.secure_channel(
    (Python) File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 2119, in secure_channel
        return _channel.Channel(
    (Python) File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 2046, in __init__
        self._channel = cygrpc.Channel(

Repro:

python -m pytest  -o log_cli=True -o log_level=Info apache_beam/examples/cookbook/bigtableio_it_test.py::BigtableIOWriteTest::test_bigtable_write   --test-pipeline-options='--runner=TestDataflowRunner --project=apache-beam-testing --temp_location=valentyn-testing  --region=us-central1 --wait_until_finish_duration=36000000' --timeout=36000

I wasn't able to repro the error locally outside of Dataflow context.

cc: @mutianf FYI given that BT client is involved. It might be a wider issue, as discussed in #30867 and grpc/grpc#36256.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[liferoad]

@mutianf Can you check this?

[tvalentyn]

gdb backtrace of the stuck that thread holds up GIL: grpc/grpc#36256 (comment)

[tvalentyn]

fixed in #31044