Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Pyarrow version to include 14.0.0 #29536

Merged
merged 7 commits into from
Dec 3, 2023

Conversation

AnandInguva
Copy link
Contributor

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@AnandInguva
Copy link
Contributor Author

AnandInguva commented Nov 27, 2023

Errors I see in https://github.com/apache/beam/actions/runs/7010246192/job/19070396821?pr=29536

self = <apache_beam.io.parquetio_test.TestParquet testMethod=test_sink_transform_int96>

    def test_sink_transform_int96(self):
      with tempfile.NamedTemporaryFile() as dst:
        path = dst.name
        # pylint: disable=c-extension-no-member
        with self.assertRaises(pl.ArrowInvalid):
          # Should throw an error "ArrowInvalid: Casting from timestamp[ns] to
          # timestamp[us] would lose data"
          with TestPipeline() as p:
>           _ = p \
            | Create(self.RECORDS) \
            | WriteToParquet(
                path, self.SCHEMA96, num_shards=1, shard_name_template='')
E           AssertionError: ArrowInvalid not raised

The above test fail could be because of apache/arrow#35656


Windows tests are running for very long time -https://github.com/apache/beam/actions/runs/7010335606/job/19070644422?pr=29536. This might be due to pyarrow update.

C:\arrow\cpp\src\arrow\filesystem\s3fs.cc:2904:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit

@AnandInguva
Copy link
Contributor Author

#28437 (comment) - following up on the comment here, I will be skipping updates for windows.

@github-actions github-actions bot added the io label Nov 29, 2023
@AnandInguva
Copy link
Contributor Author

R: @tvalentyn

@AnandInguva AnandInguva marked this pull request as ready for review November 30, 2023 18:35
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

sdks/python/setup.py Outdated Show resolved Hide resolved
@tvalentyn
Copy link
Contributor

Thanks!

@AnandInguva AnandInguva merged commit f7ae6b4 into apache:master Dec 3, 2023
78 checks passed
@AnandInguva AnandInguva linked an issue Dec 6, 2023 that may be closed by this pull request
15 tasks
Naireen pushed a commit to Naireen/beam that referenced this pull request Jan 3, 2024
* generate requirements file

* update setup.py pyarrow

* Update pyarrow except for windows

* Skip tests for version 13 or greater.

* Add reason for unittest skipIf

* Fix pyarrow lower bound

* Update sdks/python/setup.py

Co-authored-by: tvalentyn <[email protected]>

---------

Co-authored-by: tvalentyn <[email protected]>
nji302 pushed a commit to lyft/beam that referenced this pull request Jul 24, 2024
* generate requirements file

* update setup.py pyarrow

* Update pyarrow except for windows

* Skip tests for version 13 or greater.

* Add reason for unittest skipIf

* Fix pyarrow lower bound

* Update sdks/python/setup.py

Co-authored-by: tvalentyn <[email protected]>

---------

Co-authored-by: tvalentyn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support new versions of pyarrow in apache-beam
2 participants