TT-3780 Add fixes for Python 3.8/3.9 #3479

matiasinsaurralde · 2021-02-27T20:30:12Z

I've found two breaking changes, first one is related to PyTuple_ClearFreeList which was removed, mentioned here. The mapping for this binding was autogenerated and because we're not using it should be safe to remove it from binding.go.

The second change is related to the python-config tool, 3.8 modified the flags required to retrieve the library path, they suggest implementing the following behavior:

To embed Python into an application, a new --embed option must be passed to python3-config --libs --embed to get -lpython3.8 > (link the application to libpython). To support both 3.8 and older, try python3-config --libs --embed first and fallback to python3-config --libs (without --embed) if the previous command fails.

More info here.

https://tyktech.atlassian.net/browse/TT-3780

sredxny · 2021-02-27T20:37:08Z

working in python 3.9, tested in MacOS 10.15.7

matiasinsaurralde · 2021-02-27T20:43:06Z

@sredxny Nice!

buger · 2021-05-10T19:51:08Z

@matiasinsaurralde lets finally merge it to one of upcoming patches.

matiasinsaurralde · 2022-01-28T01:14:59Z

Rebased with latest master.

…https://docs.python.org/3.9/whatsnew/3.9.html?highlight=pytuple_clearfreelist Improve libdl error handler

… 3.8: https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

Tyk-ITS · 2022-02-02T01:19:34Z

API tests result: success ✅
Branch used: refs/pull/3479/merge
Commit: 214ada0
Triggered by: pull_request (@matiasinsaurralde)
Execution page

Tyk-ITS · 2022-02-02T01:20:16Z

API tests result: success ✅
Branch used: refs/pull/3479/merge
Commit: 06b233f
Triggered by: pull_request (@matiasinsaurralde)
Execution page

tbuchaillot · 2022-02-03T09:08:21Z

dlpython/binding.go

@@ -296,14 +292,10 @@ func mapCalls() error {
 defer C.free(unsafe.Pointer(s_Py_DecRef))
 C.Py_DecRef_ptr = C.Py_DecRef_f(C.dlsym(C.python_lib, s_Py_DecRef))

- s_PyTuple_ClearFreeList := C.CString("PyTuple_ClearFreeList")


@matiasinsaurralde if we delete this aren't we breaking backwards compatibility with the old versions of py which use PyTuple_ClearFreeList?

wouldn't it better to start receiving the python version in mapCalls and ignore this code for >3.8 versions?

@tbuchaillot We actually don't use PyTuple_ClearFreeList in any of the codebase, but as it was removed in one of the Python 3.x., the map call was causing issues. It's safe to remove it.

I think passing the version to mapCalls is interesting but might not be required at this time, in case Python 4 ever lands and breaks any mapping, we can incorporate it 👍

makes sense @matiasinsaurralde 👌

sredxny

lgtm! @matiasinsaurralde any idea how can we add test coverage to it? I mean like create a matrix and test that python plugin load on different python versions? what would this imply?

Tyk-ITS · 2022-02-07T16:38:03Z

API tests result: failure 🚫
Branch used: refs/pull/3479/merge
Commit: 9fa4271
Triggered by: pull_request (@tbuchaillot)
Execution page

Tyk-ITS · 2022-02-09T16:13:17Z

API tests result: success ✅
Branch used: refs/pull/3479/merge
Commit: b5a575c
Triggered by: pull_request (@tbuchaillot)
Execution page

sonarcloud · 2022-02-09T16:35:51Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

56.2% Coverage
0.0% Duplication

tbuchaillot · 2022-02-09T17:10:01Z

/release to release-4

tykbot · 2022-02-09T17:10:22Z

Working on it! Note that it can take a few minutes.

* Remove deprecated function mapping "PyTuple_ClearFreeList", reference: https://docs.python.org/3.9/whatsnew/3.9.html?highlight=pytuple_clearfreelist Improve libdl error handler * Remove PyTuple_ClearFreeList dlpython helper * Display libdl errors when calling PythonInit * Use "--embed" flag when calling python-config, this was introduced in 3.8: https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build * dlpython: ensure backwards compatibility when calling python-config Co-authored-by: Tomas Buchaillot <[email protected]> (cherry picked from commit cc9fcd4)

tykbot · 2022-02-09T17:10:39Z

@tbuchaillot Succesfully merged cc9fcd4c4ea278f68a4b6a321ebb98bc95f86b9b to release-4 branch.

tbuchaillot · 2022-02-09T17:12:40Z

/release to release-4-lts

tykbot · 2022-02-09T17:12:51Z

Working on it! Note that it can take a few minutes.

tykbot · 2022-02-09T17:13:09Z

@tbuchaillot Succesfully merged cc9fcd4c4ea278f68a4b6a321ebb98bc95f86b9b to release-4-lts branch.

* Remove deprecated function mapping "PyTuple_ClearFreeList", reference: https://docs.python.org/3.9/whatsnew/3.9.html?highlight=pytuple_clearfreelist Improve libdl error handler * Remove PyTuple_ClearFreeList dlpython helper * Display libdl errors when calling PythonInit * Use "--embed" flag when calling python-config, this was introduced in 3.8: https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build * dlpython: ensure backwards compatibility when calling python-config Co-authored-by: Tomas Buchaillot <[email protected]> (cherry picked from commit cc9fcd4)

Tyk-ITS · 2022-02-09T17:23:21Z

API tests result: success ✅
Branch used: refs/heads/master
Commit: cc9fcd4 Add fixes for Python 3.8/3.9 (#3479)

Remove deprecated function mapping "PyTuple_ClearFreeList", reference: https://docs.python.org/3.9/whatsnew/3.9.html?highlight=pytuple_clearfreelist

Improve libdl error handler

Remove PyTuple_ClearFreeList dlpython helper
Display libdl errors when calling PythonInit
Use "--embed" flag when calling python-config, this was introduced in 3.8: https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
dlpython: ensure backwards compatibility when calling python-config

Co-authored-by: Tomas Buchaillot [email protected]
Triggered by: push (@tbuchaillot)
Execution page

Tyk-ITS · 2022-02-15T13:59:35Z

API tests result: skipped 🚫
Branch used: refs/heads/master
Commit: cc9fcd4 Add fixes for Python 3.8/3.9 (#3479)

Remove deprecated function mapping "PyTuple_ClearFreeList", reference: https://docs.python.org/3.9/whatsnew/3.9.html?highlight=pytuple_clearfreelist

Improve libdl error handler

Remove PyTuple_ClearFreeList dlpython helper
Display libdl errors when calling PythonInit
Use "--embed" flag when calling python-config, this was introduced in 3.8: https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
dlpython: ensure backwards compatibility when calling python-config

Co-authored-by: Tomas Buchaillot [email protected]
Triggered by: push (@konrad-sol)
Execution page

Tyk-ITS · 2022-02-16T08:29:07Z

API tests result: skipped 🚫
Branch used: refs/heads/master
Commit: cc9fcd4 Add fixes for Python 3.8/3.9 (#3479)

Remove deprecated function mapping "PyTuple_ClearFreeList", reference: https://docs.python.org/3.9/whatsnew/3.9.html?highlight=pytuple_clearfreelist

Improve libdl error handler

Remove PyTuple_ClearFreeList dlpython helper
Display libdl errors when calling PythonInit
Use "--embed" flag when calling python-config, this was introduced in 3.8: https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
dlpython: ensure backwards compatibility when calling python-config

Co-authored-by: Tomas Buchaillot [email protected]
Triggered by: push (@konrad-sol)
Execution page

matiasinsaurralde force-pushed the python39-fix branch 2 times, most recently from d1572a1 to 54bad27 Compare February 27, 2021 21:01

matiasinsaurralde changed the title ~~[WIP] Add fixes for Python 3.9~~ [WIP] Add fixes for Python 3.8/3.9 Feb 27, 2021

matiasinsaurralde force-pushed the python39-fix branch from 54bad27 to af7c170 Compare January 28, 2022 01:14

matiasinsaurralde force-pushed the python39-fix branch from af7c170 to 214ada0 Compare February 2, 2022 01:05

matiasinsaurralde added 5 commits February 2, 2022 01:06

Remove deprecated function mapping "PyTuple_ClearFreeList", reference: …

8c031fa

…https://docs.python.org/3.9/whatsnew/3.9.html?highlight=pytuple_clearfreelist Improve libdl error handler

Remove PyTuple_ClearFreeList dlpython helper

a73e284

Display libdl errors when calling PythonInit

2be36b7

Use "--embed" flag when calling python-config, this was introduced in…

b03b762

… 3.8: https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

dlpython: ensure backwards compatibility when calling python-config

06b233f

matiasinsaurralde force-pushed the python39-fix branch from 214ada0 to 06b233f Compare February 2, 2022 01:06

TykTechnologies deleted a comment from github-actions bot Feb 2, 2022

TykTechnologies deleted a comment from Tyk-ITS Feb 2, 2022

matiasinsaurralde changed the title ~~[WIP] Add fixes for Python 3.8/3.9~~ Add fixes for Python 3.8/3.9 Feb 2, 2022

matiasinsaurralde requested review from sredxny and tbuchaillot February 2, 2022 15:18

tbuchaillot reviewed Feb 3, 2022

View reviewed changes

tbuchaillot self-requested a review February 3, 2022 13:30

tbuchaillot approved these changes Feb 3, 2022

View reviewed changes

sredxny approved these changes Feb 4, 2022

View reviewed changes

Merge branch 'master' into python39-fix

9fa4271

Merge branch 'master' into python39-fix

b5a575c

tbuchaillot merged commit cc9fcd4 into master Feb 9, 2022

tbuchaillot deleted the python39-fix branch February 9, 2022 17:09

tbuchaillot changed the title ~~Add fixes for Python 3.8/3.9~~ TT-3780 Add fixes for Python 3.8/3.9 Feb 9, 2022

This was referenced Nov 29, 2022

release 4.3 #4449

Closed

merge/cf54fe538d59c059df4e858f0faa1a7535ba970b #4450

Closed

merge/cf54fe538d59c059df4e858f0faa1a7535ba970b #4451

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TT-3780 Add fixes for Python 3.8/3.9 #3479

TT-3780 Add fixes for Python 3.8/3.9 #3479

matiasinsaurralde commented Feb 27, 2021 •

edited by tbuchaillot

Loading

sredxny commented Feb 27, 2021

matiasinsaurralde commented Feb 27, 2021

buger commented May 10, 2021

matiasinsaurralde commented Jan 28, 2022

Tyk-ITS commented Feb 2, 2022

Tyk-ITS commented Feb 2, 2022

tbuchaillot Feb 3, 2022

tbuchaillot Feb 3, 2022

matiasinsaurralde Feb 3, 2022

matiasinsaurralde Feb 3, 2022 •

edited

Loading

tbuchaillot Feb 3, 2022

sredxny left a comment

Tyk-ITS commented Feb 7, 2022

Tyk-ITS commented Feb 9, 2022

sonarcloud bot commented Feb 9, 2022

tbuchaillot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

tbuchaillot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

Tyk-ITS commented Feb 9, 2022

Tyk-ITS commented Feb 15, 2022

Tyk-ITS commented Feb 16, 2022

TT-3780 Add fixes for Python 3.8/3.9 #3479

TT-3780 Add fixes for Python 3.8/3.9 #3479

Conversation

matiasinsaurralde commented Feb 27, 2021 • edited by tbuchaillot Loading

sredxny commented Feb 27, 2021

matiasinsaurralde commented Feb 27, 2021

buger commented May 10, 2021

matiasinsaurralde commented Jan 28, 2022

Tyk-ITS commented Feb 2, 2022

Tyk-ITS commented Feb 2, 2022

tbuchaillot Feb 3, 2022

Choose a reason for hiding this comment

tbuchaillot Feb 3, 2022

Choose a reason for hiding this comment

matiasinsaurralde Feb 3, 2022

Choose a reason for hiding this comment

matiasinsaurralde Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

tbuchaillot Feb 3, 2022

Choose a reason for hiding this comment

sredxny left a comment

Choose a reason for hiding this comment

Tyk-ITS commented Feb 7, 2022

Tyk-ITS commented Feb 9, 2022

sonarcloud bot commented Feb 9, 2022

tbuchaillot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

tbuchaillot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

tykbot bot commented Feb 9, 2022

Tyk-ITS commented Feb 9, 2022

Tyk-ITS commented Feb 15, 2022

Tyk-ITS commented Feb 16, 2022

matiasinsaurralde commented Feb 27, 2021 •

edited by tbuchaillot

Loading

matiasinsaurralde Feb 3, 2022 •

edited

Loading