Thanks to the Python Software Foundation, a PandaBoard arrived on Trent Nelson’s desk just in time for the holidays! Santa dropped off the present for python-dev this morning, and there’s a Raspberry Pi not far behind it.
On Raymond Hettinger’s recent thread about the memory layout of dictionaries, Barry Warsaw and Christian Heimes shared concerns about how things might look on ARM devices. Christian mentioned the Snakebite environment, run by Trent Nelson, but without any ARM machines in the environment, Trent offered to host the boxes if someone donates them.
Based on the thread’s suggestions and the low cost of the devices, the PSF authorized purchase of a PandaBoard ES, featuring a 1.2 GHz ARM Cortex A9, along with several accessories to get it running. The PSF already had a few Raspberry Pi devices on hand, which come with a 700 MHz ARMv6, so one was dispatched to Trent.
Thanks to the PSF for making the purchase, and thanks to Trent for offering to set up the machines and add them to the environment!
Thursday, December 20, 2012
Monday, November 19, 2012
New Contributor Experience in Python and other FOSS Communities - A Survey
If you have joined the community of developers contributing to CPython since January 2010, I hope you can find a few spare minutes to participate in a survey being put on by Kevin Carillo. He’s a Ph. D. student at Victoria University of Wellington completing research on new contributors to free and open source projects. Kevin is interested in hearing from everyone from technical to non-technical contributors, whether you had positive or negative experiences.
The survey is available at https://limesurvey.sim.vuw.ac.nz/index.php?sid=65151&lang=en.
Kevin states, “The goal of this research is to understand how a person's experience as a newcomer to a Free/Open Source Software (FOSS) community influences that person's behavior and contributions within that community.” He estimates that the survey will take around 20 minutes to complete.
Throughout the nearly three year period since January 2010, over 40 committers were added and countless others contributed patches, reviews, and bug triage. Since the creation of the Core-Mentors group in March 2011, we’ve seen many first timers come through and have their work committed and released.
We hope that the mentorship group has helped introduce contributors in a positive manner, so we’re looking forward to the results of Kevin’s studies. One of the things Kevin hopes to find is whether or not formal mentorship programs work for introducing contributors. He also looks to find answers to how much community support of newcomers matters, whether formal joining processes involving sponsorship work, and if newcomer specific tasks are the way to go.
If you have the time, please fill out the survey. Python is participating in this survey along with several other groups including Debian, KDE, Gnome, openSUSE, and OpenHatch. Perhaps we can learn a few things and create an even better experience for new contributors!
The survey is available at https://limesurvey.sim.vuw.ac.nz/index.php?sid=65151&lang=en.
Kevin states, “The goal of this research is to understand how a person's experience as a newcomer to a Free/Open Source Software (FOSS) community influences that person's behavior and contributions within that community.” He estimates that the survey will take around 20 minutes to complete.
Throughout the nearly three year period since January 2010, over 40 committers were added and countless others contributed patches, reviews, and bug triage. Since the creation of the Core-Mentors group in March 2011, we’ve seen many first timers come through and have their work committed and released.
We hope that the mentorship group has helped introduce contributors in a positive manner, so we’re looking forward to the results of Kevin’s studies. One of the things Kevin hopes to find is whether or not formal mentorship programs work for introducing contributors. He also looks to find answers to how much community support of newcomers matters, whether formal joining processes involving sponsorship work, and if newcomer specific tasks are the way to go.
If you have the time, please fill out the survey. Python is participating in this survey along with several other groups including Debian, KDE, Gnome, openSUSE, and OpenHatch. Perhaps we can learn a few things and create an even better experience for new contributors!
Tuesday, October 30, 2012
Python Bug Day this Saturday
This Saturday, you have the opportunity of participating to the
Python Bug Day. How would you like to be one of the contributors of
Python? If you have ideas for improving parts of the official
documentation, the standard library, the language itself, or if you have
a patch waiting for a review that you would like to see committed, or
if you just want to come and fix an existing bug, it’s your day!
Join us for an effort at closing some Python bugs and feature requests. Get quick feedback on your patches and bugfixes, learn how to submit and examine patches, and have fun chatting with the Python developers and other contributors. You don’t need to know the CPython codebase or process to join, just Python programming knowledge.
If you live in Montreal, come at Caravan to meet fellow hackers and take part in a physical sprint!
Please register to let us know how many people to expect. People from around the world are should join the #python-dev IRC channel to participate in the bug day.
Join us for an effort at closing some Python bugs and feature requests. Get quick feedback on your patches and bugfixes, learn how to submit and examine patches, and have fun chatting with the Python developers and other contributors. You don’t need to know the CPython codebase or process to join, just Python programming knowledge.
If you live in Montreal, come at Caravan to meet fellow hackers and take part in a physical sprint!
Please register to let us know how many people to expect. People from around the world are should join the #python-dev IRC channel to participate in the bug day.
This page contains all the information you need to get set up, see the list of bugs or learn about IRC: https://wiki.python.org/moin/PythonBugDay
The goal of the bug day is to process bug reports in the Python bug tracker, trying to fix and close issues.
What to do:
- Grab a copy of the Python codebase from Mercurial, following instructions in the Developer's Guide, and compile it.
- If you have a problem that isn't in the bug tracker, announce it to the IRC channel, and if it's more than five minutes' work, create a bug report for it. See the bug reporting instructions to learn how to write bug reports.
- When you choose a bug to work on, announce it to the IRC channel (e.g. "I'm working on #123456.") or on the bug report itself. This avoids accidentally duplicating work.
- Consider providing a patch that fixes the problem, or at least a simple test case that demonstrates the bug. Please see the patch submission guidelines in the Developer's Guide before submitting a patch.
- Does the bug appear to be gone in the Python development version (the Mercurial branch "default", that will become 3.4), but not the 3.2, 3.3 or 2.7 maintenance branchs? Report that, too.
- If someone else has supplied a fix, see if this fix works for² you, and add your results to the bug.
- Read the text of proposed patches and assess them for correctness and code quality. This is usually the most time-consuming step in the bug fixing process, so reading patches is very useful.
- If there's a working fix, feel free to add a note asking for the fix to get committed. The bug tracker has a lot of items in it, and it's easy for bugs to be overlooked.
- Feature requests should be classified as type 'feature request' in the bug tracker.
Monday, October 29, 2012
Updates to docs.python.org
If you haven't already noticed, several months ago we updated the Sphinx theme for documentation of versions Python 3.2 and beyond on docs.python.org. It's a more modern look, and it also serves as an indicator that you're looking at documentation for a newer version. Thanks go out to Georg Brandl for his work on Sphinx, Python's documentation, and this new theme!
Over the weekend, PEP 430 was approved, which changes the default documentation displayed at https://docs.python.org. See the PEP for full details, but the jist is that we're now promoting the current Python 3 release as the default when you go to the docs home page. However, as the majority use case is still for Python 2 documentation, navigating straight to an unversioned page will present you with the current Python 2 documentation. For example, an unversioned link such as https://docs.python.org/library/zipfile will bring up the 2.7.3 documentation.
Supporting that change is a new feature that adds a version dropdown to the top of all documentation pages. Not only does this help when users are brought to a page which they don't expect, but switching between versions is a common operation as more and more projects work to add support for Python 3. Issue 8040 is where you'll find discussion on the change and its patches, with the bulk of the work completed by Yury Selivanov with some help from Georg.
This dropdown is especially handy as you peruse the documentation and come to a page that you want to view in another version. Choosing another version while on any page will load that page's other version, where the latest release of that version is chosen, e.g., 2.7 currently points to 2.7.3. So, as you browse the 2.7.3 built-ins page, choosing 3.3 in the dropdown will bring you to the 3.3.0 built-ins page.
We hope these changes enhance your experience when browsing the Python documentation!
PEP 430
Over the weekend, PEP 430 was approved, which changes the default documentation displayed at https://docs.python.org. See the PEP for full details, but the jist is that we're now promoting the current Python 3 release as the default when you go to the docs home page. However, as the majority use case is still for Python 2 documentation, navigating straight to an unversioned page will present you with the current Python 2 documentation. For example, an unversioned link such as https://docs.python.org/library/zipfile will bring up the 2.7.3 documentation.
Version Dropdown
Supporting that change is a new feature that adds a version dropdown to the top of all documentation pages. Not only does this help when users are brought to a page which they don't expect, but switching between versions is a common operation as more and more projects work to add support for Python 3. Issue 8040 is where you'll find discussion on the change and its patches, with the bulk of the work completed by Yury Selivanov with some help from Georg.
This dropdown is especially handy as you peruse the documentation and come to a page that you want to view in another version. Choosing another version while on any page will load that page's other version, where the latest release of that version is chosen, e.g., 2.7 currently points to 2.7.3. So, as you browse the 2.7.3 built-ins page, choosing 3.3 in the dropdown will bring you to the 3.3.0 built-ins page.
We hope these changes enhance your experience when browsing the Python documentation!
Tuesday, August 14, 2012
Python 3.3 Beta 2 Released
Release manager Georg Brandl announced on August 12 that the second beta of CPython 3.3 was released, complete with installers for both Mac and Windows. This release represents the final feature set, and the goal is to get it in the hands of users to iron out any last issues.
Following this beta will be two release candidates, coming August 25 and September 8. The final release is slated to happen on September 22.
The "What's New in Python 3.3" document is currently being finalized by curator and long time developer Raymond Hettinger. The document already contains many of the new changes, but keep an eye out for newer versions.
Here are some of the bigger changes:
- PEP 380, syntax for delegating to a subgenerator ("yield from")
- PEP 393, flexible string representation (doing away with the distinction between "wide" and "narrow" Unicode builds)
- A C implementation of the "decimal" module, with up to 80x speedup for decimal-heavy applications
- The import system (__import__) now based on importlib by default
- The new "lzma" module with LZMA/XZ support
- PEP 397, a Python launcher for Windows
- PEP 405, virtual environment support in core
- PEP 420, namespace package support
- PEP 3151, reworking the OS and IO exception hierarchy
- PEP 3155, qualified name for classes and functions
- PEP 409, suppressing exception context
- PEP 414, explicit Unicode literals to help with porting
- PEP 418, extended platform-independent clocks in the "time" module
- PEP 412, a new key-sharing dictionary implementation that significantly saves memory for object-oriented code
- PEP 362, the function-signature object
- The new "faulthandler" module that helps diagnosing crashes
- The new "unittest.mock" module
- The new "ipaddress" module
- The "sys.implementation" attribute
- A policy framework for the email package, with a provisional (see PEP 411) policy that adds much improved unicode support for email header parsing
- A "collections.ChainMap" class for linking mappings to a single unit
- Wrappers for many more POSIX functions in the "os" and "signal" modules, as well as other useful functions such as "sendfile()"
- Hash randomization, introduced in earlier bugfix releases, is now switched on by default
In total, almost 500 API items are new or improved in Python 3.3.
Be sure to check out this release at https://www.python.org/download/releases/3.3.0/ and report any issues to https://bugs.python.org.
Thursday, June 7, 2012
Mercurial Mirrors Provided by Atlassian
Long-time friends of the Python community, Atlassian (makers of Bitbucket) recently made available a mirror of https://hg.python.org, synchronized hourly, for your cloning and hacking pleasure.
Using the new mirror should be very intuitive for current users of the Hg repository -- the projects housed in the mirror follow the same naming convention as the repository they're mirroring. So, the CPython source code is mirrored at https://bitbucket.org/python_mirrors/cpython, corresponding to its canonical home at https://hg.python.org/cpython.
Since it's hosted on Bitbucket, the collaborative floodgates are effectively flung open. Not only is it dead easy to clone and submit contributions back to the project, you'll also have the ability to follow the project and receive updates in your dashboard. If RSS is more your style, Bitbucket makes it easy to stay up-to-date with changes via each repository's feed.
If you cloned the cpython repo and want to submit your changes to an issue on https://bugs.python.org, it's as simple as pasting a link to your Bitbucket clone in the "Remote hg repo" box. The default branch is automatically chosen, but appending #branchname to the end of your link will choose that branch.
See how easy it is to get your changes associated with an issue? If you're interested in getting started with CPython development, check out our developer guide.
Atlassian has been a user of Python and supporter of the Python community for some time now. They've sponsored PyCons around the world as well as events at those conferences, from the CodeWars competitions during PyCon AU to the recent PyLadies party at PyCon US!
Thanks, Atlassian!
Using the new mirror should be very intuitive for current users of the Hg repository -- the projects housed in the mirror follow the same naming convention as the repository they're mirroring. So, the CPython source code is mirrored at https://bitbucket.org/python_mirrors/cpython, corresponding to its canonical home at https://hg.python.org/cpython.
Since it's hosted on Bitbucket, the collaborative floodgates are effectively flung open. Not only is it dead easy to clone and submit contributions back to the project, you'll also have the ability to follow the project and receive updates in your dashboard. If RSS is more your style, Bitbucket makes it easy to stay up-to-date with changes via each repository's feed.
If you cloned the cpython repo and want to submit your changes to an issue on https://bugs.python.org, it's as simple as pasting a link to your Bitbucket clone in the "Remote hg repo" box. The default branch is automatically chosen, but appending #branchname to the end of your link will choose that branch.
See how easy it is to get your changes associated with an issue? If you're interested in getting started with CPython development, check out our developer guide.
Atlassian has been a user of Python and supporter of the Python community for some time now. They've sponsored PyCons around the world as well as events at those conferences, from the CodeWars competitions during PyCon AU to the recent PyLadies party at PyCon US!
Thanks, Atlassian!
Friday, June 1, 2012
Python 3.3 Alpha 4 Released
Yesterday, May 31, brought the fourth alpha release in the Python 3.3 development schedule. It's an exciting release as it introduces a number of long awaited features that we really hope the community will enjoy.
New Features
PEP 405 - Virtual Environments
Just in time for Alpha 4 comes the addition of PEP 405's support for virtual environments by way of the venv module and pyenv script.
python -m venv /home/yourname/dev/myproject
You may know this functionality through virtualenv, originally created by Ian Bicking. Thanks to Carl Meyer, Vinay Sajip, and anyone else for working on the PEP and implementation, we now have this widely used functionality available in a Python release!
PEP 420 - Namespace Packages
After a long road featuring two preceding PEPS (382 and 402), several sprints (including one sponsored by the PSF), and much discussion on python-dev, import-sig, and at PyCon language summits over the last two years, namespace packages are here. At the summit, Eric Smith stepped up to write a new PEP after the group decided to reject PEPs 382 and 402.
The result is PEP 420. The most obvious feature of a namespace package is the lack of a __init__.py file. However, there's a lot more to it, so check out the PEP!
PEP 3144 - The ipaddress Module
After discussion starting during the Python 3.2 development cycle, the ipaddress module has a new home in the standard library for 3.3. PEP 3144, authored by Peter Moody and taken up by core contributor Nick Coghlan, introduces a collection of classes for working with addresses, networks, and interfaces for both IPv4 and IPv6.
Windows Build Upgraded to Visual Studio 2010
As was recently covered, the Alpha 4 Windows installers now feature binaries produced by Visual Studio 2010, up from the 2008 version. We needed to upgrade to keep up with what most organizations and many of our contributors were using, along with the fact that not changing would mean we'd be at least two versions behind at our next opportunity to do so. With Python 3.4 not coming out until some time in 2014, we didn't want to end up eight years behind the curve and have to make that big of a version jump.
Bug Fixes
As with all of our releases, many contributors submitted patches to fix over 80 issues since last month's Alpha 3. We have fixes across a number of modules, including batches of fixes to IDLE, email, and urllib.
We Need Your Help!
As with all of our releases, backwards compatibility is important to us, so we'd love to hear if any of your projects have issues. Please help us make the best release possible by trying it out!
Python 3.3 is quickly shaping up to be the release everyone's waiting for, so run your tests and report your issues to https://bugs.python.org.
Thursday, May 24, 2012
Recent Windows Changes in Python 3.3
The Windows build of Python 3.3 has recently seen changes that could use a look from the community throughout our alpha and beta cycle. The first change is the long requested addition of Python to the system Path variable, which was completed in the installer. Secondly, the build was upgraded to Visual Studio 2010.
The harder part was deciding how to go about the change. If you're going to provide Path manipulation, the major questions are to do it by default or not, and to prepend or append to the Path.
We decided that it wasn't appropriate to make this a default feature. For one, in the dual-version state many users are running in, we run the risk of users running through the installer and putting their system into a state they aren't prepared for. We don't want to change the meaning of python when executed on the command line without the user asking for it. On one hand it's a very beginner focused feature in that it gets a first-timer successfully up and running with ease. However, it's also an advanced feature in that it takes a good understanding of what it's going to do to the users who have 2.6, 2.7, 3.2, and now 3.3 on their machines. We think the best solution for all is to leave it up to them and include an explanation.
The other part we had to think about was whether to prepend or append to the path. While some believe that appending to the path is the more friendly way to work with the system, it would seem to be of limited utility given that the feature is added this late in the game. Instead we went the route of prepending the installation folder, e.g., C:\Python33, in order to make sure this feature is actually useful to our users.
If you have questions or comments, please feel free to raise them on python-dev or see Issue 3561.
Many potential contributors as well as general Python users have long moved to work environments that use Visual Studio 2010. During a "bug day" some months ago, we had two or three patches come from interested first-timers who found our VS2008 solution not working in VS2010. Over time we received a few more contributions and bug reports on the topic, as well as some chatter in IRC about being behind the curve.
On top of that, my employer at the time moved to VS2010 as well as the employers of at least one other core maintainer, so we were already operating on ports for our companies.
When it came time to think about what to do for Python 3.3, moving to VS2010 became a must have due to our release schedule. Staying with VS2008 for 3.3 would put us into the middle of 2014 as the next time we could release on a new version. That would leave us at least two versions behind, with VS2010 as well as VS11 being available by then.
Another reason is the relative ease of porting between VS2010 and VS11. Once we got ourselves on to 2010, moving on to 11 would not be that hard. VS11 currently reads our VS2010 files without change if you want to use the IDE features of VS11. However, there'd need to be another port in order to use the VS11 compiler suite, but it seems to require minimal effort. Just following the VS11 wizard produced a functioning executable, although it didn't build cleanly.
The fine folks at Microsoft's Open Source Technology Center have provided the core contributors with MSDN licenses free of charge, allowing for access to the full versions of Visual Studio among other products. The full versions of Visual Studio support 64-bit compilation which comes in handy for our amd64 releases, which have been available since 2.5.
The last alpha release is scheduled for this weekend, and the first beta release is scheduled for June 24. You can download our 3.3.0 releases at https://www.python.org/download/releases/3.3.0/.
Python on the Path
A long requested feature, especially from beginners to those involved in education and training, has been the ability for the Python installer to place itself in the system Path environment variable. Having the following message appear when you try to run a simple exercise is not a great first experience:'python' is not recognized as an internal or external command, operable program or batch file.Because of that, the first post-install step by many users is to edit the Path environment variable manually to insert the C:Python33 directory. This allows the user to simply type python on the command line and have it open C:\\Python33\\python.exe -- a very desirable feature for a majority of users. In fact, it's such a common post-install step that there are a huge amount of tutorials either about this step by itself or tutorials where their setup introduces this step before moving on.
The easiest part of the whole thing was the code. Path manipulation in the installer consists of adding a new feature to the Feature table, then the Environment table may be updated based on selection of the Path feature. If the feature was selected, the Environment table is modified in a way that the Path is prepended to and will be correctly cleaned up on uninstallation.
The harder part was deciding how to go about the change. If you're going to provide Path manipulation, the major questions are to do it by default or not, and to prepend or append to the Path.
We decided that it wasn't appropriate to make this a default feature. For one, in the dual-version state many users are running in, we run the risk of users running through the installer and putting their system into a state they aren't prepared for. We don't want to change the meaning of python when executed on the command line without the user asking for it. On one hand it's a very beginner focused feature in that it gets a first-timer successfully up and running with ease. However, it's also an advanced feature in that it takes a good understanding of what it's going to do to the users who have 2.6, 2.7, 3.2, and now 3.3 on their machines. We think the best solution for all is to leave it up to them and include an explanation.
The other part we had to think about was whether to prepend or append to the path. While some believe that appending to the path is the more friendly way to work with the system, it would seem to be of limited utility given that the feature is added this late in the game. Instead we went the route of prepending the installation folder, e.g., C:\Python33, in order to make sure this feature is actually useful to our users.
If you have questions or comments, please feel free to raise them on python-dev or see Issue 3561.
Transition to Visual Studio 2010
In time for the last alpha release, we've updated our build tools from Visual Studio 2008 to 2010.Many potential contributors as well as general Python users have long moved to work environments that use Visual Studio 2010. During a "bug day" some months ago, we had two or three patches come from interested first-timers who found our VS2008 solution not working in VS2010. Over time we received a few more contributions and bug reports on the topic, as well as some chatter in IRC about being behind the curve.
On top of that, my employer at the time moved to VS2010 as well as the employers of at least one other core maintainer, so we were already operating on ports for our companies.
When it came time to think about what to do for Python 3.3, moving to VS2010 became a must have due to our release schedule. Staying with VS2008 for 3.3 would put us into the middle of 2014 as the next time we could release on a new version. That would leave us at least two versions behind, with VS2010 as well as VS11 being available by then.
Another reason is the relative ease of porting between VS2010 and VS11. Once we got ourselves on to 2010, moving on to 11 would not be that hard. VS11 currently reads our VS2010 files without change if you want to use the IDE features of VS11. However, there'd need to be another port in order to use the VS11 compiler suite, but it seems to require minimal effort. Just following the VS11 wizard produced a functioning executable, although it didn't build cleanly.
Where to get Visual Studio 2010?
As usual, Microsoft provides a zero-cost version of Visual Studio 2010 in the name Visual C++ Express, available at https://www.microsoft.com/visualstudio/en-us/products/2010-editions/visual-cpp-express. While there are some differences between the Express version and the for-purchase versions, the Express version is used successfully by many contributors.The fine folks at Microsoft's Open Source Technology Center have provided the core contributors with MSDN licenses free of charge, allowing for access to the full versions of Visual Studio among other products. The full versions of Visual Studio support 64-bit compilation which comes in handy for our amd64 releases, which have been available since 2.5.
Help us out -- try the alphas and betas!
With a change to the installer, a new build system, and the other great changes we have in store, the more feedback we hear from the community during the development cycle, the better we can make this release. If you have a chance to run your projects on Python 3.3, https://bugs.python.org is always open for your reports. You've even got a month to get feature requests in and completed!The last alpha release is scheduled for this weekend, and the first beta release is scheduled for June 24. You can download our 3.3.0 releases at https://www.python.org/download/releases/3.3.0/.
Wednesday, March 14, 2012
2012 Language Summit Report
This year's Language Summit took place on Wednesday March 7 in Santa Clara, CA before the start of PyCon 2012. As with previous years, in attendance were members of the various Python VMs, packagers from various Linux distributions, and members of several community projects.
As of Monday at the PyCon sprints, both PEPs have been rejected (see the Rejection Notice at the top of each PEP). Martin von Loewis posted to the import-sig list that a resolution has been found and Eric Smith will draft a new PEP on the ideas agreed upon there. Effectively, PEP 382 has been outright rejected, while portions of PEP 402 will be accepted.
Discussion began by outlining the only real existing issue, which lies in stat'ing of directories. There's a minor backwards incompatibility issue with time granularity. However, everyone agreed that it's so unlikely to be of issue that it's not a showstopper and the work can move forward. Additionally, there was an optimization made around the stat calls, which was arrived at independently by each of Brett, Antoine Pitrou, and P.J. Eby.
The topic of performance came up and Brett explained that the current pure-Python implementation is around 5% slower. Thomas Wouters exclaimed that 5% slower is actually really good, especially given some recent benchmark work he was doing showing that changing compilers sometimes shows a 5% difference in startup time. There was a shared feeling that 5% slower was not something to hold up integration of the code, which pushed discussion happily along.
Brett went on to explain what the bootstrapping actually looks like, even asserting that the implementation finds what could be the first real use of frozen modules! Guido's first response was, "you mean to tell me that after 20 years we finally found a use for freezing code?"
importlib._bootstrap is a frozen module containing the necessary builtins to operate, along with some re-implementations of a small number of functions. Some of the libraries included in the frozen module are warnings, _os (select code from posix), and marshal.
Another compatibility issue was brought up, but again, was decided to be an issue unworthy of halting the progress on this issue. There's a negative level count which is not supported in importlib, used in implicit relative imports, and it was agreed that it's acceptable to continue not supporting it.
The future will likely result in a strip down of import.c, as well as the exposure of numerous hooks as well as exposure of much of the importlib API.
As for merging with the default branch, it was pretty universally agreed upon that this should happen for 3.3 and it should happen soon in order to get mileage on the implementation throughout the alpha and beta cycles. Since this will be happening shortly, Brett is going to follow-up to python-dev with some cleanup details and look for reviews.
Immediately, the idea of splitting out the standard library to be on its own was resurrected, which could lend itself to both PEPs. Some questions remain, namely in where would the test suite live. Additionally, there may need to be some distinction between the tests which cover standard libraries versus the tests which cover language features.
The topic of versioning came up, with three distinctions needing to be made. We would seem to need a version of the language spec, a version of the implementation, and a version of the standard library.
Many commenters mentioned that these PEPs make things too complicated. Additionally, there was a question about whether there are enough users who care about either of these changes being made. Several of us stated that we could use the quicker releases, but with so many users being stuck on old versions for one reason or another, there was a wonder of who would take the releases.
Thomas Wouters mentioned a good point about the difficulty in lining up the so-called Python "LTS" releases with other Python consumers who do similar LTS-style releases. Ubuntu and their LTS schedule was a prime example, as well as the organizations who plan releases atop something like Ubuntu. Many of the Linux distribution packagers in attendance seemed to agree.
One thing that seemed to have broad agreement was that shortening the standard library turnaround time would be a good thing in terms of new contributors. Few people are interested in writing new features that might not be released for over a year -- it's just not fun. Even with bug fixes, sometimes the duration can be seen as too long, to the point where users may end up just fixing our problems from within their own code if possible.
Guido went on to make a comment about how we hope to avoid the mindset some have of "my package isn't accepted until it's in the standard library". The focus continues to be on projects being hosted on PyPI, being successful out in the wild, then vetted for acceptance in the standard library after maturity of the project and its APIs.
It was suggested that perhaps speeding up bug fix releases could be a good move, but we would need to check with release managers to ensure they're on board and willing to expend the effort to produce more frequent releases. As with the new feature releases, we need to be sure there's an audience to take the new bug fixes.
There was also some discussion about what have previously been called "sumo" releases. Given that some similar releases are already made by third-party vendors, the idea didn't seem to gain much traction.
Steve and Jesse Noller were adamant about the support not only being for all Python implementations, but also for third-party projects. What's needed to begin funding for a project is a concrete proposal on what will be accomplished. They stressed that the money is ready and waiting -- proposals are the way to unlock it.
Some ideas for how to use the funding came from Steve but also from around the room. One idea which started off the discussion was the idea of funding one-month sabbaticals. Then comes the issue of who might be available. Some suggested that freelance consultants in the development community might be the ones we should try to engage. Those with full-time employment may find it harder to acquire such a sabbatical, but the possibility is open to anyone.
Another thought was potential funding of someone to do spurts of full-time effort on the bug tracker, ideally someone already involved in the triage effort. This type of funding would hope to put an end to the times when it takes three days to fix a bug and three years for the patch to be accepted. Some thought this might be a nice idea in the short term, but it could be tough work and burn out the individual(s) involved. If anyone is up for it, they're encouraged to propose the idea to the foundation.
Along similar lines of tracker maintenance, Glyph Lefkowitz of the Twisted project had an idea to fund code reviews over code-writing efforts. Some thought this might be a good way to push forward the regex/re situation, given that the regex is very large and most felt that the only thing holding it back from some form of inclusion is an in-depth review. The cdecimal module was mentioned as another project that could use some review assistance.
The code review funding is also an idea to push forward some third-party project's ports to Python 3, specifically including Twisted, which the group felt was an effort which should receive some of this funding.
Along the way it was remarked that the core-mentors group has been a success in involving new contributors. Kudos to those involved with that list.
After discussion outside of the summit, it was additionally agreed upon that the launcher should be installed via the 3.3 Windows installer, while it can also live as a standalone installer for those not taking 3.3. Additionally, there needs to be some work done on the PEP to remove much of the low-level detail that is coupled too tightly with the implementation, e.g., explaining of the location of the py.ini file.
The project suffers not from a technological problem but from a personnel problem, which was thought to be another area that funding could be used for. However, even if money is on the table, we still need to find someone with the time, the know-how, and the drive to complete the task. Ideally the starting task would be to get PyPy and CPython implementations running and comparing. After that, there are a number of infrastructure tasks in line.
It was first suggested that documentation notes don't work -- we can't rely only on documentation to be the single notification point, especially for this type of code inclusion. Other thoughts were some type of flag on the library to specify its experimental status. Another thought was to emit a warning on import of a provisional library, but it's another thing that we'd likely want to silence by default in order to not affect user code in the hopes that developers are running their test suite with warnings enabled. However, as with other times we've gone down this path, we run the risk of developers just disabling warnings all together if they become annoying.
As has been suggested on python-dev, importing a provisional library from a special package, e.g., from __experimental__ import foo, was pretty strongly discouraged. If the library gains a consistent API, it penalizes users once it moves from provisional status to being officially accepted. Aliasing just exacerbates the problem.
The PEP boils down to being about process, and we need to be sure that libraries being included use the ability to change APIs very carefully. We also need to make people, especially the library author, aware of the need to be responsive to feedback and open to change as the code reaches a wider audience.
Looking back, Jesse Noller suggested multiprocessing would have been a good candidate for something like this PEP is suggesting. Around this time, it was suggested that Michael Foord's mock could gain some provisional inclusion within unittest, perhaps as unittest.mock. Instead, given mock's stable API and wide use among us, along with the need for a mocking library within our own test suite, it was agreed to just accept it directly into the standard library without any provisional status.
While on the topic of ``regex``'s role within the PEP came an idea from Thomas Wouters that ``regex`` be introduced into the standard library, bypassing any provisional status. From there, the previously known ``re`` module could be moved to the ``sre`` name, and there didn't appear to be any dissenting opinion there.
It should also be noted to users of provisional libraries that the library maintainers would need to exercise extreme care and be very conservative in changing of the APIs. The last thing we want to do is introduce a good library but as a moving target to its users.
In order to support this, the PyArg_ParseTuple function would need to do more work, and it's already known to be somewhat slow. Alternatively, PyArg_Parse is much faster, and the tuple version could take a thing or two from it regardless of any wide scale change to builtins.
There does exist some potential break in compatibility when replacing a builtin function with a Python one, where positional-only arguments suddenly get a potentially conflicting name.
It was widely agreed upon that we should avoid any blanket rules and keep changes to places where it makes sense rather than make wholesale changes. We also need to be mindful of documentation and doc strings being kept to match the actual keyword argument names as well as keep them in sync.
OrderedDict was suggested as the container for keyword arguments, but Guido and Gregory were unsure of use-cases for that. Whether or not we use a traditional or ordered dictionary, it was suggested that we could possibly use a decorator to handle some of this. We could even go as far as exposing something like PyArg_ParseTuple as a Python-level function.
PEP 362, a proposal for a function signature object, would help here and with decorators in general. It seems that all that's left with that PEP is another look and someone to declare on it.
It was suggested that documentation could stand to gain more examples of real-world porting examples, ideally pointing to changesets of these projects. The thought of our porting documentation gaining a cookbook-style approach seemed to get some agreement as a good idea.
It was also remarked that the only user-visible difference that this implementation brings is a difference in sort ordering, but the recent hash randomization work makes this a moot point.
If you have any questions or comments, please post to python-dev.
Thanks to Eric Snow and Senthil Kumaran for contributing to this post.
The Namespace PEPs
The summit began with a discussion on PEPs 382 and 402, with Barry Warsaw leading much of the discussion. After some discussion, the decision was ultimately deferred with what appeared to be a want for parts of both PEPs.As of Monday at the PyCon sprints, both PEPs have been rejected (see the Rejection Notice at the top of each PEP). Martin von Loewis posted to the import-sig list that a resolution has been found and Eric Smith will draft a new PEP on the ideas agreed upon there. Effectively, PEP 382 has been outright rejected, while portions of PEP 402 will be accepted.
importlib Status
Brett Cannon announced that there is a completed and available branch of CPython using importlib at https://hg.python.org/sandbox/bcannon/. See the bootstrap_importlib named branch.Discussion began by outlining the only real existing issue, which lies in stat'ing of directories. There's a minor backwards incompatibility issue with time granularity. However, everyone agreed that it's so unlikely to be of issue that it's not a showstopper and the work can move forward. Additionally, there was an optimization made around the stat calls, which was arrived at independently by each of Brett, Antoine Pitrou, and P.J. Eby.
The topic of performance came up and Brett explained that the current pure-Python implementation is around 5% slower. Thomas Wouters exclaimed that 5% slower is actually really good, especially given some recent benchmark work he was doing showing that changing compilers sometimes shows a 5% difference in startup time. There was a shared feeling that 5% slower was not something to hold up integration of the code, which pushed discussion happily along.
Brett went on to explain what the bootstrapping actually looks like, even asserting that the implementation finds what could be the first real use of frozen modules! Guido's first response was, "you mean to tell me that after 20 years we finally found a use for freezing code?"
importlib._bootstrap is a frozen module containing the necessary builtins to operate, along with some re-implementations of a small number of functions. Some of the libraries included in the frozen module are warnings, _os (select code from posix), and marshal.
Another compatibility issue was brought up, but again, was decided to be an issue unworthy of halting the progress on this issue. There's a negative level count which is not supported in importlib, used in implicit relative imports, and it was agreed that it's acceptable to continue not supporting it.
The future will likely result in a strip down of import.c, as well as the exposure of numerous hooks as well as exposure of much of the importlib API.
As for merging with the default branch, it was pretty universally agreed upon that this should happen for 3.3 and it should happen soon in order to get mileage on the implementation throughout the alpha and beta cycles. Since this will be happening shortly, Brett is going to follow-up to python-dev with some cleanup details and look for reviews.
Release Schedule PEPs
Discussion on PEPs 407 and 413 followed the importlib talk. Like the namespace PEP discussion, several ideas were tossed around but the group didn't arrive at any conclusion on acceptability of the PEPs.Immediately, the idea of splitting out the standard library to be on its own was resurrected, which could lend itself to both PEPs. Some questions remain, namely in where would the test suite live. Additionally, there may need to be some distinction between the tests which cover standard libraries versus the tests which cover language features.
The topic of versioning came up, with three distinctions needing to be made. We would seem to need a version of the language spec, a version of the implementation, and a version of the standard library.
Many commenters mentioned that these PEPs make things too complicated. Additionally, there was a question about whether there are enough users who care about either of these changes being made. Several of us stated that we could use the quicker releases, but with so many users being stuck on old versions for one reason or another, there was a wonder of who would take the releases.
Thomas Wouters mentioned a good point about the difficulty in lining up the so-called Python "LTS" releases with other Python consumers who do similar LTS-style releases. Ubuntu and their LTS schedule was a prime example, as well as the organizations who plan releases atop something like Ubuntu. Many of the Linux distribution packagers in attendance seemed to agree.
One thing that seemed to have broad agreement was that shortening the standard library turnaround time would be a good thing in terms of new contributors. Few people are interested in writing new features that might not be released for over a year -- it's just not fun. Even with bug fixes, sometimes the duration can be seen as too long, to the point where users may end up just fixing our problems from within their own code if possible.
Guido went on to make a comment about how we hope to avoid the mindset some have of "my package isn't accepted until it's in the standard library". The focus continues to be on projects being hosted on PyPI, being successful out in the wild, then vetted for acceptance in the standard library after maturity of the project and its APIs.
It was suggested that perhaps speeding up bug fix releases could be a good move, but we would need to check with release managers to ensure they're on board and willing to expend the effort to produce more frequent releases. As with the new feature releases, we need to be sure there's an audience to take the new bug fixes.
There was also some discussion about what have previously been called "sumo" releases. Given that some similar releases are already made by third-party vendors, the idea didn't seem to gain much traction.
Funding from the Python Software Foundation
PSF Chairman Steve Holden joined the group after lunch to mention that the foundation has resources available to assist development efforts, especially given the sponsorship success of this year's conference. While the foundation can't and won't dictate what should be coded up, they're open to proposals about the types of work to be funded.Steve and Jesse Noller were adamant about the support not only being for all Python implementations, but also for third-party projects. What's needed to begin funding for a project is a concrete proposal on what will be accomplished. They stressed that the money is ready and waiting -- proposals are the way to unlock it.
Some ideas for how to use the funding came from Steve but also from around the room. One idea which started off the discussion was the idea of funding one-month sabbaticals. Then comes the issue of who might be available. Some suggested that freelance consultants in the development community might be the ones we should try to engage. Those with full-time employment may find it harder to acquire such a sabbatical, but the possibility is open to anyone.
Another thought was potential funding of someone to do spurts of full-time effort on the bug tracker, ideally someone already involved in the triage effort. This type of funding would hope to put an end to the times when it takes three days to fix a bug and three years for the patch to be accepted. Some thought this might be a nice idea in the short term, but it could be tough work and burn out the individual(s) involved. If anyone is up for it, they're encouraged to propose the idea to the foundation.
Along similar lines of tracker maintenance, Glyph Lefkowitz of the Twisted project had an idea to fund code reviews over code-writing efforts. Some thought this might be a good way to push forward the regex/re situation, given that the regex is very large and most felt that the only thing holding it back from some form of inclusion is an in-depth review. The cdecimal module was mentioned as another project that could use some review assistance.
The code review funding is also an idea to push forward some third-party project's ports to Python 3, specifically including Twisted, which the group felt was an effort which should receive some of this funding.
Along the way it was remarked that the core-mentors group has been a success in involving new contributors. Kudos to those involved with that list.
virtualenv Inclusion
In about two minutes, discussion on PEP 405 came and went. Carl Meyer mentioned that a reference implementation is available and is working pretty well. A look from the OSX maintainers would be beneficial, and both Ned Deily and Ronald Oussoren were in attendance. It seemed like one of the only things left in terms of the PEP was to find someone to make a declaration on it, and Thomas Wouters put his name out there if Nick Coghlan wasn't going to do t (update: Nick will be the PEP czar).PEP 397 Inclusion
Without much of a Windows representation at the summit, discussion was fairlyquick, but it was pretty much agreed that PEP 397 was something we should accept. Brian Curtin spoke in favor of the PEP, as well as mentioning ongoing work on the Windows installer to optionally add the executable's directory to the Path.After discussion outside of the summit, it was additionally agreed upon that the launcher should be installed via the 3.3 Windows installer, while it can also live as a standalone installer for those not taking 3.3. Additionally, there needs to be some work done on the PEP to remove much of the low-level detail that is coupled too tightly with the implementation, e.g., explaining of the location of the py.ini file.
speed.python.org
After generous hardware donations, the https://speed.python.org site has gone live and is currently running PyPy benchmarks. We need to make a decision on what benchmarks can be used as well as what benchmarks should be used when it comes to creating a Python 3 suite. As we get implementations on Python 3 we'll want to scale back 2.7 testing and push forward with 3.x.The project suffers not from a technological problem but from a personnel problem, which was thought to be another area that funding could be used for. However, even if money is on the table, we still need to find someone with the time, the know-how, and the drive to complete the task. Ideally the starting task would be to get PyPy and CPython implementations running and comparing. After that, there are a number of infrastructure tasks in line.
PEP 411 Inclusion
PEP 411 proposes the inclusion of provisional packages into the standard library. The recently discussed regex and ipaddr modules were used as examples of libraries to include under this PEP. As for how this inclusion should be implemented and denoted to users was the major discussion point.It was first suggested that documentation notes don't work -- we can't rely only on documentation to be the single notification point, especially for this type of code inclusion. Other thoughts were some type of flag on the library to specify its experimental status. Another thought was to emit a warning on import of a provisional library, but it's another thing that we'd likely want to silence by default in order to not affect user code in the hopes that developers are running their test suite with warnings enabled. However, as with other times we've gone down this path, we run the risk of developers just disabling warnings all together if they become annoying.
As has been suggested on python-dev, importing a provisional library from a special package, e.g., from __experimental__ import foo, was pretty strongly discouraged. If the library gains a consistent API, it penalizes users once it moves from provisional status to being officially accepted. Aliasing just exacerbates the problem.
The PEP boils down to being about process, and we need to be sure that libraries being included use the ability to change APIs very carefully. We also need to make people, especially the library author, aware of the need to be responsive to feedback and open to change as the code reaches a wider audience.
Looking back, Jesse Noller suggested multiprocessing would have been a good candidate for something like this PEP is suggesting. Around this time, it was suggested that Michael Foord's mock could gain some provisional inclusion within unittest, perhaps as unittest.mock. Instead, given mock's stable API and wide use among us, along with the need for a mocking library within our own test suite, it was agreed to just accept it directly into the standard library without any provisional status.
While on the topic of ``regex``'s role within the PEP came an idea from Thomas Wouters that ``regex`` be introduced into the standard library, bypassing any provisional status. From there, the previously known ``re`` module could be moved to the ``sre`` name, and there didn't appear to be any dissenting opinion there.
It should also be noted to users of provisional libraries that the library maintainers would need to exercise extreme care and be very conservative in changing of the APIs. The last thing we want to do is introduce a good library but as a moving target to its users.
Keyword Arguments on all builtin functions
As recently came up on the tracker, it was suggested that wider use of keyword arguments in our APIs would likely be a good thing. Gregory P. Smith suggested that we leave single-argument APIs alone, which was agreed upon. However, the overall change got some push back as "change for change's sake".In order to support this, the PyArg_ParseTuple function would need to do more work, and it's already known to be somewhat slow. Alternatively, PyArg_Parse is much faster, and the tuple version could take a thing or two from it regardless of any wide scale change to builtins.
There does exist some potential break in compatibility when replacing a builtin function with a Python one, where positional-only arguments suddenly get a potentially conflicting name.
It was widely agreed upon that we should avoid any blanket rules and keep changes to places where it makes sense rather than make wholesale changes. We also need to be mindful of documentation and doc strings being kept to match the actual keyword argument names as well as keep them in sync.
OrderedDict was suggested as the container for keyword arguments, but Guido and Gregory were unsure of use-cases for that. Whether or not we use a traditional or ordered dictionary, it was suggested that we could possibly use a decorator to handle some of this. We could even go as far as exposing something like PyArg_ParseTuple as a Python-level function.
PEP 362, a proposal for a function signature object, would help here and with decorators in general. It seems that all that's left with that PEP is another look and someone to declare on it.
Porting to Python 3
We moved on to talk about Python 3 porting, starting with the current strategies and how they're working out. Single-codebase porting is working better than expected for most of us, although except handling is a bit messy when supporting versions like 2.4. Having a lot of options, from 3to2 to 2to3, then the single codebase through parallel trees, is a really good thing. However, it's hard for us to choose a strategy for projects, so we don't, which is why most documentation tries to lay numerous strategies out there.It was suggested that documentation could stand to gain more examples of real-world porting examples, ideally pointing to changesets of these projects. The thought of our porting documentation gaining a cookbook-style approach seemed to get some agreement as a good idea.
Hash Randomization
Release candidates are available to all branches receiving security fixes, and in the meantime, David Malcolm found and reported a security issue in the upstream expat project. However, since the upstream fix includes many other fixes at the same time, we should pick up only the security fix at this time and leave the bug fixes for the next bug fix release of the relevant branches.New dict Implementation
Since the implementation makes sense and the tests pass, it was quickly agreed upon that Mark Shannon's PEP 412 should be accepted. As with other changes agreed upon in this summit, we'd like for the change to be pushed soon in order to get mileage on it throughout the alpha and beta cycles. With this acceptance comes commit access for Mark so that he can maintain the code.It was also remarked that the only user-visible difference that this implementation brings is a difference in sort ordering, but the recent hash randomization work makes this a moot point.
New pickle Protocol
PEP 3154, mentioned by Lukasz Langa, specifies a new pickle protocol -- version 4. Lukasz mentioned exception pickling in multiprocessing as being an issue, and Antoine solved it with this PEP. While qualified names provide some help, it was agreed upon that this PEP needs more attention.If you have any questions or comments, please post to python-dev.
Thanks to Eric Snow and Senthil Kumaran for contributing to this post.
Subscribe to:
Posts (Atom)