-
-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support .format for bytes #48232
Comments
I just working on porting some networking code from 2.x to 3.x and it [I understand format is implemented with stringlib so shouldn't it be |
Yes, it would be easy to add. Maybe bring this up on python-dev (or Are we in feature freeze for 3.0? |
On Sat, Sep 27, 2008 at 12:33 PM, Eric Smith <[email protected]> wrote:
Yes, that will have to be done.
Unfortunately, yes. |
I'm skeptical. What networking code specifically are you using, and what |
On Sat, Sep 27, 2008 at 12:35 PM, Martin v. Löwis
I'm working on the tests for ftplib. [1] The dummy server uses string [1] https://svn.python.org/view/python/trunk/Lib/test/test_ftplib.py?view=markup |
I see. I propose to add a method push_string, defined as def push_string(self, s):
self.push(s.encode("ascii") In FTP, the responses are, by definition, ASCII-encoded strings. |
I don't think that b'...'.format() is a good idea. Programmers will |
b''.format() would return bytes, not a string. This is also how it works I'm also not sold on implementing it, although it would be easy and I |
As I understand, you would like to use bytes as characters, like If you need to work on bytes, it means that you will use the full |
That's indeed exactly what I had proposed - only that you shouldn't |
loewis> That's indeed exactly what I had proposed If you can only use bytes 0..127, it can not used for binary protocols I also not fan on functions having different result type |
In 3.x, str.format() and bytes.format() would be two different methods Similarly, in 2.6, str.format() has a different return type than Now the builtin format() function is another issue. In 2.6 the return |
There are many binary formats that use ASCII numbers. 'HTTP chunking' uses ASCII mixed with binary (octets). With 2.6 you could write: def chunk(block):
return b'{0:x}\r\n{1}\r\n'.format(len(block), block) With 3.0 you'd have to write this: def chunk(block):
return format(len(block), 'x').encode('ascii') + b'\r\n' + block +
b'\r\n' You cannot convert to ascii at the end of the pipeline as there are |
I wouldn't write it in such a complicated way. Instead, use def chunk(block):
return hex(len(block)).encode('ascii') + b'\r\n' + block + b'\r\n' This doesn't need any format call, and describes adequatly how the |
hex(10) returns '0xa' instead of 'a'.
The point is that need to convert to ascii for each int that you send. |
Ah, right. So I would still use '{0:x}'.format(100).encode("ascii") rather than the format builtin format function. Actually, I would ('%x' % len(bytes)).encode("ascii")
This I don't understand. What porting becomes more difficult? |
This kind of formatting is needed quite often when working on network protocols or file formats, and I think the replies here fail to address important issues. In general you can't encode after formatting, as that doesn't work with binary data, and often it's not appropriate for the low-level routines doing the formatting to know what charset the data is in even if it is text (so it should be fed in already encoded as bytes). The replies from Martin v. Löwis seem to argue that you could use methods other than formatting; that would work almost as well as an argument to remove formatting support from text strings, and IMO cases where formatting is the best option are common. Here's an example (based on real use but simplified): template = b"""
stuff here
header1: {}
header2: {}
more stuff
"""
def lowlevel_send(s, b1, b2): # s socket, b1 and b2 bytes
s.send(template.format(b1, b2)) To clarify the requirements a bit, the issue is not so much about having a .format method on byte string objects (that's just the most natural-looking way of solving it); the core requirement is to have a formatting operator that can take byte strings as *arguments* and produce byte string *output* where the arguments can be placed unchanged. |
For future reference, struct.pack, not mentioned here, is a binary bytes formatting function. It can mix ascii bytes with binary octets. It works the same in Python 2 and 3. Str.bytes does two things: convert objects to strings according to the contents of field specifiers; interpolate the resulting strings into a template string according to the locations of the field specifiers. If desired bytes represent encoded text, then encoding computed text is the obvious Py3 solution. For some mixed ascii-binary uses, struct.pack is not as elegant as a bytes.format might be. But I think such a method should use struct format codes within field specifiers to convert objects into binary bytes rather than text. |
struct.pack does not work with variable length data. Something like: b'{0:x}\r\n{1}\r\n'.format(len(block), block) or b'%x\r\n%s\r\n' % (len(block), block) is not possible with struct.pack |
You are right, I misinterpreted the meaning of 's' without a count (and opened bpo-11436 to clarify). However, for the fairly common case where a variable-length binary block is preceded by a 4 byte *binary* count, one can do something which is not too bad: >>> block = b'lsfjdlksaj'
>>> n=len(block)
>>> struct.pack('I%ds'%n, n, block)
b'\n\x00\x00\x00lsfjdlksaj' If leading blanks are acceptable for your example with count as ascii hex digits, one can do something that I admit is worse: >>> struct.pack('10s%ds2s'%n, ('%8x\r\n'%n).encode(), block, b'\r\n')
b' a\r\nlsfjdlksaj\r\n' Of course, for either of these in isolation, I would probably only use .pack for the binary conversion and otherwise use '+' or b''.join(...). |
I've hit this limitation a couple more times, and none of the proposed workarounds are adequate. Working with protocols and file formats that use human-readable markup is significantly clumsier than it was with Python 2 (using either the % operator, which also lost its support for byte strings in Python 3, or .format()). This bug report was closed by its original creator, after early posts where IMO nobody made as good a case for the feature as they could have. Is it possible to reopen this bug or is it necessary to file a new one? Is there any clear argument AGAINST having .format() for bytes, other than work needed to implement it? Some posts mention "mixing characters and bytes", but I see no reason why this would be much of a real practical concern if it's a method on bytes objects producing bytes output. |
If you want to discuss this issue further, I think you post to python-ideas list with concrete examples. |
Since Benjamin originally requested this feature, and then decided that he could accomplish his desired goal (ftplib porting, as far as I can tell) without it, I think that the "rejected" status is actually incorrect. I think that Benjamin just wanted to indicate that he no longer needed the feature. This doesn't mean that no one else will need the feature, and as it turns out the comments seem to reveal that other people do need the feature (also, I need the feature). So, adjusting the ticket metadata to reflect that this is a valid feature request just waiting for someone to implement it, not a rejected idea that is not welcome in Python. |
The proposal sounds like a good idea to me. Benjamin, what needs to be done to implement the feature? |
Formatting is a very complicated part of Python (especially after Victor's optimizations). I think no one wants to maintain this code for a long time. The price of maintaining exceeds the potential very limited benefits from the use. |
I was just logging in to make this point, but Serhiy beat me to it. When I wrote several years ago that this was "easy", it was before the (awesome) PEP-393 work. I suspect, but have not verified, that having a bytes version of this code would now require an implementation that shared very little with the str version. So I think Martin's advice to just encode to ascii is the best course of action. |
Given a spec hasn't been written (bytes.__mod__ can't support the same things as str.__mod__), and nobody seems to step up to write it, I'd say this is unlikely to appear in 3.4. |
Is there any chance we could just have it work for bytes, ints, and floats? That'd solve the immediate need, and it'd be obviously correct how to have those behave. Punting this to 3.5 basically means we'll have to either wait for 3.5, or do something awful like use cffi to grab sprintf to port Mercurial. |
If you could write up a concrete proposal, including which format specifiers would be supported, that would be helpful. Would it be extensible with something like __bformat__? There's really quite a bit of work to be done to specify how this would work. |
Also, with the PEP-393 changes, the implementation will be much more difficult. Sharing code with str (unicode) will likely be impossible, or require much refactoring of the existing code. |
You mean "%s" and "%d"?
Or write a pure Python implementation. |
On Tue, Oct 8, 2013 at 11:08 AM, Antoine Pitrou <[email protected]>wrote:
Basically, yes.
Hah. Probably too slow for anything beyond a proof of concept, no? |
On Oct 8, 2013, at 8:10 AM, Augie Fackler <[email protected]> wrote:
It should perform acceptably on PyPy ;-). |
If it's only for the Mercurial test suite, that shouldn't be a problem? |
On Tue, Oct 8, 2013 at 5:11 PM, Antoine Pitrou <[email protected]>wrote:
It's not just the testsuite though: we do this _all over_ hg itself. For sys.stdout.write('%(state)s %(path)s\n' % {'state': 'M', 'path': except we don't know the encoding of the filesystem path (Hi unix!) so we |
2013/10/8 Augie Fackler <[email protected]>:
You are doing it wrong. In Python 3, you "should" store filenames as The Unicode type became natural in Python 3, as byte string (old "str" sys.stdout.write() expects a Unicode string, not a byte string. Does it mean that Mercurial is moving to Python 3? Cool :-) |
I've lost track what we were talking about. I thought we were trying to support b'<something>'.format() in 3.4, for a restricted set of arguments. I don't see how a third-party package is going to help, if the goal is to allow 3.4 to be source compatible with 2.7. And the recent example uses %-formatting, which is not the subject of this ticket. What proposal is actually on the table here? |
On Oct 8, 2013, at 2:35 PM, Eric V. Smith wrote:
Sorry Eric, you're right, there is too much discussion here. This issue ought to be about .format, like the title says. There should be a separate ticket for %-formatting, since it seems to be an almost wholly unrelated task. While I'm sympathetic to Mercurial's issues, they're somewhat different from Twisted's, in that we're willing to adopt the "one new way" to do things in order to achieve compatibility whereas that would be too hard for Mercurial. |
On Oct 8, 2013, at 5:24 PM, STINNER Victor <[email protected]> wrote:
No, I'm not. In Mercurial, all end-user data is OPAQUE BYTES, and must remain that way. We're not able to change either our on-disk data format OR our stdout format, even to support a newer version of Python. I don't know the encoding of the filename's bytes, but I _must_ faithfully reproduce them exactly as they are or I'll break tools like make(1) and patch(1). Similarly, if a file goes from ISO-8859-1 to UTF-8, I have to emit a diff that has some ISO bytes and some UTF bytes - it's not in *any* valid encoding. Changing that is a showstopper regression.
Ouch. Is there any way to write things to stderr and stdout without decoding and hopelessly breaking user data?
Not likely, honestly. I tackle this when I've got some spare cycles and my ability to handle pain is high. As it stands, I have the test-runner barely working, but it's making wrong assumptions to get there. The best estimate is that it's a year of work to upgrade to Python 3.
|
On Oct 8, 2013, at 6:19 PM, Glyph Lefkowitz <[email protected]> wrote:
Yeah, my bad too. I suppose I should add a new bug for %-formatting on bytes objects? Note that for hg, we can't drop Python 2.6 or so (we'll only drop *2.4* if we can do 2.6 and some 3.x from a single source tree) for a while, due to supporting the system interpreter on a variety of LTS platforms. |
Augie, to understand what Viktor meant, I suggest reading |
On Oct 8, 2013, at 6:28 PM, "Terry J. Reedy" <[email protected]> wrote:
At a quick skim, likely not good enough, because https://en.wikipedia.org/wiki/Shift_JIS isn't completely ASCII-compatible, and we've got a fair number of users on weird Shift-JIS using platforms. |
On Oct 8, 2013, at 3:19 PM, Augie Fackler wrote:
The PEP-383 technique for handling file names is completely capable of round-tripping exact bytes, given one encoding for both input and output. You can still handle file names this way internally in Mercurial and not risk disturbing any observable output. You do not need to change that in order to do what Victor suggests. We should get together in some other forum and discuss file-name handling though, since you can't actually round-trip "opaque bytes" through a *filesystem* and not disturb your output.
You can use sys.stdout.buffer.write. |
Here is a proof of concept Python function, with a minimal test. It is similar to how str.format could be coded in Python, with re.split and ''.join, except that it does not allow anything before : in the format specification. By default (no format spec given), it copies bytes objects without change. If a format specification *is* given, it does not restrict the object, as this code simply uses builtin format sandwiched between decode and encode. |
Note that there's no guarantee that sys.stdout.buffer exists, e.g. if sys.stdout has been replaced with a StringIO. |
Tempting as it is to reply to the comment about 'buffer' not existing, we're way off topic here. Let's please keep further comments on this bug to issues about a 'format' methods on the 'bytes' object. |
First off, +1 for this feature. It's not just for twisted, but anyone doing anything with binary data (storage, compression, encryption and networking for me) with python since 2.6 will very likely have been using .format for building messages. I know I have and obviously others have been doing so as well. The advantages of .format to me are:
Specific comments on the patch supplied by terry.reedy:
Really this isn't a good way to solve the problem. Has a PEP been created for this? If not how can I help make that happen? Including this in 3.5 would be so helpful for us low level systems programmers out there who have lots of code using .format for binary interfaces in python 2.6/2.7 already. Also, not to add to derailment, but if we're adding a .format for python3 bytes it would be great if .format could pad with the null byte ('\0') which it currently converts to spaces internally (which is strange). Since this unexpected conversion is bad (so padding with null doesn't happen in python2) its more like a bug fix... actually - maybe that's a separate bug to file on the current .format for text... |
sorry, terry's patch does handle padding - just with the caveats i listed later. i should have removed that bullet. |
https://legacy.python.org/dev/peps/pep-0461/ Nick, I have the impression that there was a decision to not add bytes.format. Correct? If so, this issue should be closed. If not, what, if anything, has been decided? |
Right, bytes.format was considered as part of the PEP-461 discussions, and rejected as an operation that only made sense in the text domain: https://www.python.org/dev/peps/pep-0461/#proposed-variations With PEP-461 accepted, and PEP-460 withdrawn, that means we won't be adding bytes.format and bytearray.format. |
This came up in the language summit today when discussing twisted. .format() is still not supported on bytes though % is in 3.5. realistically it sounded like twisted needs to support python 3.4 for many years so they can't rely on bytes having a .format() method that also works on 2.7 anyways... but assuming .format() is only useful for text may still have been an oversight. (i'll have to go re-read PEP-460 and 461 and discussion before commenting further) |
Gregory - I'm glad that you're willing to consider this again. It still is a constant issue for me, and .format with variable width fields in binary protocols is so the right tool for the job. If there is anything I can do to help get this added to 3.6 let me know. The forward/backward compatibility issue is secondary to me to the flexibility gained from having .format available for bytes. Also padding with null bytes that don't get converted would be awesome. |
The core problem with the idea of adding bytes.format to Python 3 is that the real power of str.format actually lies in the extensible __format__ protocol and the associated format() builtin, as those rely heavily on text-specific assumptions. I interpreted Amber's comments at the language summit as referring more to our changing tune regarding mod formatting from:
Folks that followed our original "stop using mod formatting" guidance thus needed to change course when it became our recommended technique for formatting binary data. Since we now know format() and __format__ aren't suitable for binary data (PEP-361 originally included it, and it got dropped as we kept finding awkward corner cases), that means any new binary formatting proposal needs to explain:
|
I could have sworn that I also remember that Who else has this false memory? |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: