implement pseudo tty on stdout/stderr

[masenf]

Thanks for contribution

Please, make sure you address all the checklists (for details on how see
development documentation)!

• ran the linter to address style issues (tox -e fix)
• wrote descriptive pull request text
• ensured there are test(s) validating the fix
• added news fragment in docs/changelog folder
• updated/extended the documentation

pty

This is my first attempt at fixing #1773, starting with the example code posted on that issue, and adapting it for some changes that have occured since then.

For the most part the code worked really nice already.

I did encounter the one problem mentioned after setting the Popen stdout and stderr handles to pty:

but once done that the stdin echo seems to no longer buffer realtime, and is instead only line-buffered. This is bad for interacting with the GDB

After lots of poking around, I determine that the code in SyncWrite, which is supposed to write the content to the target after some minimum delay, was not restarting its timer appropriately, causing the output to "hang" until SyncWrite discovered a newline \n.

With that fix in 9b06df5, everything seems to be working fine, EXCEPT, the log files now have ANSI escape sequences in them... which may or may not be desirable. After all, that is what the program output.

After this patch, colored output and interactive use (pdb, ipython) appears to be working out of the box!

Testing

Here's my "human" test file with 4 cases, to exemplify the differences between tox 3, tox 4, and this patch.

I've made these sort of non-scientific comparisons on a macOS 10.15 machine.

[tox]
skipsdist = true
envlist = py, pytest, textual, vim

[testenv]
commands =
    python -i -c 'import sys; \
        print("sys.stdout.isatty(): %s" % sys.stdout.isatty()); \
        print("to stdout, should be colorless\n"); \
        print("sys.stderr.isatty(): %s" % sys.stderr.isatty()); \
        print("to stderr, should be red?\n", file=sys.stderr)'

[testenv:pytest]
deps = pytest
commands =
    pytest {posargs} --pdb

[testenv:textual]
deps = textual
commands =
    python -m textual

[testenv:vim]
allowlist_externals = vim
commands = vim -p /tmp/f1 /tmp/f2 /tmp/f3

tox 3 ("current behavior" expected by users)

• py
  • sys.stdout.isatty(): True
  • sys.stderr.isatty(): True
  • printed lines are in order
  • session is interactive, arrow keys work, input is echoed
  • stderr is not colored red (that's a tox4 feature)
• pytest
  • color output (green dots, red F, and red Failed: text
  • pdb session is interactive, arrow keys work, input is echoed
• textual
  • demo works, but appears to be in "low color" or "reduced feature" mode
• vim: fully interactive, working as expected

tox 3 (Using tee to pipe the output to a log file)

• py
  • sys.stdout.isatty(): False
  • sys.stderr.isatty(): False
  • printed lines out of order (after pressing <enter>)
  • session is non-interactive, arrow keys print control codes, input is echoed, but stdout is block buffered and output is delayed to screen.
• pytest
  • monochrome output
  • pdb session is non-interactive, arrow keys print control codes
• textual
  • surprisingly, demo works. still in "low color" or "reduced feature" mode and small fixed window size (toggle light/dark mode fixes window size)
• vim: fully interactive, working as expected (also surprising)

tox 4 ("new behavior" shows some regression in expected functionality)

• py
  • stderr is colored red
  • printed lines out of order (pressing <enter> shows previous output)
  • sys.stdout.isatty(): False
  • sys.stderr.isatty(): False
  • session is non-interactive, arrow keys print control codes. input is echoed, but stdout and stderr are buffered unexpectedly (for example: the prompt doesn't show up until a newline appears on stderr)
• pytest
  • familiar dot and status letters are monochrome; stderr is red (as expected)
  • pdb session is not interactive, arrow keys print control codes
• textual
  • surprisingly, demo works basically perfectly. not sure how they pulled that off 🤯
• vim
  • sort of works (better than expected), but there are still delayed output issues, sometimes nothing shows up when you type. However, switching tabs (gt and gT can sometimes trigger a refresh).

tox 4 (Using tee to pipe the output to a log file)

• py: no change using tee
• pytest: no change using tee
• textual: behaves like tox 3 with tee
• vim: no change using tee

tox 4 with this patch!

• py
  • sys.stdout.isatty(): True
  • sys.stderr.isatty(): True
  • stderr text is colored red
  • session is interactive, arrow keys work, input is echoed
• pytest
  • color output (green dots, red F, and red Failed: text
  • pdb session is interactive, arrow keys work, input is echoed
• textual
  • demo works perfectly, no change from tox 4; using high color mode, unlike tox 3.
• vim: fully interactive, working as expected

tox 4 with this patch! (Using tee to pipe the output to a log file)

• py
  • sys.stdout.isatty(): False
  • sys.stderr.isatty(): True
  • monochrome output
  • session is non-interactive, arrow keys print control codes. input is echoed; not observing any buffering issues (likey SyncWrite fix at work)
• pytest
  • monochrome output
  • pdb session is non-interactive, arrow keys print control codes
• textual: behaves like tox 3 with tee
• vim: fully interactive, working as expected (again, not sure how, but i'll take it)

TODO

• Failing test to investigate turned out to be a macOS issue with homebrew-installed python interpreters. The test was expecting the executable to be /usr/local/opt/[email protected]/bin/python3.11, but when it runs throught the virtualenv, the path name comes out as /usr/local/Cellar/[email protected]/3.11.0/Frameworks/Python.framework/Versions/3.11/bin/python3.11... These are both hardlinks to the same inode. Not reproducing on linux.
• Additional test coverage for this feature added to existing test_local_subprocess_tty
• Testing on other platforms:
  • windows
  • linux
  • pypy