engine: make output Readline-friendly

[kyrylo]

The motivation behind this commit is to make Foreman respect
GNU Readline 1.

As Foreman is often used with Pry (or at least many people want to), and
Pry is dependent on Readline, using these tools together is very
problematic 2,3.

The problem is that when you start Pry by means of Foreman, Foreman
supresses output from Pry, so you don't see what you type. The output
can only be seen after pressing carriage return. Demonstration (a bit
hard to follow, but you may want to try it yourself; clone the Pry repo
and execute the same command):

http:https://showterm.io/3eb716461d6602ac90b09

Although this problem was reported by Pry users, any Readline
application is affected. This commit uses IRB for tests, because it also
depends on Readline, hence it suits for testing perfectly. Other
Readline dependent applications that I tested were GHCi, the Lua REPL,
Eshell (erlang shell).

Previously, Pry was using a trick, which allowed us to bypass this
problem 4. The fix was featured in Pry v0.9.12.6, but then it was
removed by accident. It's been readded on HEAD recently (not released
yet). However, the fix we currently have is still not perfect. Although
you can see your input immediately, you can't see your output now :D
Demo:

http:https://showterm.io/f648de27568d96a02f1b3

The fix breaks correct piping. Now, when you pipe Pry's output, it
doesn't output Readline's prompt and user's input. What's being piped is
only return values of executed expressions.
Demo:

http:https://showterm.io/3651389faf1fdc0d18211

To fix the missing output and pipe everything, including the user's
input, we need to set Readline.output correctly. However, if we do
that, we break minimal support for Foreman. So when I fixed Pry 5, I
figured it would be interesting to try to fix Foreman.

Now, it's time to talk about the changes here. I'm not very familiar
with Foreman, so take the code with a grain of salt. Basically, the
whole point of the fix is to read one character at a time and print it
immediately, instead of waiting for a newline character. This technique
allows us to prepend so-called markers (timestamps, if you wish) to a
Readline prompt and display the user's input. It works for all the
Readline apps that I mentioned above and doesn't break piping (that is,
tee).

Note. I had to remove $stdout.flush from the output method, because it
was causing thread deadlocks with the flush in #handle_io. The
deadlock appears when you use Foreman with Pry, pipe the output and
trigger SIGINT (control-c press). Example:

% foreman start -f Procfile | tee log
15:50:05 pryhere.1 | started with pid 26881
15:50:28 pryhere.1 | 1 pry(main)> <CTRL+C><CTRL+D>
15:50:49 pryhere.1 | 2 pry(main)> foreman/engine.rb:323:in
`synchronize': deadlock; recursive locking (ThreadError)

This is probably because of the race condition, when on SIGINT Foreman
prints stuff here 6, which flushes $stdout.

[ddollar]

Foreman intentionally does not handle stdin on the terminal. If you are running multiple processes which of them should get the data from stdin?

[kyrylo]

That's a very good point. I just tested both IRB and Pry together in one Procfile and it is unusable in both cases, with and without this patch (in a slightly different manner, though). I cannot easily describe the behaviour, because it's unpredictable.

I assume if you add multiple applications that read stdin to Procfile, you know what you're doing. On the other hand, maybe it makes sense to read input sequentially from each application. So the first time I enter input for Pry, press Enter, then I enter input for IRB, then for Pry again.

Among Rubyists Pry is probably the only thing that reads from stdin. If you think this is still a bad idea, would you be so kind and drop a comment here explaining the situation, so I could close this issue and label it "WONTFIX" :)

P.S.: the Travis fail shouldn't be hard to fix, they just set their custom prompt.

[deivid-rodriguez]

I haven't read through this in detail nor researched this issue a lot, but I have a couple of comments.

• Thanks @kyrylo for you work on this! This has been a long stading issue in byebug as well... In particular Terminal echoing permanently turned off deivid-rodriguez/byebug#59 and Backspace fails when using Open3 deivid-rodriguez/byebug#63. If you would like to help with those, it would be really appreciated. Otherwise, I might have some questions when I get there!
• @ddollar Does it make any sense to broadcast the input to every process? Is it possible at all?

[edwardloveall]

I'd love to have support for pry when using foreman. Debugging is of course very useful when developing applications but foreman doesn't handle it well. Maybe this isn't the way to fix it, but I'd love to see something done about it.

[ddollar]

I don't think it makes sense to send stdin to all processes since that will almost certainly never be what you want. Your typing is intended for one process.

My advice would be to not run irb or pry as part of your Procfile and instead use a separate shell to foreman run pry if you need an interactive console.

[deivid-rodriguez]

@ddollar Note that most people is not adding irb to their proc files, they're just dropping a binding.pry line in their code. What I don't understand is that foreman seems to work fine with pry or byebug (the input entered is correctly received and processed). The only problem is that the user input is not displayed while the user is typing, but after the command is processed, which make the console unusable. This patch seems to fix that, what's the problem with it?

[juliosantos]

+1

[LaurensN]

+1

[rramsden]

+1

[gtd]

👍

@ddollar, any chance of reconsidering based on @deivid-rodriguez's message? This is indeed an extremely common use case for pry and it's a shame to have to keep the main server process out of foreman in order to attach a debugger.

[kyletolle]

+1

I've been using binding.pry with foreman in the same way that @deivid-rodriguez mentioned. It'd certainly be nice to see what I type as I type it. I hope this'll be reconsidered.

(edit: I mistyped the pry command)

[deivid-rodriguez]

Guys, have you actually tried this patch and verify it works? If not, could someone do it?

[kyletolle]

Yes, I have tested this patch and it does work for me.

[deivid-rodriguez]

Cool.

[tadp]

+1

@deivid-rodriguez thanks for spelling out the main use case for this so that @ddollar can review this. Lack of binding.pry support is one of the primary reasons I use standard rails s for troubleshooting.

[eric-hu]

For those looking for a workaround to using pry with foreman: pry-remote. The biggest caveat for me is that tab completion doesn't work.

[mfpiccolo]

👍

[michaelkeenan]

I couldn't get pry-remote working, but I got this patch working by using kyrylo's branch of foreman. This is how:

gem uninstall foreman
git clone -b readline-support https://github.com/kyrylo/foreman.git
cd foreman
gem build foreman.gemspec
gem install foreman-0.77.0.gem

It's so great to be able to see what I'm typing again!

[sethk]

I strongly believe that all Ruby REPLs should read/write from /dev/tty the way LLDB and GDB do. If this happens via Readline, those apps will need to call the input= and output= writers with IO.console. The Python debugger does this also, and it drives me bananas! You can't even debug an app that reads from stdin this way.

[drewish]

It'd be great if we could reconsider this. The screwy interaction with pry is the only downside of foreman.

[sethk]

@drewish The problem is with pry, not foreman. Foreman has no way to demultiplex the standard I/O with multiple children. The solution is for only one process at a time to run pry against the terminal using IO.console (see above) or to directly run the command from the Procfile without forman while using pry. You can see a similar problem running better_errors against multiple web processes.