Research item: Remote debugging

[alexellis]

Research into debugging deployed functions remotely.

Idea - if a function container image contains additional tools and exposes an additional port then breakpoints can be set and hit and a remote debugger can be attached.

Workflow - configure metadata - build image, deploy function, open VSCode and attach to remote debugger. Hit breakpoint inspect values etc and disconnect.

Initial languages to explore:

• Python:

https://pypi.python.org/pypi/remote-pdb

• Node.js debugging:

https://nodejs.org/api/debugger.html

https://code.visualstudio.com/docs/nodejs/nodejs-debugging

• .NET Core

https://github.com/Microsoft/MIEngine/wiki/Offroad-Debugging-of-.NET-Core-on-Linux---OSX-from-Visual-Studio

[alexellis]

Derek add label: hacktoberfest

[austinfrey]

So this is a little rough, but it could be a start.

Using Node 8+ I was able to add debugging with Chrome dev-tools by exposing a new port and overriding the default fprocess with a new one.

I used the Node template from the CLI but swapped the base image for Node 8 to make use of the --inspect-brk flag, added the function to the docker-compose.yml and added the additional port mapping and fprocess environment variable.

To start debugging just invoke the function with curl or the cli, open chrome browser to chrome:https://inspect, there should be a Node icon at the bottom with the option to inspect.

#docker-compose.yml
function:
    ...
    ports:
        - 9222:9222
    environment:
        fprocess: "node --inspect-brk=0.0.0.0:9222 index.js"
    ....

I don't have VSCode but dev-tools does allow break-points and stepping through the code.

I think you could do this all via the CLI, but I dont think you can map ports? so something like this might work if ports are mappable from the cli.

#stack.yml
 provider:                                                                                                                       
  name: faas                                                                                                                    
    gateway: https://user:password@localhost                                                                                       
                                                                                                                                   
  functions:                                                                                                                      
    inspect:                                                                                                                      
      lang: node                                                                                                                  
      handler: ./inspect                                                                                                          
      image: aafrey/inspect                                                                                                       
      ports:                                                                                                                      
        - 9222:9222                                                                                                               
      environment:                                                                                                                
        fprocess: "node --inspect-brk=0.0.0.0:9222 index.js"

[ericstoekl]

With some help from Austin I figured out how to do this in Python. I created a python function, and modified it to look like this:

https://github.com/ericstoekl/misc-functions/blob/master/python-dbg/handler.py

Then I built the function into a docker image.

Next I re-deployed the whole swarm, modifying docker-compose.yml appending the function to the list of functions:

    python-dbg:
        image: ems5311/python-dbg:latest
        labels:
            function: "true"
        depends_on:
            - gateway
        networks:
            - functions
        ports:
            - 4444:4444
        environment:
            fprocess: python index.py
            no_proxy: "gateway"
            https_proxy: $https_proxy
        deploy:
            placement:
                constraints:
                    - 'node.platform.os == linux'

After the stack is deployed, if I call the func_python-dbg function with any input, the debugger will start listening on 0.0.0.0:4444. I can then run:

telnet localhost 4444

And it will connect to the listening debugger.

In conclusion, maybe we can add an option/flag to the python language template to activate this debugging feature?

[alexellis]

The issue I ran into with debugging was:

• Deploy the function (fprocess hasn't been run yet)
• Hit function over HTTP (function forks - starts, then runs then stops)
• Then try to attach debugger, but process has already finished

The afterburn example re-uses the process so that could be a way of debugging since when you attach the process is already running and will stay running. Otherwise some deliberate sleep etc. How did you go about it guys?

[austinfrey]

Using the inspect-brk flag in node stops the node function before it runs, so you have all the time you need to attach and step through the function. So I'm not sure if there would be a standard way of implementing this across languages?

[kenfdev]

Following @austinfrey 's steps, I've confirmed debugging works with VSCode, too. The launch.json will look like this:

{
    "version": "0.2.0",
    "configurations": [{
        "type": "node",
        "request": "attach",
        "name": "Attach to Remote",
        "address": "<REMOTE IP ADDRESS>",
        "port": 9222,
        "localRoot": "${workspaceRoot}/build/github-stats",
        "remoteRoot": "/root",
        "protocol": "inspector",
        "restart": true
    }]
}

Be sure localRoot and remoteRoot match.

[austinfrey]

thanks for testing @kenfdev , @alexellis if this is the route we want to take for NodeJS would it be helpful to add ti to the NodeJS starter guide, or would it be best as a a separate guide?

[alexellis]

I think we still need additional research since doesn't this currently require us to docker run rather than deploy via the gateway? Let's see what .NET Core requires also.

In terms of a guide, we should write this up for each language - perhaps in a single place when we have dug deep enough.

[alanjds]

As my 2 cents:

Got to debug FaaS via WDB: https://github.com/Kozea/wdb

You need to poke a hole from the FaaS machine to your machine. I am opening the hole via Ngrok service for now. Then you tell the debugger to attach to the hole on their end, opening the breakpoint on your machine.

Needed options on the FaaS environment:

• WDB_SOCKET_SERVER: The Ngrok hole entrypoint/hostname
• WDB_SOCKET_PORT=19840: The WDB server default socket port
• WDB_NO_BROWSER_AUTO_OPEN=True: The browser will not open itself

You need to keep an eye on localhost:1984 and open the stopped session, but this is very minor.

[alexellis]

I'll take a look at this @alanjds

I used the instructions from @kenfdev - they still work but I ran the container with the process of node --inspect-brk=0.0.0.0:9222 index.js instead of using it as the fprocess.

[alanjds]

Forgot to mention: WDB is a Python package. Use with Python functions.

[alexellis]

https://www.youtube.com/watch?v=uZPXwuKxsEU

Video: How to debug a Node.js function on Kubernetes with OpenFaaS

[thedemoncat]

Has anyone set up debugging for golang?

[alexellis]

/lock: inactivity