ExecutePreprocessor error display and save

[tritemio]

In using nbconvert 4.0 to execute a notebook (many notebook) from a single "master" notebook.

What I'm executing is something like:

nb = nbformat.read(nb_name_full, as_version=4)
ep = ExecutePreprocessor(timeout=3600, allow_errors=False)
out = ep.preprocess(nb, {'metadata': {'path': './'}})
nbformat.write(nb, out_nb_name)

When there is an error in the executed notebook, I get this stack trace with the actual error message encoded in a unlegible way at the end:

---------------------------------------------------------------------------
CellExecutionError                        Traceback (most recent call last)
<ipython-input-7-78bb05211e53> in <module>()
----> 1 out = ep.preprocess(nb, {'metadata': {'path': './'}})

C:\Users\laser2002j\Anaconda\envs\py3\lib\site-packages\nbconvert\preprocessors\execute.py in preprocess(self, nb, resources)
     81 
     82         try:
---> 83             nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
     84         finally:
     85             self.kc.stop_channels()

C:\Users\laser2002j\Anaconda\envs\py3\lib\site-packages\nbconvert\preprocessors\base.py in preprocess(self, nb, resources)
     68         """
     69         for index, cell in enumerate(nb.cells):
---> 70             nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
     71         return nb, resources
     72 

C:\Users\laser2002j\Anaconda\envs\py3\lib\site-packages\nbconvert\preprocessors\execute.py in preprocess_cell(self, cell, resources, cell_index)
     95             return cell, resources
     96         try:
---> 97             outputs = self.run_cell(cell)
     98         except Exception as e:
     99             self.log.error("failed to run cell: " + repr(e))

C:\Users\laser2002j\Anaconda\envs\py3\lib\site-packages\nbconvert\preprocessors\execute.py in run_cell(self, cell)
    127             if msg['parent_header'].get('msg_id') == msg_id:
    128                 if msg['metadata']['status'] == 'error' and not self.allow_errors:
--> 129                     raise CellExecutionError(msg['content']['traceback'])
    130                 else:
    131                     break

CellExecutionError: ['\x1b[1;31m---------------------------------------------------------------------------\x1b[0m', '\x1b[1;31mNameError\x1b[0m                                 Traceback (most recent call last)', "\x1b[1;32m<ipython-input-6-d71af90bd51c>\x1b[0m in \x1b[0;36m<module>\x1b[1;34m()\x1b[0m\n\x1b[0;32m      1\x1b[0m ph_sel_map = {'all-ph': Ph_sel('all'), 'Dex': Ph_sel(Dex='DAem'), \n\x1b[0;32m      2\x1b[0m               'DexDem': Ph_sel(Dex='Dem')}\n\x1b[1;32m----> 3\x1b[1;33m \x1b[0mph_sel\x1b[0m \x1b[1;33m=\x1b[0m \x1b[0mph_sel_map\x1b[0m\x1b[1;33m[\x1b[0m\x1b[0mph_sel_name\x1b[0m\x1b[1;33m]\x1b[0m\x1b[1;33m\x1b[0m\x1b[0m\n\x1b[0m\x1b[0;32m      4\x1b[0m \x1b[1;33m\x1b[0m\x1b[0m\n\x1b[0;32m      5\x1b[0m \x1b[0mdata_id\x1b[0m\x1b[1;33m,\x1b[0m \x1b[0mph_sel_name\x1b[0m\x1b[1;33m\x1b[0m\x1b[0m\n", "\x1b[1;31mNameError\x1b[0m: name 'ph_sel_name' is not defined"]

I also get a similarly cryptic message in the terminal:

[IPKernelApp] ERROR | failed to run cell: CellExecutionError(['\x1b[1;31m---------------------------------------------------------------------------\x1b[0m', '\x1b[1;31mNameError\x1b[0m                                 Traceback (most recent call last)', "\x1b[1;32m<ipython-input-6-d71af90bd51c>\x1b[0m in \x1b[0;36m<module>\x1b[1;34m()\x1b[0m\n\x1b[0;32m      1\x1b[0m ph_sel_map = {'all-ph': Ph_sel('all'), 'Dex': Ph_sel(Dex='DAem'), \n\x1b[0;32m      2\x1b[0m               'DexDem': Ph_sel(Dex='Dem')}\n\x1b[1;32m----> 3\x1b[1;33m \x1b[0mph_sel\x1b[0m \x1b[1;33m=\x1b[0m \x1b[0mph_sel_map\x1b[0m\x1b[1;33m[\x1b[0m\x1b[0mph_sel_name\x1b[0m\x1b[1;33m]\x1b[0m\x1b[1;33m\x1b[0m\x1b[0m\n\x1b[0m\x1b[0;32m      4\x1b[0m \x1b[1;33m\x1b[0m\x1b[0m\n\x1b[0;32m      5\x1b[0m \x1b[0mdata_id\x1b[0m\x1b[1;33m,\x1b[0m \x1b[0mph_sel_name\x1b[0m\x1b[1;33m\x1b[0m\x1b[0m\n", "\x1b[1;31mNameError\x1b[0m: name 'ph_sel_name' is not defined"],)
[IPKernelApp] ERROR | ph_sel_map = {'all-ph': Ph_sel('all'), 'Dex': Ph_sel(Dex='DAem'),
              'DexDem': Ph_sel(Dex='Dem')}
ph_sel = ph_sel_map[ph_sel_name]

data_id, ph_sel_name

Moreover, the biggest issue is that saving the notebook (4th line) after the error does not save the stack trace like, so the error is hard to debug.

In IPython 3.x I used to monkey-patch ExecutePreprocessor.preprocess_cell() with:

def preprocess_cell(self, cell, resources, cell_index):
    """
    Apply a transformation on each code cell. See base.py for details.
    """
    if cell.cell_type != 'code':
        return cell, resources

    outputs = self.run_cell(cell)
    cell.outputs = outputs

    for out in outputs:
        if out.output_type == 'error':
            msg = 'Error executing the following the notebook cell:\n'
            msg += str(cell.source)
            raise RuntimeError(msg)

    return cell, resources

which gives me nicer error messages in the master notebook and the stack trace was saved in the executed notebook.

For the record, the error message before looked like:

RuntimeError: Error executing the following the notebook cell:
from fretbursts import *

How can I achieve the old behaviour?

Related issues: ipython/ipython#8286, #17

[tritemio]

I tried allow_errors = True and the saved notebook has the stack traces of all the failing cells. I'd like to stop at the first error and be able to save the stack trace.

Any suggestion is appreciated.

[tritemio]

Previous PR #33 introduced the stop-on-error option via allow_errors flag.

[tritemio]

I think the problem is that by raising CellExecutionError so early, the second part of run_cell() which retrieves and returns the output is not executed. Therefore preprocess_cell() has no way to capture the output which contains the traceback to be saved. Also, returning the output on error, allows to print a meaningful message, for example which line caused the error.

[tritemio]

I hacked this PR #120 to includes the stack trace in the notebook (even with allow_error = False) and to print a meaningful error message.

The implementation raises CellExecutionError in preprocess_cell() instead of run_cell() and this give the opportunity to save the output (including the stack trace).

[tritemio]

Closed by #120