Close files of engine logger

[flandweber]

When running evolve.py the file descriptors for the individual logfiles of all started engine instances are not closed on exit of the instance. The open files can be viewed with ls -l /proc/<pid OR "self">/fd/.

This makes the engine collect hundreds of open files. The default limit is 1024 (ulimit -n) after which evolve.py (python evolve.py --test-type http --port 80 --log debug) crashes with

OSError: [Errno 24] Too many open files
Traceback (most recent call last):
  File "/home/vagrant/geneva/evolve.py", line 835, in <module>
    driver(sys.argv[1:])
  File "/home/vagrant/geneva/evolve.py", line 814, in driver
    hall_of_fame = genetic_solve(logger, options, ga_evaluator)
  File "/home/vagrant/geneva/evolve.py", line 552, in genetic_solve
    offspring = fitness_function(logger, offspring, ga_evaluator)
  File "/home/vagrant/geneva/evolve.py", line 207, in fitness_function
    return ga_evaluator.evaluate(population)
  File "/home/vagrant/geneva/evaluator.py", line 184, in evaluate
    self.worker(ind_list, "main", environment)
  File "/home/vagrant/geneva/evaluator.py", line 808, in worker
    eid, fitness = self.run_test(environment, ind)
  File "/home/vagrant/geneva/evaluator.py", line 239, in run_test
    self.plugin.start(self.args, self, environment, ind, self.logger)
  File "/home/vagrant/geneva/plugins/http/plugin.py", line 143, in start
    fitness = evaluator.run_client(evaluator.client_args, environment, logger)
  File "/home/vagrant/geneva/evaluator.py", line 301, in run_client
    self.run_local_client(args, environment, logger)
  File "/home/vagrant/geneva/evaluator.py", line 481, in run_local_client
    self.exec_cmd(command)
  File "/home/vagrant/geneva/evaluator.py", line 494, in exec_cmd
    subprocess.check_call(command, timeout=60)
  File "/usr/lib/python3.9/subprocess.py", line 368, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.9/subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1722, in _execute_child
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

This PR fixes this by employing the context manager in the main method and closing the logging handler upon exit.

[Kkevsterrr]

i think we might still want the finally to handle ^C or other catchable exceptions when the engine is invoked directly

[flandweber]

The __exit__-method of Engine is always called - even in case of an exception: https://book.pythontips.com/en/latest/context_managers.html#handling-exceptions

Since shutdown_nfqueue is called from there, we can (and should) remove the finally-block in my opinion.

[Kkevsterrr]

Ah you're right - I was remembering that this code did not use the context manager, and had missed that you swapped it to using it. Thanks for doing it - I'll merge it!

[Kkevsterrr]

Hey this is great - thanks for contributing! I thought I fixed this bug a while ago, but I guess not. I left one quick comment in there for you; happy to merge once we address that. Thanks!