Manage exceptions and ensure uptime on _controller

Handle exceptions more aggressively to ensure up-time in controller.
Notably the h/w read from thermostat fails fairly often in production,
so need to ignore it. Might as well ignore/log everything since the
alternative is pretty bad (system just stops working).

thermoclient.rb

@@ -15,12 +15,14 @@
 
 module Thermo
  class ThermoError < RuntimeError; end
- class ConfigFileNotFound < ThermoError; end
- class UnknownSchedule < ThermoError; end
- class InvalidTemperature < ThermoError; end
- class HWTempRead < ThermoError; end
- class InitializeFailed < ThermoError; end
- class UnknownRunMode < ThermoError; end
+ class ThermoCriticalError < ThermoError; end
+ class InitializeFailed < ThermoCriticalError; end
+ class ConfigFileNotFound < ThermoCriticalError; end
+ class ThermoRuntimeError < ThermoError; end
+ class InvalidTemperature < ThermoRuntimeError; end
+ class UnknownSchedule < ThermoRuntimeError; end
+ class HWTempRead < ThermoRuntimeError; end
+ class UnknownRunMode < ThermoRuntimeError; end
 
  BOOT_FILE_NAME = "boot.json"
  HYSTERESIS_DURATION_DEFAULT = "5 minutes"

thermoclient_controller.rb

@@ -1,7 +1,7 @@
 # run thermoclient repeatedly, try to turn off heater when thermoclient exits
 
 # amount of time to re-run thermoclient before exiting
-DEFAULT_RUN_TIME = '12 hours'
+DEFAULT_RUN_TIME = '24 hours'
 DEFAULT_INTERVAL_BETWEEN_RUNS = '15 seconds'
 DEFAULT_WORKING_FOLDER = './config'
 
@@ -19,20 +19,22 @@
 puts Time::now
 begin
  while start_time < end_time
- #puts "Processing schedule"
- thermostat.process_schedule
- #puts "Sleeping"
- sleep(wait_time)
- end
-# on any exception attempt to turn off heater
-rescue Exception => e
- begin
- thermostat = nil
- thermo_rescue = Thermo::Thermostat.new
- thermo_rescue.set_heater_state(false)
- ensure
- raise e
+ begin
+ thermostat.process_schedule
+ sleep(wait_time)
+ rescue ThermoRuntimeError => e
+ puts "Runtime exception encountered. Retrying.."
+ puts " Exception class: #{e.class.to_s}. Msg: #{e.message}.\n Backtrace: #{e.backtrace}"
+ rescue ThermoCriticalError => e
+ puts "*** Critical failure encountered. Retrying ***"
+ puts " Exception class: #{e.class.to_s}. Msg: #{e.message}.\n Backtrace: #{e.backtrace}"
+ end
  end
+# on any exception or exit, attempt to turn off heater
+ensure
+ thermostat = nil
+ thermo_rescue = Thermo::Thermostat.new
+ thermo_rescue.set_heater_state(false)
 end
 puts "Ran to completion successfully at #{Time::now}"
 puts " started: #{start_time} // ended: #{end_time}"