Apply static routes on change to master state

[remibergsma]

Refactored static routes for private gateways so they also get loaded when the router switches to master state. Otherwise they're lost and connections drop after fail over.

[swill]

Can I get some code reviews on this PR? I have added it to my CI queue...

[rafaelweingartner]

Does the “CsHelper.execute” logs the commands being executed and their outputs?
If not, what about adding logs to show the routes that are being added? Also, after line 35, what about a log to state that the route is being revoked.

Additionally, if this is merged, is it going to be merged forward (4.8 and master)?

[rohityadavcloud]

tag:needlove

[kiwiflyer]

We'll be pulling this one in.

[jburwell]

Should the execute be checked and error thrown and/or logged on failure?

[dmabry]

So, Si (@kiwiflyer) and I have been testing this functionality against VPCs in our lab and we verified that static routes are indeed loaded when a VR goes from BACKUP to MASTER. The relevant logs showing that is works are below. We are however still having problems with the static routes not getting added on demand to the current MASTER VR. Has anyone else had this issue while running this commit?

2016-05-06 20:32:20,795 DEBUG    {u'config': {u'router_id': u'2', u'baremetalnotificationapikey': u'QtWfDPFOrqddFe3iofWCS6Yhf6g2cCTwR6n3-L7TaaKl3ReBI4219jckJ0AfdX878jo2DpW4ohhNG_6Hlad62g', u'baremetalnotificationsecuritykey': u'B99mUf0oWKL7ATA5WF9iMb17nk1ePQ6kgg3YgWA0kJE2vCaGH52Z65EWDAyCaOYiZB1YPr40pWiwTlBt7_TTmg', u'name': u'r-28-VM', u'eth0mask': u'255.255.0.0', u'type': u'vpcrouter', u'dns1': u'207.191.191.90', u'privategateway': u'192.168.4.4', u'vpccidr': u'10.15.0.0/22', u'domain': u'cs2cloud.internal', u'redundant_state': 'BACKUP', u'host': u'10.150.0.201', u'disable_rp_filter': u'true', u'redundant_router': u'true', u'router_password': u'3838383966687309145665215760380045387957133810480803993986340699863084083818544695063990662780164604461468366180051318649230056294824943648327772148445755', u'redundant_master': False, u'port': u'8080', u'eth0ip': u'169.254.3.21', u'template': u'domP'}, u'id': u'cmdline'}
2016-05-06 20:32:20,796 DEBUG    Loading data bag type ips
2016-05-06 20:32:20,796 DEBUG    Setting router to master
2016-05-06 20:32:20,796 INFO     Will proceed configuring device ==> eth2
2016-05-06 20:32:20,797 DEBUG    Executing: ip link set eth2 up
2016-05-06 20:32:20,800 INFO     Bringing public interface eth2 up
2016-05-06 20:32:20,800 INFO     Adding gateway ==> 192.168.4.1 to device ==> eth2
2016-05-06 20:32:20,801 INFO     Will proceed configuring device ==> eth1
2016-05-06 20:32:20,801 DEBUG    Executing: ip link set eth1 up
2016-05-06 20:32:20,804 INFO     Bringing public interface eth1 up
2016-05-06 20:32:20,804 INFO     Adding gateway ==> 10.152.0.1 to device ==> eth1
2016-05-06 20:32:20,804 INFO     Checking if default ipv4 route is present
2016-05-06 20:32:20,804 DEBUG    Executing: ip -4 route list 0/0
2016-05-06 20:32:20,807 WARNING  No default route found!
2016-05-06 20:32:20,807 INFO     Adding default route
2016-05-06 20:32:20,807 DEBUG    Executing: ip route show default via 10.152.0.1
2016-05-06 20:32:20,810 INFO     Add default via 10.152.0.1
2016-05-06 20:32:20,810 DEBUG    Executing: ip route add default via 10.152.0.1
2016-05-06 20:32:20,812 DEBUG    Configuring static routes
2016-05-06 20:32:20,813 DEBUG    Loading data bag type staticroutes
2016-05-06 20:32:20,813 DEBUG    Processing CsStaticRoutes file ==> {u'id': u'staticroutes', u'10.87.0.0/16': {u'revoke': False, u'ip_address': u'192.168.4.4', u'gateway': u'192.168.4.1', u'network': u'10.87.0.0/16'}}
2016-05-06 20:32:20,813 DEBUG    Executing: ip route show | grep 10.87.0.0/16 | awk '{print $1, $3}'
2016-05-06 20:32:20,818 DEBUG    Executing: ip route add 10.87.0.0/16 via 192.168.4.1
2016-05-06 20:32:20,821 DEBUG    Executing: /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf -c
2016-05-06 20:32:20,824 DEBUG    Executing: /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf -f
2016-05-06 20:32:20,828 DEBUG    Executing: /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf -R
2016-05-06 20:32:20,831 DEBUG    Executing: /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf -B

[swill]

CI RESULTS

Tests Run: 85
  Skipped: 0
   Failed: 1
   Errors: 0
 Duration: 9h 06m 23s

Summary of the problem(s):

FAIL: Test redundant router internals
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/git/cs1/cloudstack/test/integration/smoke/test_routers_network_ops.py", line 478, in test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false
    result = check_router_command(virtual_machine, nat_rule.ipaddress, ssh_command, check_string, self)
  File "/data/git/cs1/cloudstack/test/integration/smoke/test_routers_network_ops.py", line 61, in check_router_command
    test_case.fail("Failed to SSH into the Virtual Machine: %s" % e)
AssertionError: Failed to SSH into the Virtual Machine: [Errno 110] Connection timed out
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_KAUTZA/results.txt

Associated Uploads

/tmp/MarvinLogs/DeployDataCenter__May_11_2016_00_29_07_UR1IBW:

• dc_entries.obj
• failed_plus_exceptions.txt
• runinfo.txt

/tmp/MarvinLogs/test_network_KAUTZA:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

/tmp/MarvinLogs/test_vpc_routers_N2UD4H:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

Uploads will be available until 2016-07-11 02:00:00 +0200 CEST

Comment created by upr comment.

[kiwiflyer]

LGTM. Tested in a hardware lab.

[swill]

@rafaelweingartner yes, it will be forward merged. Sorry I only just got around to answering your question. I need one more code review on this one. Thanks...

[dmabry]

LGTM. We tested this in our Lab on HW and all static routes on each PG came up as expected.

[kiwiflyer]

tag:mergeready

[swill]

Thanks gents. I will get this merged...