UISP Timeouts

[lakelinx]

was having an issue where customer plan speeds werent being updated. I looked and our shaped devices file hadnt been updated since we updated to latest main branch 3 days ago. I tried manually running the uisp integration just to see what it was saying:

lqos@lqos:/opt/libreqos/src$ python3 integrationUISP.py
Running Python Version 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
Loading Data from UISP
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1130, in read
    return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='portal.lakelinx.net', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/libreqos/src/integrationUISP.py", line 486, in <module>
    importFromUISP()
  File "/opt/libreqos/src/integrationUISP.py", line 479, in importFromUISP
    case "full": buildFullGraph()
  File "/opt/libreqos/src/integrationUISP.py", line 315, in buildFullGraph
    devices = uispRequest("devices?withInterfaces=true&authorized=true")
  File "/opt/libreqos/src/integrationUISP.py", line 21, in uispRequest
    r = requests.get(url, headers=headers, timeout=10)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='portal.lakelinx.net', port=443): Read timed out. (read timeout=10)

Im able to ping the uisp host with no issues.

lqos@lqos:/opt/libreqos/src$ ping portal.lakelinx.net
PING portal.lakelinx.net (130.44.211.12) 56(84) bytes of data.
64 bytes from 130-44-211-12.client.lakelinx.net (130.44.211.12): icmp_seq=1 ttl=62 time=43.7 ms
64 bytes from 130-44-211-12.client.lakelinx.net (130.44.211.12): icmp_seq=2 ttl=62 time=0.460 ms
64 bytes from 130-44-211-12.client.lakelinx.net (130.44.211.12): icmp_seq=3 ttl=62 time=0.557 ms
64 bytes from 130-44-211-12.client.lakelinx.net (130.44.211.12): icmp_seq=4 ttl=62 time=0.491 ms
64 bytes from 130-44-211-12.client.lakelinx.net (130.44.211.12): icmp_seq=5 ttl=62 time=0.435 ms
64 bytes from 130-44-211-12.client.lakelinx.net (130.44.211.12): icmp_seq=6 ttl=62 time=0.253 ms
^C
--- portal.lakelinx.net ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5036ms
rtt min/avg/max/mdev = 0.253/7.648/43.692/16.119 ms

After being stupid and trying to run it 10 times in a row, the 4th attempt it succeeded and rebuilt the file. Upon running the command again several times in a row, it timed out, but every (n)th run would succeed.

Running the latest UISP Beta if that means anything. We regenerated the API Key just to be on the safe side and that did not change the behavior any.

[trendal]

No issues with your SSL cert on UISP?

[lakelinx]

Updated above with complete uisp integration output.

No cert issues that im aware of. Expires June of 2023

[trendal]

If it's easily repeatable tcpdump the interface on the UISP server while running the (failing) script. I do that just for a sanity check mostly.