Improperly Formatted NTRIP sourcetable response

[dclemmon]

Improperly Formatted NTRIP sourcetable response

Describe the bug

The NTRIP sourcetable HTTP response from socket_server.py is missing the \r\n separator line that should exist between the http header section and the body. This causes some http clients to treat the sourcetable data as a header, and can result is some clients hanging and timing out while they try to read the additional bytes specified in the Content-Length: header.

Running pygnssutils version 1.0.21, but issue exists in main branch

To Reproduce

Steps to reproduce the behaviour:

1. Start the socket_server in NTRIP mode
2. Use a HTTP client such as cURL to request the sourcetable

$ curl http:https://127.0.0.1:2101/ -H 'Connection: close' -H 'User-Agent: NTRIP pygnssutils' -H 'Accept: */*' -vvv -m 3

Here is the current output when running in verbose mode (-vvv):

:~$ curl http:https://127.0.0.1:2101/ -H 'Connection: close' -H 'User-Agent: NTRIP pygnssutils' -H 'Accept: */*' -vvv -m 3
*   Trying 127.0.0.1:2101...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 2101 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:2101
> Connection: close
> User-Agent: NTRIP pygnssutils
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Ntrip-Version: Ntrip/2.0
< Ntrip-Flags:
< Server: pygnssutils_NTRIP_Caster_1.0.21/of:10 Apr 2024
< Date: Wed, 10 Apr 2024 20:17:32 UTC
< Connection: close
< Content-Type: gnss/sourcetable
< Content-Length: 241
< STR;pygnssutils;PYGNSSUTILS;RTCM 3.3;1005(5),1077(1),1087(1),1097(1),1127(1),1230(1);0;GPS+GLO+GAL+BEI;SNIP;SRB;40.0;-105.0;0;0;sNTRIP;none;B;N;0;
< NET;SNIP;pygnssutils;N;N;pygnssutils;0.0.0.0:2101;[email protected];;
< ENDSOURCETABLE
* Operation timed out after 3001 milliseconds with 0 out of 241 bytes received
* Closing connection 0
curl: (28) Operation timed out after 3001 milliseconds with 0 out of 241 bytes received

The sourcetable content can be seen in the response headers, and cURL times out trying to read data that will not arrive.

Expected Behaviour

I confirmed in the NTRIP 2.0 Spec that the expected output is the sourcetable in the HTTP body.

Running cURL in verbose mode (-vvv) the expected output would be

:~$ curl http:https://127.0.0.1:2101/ -H 'Connection: close' -H 'User-Agent: NTRIP pygnssutils' -H 'Accept: */*' -vvv
*   Trying 127.0.0.1:2101...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 2101 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:2101
> Connection: close
> User-Agent: NTRIP pygnssutils
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Ntrip-Version: Ntrip/2.0
< Ntrip-Flags:
< Server: pygnssutils_NTRIP_Caster_1.0.21/of:10 Apr 2024
< Date: Wed, 10 Apr 2024 20:14:40 UTC
< Connection: close
< Content-Type: gnss/sourcetable
< Content-Length: 241
<
STR;pygnssutils;PYGNSSUTILS;RTCM 3.3;1005(5),1077(1),1087(1),1097(1),1127(1),1230(1);0;GPS+GLO+GAL+BEI;SNIP;SRB;40.0;-105.0;0;0;sNTRIP;none;B;N;0;
NET;SNIP;pygnssutils;N;N;pygnssutils;0.0.0.0:2101;[email protected];;
ENDSOURCETABLE
* Closing connection 0

In the above example, the sourcetabe data is not returned in the headers, and the client does not timeout waiting for data.

Desktop (please complete the following information):

• Windows 11 with WSL2

[semuadmin]

Hi @dclemmon,

Thanks for the shout.

I've uploaded a new branch fix-NTRIP-caster-HTTP-header-issue which should resolve this issue (tested on Windows, MacOS & Linux).

Results before 88bd8e1:

curl http:https://127.0.0.1:2101/ -H 'Connection: close' -H 'User-Agent: NTRIP pygnssutils' -H 'Accept: */*' -vvv -m 3 -u anon

Enter host password for user 'anon':
*   Trying 127.0.0.1:2101...
* Connected to 127.0.0.1 (127.0.0.1) port 2101
* Server auth using Basic with user 'anon'
> GET / HTTP/1.1
> Host: 127.0.0.1:2101
> Authorization: Basic YW5vbjpwYXNzd29yZA==
> Connection: close
> User-Agent: NTRIP pygnssutils
> Accept: */*
> 
< HTTP/1.1 200 OK
< Ntrip-Version: Ntrip/2.0
< Ntrip-Flags: 
< Server: pygnssutils_NTRIP_Caster_1.0.22/of:11 Apr 2024
< Date: Thu, 11 Apr 2024 06:55:41 UTC
< Connection: close
< Content-Type: gnss/sourcetable
< Content-Length: 251
* Header without colon
* Closing connection
curl: (8) Header without colon

Results after 88bd8e1:

curl http:https://127.0.0.1:2101/ -H 'Connection: close' -H 'User-Agent: NTRIP pygnssutils' -H 'Accept: */*' -vvv -m 3 -u anon

Enter host password for user 'anon':
*   Trying 127.0.0.1:2101...
* Connected to 127.0.0.1 (127.0.0.1) port 2101
* Server auth using Basic with user 'anon'
> GET / HTTP/1.1
> Host: 127.0.0.1:2101
> Authorization: Basic YW5vbjpwYXNzd29yZA==
> Connection: close
> User-Agent: NTRIP pygnssutils
> Accept: */*
> 
< HTTP/1.1 200 OK
< Ntrip-Version: Ntrip/2.0
< Ntrip-Flags: 
< Server: pygnssutils_NTRIP_Caster_1.0.22/of:11 Apr 2024
< Date: Thu, 11 Apr 2024 06:57:24 UTC
< Connection: close
< Content-Type: gnss/sourcetable
< Content-Length: 251
< 
STR;pygnssutils;PYGNSSUTILS;RTCM 3.3;1005(5),1077(1),1087(1),1097(1),1127(1),1230(1);0;GPS+GLO+GAL+BEI;SNIP;SRB;50.0;-1.50;0;0;sNTRIP;none;B;N;0;
NET;SNIP;pygnssutils;N;N;pygnssutils;0.0.0.0:2101;[email protected];;
ENDSOURCETABLE
* Closing connection

I'd be grateful if you could install this branch yourself and confirm whether or not you get the same result:

python3 -m pip install git+https://github.com/semuconsulting/pygnssutils@fix-NTRIP-caster-HTTP-header-issue

Thanks

[dclemmon]

Thanks for the quick response! I was able to test it out and your code change is working perfectly! Would you like me to close the issue from my side, or would you like to close it?

[semuadmin]

Thanks @dclemmon . The issue wil be closed automatically once the branch is marged with main, which will hopefully be in the next week or so.

Thanks again for the comprehensive issue report - most helpful.