Reworked PDU parsing; Delivery reports

[magcks]

Hey there again :)

In this PR, I reworked the whole PDU parsing and building code. This includes:

• Timestamp are parsed now and forwarded to the dial plan and the manager
• SMS-STATUS-REPORT PDUs can be processed from now on

To handle the status reports correctly, this PR creates a database row for each message part of a concatenated SM (i.e. for each single SM) including the message reference returned from the modem. Once, a status report is received, a flag is updated in the database. When all reports are received, a "report" will be issued to the dial plan. This PR also gives you the opportunity to pass a payload to each message to identify it when the delivery report is completed (useful for a Mail-to-SMS gateway).

I also included commit 3a85d35 @miopa to support sending to national numbers (e.g. 0049... instead of +49...) and 'fixed' the USSD sending functions.

Major things that changed:

• The SMSDB file (I didn't include an migration script for Concatenated SMS support; Complete locking and single shift tables #75, just do $ sudo rm /var/lib/asterisk/smsdb.sqlite)
• You have to apply an identifier string to DongleSendSMS for the delivery reports
• The manager events for delivery reports are handled differently from now on (I assume nobody used them, as they were just a complete hack)
• This PR drops support for sending raw PDUs from the manager
• Disable TEXT mode for sending and receiving (now PDU-only)

I hope there are some people that can test this PR, as it changes almost the whole SMS and USSD part of the project. I only tested with an E1762 and didn't used my code in 'production' yet.

@wdoekes: Thanks for reviewing #75 :)

[magcks]

..ah, and sorry for that large commit. I usually have that strange workflow that I code everything that is in my mind and afterwards I remove the compiler issues and do only fixup commits ;)

[wdoekes]

..ah, and sorry for that large commit. I usually have that strange workflow that I code everything that is in my mind and afterwards I remove the compiler issues and do only fixup commits ;)

I don't think it's that strange. But it is a pain to review ;)

I don't think I got everything, I probably skimmed over most. There are some style issues, and a few things with buffers of unknown size.

(And as always: I don't own a dongle, so I tested none of it.)

[magcks]

I think I've fixed all your comments. Let's wait for the feedback from #78.

[hselasky]

Use "static const" before:

channel_var_t vars[] =

[hselasky]

Can we keep parse_cmgr_text() even if it is deprecated?

[magcks]

Thanks for your comments. I dropped it, as I was too lazy to fully implement it (the initial version didn't parse the SCTS field). Theoretically there is no reason to use the text mode as SMs are always transmitted as PDUs. The text mode is only for users that use the modem directly via the TTY port.

Does your modem work correctly (does the PDU mode work)? It would be great if you can tell me which model you use and if voice, USSD and Multi-SMS (sending and receiving) works.

[magcks]

Use "static const" before:

channel_var_t vars[] =

I can do that. But static won't work as the content is basically not static, right?

[hselasky]

Use "static const" before:
channel_var_t vars[] =

I can do that. But static won't work as the content is basically not static, right?

Isn't vars[] just an array of constants?

[magcks]

Nope, at least the payload for delivery reports is always included and not constant.

[hselasky]

You're right. Now I see.

[m7mdcc]

I just come to say , Good job @magcks & @wdoekes

I have test the PR , there is issue let me explain it ..
its seems same issue like #75 (comment)

if you send concatenated message its will be store as 1 message , but if you send another short message after the concatenated message its will contain text from previous message "concatenated message" .

my dongle.conf is

`

[general]
interval=15
[defaults]
context=dongle-incoming
group=0
rxgain=0
txgain=0
autodeletesms=yes
resetdongle=yes
u2diag=-1
usecallingpres=yes
callingpres=allowed_passed_screen
disablesms=no
language=en
mindtmfgap=45
mindtmfduration=80
mindtmfinterval=200
callwaiting=auto
disable=no
;exten=+1234567890
dtmf=relax unaltered diaplan
[dongle0]
audio=/dev/ttyUSB1
data=/dev/ttyUSB2

`

[wdoekes]

So, is this ready to ship @magcks ?

Waiting for feedback from #78 ?

[magcks]

Thanks for your comment @m7mdcc . Can you please tell me the content of the PDU? You can log the content using
ast_log(LOG_WARNING, "PDU: %s\n", str);

in at_parse.c#322

Does the second SM contain the complete content from the first SM or only parts?

Thanks! :)

[m7mdcc]

@magcks , do you want to add this line in 322 in file at_parse.c before compiling ?
`
EXPORT_DEF int at_parse_cmgr(char *str, size_t len, int *tpdu_type, char *sca, size_t sca_len, char *oa, size_t oa_len, char *scts, int *mr, int *st, char *dt, char *msg, size_t *msg_len, pdu_udh_t udh)
{
==> here ? / skip "+CMGR:" */
str += 6;
len -= 6;

`

the second message contain only one part of first message .

[magcks]

Yes, please. It will print the PDU content as a warning to the asterisk console before parsing it.

[m7mdcc]

Ok here is the log , first i receive concatenated message then i receive short message which only contain "Test" , if you see the log its say : [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'Test'; Concatenated [ref,parts,order]: 0 1 2
its marked as Concatenated since its not .

[2020-05-04 21:58:21] VERBOSE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,159 0791695650309199400C916956854351020008025040228201218C05000325020106450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002000200020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D0628062700200645' [2020-05-04 21:58:21] NOTICE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,159 0791695650309199400C916956854351020008025040228201218C05000325020106450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002000200020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D0628062700200645' [2020-05-04 21:58:21] WARNING[2164] at_parse.c: PDU: +CMGR: 0,,159 0791695650309199400C916956854351020008025040228201218C05000325020106450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002000200020002006450631062D06280627002006450631062D06280627002006450631062D062806270020002006450631062D0628062700200645 [2020-05-04 21:58:21] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا م'; Concatenated [ref,parts,order]: 37 2 1 [2020-05-04 21:58:21] VERBOSE[2164] at_response.c: [dongle0] Waiting for following parts [2020-05-04 21:58:23] VERBOSE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,49 0791695650309199400C916956854351020008025040228211211E0500032502020631062D06280627002006450631062D0628062700200020' [2020-05-04 21:58:23] NOTICE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,49 0791695650309199400C916956854351020008025040228211211E0500032502020631062D06280627002006450631062D0628062700200020' [2020-05-04 21:58:23] WARNING[2164] at_parse.c: PDU: +CMGR: 0,,49 0791695650309199400C916956854351020008025040228211211E0500032502020631062D06280627002006450631062D0628062700200020 [2020-05-04 21:58:23] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'رحبا مرحبا '; Concatenated [ref,parts,order]: 37 2 2 [2020-05-04 21:58:23] VERBOSE[2164] at_response.c: [dongle0] Got full SMS from +XXXXXXXXXXXX: 'مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Executing [sms@dongle-incoming:1] Verbose("Local/sms@dongle-incoming-00000000;1", "Incoming SMS from +XXXXXXXXXXXX مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ") in new stack [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] app_verbose.c: Incoming SMS from +XXXXXXXXXXXX مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Executing [sms@dongle-incoming:2] System("Local/sms@dongle-incoming-00000000;1", "echo '2020-05-04 21:58:23 - - +XXXXXXXXXXXX: مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' >> /var/log/asterisk/sms.txt") in new stack [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Executing [sms@dongle-incoming:3] Hangup("Local/sms@dongle-incoming-00000000;1", "") in new stack [2020-05-04 21:58:23] VERBOSE[2403][C-00000001] pbx.c: Spawn extension (dongle-incoming, sms, 3) exited non-zero on 'Local/sms@dongle-incoming-00000000;1' [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,23 0791695650309199040C9169568543510200000250500085512104D4F29C0E' [2020-05-04 21:58:24] NOTICE[2164] at_response.c: [dongle0] Got Response for user's command:'+CMGR: 0,,23 0791695650309199040C9169568543510200000250500085512104D4F29C0E' [2020-05-04 21:58:24] WARNING[2164] at_parse.c: PDU: +CMGR: 0,,23 0791695650309199040C9169568543510200000250500085512104D4F29C0E [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got SMS part from +XXXXXXXXXXXX: 'Test'; Concatenated [ref,parts,order]: 0 1 2 [2020-05-04 21:58:24] VERBOSE[2164] at_response.c: [dongle0] Got full SMS from +XXXXXXXXXXXX: 'Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Executing [sms@dongle-incoming:1] Verbose("Local/sms@dongle-incoming-00000001;1", "Incoming SMS from +XXXXXXXXXXXX Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ") in new stack [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] app_verbose.c: Incoming SMS from +XXXXXXXXXXXX Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Executing [sms@dongle-incoming:2] System("Local/sms@dongle-incoming-00000001;1", "echo '2020-05-04 21:58:24 - - +XXXXXXXXXXXX: Testحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا مرحبا ' >> /var/log/asterisk/sms.txt") in new stack [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Executing [sms@dongle-incoming:3] Hangup("Local/sms@dongle-incoming-00000001;1", "") in new stack [2020-05-04 21:58:24] VERBOSE[2406][C-00000002] pbx.c: Spawn extension (dongle-incoming, sms, 3) exited non-zero on 'Local/sms@dongle-incoming-00000001;1

[magcks]

Okay, thanks. Looks like an uninitialized value. I will have a look.

[magcks]

Could you please check now. The code at least compiles now, but I currently don't have time to test and I don't know where I put my sticks :D

[m7mdcc]

@magcks , Thanks .. I can confirm issue is solved now .

[magcks]

@m7mdcc thank you very much for testing :)
@wdoekes alrighty. in this case it should be ready to merge.

[wdoekes]

Can you get these conflicts resolved? Thanks!

[magcks]

I resolved the conflicts, but didn't test if #79 still works. Nevertheless, I recommend to merge this PR as soon as possible, as there are serious bugs (smsdb threads and string termination) in #75 that are fixed with this PR.

[hselasky]

Looks good to me.

[hselasky]

Maybe you could squash that last change commit. Minor nit though.