Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unittests/uri_parser: Rework tests to be more verbose #18734

Merged
merged 1 commit into from
Nov 29, 2022

Conversation

Teufelchen1
Copy link
Contributor

@Teufelchen1 Teufelchen1 commented Oct 13, 2022

Hi! 👋

As prototyped and reviewed in #18629, this is the start of rewriting the unit tests of the uri_parser.

I believe it can safely be merged into RIOT there are however some things worth noting:

  • I wanted to tests for invalid schemes but failed since the uri_parser just can't 😢
  • Some minor coverage is lost compared to the old test but a lot is gained including new (different) coverage. (Again, see my comments)
  • This only partially rewrites the tests, some old tests remain.
  • The test size (as in bytes & execution time) has increased. Be aware of that when testing via HIL.

This is related work to #18702
The testing of the uri_parser remains far from being complete after this is merged.

@github-actions github-actions bot added the Area: tests Area: tests and testing framework label Oct 13, 2022
char *simple_uri_19 = "coap:https:///R@[2008::1]:5own//R@[2008::1]:5own/?v=1";
char *simple_uri_20 = "coaP:https://R/RZ[2001[8:01[8::1]:5o:1]:5oTMv=1";
char *simple_uri_21 = "coap:https://R@////////////////7///v=1";
char *simple_uri_22 = "coa[:https:////[2001:db5ow:5own/Ov=1";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A ton of the old test vectors are just like "mh okay? What exactly is being tested?". If you got any ideas, please drop them so I can rename them from simple_uri to something more helpful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess @cgundogan is the best person to ask here.

* The invalid uri are in the next test. The numbering represents the
* sequence / order in the old test.
*/
char *simple_uri_05 = "coap:https://RIOT:test@[fe80:db8::1%tap0]:5683/.well-known/core?v=1";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old tests used to also check every component in the result after parsing. This is no longer the case.

@Teufelchen1 Teufelchen1 force-pushed the tests/uri_parser branch 2 times, most recently from a1fd37b to 6c24d62 Compare October 13, 2022 13:03
Copy link
Member

@PeterKietzmann PeterKietzmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor inline comments from a very rough review. Overall, the changes look well justified. In particular, the code gains readability and newly introduced cases extend the test scope of the parser. But I did not check if (i) existing test cases were translated properly, nor (ii) if "relevant" test cases have been removed.

The unit test still passes on native as well as nrf52840dk.

tests/unittests/tests-uri_parser/tests-uri_parser.c Outdated Show resolved Hide resolved
char *simple_uri_03 = "https://riot-os.org:99/bar/foo";
char *simple_uri_04 = "https://riot-os.org/";

/* these are the old test vectors which contained valid uri.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this go to the code bass (but nice for reviewing)

char *failure_msg = "Failure: The uri_parser failed to reject an invalid uri.";
int expected_ret = -1; /* Indicates an invalid uri */

/* these are the old test vectors containing invalid uri */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

(strncmp(actual_str, expected_str, actual_len) == 0)) {
return 0;
}
printf(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, you have actually looked at the output of this, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes ofc.
It looks like this:

main(): This is RIOT! (Version: 2018.01-devel-26059-g6c24d-tests/uri_parser)
Help: Press s to start test, r to print it is ready
s
START
......
URI-Input: co ap:https://example.org/
Expected return: -1
Observed return: 0
uri_parser_tests._error_if_scheme_is_invalid (tests/unittests/tests-uri_parser/tests-uri_parser.c 371) Failure: The uri_parser failed to reject an invalid scheme.
.....
run 11 failures 1

tests/unittests/tests-uri_parser/tests-uri_parser.c Outdated Show resolved Hide resolved
tests/unittests/tests-uri_parser/tests-uri_parser.c Outdated Show resolved Hide resolved
ret = uri_parser_process_string(&uri_res, test_vec[i]);
if (ret != expected_ret) {
/* At the moment, the uri_parser is not capable of detecting invalid schemes
*_print_return_expectation(test_vec[i], expected_ret, ret);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have some kind of TODO tag so this doesn't get lost?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed from the PR. I will add this test later as soon as the uri-parser gets an appropriate update :)

tests/unittests/tests-uri_parser/tests-uri_parser.c Outdated Show resolved Hide resolved
tests/unittests/tests-uri_parser/tests-uri_parser.c Outdated Show resolved Hide resolved
@Teufelchen1
Copy link
Contributor Author

Thanks for your review @PeterKietzmann addressed most of your comments and already marked the spelling errors as resolved.

@Teufelchen1 Teufelchen1 force-pushed the tests/uri_parser branch 2 times, most recently from a05f9ab to 49efb72 Compare November 29, 2022 15:44
@RIOT-OS RIOT-OS deleted a comment from riot-hil-bot Nov 29, 2022
@riot-hil-bot
Copy link

HiL Test Results

PASS FAIL SKIP
10 1 12
  😬 frdm-k64f (1 fail flash)
PASS FAIL SKIP
0 1 0
TEST RESULT
tests/unittests 😬 flash fail
  ✅ frdm-k22f
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ nucleo-f091rc
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ nucleo-f303re
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ nucleo-f767zi
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ nucleo-l152re
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ arduino-mega2560
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ nucleo-l432kc
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ saml10-xpro
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ saml11-xpro
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ samr21-xpro
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ nrf52dk
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ remote-revb
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ nucleo-l073rz
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ slstk3401a
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ nucleo-f411re
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ nucleo-f103rb
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ frdm-kw41z
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ esp8266-esp-12x
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ esp32-wroom-32
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ z1
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip
  ✅ arduino-due
PASS FAIL SKIP
1 0 0
TEST RESULT
tests/unittests ✅ pass
  ✅ slstk3400a
PASS FAIL SKIP
0 0 1
TEST RESULT
tests/unittests 🙈 skip

Copy link
Contributor

@MrKevinWeiss MrKevinWeiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearly an improvement for test documentation. I have run locally and let the CI run through things. No need to delay as all comments seem to be addressed.

ACK.

@MrKevinWeiss MrKevinWeiss added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Nov 29, 2022
@riot-ci
Copy link

riot-ci commented Nov 29, 2022

Murdock results

✔️ PASSED

49efb72 unittests/uri_parser: Rework tests to be more verbose

Success Failures Total Runtime
1 0 1 50s

Artifacts

This only reflects a subset of all builds from https://ci-prod.riot-os.org. Please refer to https://ci.riot-os.org for a complete build for now.

@MrKevinWeiss MrKevinWeiss merged commit 16b6162 into RIOT-OS:master Nov 29, 2022
@MrKevinWeiss
Copy link
Contributor

Somehow murdock didn't really run any tests huh...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants