Run Erlang/OTP smoke tests before the build will be installed

[drobakowski]

From the documentation: Testing

[drobakowski]

For now I'm getting three test case failures. As I'm seeing this failures also in other installations, I'm not sure if these once could be ignored for now.

ALL RUNS

                                             Test Name                            Label     Test Run Started     Ok Failed   Skipped
                                                                                                                           (User/Auto) Missing
                                                                                                                                       Suites           Node           CT Log Old Runs
                  tests.emulator_test.crypto_SUITE.t_md5                            -   Thu Dec 10 2015 18:34:11  1      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.emulator_test.float_SUITE.cases                             -   Thu Dec 10 2015 18:34:11  2      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.emulator_test.smoke_test_SUITE                              -   Thu Dec 10 2015 18:34:11  3      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.emulator_test.time_SUITE                                    -   Thu Dec 10 2015 18:34:11 14      3     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.erl_interface_test.ei_decode_encode_SUITE                   -   Thu Dec 10 2015 18:35:43  1      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.ic_test.ic_SUITE                                            -   Thu Dec 10 2015 18:35:48 35      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.jinterface_test.jinterface_SUITE.java_erlang_send_receive   -   Thu Dec 10 2015 18:35:46  0      0     1 (1/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.kernel_test.gen_tcp_echo_SUITE.active_echo                  -   Thu Dec 10 2015 18:35:20  1      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.kernel_test.heart_SUITE.reboot                              -   Thu Dec 10 2015 18:35:20  1      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.kernel_test.inet_SUITE.cases                                -   Thu Dec 10 2015 18:35:20  2      0     4 (0/4)       0 test_server@9a392ddf40bf CT Log none
                  tests.kernel_test.inet_res_SUITE.cases                            -   Thu Dec 10 2015 18:35:20  0      0     5 (1/4)       0 test_server@9a392ddf40bf CT Log none
                  tests.os_mon_test.cpu_sup_SUITE.cases                             -   Thu Dec 10 2015 18:35:54  2      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.os_mon_test.disksup_SUITE.api                               -   Thu Dec 10 2015 18:35:54  1      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.os_mon_test.memsup_SUITE.api                                -   Thu Dec 10 2015 18:35:54  1      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.system_test.ethread_SUITE                                   -   Thu Dec 10 2015 18:35:59 13      0     1 (1/0)       0 test_server@9a392ddf40bf CT Log none
                  tests.system_test.otp_SUITE.undefined_functions                   -   Thu Dec 10 2015 18:35:59  1      0     0 (0/0)       0 test_server@9a392ddf40bf CT Log none
                  Total                                                                                          78      3    11 (3/8)       0

   Copyright © 2015 Open Telecom Platform
   Updated: Thu Dec 10 2015 18:36:47

More details of the failing test cases:

    Test started at 2015-12-10 18:34:15

   Host info:
   Run on 9a392ddf40bf
   Used Erlang v7.1 in "/usr/src/otp_src_18.1"

     * Full textual log
     * Coverage log
     * Unexpected I/O log

   Executing 17 test cases...

   Num   Module   Group                 Case                 Log  Time   Result               Comment
       time_SUITE       init_per_suite                       < > 0.000s  Ok
   1   time_SUITE       univ_to_local                        < > 0.048s  FAILED {time_SUITE,test_univ_to_local,145}
                                                                                {badmatch,{{1996,1,30},{12,45,7}}}
   2   time_SUITE       local_to_univ                        < > 0.050s  FAILED {time_SUITE,test_local_to_univ,157}
                                                                                {badmatch,{{1996,1,30},{13,45,7}}}
   3   time_SUITE       local_to_univ_utc                    < > 0.250s  Ok
   4   time_SUITE       bad_univ_to_local                    < > 0.001s  Ok
   5   time_SUITE       bad_local_to_univ                    < > 0.001s  Ok
   6   time_SUITE       univ_to_seconds                      < > 0.001s  Ok
   7   time_SUITE       seconds_to_univ                      < > 0.001s  Ok
   8   time_SUITE       consistency                          < > 0.706s  FAILED {time_SUITE,consistency,231}
                                                                                {badmatch,error}
       time_SUITE now   init_per_group                       < > 0.000s  Ok
   9   time_SUITE now   now_unique                           < > 0.007s  Ok
   10  time_SUITE now   now_update                           < > 2.018s  Ok
       time_SUITE now   end_per_group                        < > 0.000s  Ok
   11  time_SUITE       timestamp                            < > 0.026s  Ok
   12  time_SUITE       time_warp_modes                      < > 13.403s Ok
   13  time_SUITE       monotonic_time_monotonicity          < > 10.001s Ok
   14  time_SUITE       monotonic_time_monotonicity_parallel < > 0.036s  Ok
   15  time_SUITE       time_unit_conversion                 < > 15.640s Ok
   16  time_SUITE       signed_time_unit_conversion          < > 7.384s  Ok
   17  time_SUITE       erlang_timestamp                     < > 10.001s Ok
       time_SUITE       end_per_suite                        < > 0.000s  Ok
       TOTAL                                                     60.297s FAILED 14 Ok, 3 Failed of 17
     _____________________________________________________________________________________________________________________________________________________________________________________________

                                                                                Test run history | Top level test index

   Copyright © 2015 Open Telecom Platform
   Updated: Thu Dec 10 2015 18:34:15

[drobakowski]

WIP:

export ERL_TOP=`pwd`
make release_tests
cd release/tests/test_server
$ERL_TOP/bin/erl -s ts install -s ts smoke_test batch -s init stop
lynx index.html

[jakirkham]

Interesting. I have seen the exact same failures in my own builds.

With 1 and 2, they apparently make some assumption about what the local timezone is which is inaccurate. See the thread. However, I really don't have a good read on what 8 is testing or why it wouldn't work.