Source code not found warning after upgrading to rebar 3.14.3

[oleglite]

I am trying to upgrade to the latest version of rebar3, but I am getting following messages during release:

===> Warnings generating release:
*WARNING* prometheus: Source code not found: prometheus_boolean.erl
*WARNING* prometheus: Source code not found: prometheus_counter.erl
*WARNING* prometheus: Source code not found: prometheus_gauge.erl

I found out that this warning comes from https://github.com/erlang/otp/blob/OTP-21.3/lib/sasl/src/systools_make.erl#L1151
And may be caused be prometheus lib structure:
https://github.com/deadtrickster/prometheus.erl/tree/v4.6.0/src

$ tree lib/prometheus/src/
lib/prometheus/src/
|-- collectors
|   |-- mnesia
|   |   `-- prometheus_mnesia_collector.erl
|   `-- vm
|       |-- prometheus_vm_dist_collector.erl
|       |-- prometheus_vm_memory_collector.erl
|       |-- prometheus_vm_statistics_collector.erl
|       `-- prometheus_vm_system_info_collector.erl
|-- contrib
|   |-- prometheus_http.erl
|   |-- prometheus_mnesia.erl
|   `-- prometheus_test_instrumenter.erl
|-- formats
|   |-- prometheus_protobuf_format.erl
|   `-- prometheus_text_format.erl
|-- metrics
|   |-- prometheus_boolean.erl
|   |-- prometheus_counter.erl
|   |-- prometheus_gauge.erl
|   |-- prometheus_histogram.erl
|   `-- prometheus_summary.erl
|-- model
|   |-- prometheus_model.erl
|   |-- prometheus_model.proto
|   `-- prometheus_model_helpers.erl
|-- prometheus.app.src
|-- prometheus.erl
|-- prometheus_buckets.erl
|-- prometheus_collector.erl
|-- prometheus_format.erl
|-- prometheus_instrumenter.erl
|-- prometheus_metric.erl
|-- prometheus_metric_spec.erl
|-- prometheus_misc.erl
|-- prometheus_registry.erl
|-- prometheus_sup.erl
`-- prometheus_time.erl

My question is it a problem with prometheus lib, with my rebar config or something else? How can I fix this warning?

I reproduced it with bare umbrella project:

$ ./rebar3 --version
rebar 3.14.3 on Erlang/OTP 21 Erts 10.3
$ ./rebar3 new umbrella myproj
===> Writing myproj/apps/myproj/src/myproj_app.erl
===> Writing myproj/apps/myproj/src/myproj_sup.erl
===> Writing myproj/apps/myproj/src/myproj.app.src
===> Writing myproj/rebar.config
===> Writing myproj/config/sys.config
===> Writing myproj/config/vm.args
===> Writing myproj/.gitignore
===> Writing myproj/LICENSE
===> Writing myproj/README.md
$ cd myproj/
$ cat myproj/rebar.config
{erl_opts, [debug_info]}.
{deps, [
     {prometheus, ".*",
        {git, "https://github.com/deadtrickster/prometheus.erl.git", {tag, "v4.6.0"}}}
]}.

{relx, [{release, {myproj, "0.1.0"},
         [myproj,
          sasl,
          prometheus]},

        {mode, dev},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.

{profiles, [{prod, [{relx,
                     [%% prod is the default mode when prod
                      %% profile is used, so does not have
                      %% to be explicitly included like this
                      {mode, prod}

                      %% use minimal mode to exclude ERTS
                      %% {mode, minimal}
                     ]
            }]}]}.
$ ../rebar3 compile
===> Verifying dependencies...
===> Fetching prometheus (from {git,"https://github.com/deadtrickster/prometheus.erl.git",
                      {tag,"v4.4.0"}})
===> Fetching rebar3_archive_plugin v0.0.1
===> Fetching rebar3_elvis_plugin (from {git,"https://github.com/deadtrickster/rebar3_elvis_plugin.git",
                               "master"})
===> WARNING: It is recommended to use {branch, Name}, {tag, Tag} or {ref, Ref}, otherwise updating the dep may not work as expected.
===> Fetching katana_code v0.1.0
===> Fetching zipper v1.0.1
===> Fetching aleppo v1.0.0
===> Analyzing applications...
===> Compiling aleppo
_build/default/plugins/aleppo/src/aleppo.erl:6: Warning: record ale_context has field(s) without type information
_build/default/plugins/aleppo/src/aleppo.erl:12: Warning: missing specification for function process_file/1
_build/default/plugins/aleppo/src/aleppo.erl:15: Warning: missing specification for function process_file/2
_build/default/plugins/aleppo/src/aleppo.erl:25: Warning: missing specification for function process_tokens/1
_build/default/plugins/aleppo/src/aleppo.erl:32: Warning: missing specification for function process_tokens/2
_build/default/plugins/aleppo/src/aleppo.erl:307: Warning: missing specification for function scan_file/1

===> Compiling katana_code
===> Compiling rebar3_elvis_plugin
===> Compiling zipper
===> Analyzing applications...
===> Compiling rebar3_archive_plugin
===> Analyzing applications...
===> Compiling prometheus
===> Analyzing applications...
===> Compiling myproj
$ ../rebar3 release
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling myproj
===> Assembling release myproj-0.1.0...
===> Warnings generating release:
*WARNING* prometheus: Source code not found: prometheus_boolean.erl
*WARNING* prometheus: Source code not found: prometheus_counter.erl
*WARNING* prometheus: Source code not found: prometheus_gauge.erl
*WARNING* prometheus: Source code not found: prometheus_histogram.erl
*WARNING* prometheus: Source code not found: prometheus_http.erl
*WARNING* prometheus: Source code not found: prometheus_mnesia.erl
*WARNING* prometheus: Source code not found: prometheus_mnesia_collector.erl
*WARNING* prometheus: Source code not found: prometheus_model.erl
*WARNING* prometheus: Source code not found: prometheus_model_helpers.erl
*WARNING* prometheus: Source code not found: prometheus_protobuf_format.erl
*WARNING* prometheus: Source code not found: prometheus_summary.erl
*WARNING* prometheus: Source code not found: prometheus_test_instrumenter.erl
*WARNING* prometheus: Source code not found: prometheus_text_format.erl
*WARNING* prometheus: Source code not found: prometheus_vm_dist_collector.erl
*WARNING* prometheus: Source code not found: prometheus_vm_memory_collector.erl
*WARNING* prometheus: Source code not found: prometheus_vm_statistics_collector.erl
*WARNING* prometheus: Source code not found: prometheus_vm_system_info_collector.erl
===> Release successfully assembled: _build/default/rel/myproj

[ferd]

I guess part of the issue is that the beam files are flat in ebin/ but the source files are not searched recursively by systools whereas rebar3 will do so.

[oleglite]

Yes, beam files are flat in /ebin and source files are not. It looks like there are two solutions:

1. The structure of source files in libs like prometheus must be changed to flat
2. systools must support recursive search

Also I reproduced it with the latest erlang 23.2.

[ferd]

23.2 has SSL issues I believe, so you might want to update.

The systools recursive search punts this problem into OTP rather than rebar3, if it is the component that complains. the only thing rebar3 could do is flatten the source directory on behalf of users before compiling, but that would probably be annoyingly slow.

[bryanhughes]

I am having a similar issue and I am not sure if there is a solution being presented that I am being too dense and missing. I am working on an opensource tool that will allow starting from a database schema, and then generate protobuffers and all the CRUDL (Create, Read, Update, Delete, List/Lookup) code. The tool then ties in with gpb to compile the protos that are generated from the schema to Erlang source codec which in turn is used by the generated code. The end result is that you run the tool against your database and you then have all the code and protos generated to seamlessly round trip from protobuffers across the wire to and from your database just using the maps/protos and back, and across your stack.

In my case, the tool repo was generated with rebar3 and is an escript. It includes an example directory with an example application also generated by rebar3 to build a working HTTP example demonstrating round tripping a proto across the wire to the database and back.

Everything compiles but does not run. My issue is a little more extensive:

example git:(main) ✗ rebar3 run
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling example
===> Assembling release example-0.1.0...
===> Warnings generating release:
*WARNING* example: Source code not found: example_app.erl
*WARNING* example: Source code not found: example_http.erl
*WARNING* example: Source code not found: example_sup.erl
*WARNING* example: Source code not found: example_utils.erl
*WARNING* example: Source code not found: public_example_a_db.erl
*WARNING* example: Source code not found: public_example_b_db.erl
*WARNING* example: Source code not found: public_example_c_db.erl
*WARNING* example: Source code not found: public_part_db.erl
*WARNING* example: Source code not found: public_part_part_db.erl
*WARNING* example: Source code not found: public_product_db.erl
*WARNING* example: Source code not found: public_product_parts_db.erl
*WARNING* example: Source code not found: test_schema_address_db.erl
*WARNING* example: Source code not found: test_schema_test_table_no_pkey_db.erl
*WARNING* example: Source code not found: test_schema_test_table_pkey_db.erl
*WARNING* example: Source code not found: test_schema_user_db.erl
*WARNING* example: Source code not found: test_schema_user_product_part_db.erl
*WARNING* stdlib: Object code (filename) out of date
*WARNING* inets: Source code not found: inets.erl
*WARNING* inets: Source code not found: inets_sup.erl
*WARNING* inets: Source code not found: inets_app.erl
*WARNING* inets: Source code not found: inets_service.erl
*WARNING* inets: Source code not found: inets_trace.erl
*WARNING* inets: Source code not found: inets_lib.erl
*WARNING* inets: Source code not found: inets_ftp_wrapper.erl
*WARNING* inets: Source code not found: httpc.erl
*WARNING* inets: Source code not found: httpc_handler.erl
*WARNING* inets: Source code not found: httpc_handler_sup.erl
*WARNING* inets: Source code not found: httpc_manager.erl
*WARNING* inets: Source code not found: httpc_profile_sup.erl
*WARNING* inets: Source code not found: httpc_request.erl
*WARNING* inets: Source code not found: httpc_response.erl
*WARNING* inets: Source code not found: httpc_sup.erl
*WARNING* inets: Source code not found: httpc_cookie.erl
*WARNING* inets: Source code not found: http_chunk.erl
*WARNING* inets: Source code not found: http_request.erl
*WARNING* inets: Source code not found: http_response.erl
*WARNING* inets: Source code not found: http_transport.erl
*WARNING* inets: Source code not found: http_util.erl
*WARNING* inets: Source code not found: http_uri.erl
*WARNING* inets: Source code not found: httpd.erl
*WARNING* inets: Source code not found: httpd_acceptor.erl
*WARNING* inets: Source code not found: httpd_acceptor_sup.erl
*WARNING* inets: Source code not found: httpd_cgi.erl
*WARNING* inets: Source code not found: httpd_connection_sup.erl
*WARNING* inets: Source code not found: httpd_conf.erl
*WARNING* inets: Source code not found: httpd_custom.erl
*WARNING* inets: Source code not found: httpd_custom_api.erl
*WARNING* inets: Source code not found: httpd_esi.erl
*WARNING* inets: Source code not found: httpd_example.erl
*WARNING* inets: Source code not found: httpd_file.erl
*WARNING* inets: Source code not found: httpd_instance_sup.erl
*WARNING* inets: Source code not found: httpd_log.erl
*WARNING* inets: Source code not found: httpd_logger.erl
*WARNING* inets: Source code not found: httpd_manager.erl
*WARNING* inets: Source code not found: httpd_misc_sup.erl
*WARNING* inets: Source code not found: httpd_request.erl
*WARNING* inets: Source code not found: httpd_request_handler.erl
*WARNING* inets: Source code not found: httpd_response.erl
*WARNING* inets: Source code not found: httpd_script_env.erl
*WARNING* inets: Source code not found: httpd_socket.erl
*WARNING* inets: Source code not found: httpd_sup.erl
*WARNING* inets: Source code not found: httpd_util.erl
*WARNING* inets: Source code not found: mod_actions.erl
*WARNING* inets: Source code not found: mod_alias.erl
*WARNING* inets: Source code not found: mod_auth.erl
*WARNING* inets: Source code not found: mod_auth_dets.erl
*WARNING* inets: Source code not found: mod_auth_mnesia.erl
*WARNING* inets: Source code not found: mod_auth_plain.erl
*WARNING* inets: Source code not found: mod_auth_server.erl
*WARNING* inets: Source code not found: mod_cgi.erl
*WARNING* inets: Source code not found: mod_dir.erl
*WARNING* inets: Source code not found: mod_disk_log.erl
*WARNING* inets: Source code not found: mod_esi.erl
*WARNING* inets: Source code not found: mod_get.erl
*WARNING* inets: Source code not found: mod_head.erl
*WARNING* inets: Source code not found: mod_log.erl
*WARNING* inets: Source code not found: mod_range.erl
*WARNING* inets: Source code not found: mod_responsecontrol.erl
*WARNING* inets: Source code not found: mod_security.erl
*WARNING* inets: Source code not found: mod_security_server.erl
*WARNING* inets: Source code not found: mod_trace.erl
*WARNING* inets: Source code not found: inets_tftp_wrapper.erl
===> Release successfully assembled: _build/default/rel/example
Error spawning "/usr/local/Cellar/erlang/23.2.3/lib/erlang/erts-11.1.7/bin/epmd" -daemon (error 10)

Note, the same is true for rebar3 release, just doesnt try to spawn epmd.

Here is my rebar3.config

{erl_opts, [debug_info,
            {i, "./_build/default/plugins/gpb/include/"},
            {i, "apps/example/include"}]}.
{deps, [{elli, {git, "https://github.com/elli-lib/elli.git", {branch, "develop"}}},
        elli_cookie,
        elli_date,
        mimerl,
        {pgo, {git, "https://github.com/bryanhughes/pgo.git", {branch, "master"}}}]}.

{plugins, [{rebar3_gpb_plugin, "2.19.0"}, {rebar3_autotest, "0.1.1"}, rebar3_run, rebar_alias, rebar3_hex]}.

{gpb_opts, [{i, "proto"},
            {o_erl, "src"},
            {o_hrl, "include"},
            use_packages,
            {module_name_suffix, "_pb"},
            {strings_as_binaries, true},
            {recursive, true},
            maps,
            type_specs]}.

{provider_hooks, [
    {pre, [
        {compile, {protobuf, compile}},
        {clean, {protobuf, clean}}
    ]}
]}.

{relx, [{release, {example, "0.1.0"}, [example, sasl]},
        {mode, dev},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.

{shell, []}.
{dist_node, [{name, '[email protected]'}]}.

{profiles, [
    {prod, [
        {include_erts, true},
        {erl_opts, [no_debug_info, warnings_as_errors]},
        {relx, [{mode, prod}]}
    ]},
    {test, [
        {deps, [meck]},
        {erl_opts, [debug_info]}
    ]}
]}.

When I investigate the _build/default/lib/example/ebin all the beams are there, but not all the sources are in _build/default/lib/example/src. In any case, the WARNINGS list pretty much all the sources in the project, including those from one of the dependencies.

I am using

Erlang/OTP 23 [erts-11.1.7] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe] [dtrace]

and

rebar 3.14.3 on Erlang/OTP 23 Erts 11.1.7

The other thing that stands out is *WARNING* stdlib: Object code (filename) out of date

Here is the debug output...
example-debug.txt

Any help is much appreciate.

Thanks!
Bryan

[bryanhughes]

I have been futzing around with trying to see if I can make the release and found that if I change {mode, dev} to {mode, prod} then it gets rid of all the warnings.

        {mode, prod},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.```

Now I just have to figure out why I am getting...

➜ example git:(main) ✗ rebar3 run
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling example
===> Assembling release example-0.1.0...
===> Release successfully assembled: _build/default/rel/example
Error spawning "/usr/local/Cellar/erlang/23.2.3/lib/erlang/erts-11.1.7/bin/epmd" -daemon (error 10)

[ferd]

we don't have anything to offer at this point -- none of the maintainers recently had time to solve this one yet. Thanks for the additional information though. No idea yet when I'll have the time to take a look, unfortunately, but the extra data will likely be useful.

[bryanhughes]

Bummer. Just wanted to give some more information. I have created a second new umbrella project and ran into the very same situation, but I figured out how to run rebar3 release without the warnings, and thus broken application because it fails on stdlib.

To work around, add {dev_mode, false}, to your relx config. if it is set to true, then switch it to false:

{relx, [{release, {traphic, "0.1.0"}, [traphic, sasl]},
        {mode, prod},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "config/sys.config"},
        {vm_args, "config/vm.args"},

        {dev_mode, false},   <--------
        {include_erts, false},

        {extended_start_script, true}
]}.

Hope this helps...

[ferd]

Might have something to do with symlinks then, but that's a good pointer to solve it, thanks for the info

[reganheath]

I have a similar (the same?) issue.

I have just upgraded from rebar3 v3.9.1 to rebar3 v3.15.2 (for the hex.pm TLS fix). With no other changes to OTP version (21). rebar.config or the relx.config (we use a separate file and run rebar3 release -c relx.conf) we now get these same "Source code not found" warnings.

The files it is complaining about all all external deps and the src tree/folder structure for these has not changed (they're pinned).

So.. it feels like something has changed in the defaults or behaviour of rebar3/relx to trigger these warnings?

The relx.config is minimal (replaced app names with X, Y, and Z), e.g.

{default_release, {X, "0.3.4"}}.

{release, {X, "0.3.4"}, ["X>=0.3.4",
                            "Y>=0.3.4",
                            "Z>=0.1.0",
                            sasl,
                            redbug,
                            recon,
                            tools,
                            wx,
                            observer,
                            debugger,
                            runtime_tools,
                            debugger]}.

{sys_config, "conf/sys.config"}.
{vm_args, "conf/vm.args"}.
{extended_start_script, true}.

As you can see, it has no explicit "mode" specified, nor any of the things that mode is expanded to by relx (include_src, debug_info, include_erts, dev_mode).

Inspecting our previous releases, they appears to be the result of "{mode, dev}" or equivalent.

In addition to these new warnings, the relx.config above is no longer valid.

default_release - no longer supported? (but still documented here https://github.com/erlware/relx/wiki/Configuration)

release tuple, applications list, format of applications WRT required version changed from "X>=0.3.4" to {X, "0.3.4"} (and I cannot see whether >= is still supported in some form?)

[tsloughter]

rebar3 upgraded to relx 4.x and this caused the changes you are experiencing.

Adding {src_tests, false} should get rid of the warnings about source code. It is also only done if include_src is true which happens with the new default of mode being dev.

The constraints like X>=0.3.4 were removed because they didn't do anything, rebar3 is in charge of what app versions are in the release. Were you seeing otherwise?

I don't recall why default_release was removed.. Thanks for calling out it is still in the wiki.

[DominicGame]

I have some test result about it
env:

win10
rebar 3.20.0 on Erlang/OTP 24 Erts 12.3.2

flow:

rebar3 new release test
cd test
rebar3 release as prod
rebar3 tar as prod
(move tar to some where and decompression)
./bin/test console

default config:

{undef,[{test_app,.....

when I remove otp(right?), it can work:

{relx, [{release, {test, "0.1.0"},
         [test,
          sasl]},

        {mode, dev},
        {system_libs, false},
        {include_erts, false},
        {include_src, true},

        %% automatically picked up if the files
        %% exist but can be set manually, which
        %% is required if the names aren't exactly
        %% sys.config and vm.args
        {sys_config, "./config/sys.config"},
        {vm_args, "./config/vm.args"}

        %% the .src form of the configuration files do
        %% not require setting RELX_REPLACE_OS_VARS
        %% {sys_config_src, "./config/sys.config.src"},
        %% {vm_args_src, "./config/vm.args.src"}
]}.

{profiles, [{prod, [{relx,
                     [%% prod is the default mode when prod
                      %% profile is used, so does not have
                      %% to be explicitly included like this
                      %% use minimal mode to exclude ERTS
                      {mode, minimal}
                     ]
            }]}]}.

release warning:

*WARNING* test: Source code not found: test_app.erlrelease start script created*WARNING* Not creating RELEASES file because system libs are not included in the release.

tar warning:

*WARNING* Not creating RELEASES file because system libs are not included in the release. The RELEASES file is required before release_handler can be used to install a release.

it can work with recursive dir in src