build: fix ninja builds on Windows #60

agrieve · 2018-04-17T13:52:44Z

Fixed several hardcoded paths to use <(STATIC_LIB_PREFIX) &
<(STATIC_LIB_SUFFIX)
Added GENERATOR=='ninja' conditionals to /WHOLEARCHIVE flags (msbuild
puts .lib files under separate directories).
Add missing link_settings for v8_monolith
Set use_sysroot=false only for non-windows builds.

To build using clang-cl:

set CC="%cd%\deps\v8\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe"
python configure --build-v8-with-gn --ninja
ninja -C out\Release

agrieve · 2018-04-17T13:53:06Z

Not sure how to make this not include the other commit...

mi-ac · 2018-04-18T12:00:08Z

deps/v8/gypfiles/v8-monolithic.gyp

+ 'conditions': [
+ ['OS=="win"', {
+ 'libraries': ['-ldbghelp.lib', '-lshlwapi.lib', '-lwinmm.lib', '-lws2_32.lib'],
+ }, 'OS=="linux"', {


Most gyp code I saw uses separate conditions for the separate cases, i.e.:
['OS=="win"', {
}],
['OS=="linux"', {
}],

Maybe do the same for consistency?

mi-ac · 2018-04-18T12:03:05Z

deps/v8/gypfiles/v8-monolithic.gyp

+ ],
+ # Specify a non-existent output to make the target permanently dirty.
+ # Alternatively, a depfile could be used, but then two dirty checks
+ # would run: one by outer the build tool, and one by build_gn.py.


s/one by outer the build tool/one by the outer build tool/

mi-ac · 2018-04-18T12:10:35Z

deps/v8/tools/node/build_gn.py

+
+ with open(os.path.join(options.build_path, 'args.gn')) as f:
+ if 'use_goma = true' in f.read():
+ args += ["-j500"]


In order to use goma on chrome infrastructure, this should be configurable. Chrome infra sets 50 on standard bots currently.

Added more flags.

mi-ac · 2018-04-18T12:14:54Z

deps/v8/gypfiles/v8-monolithic.gyp

@@ -49,7 +61,31 @@
 '--flag', 'v8_optimized_debug=<(v8_optimized_debug)',
 '--flag', 'v8_enable_disassembler=<(v8_enable_disassembler)',
 '--flag', 'v8_postmortem_support=<(v8_postmortem_support)',
+ '--flag', 'use_goma=<(build_v8_with_gn_use_goma)',


In order to use goma on chrome infrastructure also the goma_dir must be configurable somehow, since it's not at the default location.

Added more flags

mi-ac · 2018-04-18T12:22:00Z

node.gyp

+ 'conditions': [
+ ['GENERATOR=="ninja"', {
+ 'AdditionalOptions': [
+ '/WHOLEARCHIVE:<(obj_dir)/<(STATIC_LIB_PREFIX)'


Just clarification: We don't need \ in the paths?

Windows accepts /s for path separators. I've changed them to \s though, since that's more correct.

mi-ac · 2018-04-18T12:44:28Z

configure

@@ -572,6 +583,9 @@ options.prefix = os.path.expanduser(options.prefix or '')
 # set up auto-download list
 auto_downloads = nodedownload.parse(options.download_list)

+if options.use_clang_cl:
+ CC = os.path.abspath(os.path.join(
+ 'deps', 'v8', 'third_party', 'llvm-build', 'Release+Asserts', 'bin', 'clang-cl.exe'))


How does this end up there? Is it copied from V8's deps?

Never mind. It seems to get there during fetch_deps. I checked on our bots and it's there.

This is too V8-specific, IMO. How about turning this into a flag that takes the path to clang-cl as an argument? It should be possible for people to use an external clang-cl.

I've backed out the flag entirely. All it does is set the CC environment variable, so making it any more generic makes it no different than CC.

bnoordhuis · 2018-04-18T18:52:16Z

configure

@@ -568,10 +593,14 @@ parser.add_option('-C',

 # Expand ~ in the install prefix now, it gets written to multiple files.
 options.prefix = os.path.expanduser(options.prefix or '')
+options.build_v8_with_gn = options.build_v8_with_gn or options.build_v8_with_gn_use_goma


Can you keep lines under 80 columns in Node.js files? I know not everything is 100% clean in that respect but we try to keep new code on the straight and narrow.

bnoordhuis · 2018-04-18T18:58:07Z

configure

@@ -572,6 +583,9 @@ options.prefix = os.path.expanduser(options.prefix or '')
 # set up auto-download list
 auto_downloads = nodedownload.parse(options.download_list)

+if options.use_clang_cl:
+ CC = os.path.abspath(os.path.join(
+ 'deps', 'v8', 'third_party', 'llvm-build', 'Release+Asserts', 'bin', 'clang-cl.exe'))


This is too V8-specific, IMO. How about turning this into a flag that takes the path to clang-cl as an argument? It should be possible for people to use an external clang-cl.

bnoordhuis · 2018-04-18T19:02:06Z

configure

+parser.add_option('--build-v8-with-gn-goma-dir',
+ dest='build_v8_with_gn_goma_dir',
+ default='',
+ help='For Googlers: Value for the goma_dir GN arg.')


I think I'd prefer to keep these undocumented (help=optparse.SUPPRESS_HELP) since they're only relevant to Googlers.

mi-ac

lgtm if Ben's comments addressed and if tested locally hat this still works with linux.

mi-ac · 2018-04-19T07:22:15Z

configure

@@ -1054,6 +1083,10 @@ def configure_v8(o):
 print('Fetching dependencies to build V8 with GN')
 options.build_v8_with_gn = FetchDeps(v8_path)
 o['variables']['build_v8_with_gn'] = b(options.build_v8_with_gn)
+ o['variables']['build_v8_with_gn_use_goma'] = b(options.build_v8_with_gn_use_goma)
+ o['variables']['build_v8_with_gn_goma_dir_'] = options.build_v8_with_gn_goma_dir


Is the trailing _ indicating this field as private? Is it different to the other three?

Added a comment.

mi-ac · 2018-04-19T07:29:53Z

deps/v8/tools/node/build_gn.py

+ args += ["-j" + options.max_jobs]
+ else:
+ with open(os.path.join(options.build_path, "args.gn")) as f:
+ if "use_goma = true" in f.read():


Maybe make this more robust to spaces? Above all gn args are passed now without spaces. Also goma might be passed as:
use_goma=true

The script now uses gn gen --args=, which means GN will format the file to always have one spaces on each side of the "="

seishun · 2018-04-21T14:02:16Z

To build using clang-cl:

set CC="%cd%\deps\v8\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe"
python configure --build-v8-with-gn

This command failed for me:

C:\Users\Nikolai\node> python configure --build-v8-with-gn
Fetching dependencies to build V8 with GN
Checking out depot_tools.
Cloning into 'C:\Users\Nikolai\node\deps\v8\_depot_tools'...
remote: Sending approximately 31.60 MiB ...
remote: Total 25994 (delta 16723), reused 25994 (delta 16723) eceiving objects: 100% (25994/25994), 30.56 MiB | 15.24 MiReceiving objects: 100% (25994/25994), 31.60 MiB | 14.77 MiB/s, done.

Resolving deltas: 100% (16723/16723), done.
Using depot tools in C:\Users\Nikolai\node\deps\v8\_depot_tools
Initializing temporary git repository in v8.
Initialized empty Git repository in C:/Users/Nikolai/node/deps/v8/.git/
[master (root-commit) 78f1b11] init
Fetching dependencies.
Syncing projects: 100% (22/22), done.

________ running 'download_from_google_storage --no_resume --platform=win32 --no_auth --bucket chromium-clang-format -s v8/buildtools/win/clang-format.exe.sha1' in '.'
0> Downloading v8/buildtools/win/clang-format.exe...
Downloading 1 files took 143.674000 second(s)
Hook 'download_from_google_storage --no_resume --platform=win32 --no_auth --bucket chromium-clang-format -s v8/buildtools/win/clang-format.exe.sha1' took 143.76 secs
Running hooks:  37% (10/27) gn_win
________ running 'download_from_google_storage --no_resume --platform=win32 --no_auth --bucket chromium-gn -s v8/buildtools/win/gn.exe.sha1' in '.'
0> Downloading v8/buildtools/win/gn.exe...
Downloading 1 files took 4.563000 second(s)
Running hooks:  48% (13/27) wasm_spec_tests
________ running 'download_from_google_storage --no_resume --no_auth -u --bucket v8-wasm-spec-tests -s v8/test/wasm-spec-tests/tests.tar.gz.sha1' in '.'
0> Downloading v8/test/wasm-spec-tests/tests.tar.gz...
0> Extracting 72 entries from v8/test/wasm-spec-tests/tests.tar.gz to v8/test/wasm-spec-tests/tests
Downloading 1 files took 7.341000 second(s)
Running hooks:  51% (14/27) closure_compiler
________ running 'download_from_google_storage --no_resume --no_auth -u --bucket chromium-v8-closure-compiler -s v8/src/inspector/build/closure-compiler.tar.gz.sha1' in '.'
0> Downloading v8/src/inspector/build/closure-compiler.tar.gz...
0> Extracting 4 entries from v8/src/inspector/build/closure-compiler.tar.gz to v8/src/inspector/build/closure-compiler
Downloading 1 files took 4.524000 second(s)
Username for 'https://chrome-internal.googlesource.com':

________ running 'C:\Python27\python.exe v8/build/vs_toolchain.py update' in '.'



Please follow the instructions at https://www.chromium.org/developers/how-tos/build-instructions-windows


Traceback (most recent call last):
  File "v8/build/vs_toolchain.py", line 477, in <module>
    sys.exit(main())
  File "v8/build/vs_toolchain.py", line 473, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "v8/build/vs_toolchain.py", line 419, in Update
    subprocess.check_call(get_toolchain_args)
  File "C:\Python27\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Python27\\python.exe', 'C:\\Users\\Nikolai\\node\\deps\\v8\\_depot_tools\\win_toolchain\\get_toolchain_if_necessary.py', '--output-json', 'C:\\Users\\Nikolai\\node\\deps\\v8\\build\\win_toolchain.json', '1180cb75833ea365097e279efb2d5d7a42dee4b0']' returned non-zero exit status 1
Error: Command 'C:\\Python27\\python.exe v8/build/vs_toolchain.py update' returned non-zero exit status 1 in .
Uninitializing temporary git repository
>> Cleaning up C:\Users\Nikolai\node\deps\v8\.git
Traceback (most recent call last):
  File "configure", line 1490, in <module>
    configure_v8(output)
  File "configure", line 1073, in configure_v8
    options.build_v8_with_gn = FetchDeps(v8_path)
  File "deps\v8\tools\node\fetch_deps.py", line 78, in FetchDeps
    env=env)
  File "C:\Python27\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Python27\\python.exe', 'C:\\Users\\Nikolai\\node\\deps\\v8\\_depot_tools\\gclient.py', 'sync', '--spec', "solutions = [{'url': 'https://chromium.googlesource.com/v8/v8.git', 'managed': False, 'name': 'v8', 'deps_file': 'DEPS', 'custom_deps': {'v8/third_party/catapult': None, 'v8/third_party/colorama/src': None, 'v8/testing/gmock': None, 'v8/third_party/markupsafe': None, 'v8/third_party/jinja2': None, 'v8/tools/swarming_client': None, 'v8/third_party/googletest/src': None, 'v8/base/trace_event/common': None, 'v8/third_party/instrumented_libraries': None, 'v8/third_party/android_tools': None, 'v8/test/benchmarks/data': None, 'v8/test/mozilla/data': None, 'v8/tools/luci-go': None, 'v8/test/test262/data': None, 'v8/test/test262/harness': None}}]"]' returned non-zero exit status 2

agrieve · 2018-04-23T11:05:43Z

I think you may have hit the same issue I did with git.exe not being in your path.

I've re-uploaded with the fix as another commit in this pull request.

seishun · 2018-04-23T11:34:55Z

git is definitely in my PATH (otherwise I wouldn't be able to check this out...). Anyway, now I get the following error:

C:\Users\nvavilovs\Downloads\node-v8>python configure --build-v8-with-gn
Traceback (most recent call last):
  File "configure", line 60, in <module>
    from fetch_deps import FetchDeps
  File "deps\v8\tools\node\fetch_deps.py", line 16, in <module>
    import node_common
  File "deps\v8\tools\node\node_common.py", line 29
    pipes.quote(depot_tools)), shell=True)
        ^
SyntaxError: invalid syntax

agrieve · 2018-04-23T11:40:54Z

Gah! Sorry, patch fail :(. Fixed now.

seishun · 2018-04-23T11:49:59Z

Now it's back to the old error:

C:\Users\nvavilovs\Downloads\node-v8>python configure --build-v8-with-gn
Fetching dependencies to build V8 with GN
Using depot tools in C:\Users\nvavilovs\Downloads\node-v8\deps\v8\_depot_tools
Initializing temporary git repository in v8.
Fetching dependencies.
Syncing projects: 100% (22/22), done.
Running hooks:  81% (22/27) win_toolchain
________ running 'C:\Python27\python.exe v8/build/vs_toolchain.py update' in '.'



Please follow the instructions at https://www.chromium.org/developers/how-tos/build-instructions-windows


Traceback (most recent call last):
  File "v8/build/vs_toolchain.py", line 477, in <module>
    sys.exit(main())
  File "v8/build/vs_toolchain.py", line 473, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "v8/build/vs_toolchain.py", line 419, in Update
    subprocess.check_call(get_toolchain_args)
  File "C:\Python27\lib\subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Python27\\python.exe', 'C:\\Users\\nvavilovs\\Downloads\\node-v8\\deps\\v8\\_depot_tools\\win_toolchain\\get_toolchain_if_necessary.py', '--output-json', 'C:\\Users\\nvavilovs\\Downloads\\node-v8\\deps\\v8\\build\\win_toolchain.json', '1180cb75833ea365097e279efb2d5d7a42dee4b0']' returned non-zero exit status 1
Error: Command 'C:\\Python27\\python.exe v8/build/vs_toolchain.py update' returned non-zero exit status 1 in .
Uninitializing temporary git repository
>> Cleaning up C:\Users\nvavilovs\Downloads\node-v8\deps\v8\.git
Traceback (most recent call last):
  File "configure", line 1490, in <module>
    configure_v8(output)
  File "configure", line 1073, in configure_v8
    options.build_v8_with_gn = FetchDeps(v8_path)
  File "deps\v8\tools\node\fetch_deps.py", line 81, in FetchDeps
    env=env)
  File "C:\Python27\lib\subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Python27\\python.exe', 'C:\\Users\\nvavilovs\\Downloads\\node-v8\\deps\\v8\\_depot_tools\\gclient.py', 'sync', '--spec', "solutions = [{'url': 'https://chromium.googlesource.com/v8/v8.git', 'managed': False, 'name': 'v8', 'deps_file': 'DEPS', 'custom_deps': {'v8/third_party/catapult': None, 'v8/third_party/colorama/src': None, 'v8/testing/gmock': None, 'v8/third_party/markupsafe': None, 'v8/third_party/jinja2': None, 'v8/tools/swarming_client': None, 'v8/third_party/googletest/src': None, 'v8/base/trace_event/common': None, 'v8/third_party/instrumented_libraries': None, 'v8/third_party/android_tools': None, 'v8/test/benchmarks/data': None, 'v8/test/mozilla/data': None, 'v8/tools/luci-go': None, 'v8/test/test262/data': None, 'v8/test/test262/harness': None}}]"]' returned non-zero exit status 2

agrieve · 2018-04-23T13:31:04Z

I think this may be an issue that's tricky for me to debug on my machines...
Does running the hook directly result in a more informative error message?:

python deps\v8\build\vs_toolchain.py update

agrieve · 2018-04-23T13:32:39Z

Also - do you get the same error when trying --build-v8-with-gn without this patch?

seishun · 2018-04-23T13:32:51Z

I get a different error:

C:\Users\nvavilovs\Downloads\node-v8>python deps\v8\build\vs_toolchain.py update
Failed to find depot_tools
Traceback (most recent call last):
  File "deps\v8\build\vs_toolchain.py", line 477, in <module>
    sys.exit(main())
  File "deps\v8\build\vs_toolchain.py", line 473, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "deps\v8\build\vs_toolchain.py", line 380, in Update
    import find_depot_tools
  File "C:\Users\nvavilovs\Downloads\node-v8\deps\v8\build\find_depot_tools.py", line 62, in <module>
    import breakpad
ImportError: No module named breakpad

seishun · 2018-04-23T13:36:41Z

Also - do you get the same error when trying --build-v8-with-gn without this patch?

You mean on vee-eight-lkgr? Yes, the same error as in #60 (comment).

agrieve · 2018-04-23T13:42:22Z

Aha - the breakpad error was meaningful.

Try adding node\deps\v8_depot_tools to your PATH as a work-around for now, and I'll see about fixing the check to look for depot_tools there rather than elsewhere.

seishun · 2018-04-23T13:51:54Z

Adding it to PATH fixed the breakpad error, but now it results in the same error as configure:

C:\Users\nvavilovs\Downloads\node-v8>python deps\v8\build\vs_toolchain.py update



Please follow the instructions at https://www.chromium.org/developers/how-tos/build-instructions-windows


Traceback (most recent call last):
  File "deps\v8\build\vs_toolchain.py", line 477, in <module>
    sys.exit(main())
  File "deps\v8\build\vs_toolchain.py", line 473, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "deps\v8\build\vs_toolchain.py", line 419, in Update
    subprocess.check_call(get_toolchain_args)
  File "C:\Python27\lib\subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Python27\\python.exe', 'C:\\Users\\nvavilovs\\Downloads\\node-v8\\deps\\v8\\_depot_tools\\win_toolchain\\get_toolchain_if_necessary.py', '--output-json', 'C:\\Users\\nvavilovs\\Downloads\\node-v8\\deps\\v8\\build\\win_toolchain.json', '1180cb75833ea365097e279efb2d5d7a42dee4b0']' returned non-zero exit status 1

Same with and without this patch.

mi-ac · 2018-04-23T13:58:51Z

The vs_toolchain update in depot tools has a default of 1 for DEPOT_TOOLS_WIN_TOOLCHAIN:
https://cs.chromium.org/chromium/src/build/vs_toolchain.py?q=vs_toolch&sq=package:chromium&l=39

I think in the node use case we should have a 0 default.

seishun · 2018-04-23T14:05:06Z

If I set DEPOT_TOOLS_WIN_TOOLCHAIN to 1, then python configure --build-v8-with-gn succeeds. But how do I actually build it? And how do I run tests?

mi-ac · 2018-04-23T14:18:07Z

You've set it to 1? Not 0 by any chance? I thought 1 is the default, that makes the toolchain download start. And 0 omits it.

seishun · 2018-04-23T14:19:23Z

Sorry, I meant I set it to 0.

agrieve · 2018-04-23T14:21:57Z

Added a commit to set DEPOT_TOOLS_WIN_TOOLCAHIN=0 by default.
If I understand things correctly, this just means that you need to install visual studio /w correct SDKs yourself rather than having depot_tools download a googler-only copy of the sdks.

agrieve · 2018-04-23T14:23:26Z

Once you've run configure, you build in the usual way (with make).

This patch is mainly about building with clang-cl though, which works only with ninja (I think). So to test, do configure --build-v8-with-gn --ninja && ninja -C out\Release

And also:
set CC="%cd%\deps\v8\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe"

seishun · 2018-04-23T17:36:16Z

C:\Users\Nikolai\node>ninja -C out\Release
'ninja' is not recognized as an internal or external command,
operable program or batch file.

After I added deps\v8\_depot_tools to PATH and tried the command again, the build failed:

C:\Users\Nikolai\node> ninja -C out\Release
ninja: Entering directory `out\Release'
[1/94] ACTION v8_monolith: build_with_gn_generate_build_files_f3270641800ff32515e443515bf752cc
FAILED: obj/deps/v8/gypfiles/v8_monolith.gen/gn/build.ninja
C:\Python27\python.exe gyp-win-tool action-wrapper environment.x64 v8_monolith_target_build_with_gn_generate_build_files_f3270641800ff32515e443515bf752cc..rsp ..\..\deps\v8\gypfiles
Failed to find depot_tools
Traceback (most recent call last):
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 477, in <module>
    sys.exit(main())
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 473, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 448, in GetToolchainDir
    runtime_dll_dirs = SetEnvironmentAndGetRuntimeDllDirs()
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 44, in SetEnvironmentAndGetRuntimeDllDirs
    update_result = Update()
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 380, in Update
    import find_depot_tools
  File "C:\Users\Nikolai\node\deps\v8\build\find_depot_tools.py", line 62, in <module>
    import breakpad
ImportError: No module named breakpad
ERROR at //build/config/win/visual_studio_version.gni:27:7: Script returned non-zero exit code.
      exec_script("../../vs_toolchain.py", [ "get_toolchain_dir" ], "scope")
      ^----------
Current dir: C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn/
Command: C:/Python27/python.exe -- C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py get_toolchain_dir
Returned 1.
See //build/config/win/BUILD.gn:10:1: whence it was imported.
import("//build/config/win/visual_studio_version.gni")
^----------------------------------------------------
See //build/config/BUILDCONFIG.gn:567:9: which caused the file to be included.
      [ "//build/config/win:default_cygprofile_instrumentation" ]
        ^------------------------------------------------------
Traceback (most recent call last):
  File "..\tools\node\build_gn.py", line 122, in <module>
    GenerateBuildFiles(options)
  File "..\tools\node\build_gn.py", line 72, in GenerateBuildFiles
    subprocess.check_call(args)
  File "C:\Python27\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\Nikolai\\node\\deps\\v8\\buildtools\\win\\gn', 'gen', 'C:\\Users\\Nikolai\\node\\out\\Release\\obj\\deps\\v8\\gypfiles\\v8_monolith.gen\\gn', '-q', '--args=v8_monolithic=true is_component_build=false v8_use_external_startup_data=false use_custom_libcxx=false v8_promise_internal_field_count=true target_cpu="x64" target_os="win" v8_target_cpu="x64" v8_embedder_string="-node.4" v8_use_snapshot=true v8_optimized_debug=false v8_enable_disassembler=true v8_postmortem_support=false use_goma=false is_debug=false']' returned non-zero exit status 1
[2/94] ACTION node_js2c: node_js2c_bce0facc3e66b523ccf35ab2d859c2d9
ninja: build stopped: subcommand failed.

Then I ran ninja -C out\Release again and got this:

C:\Users\Nikolai\node> ninja -C out\Release
ninja: Entering directory `out\Release'
Username for 'https://chrome-internal.googlesource.com': iles_f3270641800ff32515e443515bf752cc

agrieve · 2018-04-23T19:37:02Z

Did you add _depot_tools to your PATH as an absolute path?

For the username one, I wonder if globally set:

set  DEPOT_TOOLS_WIN_TOOLCHAIN=0

then run ninja if it will still prompt?

If so, it would be helpful to run ninja -v -C out\Release to have it print out all of the commands that it's running to see which command is asking for credentials.

seishun · 2018-04-23T19:43:36Z

Did you add _depot_tools to your PATH as an absolute path?

Yes

For the username one, I wonder if globally set:

set DEPOT_TOOLS_WIN_TOOLCHAIN=0

then run ninja if it will still prompt?

No, but:

C:\Users\Nikolai\node>ninja -C out\Release
ninja: Entering directory `out\Release'
[1/93] ACTION v8_monolith: build_with_gn_generate_build_files_f3270641800ff32515e443515bf752cc
FAILED: obj/deps/v8/gypfiles/v8_monolith.gen/gn/build.ninja
C:\Python27\python.exe gyp-win-tool action-wrapper environment.x64 v8_monolith_target_build_with_gn_generate_build_files_f3270641800ff32515e443515bf752cc..rsp ..\..\deps\v8\gypfiles
Traceback (most recent call last):
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 477, in <module>
    sys.exit(main())
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 473, in main
    return commands[sys.argv[1]](*sys.argv[2:])
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 305, in CopyDlls
    _CopyDebugger(target_dir, target_cpu)
  File "C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py", line 336, in _CopyDebugger
    ' 10 SDK.' % (debug_file, full_path))
Exception: dbghelp.dll not found in "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\dbghelp.dll"
You must install the "Debugging Tools for Windows" feature from the Windows 10 SDK.
ERROR at //build/toolchain/win/BUILD.gn:49:3: Script returned non-zero exit code.
  exec_script("../../vs_toolchain.py",
  ^----------
Current dir: C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn/
Command: C:/Python27/python.exe -- C:/Users/Nikolai/node/deps/v8/build/vs_toolchain.py copy_dlls C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn Release x64
Returned 1 and printed out:

Copying C:\WINDOWS\Sysnative\msvcp140.dll to C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn\msvcp140.dll...
Copying C:\WINDOWS\Sysnative\vccorlib140.dll to C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn\vccorlib140.dll...
Copying C:\WINDOWS\Sysnative\vcruntime140.dll to C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn\vcruntime140.dll...
Copying C:\WINDOWS\Sysnative\ucrtbase.dll to C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn\ucrtbase.dll...
Copying C:\Program Files (x86)/Microsoft Visual Studio/2017/Community\VC\Tools\MSVC\14.13.26128\bin\HostX64\x64\pgort140.dll to C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn\pgort140.dll...
Copying C:\Program Files (x86)/Microsoft Visual Studio/2017/Community\VC\Tools\MSVC\14.13.26128\bin\HostX64\x64\pgosweep.exe to C:/Users/Nikolai/node/out/Release/obj/deps/v8/gypfiles/v8_monolith.gen/gn\pgosweep.exe...

See //BUILD.gn:598:1: which caused the file to be included.
action("js2c") {
^---------------
Traceback (most recent call last):
  File "..\tools\node\build_gn.py", line 122, in <module>
    GenerateBuildFiles(options)
  File "..\tools\node\build_gn.py", line 72, in GenerateBuildFiles
    subprocess.check_call(args)
  File "C:\Python27\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Users\\Nikolai\\node\\deps\\v8\\buildtools\\win\\gn', 'gen', 'C:\\Users\\Nikolai\\node\\out\\Release\\obj\\deps\\v8\\gypfiles\\v8_monolith.gen\\gn', '-q', '--args=v8_monolithic=true is_component_build=false v8_use_external_startup_data=false use_custom_libcxx=false v8_promise_internal_field_count=true target_cpu="x64" target_os="win" v8_target_cpu="x64" v8_embedder_string="-node.4" v8_use_snapshot=true v8_optimized_debug=false v8_enable_disassembler=true v8_postmortem_support=false use_goma=false is_debug=false']' returned non-zero exit status 1
ninja: build stopped: subcommand failed.

* Split "gn gen" and "ninja" invocations into two separate actions * Make the ninja action use console pool so that progress is shown * Set DEPOT_TOOLS_WIN_TOOLCHAIN=0 by default (to avoid non-googlers being asked for credentials) * Propagate depot_tools path to "gn gen" step (it runs vs_toolchain.py) * Add flags for clang-cl, setting gn args, sub-ninja -j and -l parameters

* Fixed several hardcoded paths to use <(STATIC_LIB_PREFIX) & <(STATIC_LIB_SUFFIX) * Added GENERATOR=='ninja' conditionals to /WHOLEARCHIVE flags (msbuild puts .lib files under separate directories). * Add missing link_settings for v8_monolith * Set use_sysroot=false only for non-windows builds. To build using clang-cl: python configure --build-v8-with-gn --use-clang-cl --ninja

agrieve · 2018-04-24T11:08:56Z

Made a few changes:

set DEPOT_TOOLS_WIN_TOOLCHAIN=0 by default, and propagate it to the ninja step
propagate depot_tools path from configure to gn gen
Added back flag --use-clang-cl and also --clang-cl-base-path in case someone wants to use the non-v8-bundled clang-cl.exe
- Propagate --clang-cl-base-path to gn gen

With these changes, I'm now seeing the same error about dbghelp.dll not existing. I tried installing various other options for Visual Studio Community 2017, but nothing I tried added the file in the spot GN is expecting it to be.

agrieve · 2018-04-24T11:38:22Z

Update: Managed to appease the check by installing the windows 10 sdk from:
https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk

agrieve · 2018-04-24T11:42:05Z

Note: There are instructions for installing the correct MS deps here:
https://chromium.googlesource.com/chromium/src/+/master/docs/windows_build_instructions.md#visual-studio

agrieve · 2018-04-24T12:50:53Z

And, can now confirm that I was able to build with DEPOT_TOOLS_WIN_TOOLCHAIN=0 (now the default). I was missing:
“Debugging Tools For Windows” <-- This I installed via windows sdk installer
10.0.15063 Windows 10 SDK <-- This I installed via visual studio

hashseed · 2018-04-24T13:25:35Z

I'm landing this now. Any further issues can be solved in further PRs.

seishun · 2018-04-24T13:54:46Z

I tried on a different computer and got this:

C:\Users\nvavilovs\Downloads\node-v8> ninja -C out\Release
ninja: Entering directory `out\Release'
[0/704] ACTION v8_monolith: build_with_gn_f3270641800ff32515e443515bf752cc
Using depot tools in ..\_depot_tools
ninja: Entering directory `C:\Users\nvavilovs\Downloads\node-v8\out\Release\obj\deps\v8\gypfiles\v8_monolith.gen\gn'
[1/983] STAMP obj/src/inspector/protocol_compatibility.stamp
FAILED: obj/src/inspector/protocol_compatibility.stamp
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py stamp obj/src/inspector/protocol_compatibility.stamp
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
[2/983] STAMP obj/build/config/exe_and_shlib_deps.stamp
FAILED: obj/build/config/exe_and_shlib_deps.stamp
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py stamp obj/build/config/exe_and_shlib_deps.stamp
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
[3/983] COPY ../../../../../../../../deps/v8/third_party/icu/common/icudtl.dat icudtl.dat
FAILED: icudtl.dat
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py recursive-mirror ../../../../../../../../deps/v8/third_party/icu/common/icudtl.dat icudtl.dat
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
[4/983] STAMP obj/js2c_experimental_extras.stamp
FAILED: obj/js2c_experimental_extras.stamp
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py stamp obj/js2c_experimental_extras.stamp
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
[5/983] STAMP obj/v8_dump_build_config.stamp
FAILED: obj/v8_dump_build_config.stamp
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py stamp obj/v8_dump_build_config.stamp
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
[6/983] STAMP obj/js2c_extras.stamp
FAILED: obj/js2c_extras.stamp
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py stamp obj/js2c_extras.stamp
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
[7/983] STAMP obj/src/inspector/inspector_injected_script.stamp
FAILED: obj/src/inspector/inspector_injected_script.stamp
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py stamp obj/src/inspector/inspector_injected_script.stamp
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
[8/983] STAMP obj/js2c.stamp
FAILED: obj/js2c.stamp
C:/Python27/python.exe ../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py stamp obj/js2c.stamp
Traceback (most recent call last):
  File "../../../../../../../../deps/v8/build/toolchain/win/tool_wrapper.py", line 29, in <module>
    import win32file    # pylint: disable=import-error
ImportError: No module named win32file
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "..\tools\node\build_gn.py", line 135, in <module>
    Build(options)
  File "..\tools\node\build_gn.py", line 92, in Build
    subprocess.check_call(args)
  File "C:\Python27\lib\subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['..\\_depot_tools\\ninja.exe', '-C', 'C:\\Users\\nvavilovs\\Downloads\\node-v8\\out\\Release\\obj\\deps\\v8\\gypfiles\\v8_monolith.gen\\gn', 'v8_monolith']' returned non-zero exit status 1
[5/704] CC obj\deps\openssl\openssl\crypto\cms\openssl.cms_kari.obj
FAILED: obj/deps/v8/gypfiles/v8_monolith.gen/gn/obj/v8_monolith.lib ../../deps/v8/gypfiles/does-not-exist
C:\Python27\python.exe gyp-win-tool action-wrapper environment.x64 v8_monolith_target_build_with_gn_f3270641800ff32515e443515bf752cc..rsp ..\..\deps\v8\gypfiles
[15/704] LINK_EMBED icupkg.exe
All 6520 functions were compiled because no usable IPDB/IOBJ from previous compilation was found.

agrieve · 2018-04-24T14:02:11Z

Sounds like maybe the install of python in C:\Python27 is missing this file? It's supposed to live under site-packages/win32/win32file.pyd

seishun · 2018-04-24T14:28:57Z

Python 2.7 doesn't come with that file out of box. It's part of pywin32. Is that another requirement?

hashseed · 2018-04-30T14:33:06Z

For posterity. For this to work, you have to install

Git
Python 2.7
Pywin32
Visual Studio 2017, "Desktop development with C++". Make sure to include:
- Windows 10 SDK (10.0.15063) for Desktop C++. This version is hardcoded in to deps/v8/build/toolchain/win/setup_toolchain.py.
- Visual C++ ATL support

agrieve mentioned this pull request Apr 17, 2018

Fix ninja build on Windows & --build-v8-with-gn improvements #56

Closed

mi-ac reviewed Apr 18, 2018

View reviewed changes

agrieve force-pushed the win-ninja branch 2 times, most recently from d018663 to 534f06d Compare April 18, 2018 14:08

bnoordhuis reviewed Apr 18, 2018

View reviewed changes

mi-ac approved these changes Apr 19, 2018

View reviewed changes

agrieve force-pushed the win-ninja branch 2 times, most recently from 44dee42 to 5962cc2 Compare April 19, 2018 10:49

agrieve force-pushed the win-ninja branch from a5a95c8 to 620020c Compare April 23, 2018 11:40

agrieve force-pushed the win-ninja branch from 673b950 to 620020c Compare April 23, 2018 14:09

agrieve added 3 commits April 24, 2018 12:50

Fix FetchDeps on windows not finding git.bat

25084b1

agrieve force-pushed the win-ninja branch from f1cf15a to 25084b1 Compare April 24, 2018 10:55

hashseed changed the title ~~Fix ninja builds on Windows~~ build: fix ninja builds on Windows Apr 24, 2018

hashseed merged commit 09d52e9 into v8:vee-eight-lkgr Apr 24, 2018

agrieve deleted the win-ninja branch May 8, 2018 00:07

seishun mentioned this pull request Jun 19, 2018

Fix building on Windows denoland/deno#275

Closed

saurabharch mentioned this pull request May 14, 2022

[Snyk] Fix for 1 vulnerabilities saurabharch/node#26

Open

build: fix ninja builds on Windows #60

build: fix ninja builds on Windows #60

Conversation

agrieve commented Apr 17, 2018 • edited Loading

agrieve commented Apr 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mi-ac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seishun commented Apr 21, 2018

agrieve commented Apr 23, 2018

seishun commented Apr 23, 2018

agrieve commented Apr 23, 2018

seishun commented Apr 23, 2018

agrieve commented Apr 23, 2018

agrieve commented Apr 23, 2018

seishun commented Apr 23, 2018

seishun commented Apr 23, 2018

agrieve commented Apr 23, 2018

seishun commented Apr 23, 2018

mi-ac commented Apr 23, 2018

seishun commented Apr 23, 2018

mi-ac commented Apr 23, 2018

seishun commented Apr 23, 2018

agrieve commented Apr 23, 2018

agrieve commented Apr 23, 2018 • edited Loading

seishun commented Apr 23, 2018

agrieve commented Apr 23, 2018

seishun commented Apr 23, 2018

agrieve commented Apr 24, 2018

agrieve commented Apr 24, 2018

agrieve commented Apr 24, 2018

agrieve commented Apr 24, 2018

hashseed commented Apr 24, 2018

seishun commented Apr 24, 2018

agrieve commented Apr 24, 2018

seishun commented Apr 24, 2018

hashseed commented Apr 30, 2018

agrieve commented Apr 17, 2018 •

edited

Loading

agrieve commented Apr 23, 2018 •

edited

Loading