Introduce vcpkg for easier development on Windows

[Megaf]

Issue type

• Feature request

Minetest version

5+

OS / Hardware

Operating system: Windows

Summary

Hi folks, hows everyone been?
I just learned about Vcpkg. I think this could help us a lot in making the windows binaries.

You'll have to forgive the crudeness of this model, I didn't have time to paint it or build it to scale.

Quick Start
Vcpkg helps you get C and C++ libraries on Windows. This tool and ecosystem are currently in a preview state; your involvement is vital to its success.

The following was take from here.

Example: Using Sqlite

Step 1: Install

First, we need to know what name Sqlite goes by in the ports tree. To do that, we'll run the search command and inspect the output:

S D:\src\vcpkg> .\vcpkg search sqlite
libodb-sqlite        2.4.0            Sqlite support for the ODB ORM library
sqlite3              3.15.0           SQLite is a software library that implements a se...

If your library is not listed, please open an issue at:
    https://github.com/Microsoft/vcpkg/issues

Looking at the list, we can see that the port is named "sqlite3". You can also run the search command without arguments to see the full list of packages.

Installing is then as simple as using the install command.

PS D:\src\vcpkg> .\vcpkg install sqlite3
-- CURRENT_INSTALLED_DIR=D:/src/vcpkg/installed/x86-windows
-- DOWNLOADS=D:/src/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=D:/src/vcpkg/packages/sqlite3_x86-windows
-- CURRENT_BUILDTREES_DIR=D:/src/vcpkg/buildtrees/sqlite3
-- CURRENT_PORT_DIR=D:/src/vcpkg/ports/sqlite3/.
-- Downloading https://sqlite.org/2016/sqlite-amalgamation-3150000.zip...
-- Downloading https://sqlite.org/2016/sqlite-amalgamation-3150000.zip... OK
-- Testing integrity of downloaded file...
-- Testing integrity of downloaded file... OK
-- Extracting source D:/src/vcpkg/downloads/sqlite-amalgamation-3150000.zip
-- Extracting done
-- Configuring x86-windows-rel
-- Configuring x86-windows-rel done
-- Configuring x86-windows-dbg
-- Configuring x86-windows-dbg done
-- Build x86-windows-rel
-- Build x86-windows-rel done
-- Build x86-windows-dbg
-- Build x86-windows-dbg done
-- Package x86-windows-rel
-- Package x86-windows-rel done
-- Package x86-windows-dbg
-- Package x86-windows-dbg done
-- Performing post-build validation
-- Performing post-build validation done
Package sqlite3:x86-windows is installed

We can check that sqlite3 was successfully installed for x86 windows desktop by running the list command.

PS D:\src\vcpkg> .\vcpkg list
sqlite3:x86-windows         3.15.0           SQLite is a software library that implements a se...

To install for other architectures and platforms such as Universal Windows Platform or x64 Desktop, you can suffix the package name with :<target>.
PS D:\src\vcpkg> .\vcpkg install sqlite3:x86-uwp zlib:x64-windows
See .\vcpkg help triplet for all supported targets.

Step 2: Use

VS/MSBuild Project (User-wide integration)

The recommended and most productive way to use vcpkg is via user-wide integration, making the system available for all projects you build. The user-wide integration will prompt for administrator access the first time it is used on a given machine, but afterwords is no longer required and the integration is configured on a per-user basis.

PS D:\src\vcpkg> .\vcpkg integrate install
Applied user-wide integration for this vcpkg root.

All C++ projects can now #include any installed libraries.
Linking will be handled automatically.
Installing new libraries will make them instantly available.

Note: You will need to restart Visual Studio or perform a Build to update intellisense with the changes.
You can now simply use File -> New Project in Visual Studio 2015 or Visual Studio 2017 and the library will be automatically available. For Sqlite, you can try out their C/C++ sample.

To remove the integration for your user, you can use .\vcpkg integrate remove.

CMake (Toolchain File)

The best way to use installed libraries with cmake is via the toolchain file scripts\buildsystems\vcpkg.cmake. To use this file, you simply need to add it onto your CMake command line as -DCMAKE_TOOLCHAIN_FILE=D:\src\vcpkg\scripts\buildsystems\vcpkg.cmake.

If you are using CMake through Open Folder with Visual Studio 2017 you can define CMAKE_TOOLCHAIN_FILE by adding a "variables" section to each of your CMakeSettings.json configurations:

{
  "configurations": [{
    "name": "x86-Debug",
    "generator": "Visual Studio 15 2017",
    "configurationType" : "Debug",
    "buildRoot":  "${env.LOCALAPPDATA}\\CMakeBuild\\${workspaceHash}\\build\\${name}",
    "cmakeCommandArgs": "",
    "buildCommandArgs": "-m -v:minimal",
    "variables": [{
      "name": "CMAKE_TOOLCHAIN_FILE",
      "value": "D:\\src\\vcpkg\\scripts\\buildsystems\\vcpkg.cmake"
    }]
  }]
}

Now let's make a simple CMake project with a main file.

# CMakeLists.txt
cmake_minimum_required(VERSION 3.0)
project(test)

find_package(Sqlite3 REQUIRED)

add_executable(main main.cpp)
target_link_libraries(main sqlite3)

// main.cpp
#include <sqlite3.h>
#include <stdio.h>

int main()
{
    printf("%s\n", sqlite3_libversion());
    return 0;
}

Then, we build our project in the normal CMake way:

PS D:\src\cmake-test> mkdir build 
PS D:\src\cmake-test> cd build
PS D:\src\cmake-test\build> cmake .. "-DCMAKE_TOOLCHAIN_FILE=D:\src\vcpkg\scripts\buildsystems\vcpkg.cmake"
    // omitted CMake output here //
-- Build files have been written to: D:/src/cmake-test/build
PS D:\src\cmake-test\build> cmake --build .
    // omitted MSBuild output here //
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.38
PS D:\src\cmake-test\build> .\Debug\main.exe
3.15.0

Note: The correct sqlite3.dll is automatically copied to the output folder when building for x86-windows. You will need to distribute this along with your application.

Handling libraries without native cmake support

Unlike other platforms, we do not automatically add the include\ directory to your compilation line by default. If you're using a library that does not provide CMake integration, you will need to explicitly search for the files and add them yourself using find_path() and find_library().

# To find and use catch
find_path(CATCH_INCLUDE_DIR catch.hpp)
include_directories(${CATCH_INCLUDE_DIR})

# To find and use azure-storage-cpp
find_path(WASTORAGE_INCLUDE_DIR was/blob.h)
find_library(WASTORAGE_LIBRARY wastorage)
include_directories(${WASTORAGE_INCLUDE_DIR})
link_libraries(${WASTORAGE_LIBRARY})

# Note that we recommend using the target-specific directives for a cleaner cmake:
#     target_include_directories(main ${LIBRARY})
#     target_link_libraries(main PRIVATE ${LIBRARY})

[style-nine]

Has anyone actually built it this way, or is this still more in the experimental stage? Could be interesting to compare the different types of builds.

[rubenwardy]

This can easily be done without modifying minetest, just with a build script

[Megaf]

@rubenwardy precisely, that's one of the things that I find very interesting in this thing.

I'd be glad if someone could test it.

[adrido]

I recently created a irrlicht port to vcpkg. https://github.com/adrido/irrlicht-vcpkg

It brings the goal a bit closer to build Minetest using vcpkg, but it is not enough.
It is possible to build Minetest with a bat file (tested), but the idea of vcpkg is to not require a separate build file for every platform. It should just work with the normal cmakelists file.

Minetests CMakeLists.txt is messed up and overcomplicated. (Are 800+ lines of cmake script realy necessary?)
The issues why its currently not possible to build minetest with vcpkg:
Minetest sets up some random paths instead of using find_package https://github.com/minetest/minetest/blob/master/src/CMakeLists.txt#L267-L282

Minetest requires to select the curl.dll in order to enable curl. Dlls are handled well with vcpkg and are automatically copied to the bin directory. So its not required to hard depend on a dll file.

There are probably even more issues, but I did not further investigate them.

I hope some are able to test the vcpkg port. Ill also anounce it to the irrlicht community, and if there are no objections ill create a PR to vcpkg directly.

The goal (best case scenario) would be to call the 2 line script:

vcpkg install irrlicht zlib sqlite3 luajit --triplet x64-windows
cmake . -DCMAKE_TOOLCHAIN_FILE=D:\vcpkg\scripts\buildsystems\vcpkg.cmake

On Windows 10 with Powershell even a one-line script could be possible (like the linux one-line script)

[nerzhul]

i don't see any interest on this if upstream projects doesn't does it themsleves

[rubenwardy]

I see massive interest in this, it'll help gain contributors on Windows. I started doing this, but my laptop then died and I lost the progress (it was in a VM and not backed up)

[adrido]

i don't see any interest on this if upstream projects doesn't does it themsleves

What do you mean by "upstream projects"? If you mean Irrlicht, there will probably never be a CMakeLists.txt file. So its perfectly fine to copy that one from the vcpkg port to the Irrlicht source.
If you mean vcpkg, my plan is to create a pull request to add this port after this script have been tested. Lots of PR's are merged within one or two Weeks.

The great thing with vcpkg is that it creates a unique interface to manage different libraries. All of Minetests dependencies can be installed with a single line, without messing up with different build-scripts (cmake, msbuild, nothing) , output & include paths, linking- or compileoptions, dependencies of the dependencies, etc.

I see massive interest in this, it'll help gain contributors on Windows. I started doing this, but my laptop then died and I lost the progress (it was in a VM and not backed up)

Sad that you loosed your progress, but if you already have some experience I would be thankful about feedback or help.

[Ezhh]

Anything at all that makes developing on Windows less of a soul-crushingly migraine-inducing trip through multiple hells that makes the lobbing of computers from upper floor windows seem like a suddenly great and fun idea.... is very welcome as far as I'm concerned.

[rubenwardy]

i don't see any interest on this if upstream projects doesn't does it themsleves

If you're referring to Irrlicht, then it is because Irrlicht is largely dead and only has 1 developer

[Megaf]

I've been reading issues regarding Irrlicht and us blaming on it for the past 7 years. Until when are we going to continue doing that?

Anyway, back to this issue. @adrido Could you please streamline and cmake? Would you be willing to do that? I totally agree with you, our cmake is just huge, I think a lot of that stuff was indeed required at some stage, though I think now things should be a bit better.

[adrido]

Update:
vcpkg supports now irrlicht (microsoft/vcpkg@ecff299)

Im currently working on the CMake buildsystem to support vcpkg https://github.com/adrido/minetest/tree/vcpkg-buildsystem

It simplified the cmake system a little bit, but also complicated some parts. But I will take a look into tiding up cmake in another step and separate PR.

It would be really great if someone could follow the documentation https://github.com/adrido/minetest/tree/vcpkg-buildsystem#compiling-on-windows and try if it is complete and understandable.
Im uncertain whether I should keep the old outdated documentation or not.
I would be really thankful about feedback.

[Megaf]

Thanks for your work and time adrido! As a Minetest player and server admin I do really appreciate it. I can't test it myself since I don't use windows myself, but so far things are looking good. About the CMake, yep, it could do with lots of cleaning, and that will for another PR for sure, for now let's just work on vcpkg.

…

On Sat, 2 Mar 2019 at 15:49, adrido ***@***.***> wrote: Update: vcpkg supports now irrlicht ***@***.*** <microsoft/vcpkg@ecff299> ) Im currently working on the CMake buildsystem to support vcpkg https://github.com/adrido/minetest/tree/vcpkg-buildsystem It simplified the cmake system a little bit, but also complicated some parts. But I will take a look into tiding up cmake in another step and separate PR. It would be really great if someone could follow the documentation https://github.com/adrido/minetest/tree/vcpkg-buildsystem#compiling-on-windows and try if it is complete and understandable. Im uncertain whether I should keep the old outdated documentation or not. I would be really thankful about feedback. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7642 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF6gqL69dF3bwUxrGqsE4024_i48sQUcks5vSp1vgaJpZM4V70fT> .