Research alternatives to Hetzner for testnet hosting #132

jakubgs · 2022-11-14T08:54:18Z

Since Hetzner hates crypto mining and states so in their policy:

Therefore the following actions are prohibited:

Operating applications that are used to mine crypto currencies

https://www.hetzner.com/legal/dedicated-server/

We will need to migrate out hosts to an alternative server provider soon. To do that we need to find a suitable alternative.

The general requirement involve:

CPU: 8+ CPUs
RAM: 32+ GB
DISK: ~200 GB OS(mirror), 1+ TB SSD data
Ability to add new drives as storage requirements change.

We'll need about 21 hosts with similar specs.

jakubgs · 2022-11-14T09:08:12Z

Here's a lost of some possible hosting services which provide metal/root servers:

jakubgs · 2022-11-15T16:26:09Z

I'm also discussing other options for hosting with a representative that can help us get a contract with one of these:

jakubgs · 2022-11-15T16:37:33Z

The bandwidth requirements appear to be currently ~150 Mb/s ingress and 250 Mb/s egress:

This min/max graph actually shows it better:

But that will probably change with time.

jakubgs · 2022-11-15T16:42:16Z

Actually, if we look at 30 days there are some big 600 Mb/s spikes:

jakubgs · 2022-11-22T10:38:30Z

According to the OVH Cloud policy for Virtual Private Server ("VPS") in Customer Obligations section:

Customer is prohibited from using or allowing the Services to be used for any intrusive activity or any intrusion attempts (including, but not limited to port scans, sniffing, spoofing), and any activity or contentious behavior such as traffic exchanging (Hitleap, Jingling), Black Hat SEO (downloading and uploading videos from and to online gaming platforms), crypto-currency mining, video game bots, etc. Anonymization services or public proxy (including VPN, Tor, P2P, IRC) and cardsharing (CCCam or equivalent) are not permitted on the Services.

https://us.ovhcloud.com/legal/service-specific-terms

But I see no such policy in the Customer Obligations section for dedicated servers, only:

The Customer accepts sole responsibility for the performance of any and all maintenance operations in reference to the Service(s), including but not limited to any updates to the distribution installed on the Service(s) and upgrades.

But sending a separate email to their sales team is probably a good idea to verify.

jakubgs · 2022-11-22T11:06:00Z

I have called OVH Cloud sales team in Poland and they have told me that the policy has not been updated but the same rules as for VPS also applies to dedicated servers. Just to be sure I also sent an email to sales with the same query:

I'd would like to know what is your policy regarding services related to blockchain networks(testnets specifically) on your dedicated servers. I see you have a restriction in your policy in the VPS section, but no such restriction exists in the dedicated server section. Is that intentional?

But it looks like OVH Cloud will not work out.

EDIT: I've also called their sales team in UK and they checked the French entity policy based on our org being from Switzerland and they said it's against the policy. I've been sent this link as reference:

The Client refrains, as part of or through the Service, from making any activity or behaviour that is
contentious such as traffic exchanges (Hitleap, Jingling, etc.), Black Hat SEO (Download, Reupload of videos
on online video platforms, etc.), cryptocurrency mining, video game bots, etc.

https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/df42916-contrat_partDedie-GB-9.1.pdf
https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/2fbf086-contrat_partDedie-FR-20.0.pdf

jakubgs · 2022-11-22T11:53:38Z

Scaleway seems to have sensible pricing:

https://www.scaleway.com/en/elastic-metal/aluminium/

And the Terms of Use do not seem to have anything about crypto, but I'd need to verify: https://www.scaleway.com/en/terms/

jakubgs · 2022-11-22T12:04:34Z

I've sent their sales team a question:

I'd would like to know what is your policy regarding services related to blockchain networks(Ethereum testnets specifically) on your dedicated servers. I cannot find any restriction in your Terms of Service relate to blockchain or mining, but I would like to verify that you have no such policy, nor are you planning to institute such a policy in the foreseeable future. We are looking for a service to migrate our Ethereum development servers from Hetzner due to their anti-mining policy, but we wouldn't want to jump out of the frying pan into the fire.

jakubgs · 2022-11-22T15:04:51Z

Innovahosting seems to have some sensible prices for dedicated servers:

https://innovahosting.net/dedicated-server-hosting-in-europe

And I see no references to crypto or mining in ther Terms of Service: https://innovahosting.net/terms

Sent a query to their sales team:

I would like to know what is your policy regarding services related to blockchain networks(Ethereum testnets specifically) on your dedicated servers. I cannot find any restriction in your Terms of Service relates to blockchain or mining, but I would like to verify that you have no such policy, nor are you planning to institute such a policy in the foreseeable future.

We are looking for a service to migrate our Ethereum development servers from Hetzner due to their anti-mining policy, but we wouldn't want to jump out of the frying pan into the fire.

jakubgs · 2022-11-22T15:35:31Z

I've received a response from InnovaHosting:

Hello,

WE allowing crypto mining on servers, but we will increase price, due to increased typical electricity usage.

Best regards

Technical Support, IP HOST

They are probably thinking of proof-of-work, so that comment about electricity usage is probably not applicable to us.

jakubgs · 2023-01-04T12:17:44Z

I've received a response from Scaleway support:

Thank you very much for your request.
Regarding your question about the policy related to blockchain networks(Ethereum testnets specifically and mining) i can say to you that we do have a policy about mining and mining is a no go at this moment.
only chia mining is a option but there should be some more detailed conversations about that.

For any other questions just let me know, if i can help you with other services just send me a email or we can schedule a short meeting online.

Wish you a nice evening!

No idea why they think Chia is okay, but all other mining is bad, even proof-of-stake testnets.

jakubgs · 2023-01-04T12:25:36Z

I've also contacted NetCup support: https://www.netcup.eu/support/

Their prices looks sensible:

EDIT: Looks like they have a policy against it: https://www.netcup.eu/bestellen/agb.php

The following activities are expressly prohibited:

Operation of mining-services such as "Bitcoin", "Ethereum", "OneCoin" or "Monero".

jakubgs · 2023-01-04T13:21:12Z

I've also contacted the ServeRhino support: https://serverhino.com/support/

The R-Hexa and R-Octo root servers look decent::

EDIT: Actually, there is a policy against it in "Restrictions When Using Services":

4.3. use of the services to receive cryptocurrencies (mining) is prohibited for all the services, except the designated physical servers.

Not sure what they mean by "designated physical servers", so I'll ask about that:

I would like to know what is your policy regarding services related to blockchain networks - specifically Ethereum proof-of-stake testnets - on your dedicated root servers.
I found this line in your policy which I'd like to clarify:

4.3. use of the services to receive cryptocurrencies (mining) is prohibited for all the services, except the designated physical servers.

We are looking for a service to migrate our Ethereum development servers from Hetzner due to their anti-cryptocurrency policy to your root servers, but we wouldn't want to jump out of the frying pan into the fire. We are NOT looking to do "mining", we intend to run Ethereum test networks(testnets) for the sake of development of Ethereum software, and not "mining". There is no actual valuable cryptocurrencies being generated on our testnet servers.

Would that be compatible with your policy?

Cheers

jakubgs · 2023-01-04T14:20:42Z

I've checked the Terms of Service for CtrlServers and found this: https://www.ctrlservers.com/terms-of-service/

Payments made via cryptocurrency / virtual currency will not be refunded via cryptocurrency / virtual currency due to the fluctuating nature of cryptocurrency / virtual currency. Instead a credit will be placed on account. This applies to any and all cryptocurrency / virtual currency.

And nothing else related to cryptocurrencies, so I guess they might be fine with mining.
But I sent an email to [email protected] anyway.

jakubgs · 2023-01-04T15:24:05Z

Response from CtrlServers:

We don't have any such policy to restrict blockchain development or mining but we don't allow projects or activities which involved in steeling, fraud, gambling or similar coins / mining / projects which attract complainants. Also we don't allow to change power profile of servers which damage hardware in long run.

If above terms suits you, may i know what sort of hardware requirement you have and how many servers you are looking?

I provided them with our requirements.

jakubgs · 2023-01-05T11:39:44Z

It appears they could give us a discount on a bulk order of 22 servers:

Good morning, Please provide details about your avg bandwidth consumption per server and do you plan to order 22 servers at once? So i can get approval for special discount in this case.

We can offer you something like

Dual Intel Xeon (6 or 8 cores per processor)
64GB RAM DDR3 / DDR4
2x 480GB Write Intensive SSD (Hardware RAID1 - Mirror)
1x 960GB Mix Use SSD
1IP
IPMI Access

Please let me know if you want any specific change in above configuration.

Kind Regards,

Though the extra storage SSD will need adjustment. We'd probably need:

Sepolia - 2x 1 TB SSD
Prater - 2x 1 TB SSD
Mainnet - 2x 2 TB SSD

Sent them also rough bandwidth requirements:

Sepolia - ~10 Mb/s igress, ~20 Mb/s egress (3 node)
Prater - ~60 Mb/s igress, ~100 Mb/s egress (10 nodes)
Mainnet - ~80 Mb/s igress, ~300 Mb/s egress (8 nodes)

jakubgs · 2023-01-05T12:48:21Z

Their offer is 169EUR per month for this:

Dual Intel E5-2620 Xeon (2x 6 cores @ 2.00 Ghz)
64GB RAM DDR3
2x 960GB Mix Use SSD
2x 1.92TB Mix Use SSD
100Tb Data Traffic
1Gbps Dedicated Port
1IP
IPMI Access

Which is not amazing, considering we pay 37.3 EUR(AX-41) + 2x 20.9 EUR(2 TB SSD) = 79.1 EUR, but best so far.

Ticket URL: https://www.ctrlservers.com/viewticket.php?tid=781855&c=XuSzHsLy

jakubgs · 2023-01-05T13:09:28Z

Zahary pointed out that 2 Ghz is kinda low and he had issues syncing on a server like that:

Aim for the fastest single-core performance. The number of cores is less important

I asked them about other available CPUs:

As for CPU, we're more interested in higher single core performance than number of cores. The 2 Ghz of E5-2620 is kinda low for us.
For that reason I'd be more interested in CPUs like E3-1270 v6, E-2186G, or E-2274G. Are those also possible in those combinations?

jakubgs · 2023-01-05T14:12:00Z

Another offer for 139EUR per month:

Intel E3-1270v6 (4 cores, 8 threads @ 4.20 GHz)
32GB DDR4 RAM
2x 240GB SSD (Software RAID1)
2x 960GB SSD
100TB Traffic
1Gbps Dedicated Port
1IP

That's pretty nice. But I think we'd need 64 GB RAM, which would be 162EUR.

jakubgs · 2023-01-10T18:07:32Z

I called OVHCloud UK sales team to confirm, and according to them it is prohibited:

The Client refrains, as part of or through the Service, from making any activity or behaviour that is
contentious such as traffic exchanges (Hitleap, Jingling, etc.), Black Hat SEO (Download, Reupload of videos
on online video platforms, etc.), cryptocurrency mining, video game bots, etc.

https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/df42916-contrat_partDedie-GB-9.1.pdf
https://storage.gra.cloud.ovh.net/v1/AUTH_325716a587c64897acbef9a4a4726e38/contracts/2fbf086-contrat_partDedie-FR-20.0.pdf

jakubgs · 2023-01-11T14:04:23Z

I tried paying for the invoice created by CtrlServers staff, but it seems to have only option to pay via paypal:

And it seems to force you to create an account:

Normally there's a checkbox to pay as a guest, but not here. I'm starting to reconsider using this provider.

jakubgs · 2023-01-11T18:28:33Z

I explained my issues with payment and this is what I got:

I am sorry but we have no control over paypal flow. It's all depends upon location. May be you can try using "Credit Card" instead of selecting paypal? but it also available location wise. Most of user can see Paypal and Credit Card both options on 2checkout page and some only Paypal.

It might be that we have to pay with a bank transfer because I'm not creating a Paypal account.

jakubgs · 2023-01-24T12:42:12Z

I have received a Stripe link from CtrlServers and that worked well with a Revolut card.

We should get a server shortly and then we can test it by deploying some Prater nodes without validators.

Test for replacement of Hetzner hosts: #132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs · 2023-01-30T20:35:45Z

We received the host, but since it has two separate data volumes I had to refactor the bootstrap role:

https://github.com/status-im/infra-role-bootstrap-linux/commit/43b550d7 - volumes: move DC specific files to dcs folder
https://github.com/status-im/infra-role-bootstrap-linux/commit/9e1e1067 - volumes: extract volume creation to separate file
https://github.com/status-im/infra-role-bootstrap-linux/commit/4f0709af - volumes: generic volume finder for most providers
https://github.com/status-im/infra-role-bootstrap-linux/commit/9cbaa6c4 - volumes: support multiple extra data volumes

And now I can finally add the new host with two volumes:

c05fad0e - add linux-01.cs-eu-nld1.nimbus.prater host

And the volumes appear to be mounted correctly:

[email protected]:~ % df -h /data /docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        880G   28K  835G   1% /data
/dev/sdd        880G   28K  835G   1% /docker

Although maybe I should have went with 2TB from the start.

I added it without any validators to nimbus.prater fleet configuration:

infra-nimbus/ansible/group_vars/nimbus.prater.yml

Lines 154 to 158 in c05fad0

 'linux-01.cs-eu-nld1.nimbus.prater': # 0 each 

 - { branch: 'stable', build_freq: '*-*-* 13:00:00' } 

 - { branch: 'testing', build_freq: '*-*-* 15:00:00', nim_commit: 'version-1-6' } 

 - { branch: 'unstable', build_freq: '*-*-* 17:00:00', public_api: true } 

 - { branch: 'libp2p', build_freq: '*-*-* 19:00:00' }

jakubgs · 2023-01-30T21:38:01Z

The nodes are up and syncing:

[email protected]:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"17960","sync_distance":"4865917","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"16916","sync_distance":"4866961","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"6120","sync_distance":"4877757","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"440","sync_distance":"4883437","is_syncing":true,"is_optimistic":false}}

[email protected]:~ % for dir in /docker/geth-*; do $dir/rpc.sh eth_syncing | jq -c; done         
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}

Disk utilization is fairly low:

jakubgs · 2023-01-31T09:24:52Z

The beacon nodes are syncing:

[email protected]:~ % for port in $(seq 9300 9303); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"361644","sync_distance":"4525781","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"498069","sync_distance":"4389356","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"344957","sync_distance":"4542468","is_syncing":true,"is_optimistic":false}}
{"data":{"head_slot":"331474","sync_distance":"4555951","is_syncing":true,"is_optimistic":false}}

With CPU load of about 5.0 on a machine with 8 cores:

Low memory usage but plenty of caching and no SWAPing so far:

And traffic of about 10 Mb/s:

jakubgs · 2023-02-09T11:22:57Z

Looks like sync failed because one of the drives dedicated to Geth got filled up:

[email protected]:/docker % sudo du -hsc /docker/*        
206G	/docker/geth-goerli-01
203G	/docker/geth-goerli-02
235G	/docker/geth-goerli-03
192G	/docker/geth-goerli-04
16K	/docker/lost+found
835G	total

[email protected]:/docker % df -h /docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd        880G  835G     0 100% /docker

We probably should go with just 2 TB drives for all hosts.

jakubgs · 2023-08-21T17:42:51Z

Nodes are still syncing:

[email protected]:~ % for port in $(seq 9300 9301); do c 0:$port/eth/v1/node/syncing | jq -c; done 
{"data":{"head_slot":"106853","sync_distance":"6237459","is_syncing":true,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"122072","sync_distance":"6222240","is_syncing":true,"is_optimistic":false,"el_offline":false}}

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs · 2023-08-21T17:47:45Z

Disabled resync timer for now since it's not necessary;

0cdb2d6e - macos-01.nimbus.prater: disable resync timer

jakubgs · 2023-08-23T14:38:47Z

For some reason the beacon-node-prater-testing node is syncing much slower than the unstable one:

[email protected]:~ % sudo du -hsc /Users/nimbus/beacon-node-prater-*
3.9G	/Users/nimbus/beacon-node-prater-testing
 24G	/Users/nimbus/beacon-node-prater-unstable
 27G	total

jakubgs · 2023-08-23T14:54:48Z

I found these in logs:

Unable to download checkpoint state
  error="Communication failed while sending/receiving request, http error [HttpConnectionError]: Connection timed out"
  syncTarget=finalized
  restUrl=http:https://linux-01.he-eu-hel1.nimbus.prater.wg:9300/ stateId=finalized

And realized the trusted node URL is wrong:

9df1b95f - nimbus.prater: fix trusted node URL for MacOS hosts

I also had to resolve a firewall issue where the port 9300 was showing up as filtered, but a reboot fixed it.

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs · 2023-08-29T14:57:39Z

It appears one node has been fully synced but the other has not:

[email protected]:~ % for port in $(seq 9300 9301); do c 0:$port/eth/v1/node/syncing | jq -c; done 
{"data":{"head_slot":"2304209","sync_distance":"4096875","is_syncing":true,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"6401083","sync_distance":"1","is_syncing":false,"is_optimistic":false,"el_offline":false}}

[email protected]:~ % sudo du -hsc /Users/nimbus/beacon-node-prater-*
  0B	/Users/nimbus/beacon-node-prater-stable
 23G	/Users/nimbus/beacon-node-prater-testing
118G	/Users/nimbus/beacon-node-prater-unstable
141G	total

This is pretty weird, and might be caused by an enabled resync timer.

jakubgs · 2023-09-06T08:00:52Z

MacOS M1 host is fully synced:

[email protected]:~ % for port in $(seq 9300 9301); do c 0:$port/eth/v1/node/syncing | jq -c; done 
{"data":{"head_slot":"6456601","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"6456601","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}

[email protected]:~ % sudo du -hsc /Users/nimbus/beacon-node-prater-*
  0B	/Users/nimbus/beacon-node-prater-stable
118G	/Users/nimbus/beacon-node-prater-testing
118G	/Users/nimbus/beacon-node-prater-unstable
236G	total

I have moved the validators and decomissioned the old MacOS host:

227206c8 - nimbus.prater: move validators to macm1-01
a658d312 - nimbus.prater: add stable node to macm1-01 host
16027612 - drop macos-01.ms-eu-dublin.nimbus.prater host

macm1-01:~ root# for DIR in /Users/nimbus/beacon-node-prater-*/data/secrets; do find $DIR | wc -l; done
    3001
    3001

Too bad the MacStadium host was purchased on 2022-01-24, so it won't be released until 2023-09-24.

The only thing remaining is to get Windows host working.

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

Replaced by `macm1-01.ih-eu-mda1.nimbus.prater`. #132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs · 2023-09-06T08:32:11Z

Looks like we can't handle 3000 validators per node currently:

{
  "lvl": "ERR",
  "ts": "2023-09-06 11:30:37.407+03:00",
  "msg": "Failed to read passphrase file",
  "error_msg": "Too many open files",
  "path": "/Users/nimbus/beacon-node-prater-testing/data/secrets/0x88b335a2a55c7f4e05488f8669abcc6bdff801e6ae943ec485c7a65bbabe8501f2cd893b674b88e90060bd444074e0d8"
}

This is related to open file limits not being applied due to SIP:

infra-role-bootstrap-macos#2df89a2e - consul: raise soft maxfiles limit for Consul service

status-im/infra-nimbus#132 Signed-off-by: Jakub Sokołowski <[email protected]>

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs · 2023-09-06T09:02:07Z

Fixed by raising the soft limit for the service:

infra-role-beacon-node-macos#4fd4f79d - service: raise open files limit to 16384 by default
infra-nimbus#0d323c70 - drop macos-01 file from host_vars, add missing exec url
infra-nimbus#f1ec3338 - requirements: macos role increase of soft max files

All nodes are running:

[email protected]:~ % for port in $(seq 9300 9302); do c 0:$port/eth/v1/node/syncing | jq -c; done 
{"data":{"head_slot":"25827","sync_distance":"6431082","is_syncing":true,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"6456909","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"6456909","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}

jakubgs · 2023-09-06T09:27:19Z

The windows host has trouble building nodes in the first place:

PATH=".:C:\Users\admin\bin;C:\ProgramData\scoop\apps\git\2.28.0.windows.1\mingw64\bin;C:\ProgramData\scoop\apps\git\2.28.0.windows.1\usr\lo
cal\bin;C:\ProgramData\scoop\apps\git\2.28.0.windows.1\usr\bin;C:\ProgramData\scoop\apps\git\2.28.0.windows.1\usr\bin;C:\ProgramData\scoop\apps\git
\2.28.0.windows.1\mingw64\bin;C:\ProgramData\scoop\apps\git\2.28.0.windows.1\usr\bin;C:\Users\admin\bin;C:\ProgramData\scoop\apps\gcc\current\bin;C
:\ProgramData\scoop\apps\python\current\Scripts;C:\ProgramData\scoop\apps\python\current;C:\ProgramData\scoop\shims;C:\Windows\system32;C:\Windows;
C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\WireGuard;C:\Windows\system32\conf
ig\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\Users\admin\scoop\shims;C:\Users\admin\AppData\Local\Microsoft\WindowsApps;C:\ProgramData\s
coop\apps\git\2.28.0.windows.1\usr\bin\vendor_perl;C:\ProgramData\scoop\apps\git\2.28.0.windows.1\usr\bin\core_perl" "C:/ProgramData/scoop/apps/mak
e/current/bin/make.exe" -C vendor/nim-nat-traversal/vendor/miniupnp/miniupnpc -f Makefile.mingw CC=gcc libminiupnpc.a                             
make[1]: Entering directory 'D:/beacon-node-prater-stable/repo/vendor/nim-nat-traversal/vendor/miniupnp/miniupnpc'                                
process_begin: CreateProcess(NULL, git rev-parse --short HEAD, ...) failed.                                                                       
Makefile.mingw:56: pipe: No error                                                                                                                 
gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe wingenminiupnpcstrings.c           
process_begin: CreateProcess(NULL, gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe w
ingenminiupnpcstrings.c, ...) failed.                                                                                                             
make (e=2): The system cannot find the file specified.                                                                                            
make[1]: *** [Makefile.mingw:121: wingenminiupnpcstrings.exe] Error 2                                                                             
make[1]: Leaving directory 'D:/beacon-node-prater-stable/repo/vendor/nim-nat-traversal/vendor/miniupnp/miniupnpc'                                 
make: *** [vendor/nimbus-build-system/makefiles/targets.mk:134: libminiupnpc.a] Error 2

jakubgs · 2023-09-08T10:02:23Z

According to this:

Windows Server 2019 requires license activation after 180 days, otherwise the system will shut down.

https://www.techtarget.com/searchwindowsserver/tutorial/Activate-Windows-Server-2019-with-KMS-or-the-command-line

And our current Windows host was installed on 2023/07/05:

admin@windows-01 MINGW64 ~
$ systeminfo | grep 'install date'
Original Install Date:     7/5/2023, 9:31:29 AM

That means I have time till 2023/12/31 to activate a license.

jakubgs · 2023-10-02T13:19:56Z

I found an interesting difference between the new Windows host:

`windows-01.he-eu-hel1.nimbus.prater`

CPLUS_INCLUDE_PATH=C:\ProgramData\scoop\apps\mingw-nuwen\current\include;C:\ProgramData\scoop\apps\mingw-nuwen\current\include\freetype2
C_INCLUDE_PATH=C:\ProgramData\scoop\apps\mingw-nuwen\current\include;C:\ProgramData\scoop\apps\mingw-nuwen\current\include\freetype2

$ which gcc
/c/ProgramData/scoop/apps/mingw-nuwen/current/bin/gcc

`windows-01.ih-eu-mda1.nimbus.prater`

CPLUS_INCLUDE_PATH=C:\ProgramData\scoop\apps\gcc\current\include
C_INCLUDE_PATH=C:\ProgramData\scoop\apps\gcc\current\include

$ which gcc
/c/ProgramData/scoop/apps/gcc/current/bin/gcc

So I think the issue is the use off stock GCC instead of the MinGW-Nuwen distribution.

jakubgs · 2023-10-02T13:42:14Z

I found a very well priced Windows Standard 2022 for 8 cores at CHF 69.90:
https://thekeystore.ch/en/products/microsoft-windows-server-2019-standard-8-core

Avoids issues with Windows builds like: ``` gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe wingenminiupnpcstrings.c process_begin: CreateProcess(NULL, gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe w ingenminiupnpcstrings.c, ...) failed. make (e=2): The system cannot find the file specified. ``` status-im/infra-nimbus#132 (comment) Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs · 2023-10-02T16:46:49Z

Looks like switching to mingw-nuwen helped, but it's still failing:

infra-role-beacon-node-windows#51bd279d - install: use mingw-nuwen instead of stock gcc

jakubgs · 2023-10-16T11:59:13Z

When I go repo/vendor/nim-nat-traversal/vendor/miniupnp/miniupnpc and run make it does fail:

admin@windows-01 MINGW64 .../miniupnp/miniupnpc ((17cecd5...))                                                                                                                       
$ make V=2                                                                                                                                                                           
gcc  -o build/upnpc-static build/upnpc.o build/libminiupnpc.a                                                                                                                        
c:/programdata/scoop/apps/mingw-nuwen/current/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: build/upnpc.o:upnpc.c:(.text+0x93): undefined refere
nce to `__imp_UPNP_GetConnectionTypeInfo' 
c:/programdata/scoop/apps/mingw-nuwen/current/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: build/upnpc.o:upnpc.c:(.text+0xc6): undefined refere
nce to `__imp_UPNP_GetStatusInfo'
c:/programdata/scoop/apps/mingw-nuwen/current/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: build/upnpc.o:upnpc.c:(.text+0xf8): undefined refere
nce to `__imp_UPNP_GetLinkLayerMaxBitRates'
...                                  
rence to `__imp_UPNP_GetTotalPacketsSent'                                                                                                                                            
c:/programdata/scoop/apps/mingw-nuwen/current/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: build/upnpc.o:upnpc.c:(.text+0x1e4a): undefined refe
rence to `__imp_UPNP_GetTotalPacketsReceived'                                                                                                                                        
c:/programdata/scoop/apps/mingw-nuwen/current/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: build/upnpc.o:upnpc.c:(.text+0x1ef1): undefined refe
rence to `__imp_UPNP_GetOutboundPinholeTimeout'                                                                                                                                      
c:/programdata/scoop/apps/mingw-nuwen/current/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: build/upnpc.o:upnpc.c:(.text+0x1eff): undefined refe
rence to `__imp_strupnperror'                                                                                                                                                        
collect2.exe: error: ld returned 1 exit status                                                                                                                                       
make: *** [Makefile:327: build/upnpc-static] Error 1

Which looks similar to these:

https://stackoverflow.com/questions/71864639/

jakubgs · 2023-10-16T12:00:24Z

What's interesting is that it also fails on the Hetzner host:

admin@windows-01 MINGW64 .../miniupnp/miniupnpc ((17cecd5...))
$ make
...
collect2.exe: error: ld returned 1 exit status                                                                                                                                       
make: *** [Makefile:327: build/upnpc-static] Error 1

But the exe file already exists there:

admin@windows-01 MINGW64 .../miniupnp/miniupnpc ((17cecd5...))
$ ls -l wingenminiupnpcstrings.exe
-rwxr-xr-x 1 admin 197121 237795 Feb 25  2022 wingenminiupnpcstrings.exe*

So it seems like it continued to work purely because this file already exists.

jakubgs · 2023-10-16T12:14:37Z

I copied wingenminiupnpcstrings.exe from the old host to windows-01.ih-eu-mda1.nimbus.prater to see what happens:

make[1]: Entering directory 'D:/beacon-node-prater-stable/repo/vendor/nim-nat-traversal/vendor/miniupnp/miniupnpc'                                                                   
process_begin: CreateProcess(NULL, git rev-parse --short HEAD, ...) failed.
Makefile.mingw:56: pipe: No error
gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe wingenminiupnpcstrings.c
process_begin: CreateProcess(NULL, gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe wingenminiupnpcstrings.c, ...) faile
d.
make (e=2): The system cannot find the file specified.                                                                                                                               
make[1]: *** [Makefile.mingw:121: wingenminiupnpcstrings.exe] Error 2

It's still broken.

jakubgs · 2023-10-16T12:32:11Z

When I run the gcc command myself it works fine:

admin@windows-01 MINGW64 .../miniupnp/miniupnpc ((17cecd5...))
$ gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe wingenminiupnpcstrings.c

admin@windows-01 MINGW64 .../miniupnp/miniupnpc ((17cecd5...))                                                                                                                       
$ ls -l wingenminiupnpcstrings.exe
-rwxr-xr-x 1 admin 197121 238482 Oct 16 05:22 wingenminiupnpcstrings.exe*

Which means the environment in which the command is run by make is wrong.

jakubgs · 2023-10-16T12:32:18Z

Found a relevant issue:

miniupnpc CMake build fails on Windows miniupnp/miniupnp#270

There was a fix applied in transmission/miniupnpc@fa4fa71 from this PR:

Fix msvc miniupnpc shared library build miniupnp/miniupnp#304

But it appears we already have this fix in our fork: status-im/miniupnp@d3a7441

But this comment states that it was not really fixed: miniupnp/miniupnp#270 (comment)

jakubgs · 2023-10-16T13:37:25Z

I've opened an issue in nimbus-eth2 for these Windows build problems:

Project fails to build on Windows Server 2019 nimbus-eth2#5507

I'm closing this as done.

jakubgs self-assigned this Nov 14, 2022

jakubgs added a commit that referenced this issue Jan 30, 2023

add linux-01.cs-eu-nld1.nimbus.prater host

94b0ed4

Test for replacement of Hetzner hosts: #132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Jan 30, 2023

add linux-01.cs-eu-nld1.nimbus.prater host

c05fad0

Test for replacement of Hetzner hosts: #132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Aug 21, 2023

macos-01.nimbus.prater: disable resync timer

0cdb2d6

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Aug 23, 2023

nimbus.prater: fix trusted node URL for MacOS hosts

9df1b95

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Sep 6, 2023

nimbus.prater: move validators to macm1-01

227206c

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Sep 6, 2023

nimbus.prater: add stable node to macm1-01 host

a658d31

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Sep 6, 2023

drop macos-01.ms-eu-dublin.nimbus.prater host

1602761

Replaced by `macm1-01.ih-eu-mda1.nimbus.prater`. #132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit to status-im/infra-role-beacon-node-macos that referenced this issue Sep 6, 2023

service: raise open files limit to 16384 by default

4fd4f79

status-im/infra-nimbus#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Sep 6, 2023

drop macos-01 file from host_vars, add missing exec url

0d323c7

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit that referenced this issue Sep 6, 2023

requirements: macos role increase of soft max files

f1ec333

#132 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs closed this as completed Oct 16, 2023

jakubgs mentioned this issue May 13, 2024

Upgrade storage for mainnet fleet #184

Closed

Research alternatives to Hetzner for testnet hosting #132

Research alternatives to Hetzner for testnet hosting #132

Comments

jakubgs commented Nov 14, 2022 • edited Loading

jakubgs commented Nov 14, 2022 • edited Loading

jakubgs commented Nov 15, 2022

jakubgs commented Nov 15, 2022 • edited Loading

jakubgs commented Nov 15, 2022

jakubgs commented Nov 22, 2022

jakubgs commented Nov 22, 2022 • edited Loading

jakubgs commented Nov 22, 2022

jakubgs commented Nov 22, 2022

jakubgs commented Nov 22, 2022

jakubgs commented Nov 22, 2022

jakubgs commented Jan 4, 2023

jakubgs commented Jan 4, 2023 • edited Loading

jakubgs commented Jan 4, 2023 • edited Loading

jakubgs commented Jan 4, 2023

jakubgs commented Jan 4, 2023

jakubgs commented Jan 5, 2023 • edited Loading

jakubgs commented Jan 5, 2023 • edited Loading

jakubgs commented Jan 5, 2023

jakubgs commented Jan 5, 2023 • edited Loading

jakubgs commented Jan 10, 2023

jakubgs commented Jan 11, 2023

jakubgs commented Jan 11, 2023

jakubgs commented Jan 24, 2023

jakubgs commented Jan 30, 2023

jakubgs commented Jan 30, 2023

jakubgs commented Jan 31, 2023 • edited Loading

jakubgs commented Feb 9, 2023

jakubgs commented Aug 21, 2023

jakubgs commented Aug 21, 2023

jakubgs commented Aug 23, 2023 • edited Loading

jakubgs commented Aug 23, 2023

jakubgs commented Aug 29, 2023

jakubgs commented Sep 6, 2023 • edited Loading

jakubgs commented Sep 6, 2023

jakubgs commented Sep 6, 2023

jakubgs commented Sep 6, 2023

jakubgs commented Sep 8, 2023

jakubgs commented Oct 2, 2023 • edited Loading

windows-01.he-eu-hel1.nimbus.prater

windows-01.ih-eu-mda1.nimbus.prater

jakubgs commented Oct 2, 2023

jakubgs commented Oct 2, 2023 • edited Loading

jakubgs commented Oct 16, 2023

jakubgs commented Oct 16, 2023

jakubgs commented Oct 16, 2023

jakubgs commented Oct 16, 2023

jakubgs commented Oct 16, 2023 • edited Loading

jakubgs commented Oct 16, 2023

jakubgs commented Nov 14, 2022 •

edited

Loading

jakubgs commented Nov 14, 2022 •

edited

Loading

jakubgs commented Nov 15, 2022 •

edited

Loading

jakubgs commented Nov 22, 2022 •

edited

Loading

jakubgs commented Jan 4, 2023 •

edited

Loading

jakubgs commented Jan 4, 2023 •

edited

Loading

jakubgs commented Jan 5, 2023 •

edited

Loading

jakubgs commented Jan 5, 2023 •

edited

Loading

jakubgs commented Jan 5, 2023 •

edited

Loading

jakubgs commented Jan 31, 2023 •

edited

Loading

jakubgs commented Aug 23, 2023 •

edited

Loading

jakubgs commented Sep 6, 2023 •

edited

Loading

jakubgs commented Oct 2, 2023 •

edited

Loading

`windows-01.he-eu-hel1.nimbus.prater`

`windows-01.ih-eu-mda1.nimbus.prater`

jakubgs commented Oct 2, 2023 •

edited

Loading

jakubgs commented Oct 16, 2023 •

edited

Loading