0.0.6 release

CHANGES.md

@@ -1,5 +1,12 @@
 # Changes
 
+## 0.0.6 (2022-02-11)
+
+- FEATURE: Moved all cluster configuration files, scripts and keys into hidden folder named equivalent to the cluster prefix, both local and remote.
+- FEATURE: New CLI and API for accessing the catalog of server types, their specifications, prices and data center locations.
+- FIX: Setting debug environment variable `SCHERBELBERG_DEBUG` to `1` also automatically lowers the log level to `DEBUG` (`10`) if *scherbelberg* is used via the CLI.
+- FIX: Added missing logging during cluster destruction.
+
 ## 0.0.5 (2022-02-06)
 
 - FEATURE: Dask scheduler and worker processes run as systemd services, allowing them to be restarted and the nodes to be rebooted, see #1.

README.md

@@ -16,7 +16,7 @@
 
 ## Synopsis
 
-`scherbelberg` provides both a CLI interface and a library for deploying and managing small [Dask](https://dask.org/)-based HPC clusters in the [Hetzner cloud](https://cloud.hetzner.com/). Development status *alpha*, stability *acceptable*, security also *acceptable* but implementation needs a review.
+`scherbelberg` provides both a CLI and an API for deploying and managing small [Dask](https://dask.org/)-based HPC clusters in the [Hetzner cloud](https://cloud.hetzner.com/). Development status *alpha*, stability *acceptable*, security also *acceptable* but implementation needs a review.
 
 ## Project's Name
 
@@ -49,6 +49,7 @@ Options:
  --help Show this message and exit.
 
 Commands:
+ catalog list data centers and available servers types
  create create cluster
  destroy destroy cluster
  ls list cluster nodes
@@ -57,8 +58,6 @@ Commands:
  ssh ssh into cluster node
 ```
 
-At the moment, the ssh sub-command is broken on Windows.
-
 See [chapter on CLI](https://scherbelberg.readthedocs.io/en/latest/cli.html) in `scherbelberg`'s documentation for further details.
 
 ## API

docs/source/api.rst

@@ -16,3 +16,4 @@ The application programming interface (API) is one of two modes of usage, next t
  process
  node
  sshconfig
+ catalog

docs/source/catalog.rst

@@ -0,0 +1,13 @@
+.. _catalog:
+
+Catalog
+=======
+
+*scherbelberg* offers facilities to get a list of available data center locations as well as lists of available server types per location plus their specifications and prices.
+
+Routines
+--------
+
+.. autofunction:: scherbelberg.get_datacenters
+
+.. autofunction:: scherbelberg.get_servertypes

docs/source/debugging.rst

@@ -22,7 +22,7 @@ If *scherbelberg* is used via its :ref:`API <api>`, the log level can be adjuste
 
  The used logger is, by default, named after the cluster, i.e. its ``prefix``.
 
-For additional insights and debugging output, run-time type checks based on `typeguard`_ can be activated by setting the ``SCHERBELBERG_DEBUG`` environment variable to ``1`` prior to running a CLI command or prior to importing *scherbelberg* in Python.
+For additional insights and debugging output, run-time type checks based on `typeguard`_ can be activated by setting the ``SCHERBELBERG_DEBUG`` environment variable to ``1`` prior to running a CLI command or prior to importing *scherbelberg* in Python. As a side effect, this will also automatically set the log level to ``DEBUG`` (i.e. ``10``) if *scherbelberg* is used via the command line.
 
 .. _log level: https://docs.python.org/3/library/logging.html#levels
 .. _typeguard: https://typeguard.readthedocs.io/

docs/source/index.rst

@@ -47,7 +47,7 @@ scherbelberg - HPC cluster deployment and management for the Hetzner Cloud
 User's guide
 ------------
 
-*scherbelberg* provides both a **CLI interface** and a **Python API** for deploying and managing small **Dask-based HPC clusters in the Hetzner cloud**. Development status alpha, stability acceptable, security also acceptable but implementation needs a review.
+*scherbelberg* provides both a **CLI** and a **Python API** for deploying and managing small **Dask-based HPC clusters in the Hetzner cloud**. Development status alpha, stability acceptable, security also acceptable but implementation needs a review.
 
 .. toctree::
  :maxdepth: 2

setup.py

@@ -66,6 +66,7 @@
  "dask",
  "hcloud",
  "pyyaml",
+ "tabulate",
  "typeguard",
 ]
 extras_require = {

src/scherbelberg/__init__.py

@@ -27,19 +27,24 @@
 # VERSION
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
-__version__ = "0.0.5"
+__version__ = "0.0.6"
 
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 # EXPORT
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
+from ._core.catalog import (
+ get_datacenters,
+ get_servertypes,
+)
 from ._core.cluster import (
  Cluster,
  ClusterSchedulerNotFound,
  ClusterWorkerNotFound,
  ClusterFirewallNotFound,
  ClusterNetworkNotFound,
 )
+from ._core.creator import ClusterPrefixFolderExists
 from ._core.command import Command
 from ._core.node import (
  Node,

src/scherbelberg/_cli/catalog.py

@@ -0,0 +1,109 @@
+# -*- coding: utf-8 -*-
+
+"""
+
+SCHERBELBERG
+HPC cluster deployment and management for the Hetzner Cloud
+
+https://github.com/pleiszenburg/scherbelberg
+
+ src/scherbelberg/_cli/catalog.py: List data centers and available types of servers
+
+ Copyright (C) 2021-2022 Sebastian M. Ernst <[email protected]>
+
+<LICENSE_BLOCK>
+The contents of this file are subject to the BSD 3-Clause License
+("License"). You may not use this file except in
+compliance with the License. You may obtain a copy of the License at
+https://github.com/pleiszenburg/scherbelberg/blob/master/LICENSE
+Software distributed under the License is distributed on an "AS IS" basis,
+WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the
+specific language governing rights and limitations under the License.
+</LICENSE_BLOCK>
+
+"""
+
+
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+# IMPORT
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+from asyncio import run
+from logging import ERROR
+
+import click
+from tabulate import tabulate
+
+from .._core.catalog import (
+ get_datacenters,
+ get_servertypes,
+)
+from .._core.const import TOKENVAR
+from .._core.log import configure_log
+
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+# ROUTINES
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+@click.command(short_help="list data centers and available servers types")
+@click.option("-t", "--tokenvar", default=TOKENVAR, type=str, show_default=True)
+@click.option("-l", "--log_level", default=ERROR, type=int, show_default=True)
+@click.argument("datacenter", nargs=1, required=False)
+def catalog(
+ tokenvar,
+ log_level,
+ datacenter,
+):
+
+ configure_log(log_level)
+
+ if datacenter is None:
+
+ table = run(get_datacenters(tokenvar = tokenvar))
+ columns = (
+ 'name',
+ 'city',
+ 'country',
+ 'description',
+ 'latitude',
+ 'longitude',
+ 'network_zone',
+ 'location_name',
+ 'location_description',
+ )
+ table = [
+ [row[column] for column in columns]
+ for row in table
+ ]
+ click.echo(tabulate(
+ table,
+ headers=columns,
+ tablefmt="github",
+ ))
+
+ return
+
+ table = run(get_servertypes(datacenter = datacenter, tokenvar = tokenvar))
+ columns = (
+ 'name',
+ # 'description',
+ 'cores',
+ 'cpu_type',
+ 'memory',
+ 'disk',
+ 'storage_type',
+ 'price_hourly_net',
+ 'price_hourly_gross',
+ 'price_monthly_net',
+ 'price_monthly_gross',
+ 'deprecated',
+ )
+ table = [
+ [row[column] for column in columns]
+ for row in table
+ ]
+ click.echo(tabulate(
+ table,
+ headers=columns,
+ tablefmt="github",
+ ))

src/scherbelberg/_core/catalog.py

@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+
+"""
+
+SCHERBELBERG
+HPC cluster deployment and management for the Hetzner Cloud
+
+https://github.com/pleiszenburg/scherbelberg
+
+ src/scherbelberg/_core/catalog.py: List data centers and available types of servers
+
+ Copyright (C) 2021-2022 Sebastian M. Ernst <[email protected]>
+
+<LICENSE_BLOCK>
+The contents of this file are subject to the BSD 3-Clause License
+("License"). You may not use this file except in
+compliance with the License. You may obtain a copy of the License at
+https://github.com/pleiszenburg/scherbelberg/blob/master/LICENSE
+Software distributed under the License is distributed on an "AS IS" basis,
+WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the
+specific language governing rights and limitations under the License.
+</LICENSE_BLOCK>
+
+"""
+
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+# IMPORT
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+import os
+from typing import Any, Dict, List, Optional
+
+from hcloud import Client
+
+from hcloud.datacenters.client import DatacentersClient
+from hcloud.datacenters.domain import Datacenter
+
+from hcloud.locations.domain import Location
+
+from hcloud.server_types.client import ServerTypesClient
+from hcloud.server_types.domain import ServerType
+
+from .const import HETZNER_DATACENTER, TOKENVAR
+from .debug import typechecked
+
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+# ROUTINES
+# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+@typechecked
+async def get_datacenters(tokenvar: str = TOKENVAR) -> List[Dict[str, Any]]:
+ """
+ Queries a list of data centers.
+
+ Args:
+ tokenvar : Name of the environment variable holding the cloud API login token.
+ Returns:
+ Data centers.
+ """
+
+ client = Client(token=os.environ[tokenvar])
+
+ return [
+ _parse_datacenter(datacenter.data_model)
+ for datacenter in DatacentersClient(client).get_all()
+ ]
+
+@typechecked
+async def get_servertypes(datacenter: str = HETZNER_DATACENTER, tokenvar: str = TOKENVAR) -> List[Dict[str, Any]]:
+ """
+ Queries a list of server types plus their specifications and prices.
+
+ Args:
+ datacenter : Name of data center location.
+ tokenvar : Name of the environment variable holding the cloud API login token.
+ Returns:
+ Server types plus their specifications and prices.
+ """
+
+ client = Client(token=os.environ[tokenvar])
+
+ servertypes = ServerTypesClient(client).get_all()
+
+ servertypes = [_parse_model(servertype.data_model) for servertype in servertypes]
+
+ servertypes = [_parse_prices(servertype, datacenter = datacenter) for servertype in servertypes]
+ servertypes = [servertype for servertype in servertypes if servertype is not None]
+
+ servertypes.sort(key = _sort_key)
+
+ return servertypes
+
+@typechecked
+def _parse_datacenter(location: Datacenter) -> Dict[str, Any]:
+ datacenter = {
+ attr: getattr(location, attr)
+ for attr in dir(location)
+ if not attr.startswith('_') and attr not in ('from_dict', 'id', 'id_or_name', 'server_types')
+ }
+ location = _parse_location(datacenter.pop('location').data_model)
+ location['location_description'] = location.pop('description')
+ location['location_name'] = location.pop('name')
+ datacenter.update(location)
+ return datacenter
+
+@typechecked
+def _parse_location(location: Location) -> Dict[str, Any]:
+ return {
+ attr: getattr(location, attr)
+ for attr in dir(location)
+ if not attr.startswith('_') and attr not in ('from_dict', 'id', 'id_or_name')
+ }
+
+@typechecked
+def _parse_model(model: ServerType) -> Dict[str, Any]:
+ model = {
+ attr: getattr(model, attr)
+ for attr in dir(model)
+ if not attr.startswith('_') and attr not in ('from_dict', 'id', 'id_or_name')
+ }
+ if model.get('deprecated', None) is None:
+ model['deprecated'] = False
+ return model
+
+@typechecked
+def _parse_prices(servertype: Dict[str, Any], datacenter: str) -> Optional[Dict[str, Any]]:
+ location, _ = datacenter.split('-')
+ prices = servertype.pop('prices')
+ prices = {price['location']: price for price in prices if price['location'] == location}
+ if location not in prices.keys():
+ return None
+ price = prices[location]
+ price.pop('location')
+ for price_type in ('price_hourly', 'price_monthly'):
+ price.update({f'{price_type:s}_{k:s}': v for k, v in price.pop(price_type).items()})
+ servertype.update(price)
+ return servertype
+
+@typechecked
+def _sort_key(servertype: Dict[str, Any]):
+
+ return servertype['cpu_type'].ljust(100) + f"{servertype['cores']:05d}"