HiYaPyCo - A Hierarchical Yaml Python Config
A simple python lib allowing hierarchical overlay of config files in YAML syntax, offering different merge methods and variable interpolation based on jinja2.
The goal was to have something similar to puppets hiera deeper merge_behavior
for python.
- hierarchical overlay of multiple YAML files
- multiple merge methods for hierarchical YAML files
- variable interpolation using jinja2
- PyYAML aka. python-yaml
- Jinja2 aka. python-jinja2
- ordereddict for python2.6 (if you like to use the Ordered Dict Yaml Loader / Dumper aka. ODYLDo)
HiYaPyCo was designed to run on both current major python versions without changes. Tested versions:
- 2.6
- 2.7
- 3.2
- 3.4
A simple example:
import hiyapyco
conf = hiyapyco.load('yamlfile1' [,'yamlfile2' [,'yamlfile3' [...]]] [,kwargs])
print(hiyapyco.dump(conf))
yaml1.yaml
:
---
first: first element
second: xxx
deep:
k1:
- 1
- 2
yaml2.yaml
:
---
second: again {{ first }}
deep:
k1:
- 4
- 6
k2:
- 3
- 6
load ...
>>> import pprint
>>> import hiyapyco
>>> conf = hiyapyco.load('yaml1.yaml', 'yaml2.yaml', method=hiyapyco.METHOD_MERGE, interpolate=True, failonmissingfiles=True)
>>> pprint.PrettyPrinter(indent=4).pprint(conf)
{ 'deep': { 'k1': [1, 2, 4, 6], 'k2': [3, 6]},
'first': u'first element',
'ma': { 'ones': u'12', 'sum': u'22'},
'second': u'again first element'}
>>> import hiyapyco
>>> y1="""
... yaml: 1
... y:
... y1: abc
... y2: xyz
... """
>>> y2="""
... yaml: 2
... y:
... y2: def
... y3: XYZ
... """
>>> conf = hiyapyco.load([y1, y2], method=hiyapyco.METHOD_MERGE)
>>> print (conf)
OrderedDict([('yaml', 2), ('y', OrderedDict([('y1', 'abc'), ('y2', 'def'), ('y3', 'XYZ')]))])
>>> hiyapyco.dump(conf, default_flow_style=True)
'{yaml: 2, y: {y1: abc, y2: def, y3: XYZ}}\n'
All args
are handled as file names or yaml documents. They may be strings or list of strings.
-
method
: bit (one of the listed below):hiyapyco.METHOD_SIMPLE
: replace values (except for lists a simple merge is performed) (default method)hiyapyco.METHOD_MERGE
: perform a deep merge
-
interpolate
: boolean : perform interpolation after the merge (default: False) -
castinterpolated
: boolean : try to perform a best possible match cast for interpolated strings -
usedefaultyamlloader
: boolean : force the usage of the default PyYAML loader/dumper instead of HiYaPyCos implementation of a OrderedDict loader/dumper (see: Ordered Dict Yaml Loader / Dumper aka. ODYLDo) (default: False) -
failonmissingfiles
: boolean : fail if a supplied YAML file can not be found (default: True) -
loglevel
: int : loglevel for the hiyapyco logger; should be one of the valid levels fromlogging
: 'WARN', 'ERROR', 'DEBUG', 'I NFO', 'WARNING', 'CRITICAL', 'NOTSET' (default: default oflogging
) -
loglevelmissingfiles
: int : one of the valid levels fromlogging
: 'WARN', 'ERROR', 'DEBUG', 'INFO', 'WARNING', 'CRITICAL', 'NOTSET' (default:logging.ERROR
iffailonmissingfiles = True
, elselogging.WARN
)
For using interpolation, I strongly recomend not to use the default PyYAML loader, as it sorts the dict entrys alphabetically, a fact that may break interpolation in some cases (see test/odict.yaml
and test/test_odict.py
for an example).
See Ordered Dict Yaml Loader / Dumper aka. ODYLDo
The default jinja2.Environment for the interpolation is
hiyapyco.jinja2env = Environment(undefined=Undefined)
This means that undefined vars will be ignored and replaced with a empty string.
If you like to change the jinja2 Environment used for the interpolation, set hiyapyco.jinja2env
before calling hiyapyco.load
!
If you like to keep the undefined var as string but raise no error, use
from jinja2 import Environment, Undefined, DebugUndefined, StrictUndefined
hiyapyco.jinja2env = Environment(undefined=DebugUndefined)
If you like to raise a error on undefined vars, use
from jinja2 import Environment, Undefined, DebugUndefined, StrictUndefined
hiyapyco.jinja2env = Environment(undefined=StrictUndefined)
This will raise a hiyapyco.HiYaPyCoImplementationException
wrapped arround the jinja2.UndefinedError
pointing at the string causing the error.
See: jinja2.Environment
As you must use interpolation as strings (PyYAML will weep if you try to start a value with {{
),
you can set castinterpolated
to True in order to try to get a best match
cast for the
interpolated values.
The best match
cast is currently only a q&d implementation and may not give you the expected results!
This is a simple implementation of a PyYAML loader / dumper using OrderedDict
from collections.
Because chaos is fun but order matters on loading dicts from a yaml file.
In order to use this on python 2.6, please install ordereddict:
sudo pip-2.6 install ordereddict
https://github.com/zerwes/hiyapyco
git clone https://github.com/zerwes/hiyapyco
cd hiyapyco
sudo python setup.py install
Download the latest or desired version of the source package from https://pypi.python.org/pypi/HiYaPyCo. Unpack the archive and install by executing:
sudo python setup.py install
Install the latest wheel package using:
pip install HiYaPyCo
echo "deb https://repo.zero-sys.net/hiyapyco/deb ./" > /etc/apt/sources.list.d/hiyapyco.list
gpg --keyserver keys.gnupg.net --recv-key ED7D414C
# or use:
wget https://jwhoisserver.net/key.asc -O - | gpg --import -
gpg --armor --export ED7D414C | apt-key add -
apt-get update
apt-get install python3-hiyapyco python-hiyapyco
use https://repo.zero-sys.net/hiyapyco/rpm as URL for the yum repo and https://jwhoisserver.net/key.asc as the URL for the key.
An AUR package is available.
(c) 2014 - 2016 Klaus Zerwes zero-sys.net
This package is free software.
This software is licensed under the terms of the
GNU GENERAL PUBLIC LICENSE version 3 or later,
as published by the Free Software Foundation.
See https://www.gnu.org/licenses/gpl.html
MERGED: fixes from mmariani
FIXED: issues #9 and #11
deb packages:
- removed support for python 2.6
- include examples as doc
FIXED: issue #6 import of hiyapyco version in setup.py causes pip install failures
Changed: moved to GPL
Improvements: missing files handling, doc
Implemented: castinterpolated
Implemented: loading yaml docs from string
Improved tests and bool args checks
Implemented a Ordered Dict Yaml Loader
Fixed unicode handling
Initial release