Incorrect order in php-fpm.conf

[TaiSHiNet]

When doing a state.highstate with the following pillar:

php:
  ng:
    fpm:                                                                                                                                                                                                                                                          
      service:                                                                                                                                                                                                                                                    
        enabled: True                                                                                                                                                                                                                                             
        opts:                                                                                                                                                                                                                                                     
          reload: True                                                                                                                                                                                                                                            
      config:                                                                                                                                                                                                                                                     
        ini:                                                                                                                                                                                                                                                      
          opts:
            recurse: True
          settings:
            PHP:
              engine: 'Off'

        conf:
          opts:
            recurse: True
          settings:
            global:
              pid: /var/run/php5-fpm.pid
              error_log: /var/log/php5-fpm.log
              include: /etc/php5/fpm/pool.d/*.conf

Settings will be outputted like this:

    error_log = /var/log/php5-fpm.log
    include = /etc/php5/fpm/pool.d
    pid = /var/run/php5-fpm.pid 

And will break since include -should- be at the end of the config file

[TaiSHiNet]

Tried removing all settings: and worked like a dream.
Now, if I add at least one option (settings: global: daemonize: yes) it will -only- output the "daemonize: True" into php-fpm.conf

[TaiSHiNet]

I was told to tag @cheuschober

[cheuschober]

Hi @TaiSHiNet Thank you for CC'ing me on this. I'm the mad scientist responsible for the ng state above and the pillar merge parameter that enables this pattern. Your bug report is appreciated. I'll try to give this a look on Monday. In the meantime, some answers:

Your issue with settings:global:daemonize:yes is due to the fact that the way the defaults and pillar data are currently merged is through python's dict.update() which does not perform a fully recursive merge. I could build such a function but it does get a little weird since RedHat family and Debian family default fpm configs are different so you, as an end-user, really aren't merging your pillar data into php:ng:fpm:conf:settings. It's an interesting issue. I've got an idea how it can be solved but it's going to make the map.jinja even more impenetrable than it already is. I'll need a little time to think on that. A current workaround would be merging into the defaults as found in php:ng:lookup:fpm:defaults

The issue of your parameter order is far more perplexing to me. Python dicts are unordered but (I think) the salt yaml parser for pillars and related should be throwing that data into OrderedDict objects. Pillar merging should also respect that. The default is an odict object and I see that in the map.jinja, I did not make the settings merge targets odicts as I did in the defaults. It might be as simple as changing those declarations to empty odicts

[TaiSHiNet]

Thanks for your reply.
I don't fully understand everything you mention, but I'd like to add that php.ini is working as expected (currently using the default Engine: off).
I agree that this ng is rather complex to understand :P

Once again, thanks for replying and taking a look at this

[cheuschober]

@TaiSHiNet, I've updated a branch with some fixes that might address your issues. Please test it and let me know if it works for you. If it does, I'll merge it to the main repo:

https://github.com/spsoit/php-formula/tree/fixes_unordered_fpm_params

[TaiSHiNet]

https://puu.sh/a75cc/dfc05982f4.png

It's still happening

----------
          ID: php_fpm_conf_config
    Function: file.managed
        Name: /etc/php5/fpm/php-fpm.conf
      Result: True
     Comment: File /etc/php5/fpm/php-fpm.conf updated
     Started: 16:21:34.128810
     Duration: 37 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -6,6 +6,6 @@


                   [global]
                  -    pid = /var/run/php5-fpm.pid
                       error_log = /var/log/php5-fpm.log
                       include = /etc/php5/fpm/pool.d/*.conf
                  +    pid = /var/run/php5-fpm.pid

[cheuschober]

Without stating the obvious, just to confirm, you're using the branch I specified, not master, right? If you point to it in the master config it's likely using the master branch. You'd have to pull it my branch specifically and not use the git backend (use it like a file_roots).

Also, just for the sake of argument, could you post the output of:

salt $yourhost pillar.get php:ng:fpm:config:conf:settings

I want to confirm that the data that's making it into the map.jinja is ordered correctly.

[TaiSHiNet]

I'm an idiot. I was using master.
I just tried and received the following error:

front02-dalepumas:
    Data failed to compile:
----------
    Rendering SLS 'base:php.ng.curl' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/var/cache/salt/minion/files/base/php/ng/installed.jinja", line 2, in top-level template code
    {% from "php/ng/map.jinja" import php with context %}
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'


----------
    Rendering SLS 'base:php.ng.cli.install' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/var/cache/salt/minion/files/base/php/ng/installed.jinja", line 2, in top-level template code
    {% from "php/ng/map.jinja" import php with context %}
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'


----------
    Rendering SLS 'base:php.ng.cli.ini' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'


----------
    Rendering SLS 'base:php.ng.json' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/var/cache/salt/minion/files/base/php/ng/installed.jinja", line 2, in top-level template code
    {% from "php/ng/map.jinja" import php with context %}
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'


----------
    Rendering SLS 'base:php.ng.mysql' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/var/cache/salt/minion/files/base/php/ng/installed.jinja", line 2, in top-level template code
    {% from "php/ng/map.jinja" import php with context %}
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'


----------
    Rendering SLS 'base:php.ng.fpm.config' failed: Jinja syntax error: Encountered unknown tag 'conf_settings'. Jinja was looking for the following tags: 'elif' or 'else' or 'endif'. The innermost block that needs to be closed is 'if'.; line 9

---
[...]

{% set ini_settings = php.ini.defaults %}
{% do ini_settings.update(php.fpm.config.ini.settings) %}

{%- if php.fpm.config.conf.settings.length() > 0 -%}
  {%- conf_settings = php.fpm.config.conf.settings -%}    <======================
{%- else -%}
  {%- conf_settings = php.lookup.fpm.defaults -%}
{%- endif -%}

php_fpm_ini_config:
[...]
---
----------
    Rendering SLS 'base:php.ng.fpm.service' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'


----------
    Rendering SLS 'base:php.ng.fpm.pools' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'


----------
    Rendering SLS 'base:php.ng.gd' failed: Jinja error: unhashable type: 'dict'
/var/cache/salt/minion/files/base/php/ng/map.jinja(1):
---
{% set php = salt['pillar.get']('php:ng', {    <======================
    'lookup': salt['grains.filter_by']({
        'Debian': {
            'pkgs': {
                'php': 'php5',
                'apc': 'php-apc',
[...]
---
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 275, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 2, in top-level template code
  File "/var/cache/salt/minion/files/base/php/ng/installed.jinja", line 2, in top-level template code
    {% from "php/ng/map.jinja" import php with context %}
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/var/cache/salt/minion/files/base/php/ng/map.jinja", line 1, in top-level template code
    {% set php = salt['pillar.get']('php:ng', {
  File "/usr/lib/python2.7/dist-packages/salt/modules/grains.py", line 424, in filter_by
    default, None)
TypeError: unhashable type: 'dict'

And your command returned blank result, guessing because it broke before.
Feel free to ping me on IRC @ #salt (I'm TaiSHi)

[ckng]

Running into the same issue, and same result using fixes_unordered_fpm_params branch. Any update?

[nmadhok]

I will look into refactoring some of the code once I get some time for this. Thank for filing the bug!

[ckng]

It is somehow related for php.ng. On current master branch, if any section is overwritten in pillar, the default from the section are no longer output to conf or ini.

[TaiSHiNet]

@cheuschober Any news on this?