Status of services (pushed through api) show after icinga2 service was reload

[de1m]

Describe the bug

The error happens only on first start of my container. The container start without any errors und icinga2 can receive data through NSCA (from Windows Server) without errors.
I've a cuple of scripts, written in nodejs (icinga2 modul). This scripts create icinga2 objects (hosts and services) and set state of they through icinga2 api.
The error is, that the state of this object will be seeable(in icinga2 webgui) after I run the command "service icinga2 reload"

To Reproduce

1. Start container (show the readme
2. Create an host and server of them through api
3. Show the state in webgui

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): version: r2.10.5-1
• Operating System and version:

System information:
  Platform: Ubuntu
  Platform version: 18.04.2 LTS (Bionic Beaver)
  Kernel: Linux
  Kernel version: 4.15.0-47-generic
  Architecture: x86_64`

• Enabled features (icinga2 feature list):

root@monitoring:/# icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf ido-myql influxdb livestatus mainlog opentsdb perfdata statusdata syslog
Enabled features: api checker command graphite ido-mysql notification

• Icinga Web 2 version and modules (System - About): 2.6.3
• Config validation (icinga2 daemon -C):

root@monitoring:/# icinga2 daemon -C
[2019-06-05 08:14:58 +0000] information/cli: Icinga application loader (version: r2.10.5-1)
[2019-06-05 08:14:58 +0000] information/cli: Loading configuration file(s).
[2019-06-05 08:14:58 +0000] information/ConfigItem: Committing config item(s).
[2019-06-05 08:14:58 +0000] information/ApiListener: My API identity: 30b5d7050d6f
[2019-06-05 08:14:58 +0000] warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere!
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 384 Services.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 34 Hosts.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 3 NotificationCommands.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 418 Notifications.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 3 HostGroups.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 3 Zones.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 4 ApiUsers.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 2 Users.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 215 CheckCommands.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 4 ServiceGroups.
[2019-06-05 08:14:58 +0000] information/ConfigItem: Instantiated 4 TimePeriods.
[2019-06-05 08:14:58 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-06-05 08:14:58 +0000] information/cli: Finished validating the configuration file(s).
* If you run multiple Icinga 2 instances, the `zones.conf` file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.

Additional context

I think the error happens because I create the host, then server and set state of this and all of this very fast (in script).
All logs are empty.

[dnsmichi]

set state of they through icinga2 api.

I assume that a new check result is being pushed via API action. If you're manually modifying the state attribute somehow, this won't trigger any runtime updates towards backends (IDO, etc.).

[de1m]

okay, but after I've reload the icinga2 its working correctly. Can I do somephing, for debug or testing?

[dnsmichi]

So you're modifying the state attribute in your library? This is not supported by design.

[de1m]

What do you mean with "state attribute"? "Custom Variables" in Icinga2 webgui?
or the "vars" in config file?

object Host "k8s-ext-pods-monitoring" {
        import "passive-host"

        display_name = "k8s-ext-pods-monitoring"
        vars["group"] = "adito"
        vars["server"] = "k8s"
        version = 1559720645.693325
        zone = "30b5d7050d6f"
}

[de1m]

hm...
I've changed my Script, that create now this object

object Host "k8s-ext-pods-monitoring" {
        import "passive-host"

        display_name = "k8s-ext-pods-monitoring"
        version = 1560243611.427802
        zone = "30b5d7050d6f"
}

but I get the same error

[dnsmichi]

Hm, I think I misunderstood the meaning of "set a state", which is done via the process-check-result action. The host object and its changed state is visible via REST API, right?

Then I'd point to IDO activation bugs collected in #6012.

[lippserd]

Hi @de1m,

If you still have this problem, please open a new issue. I close this one in the meantime.

For the records, I doubt that it is a good idea to add version and zone to the configuration.
But that is just a blind guess.

All the best,
Eric