Error: Validation failed for object, servicegroup not found which has been created via api #7493
Labels
area/api
REST API
Comments
|
Your checks point to |
dnsmichi
added
area/api
|
It also claims that |
|
Ok, maybe the _api package is broken for storing the runtime objects. |
|
Doesn't look like it, I already checked that since I've encountered this before. But as described: icinga creates the servicegroup in the api package but at the same time tells me it can't find servicegroup tubee-stage. |
|
Wondering how to reproduce this without the overhead of kube-icinga. Can this be reduced into single curl requests somehow? |
|
I was able to reduce it to just three api requests, basically:
SetupIcinga cluster with one default zone and three endpoints, one master, two "slaves". Curl requestsError DescriptionAfter the restart is triggered I get: The two object files exist on all three nodes: And I can reproduce it everytime, If I remove everything from /var/lib/icinga2/api/packages/_api/monitoring003-1466778742-1/conf.d on all three nodes, restart icinga on each and repeat the curl requests it will endup with the error above. Some questions:
full icinga endpoint-1 log (where I executed the requests):Logs from the other two endpointsendpoint-2: endpoint-3: |
|
Ah, so the error originates from a secondary master instance, not the one where you are creating the objects initially via API? Then this would point to #7362. |
The error is logged on the instance where I created the objects. |
But only after you either restart the instance, or the secondary master in the same zone reconnects to this instance. The error originates from the runtime config object sync on connect, as can be seen in the logs. This behaviour of "return to sender" is a different bug, long term and now fixed for 2.11. You may want to test the snapshot packages, or you'll wait for 2.11. |
|
It is not fixed in the RC, we've fixed this afterwards from customer's feedback. AFAIK 19.8.2019 or so. In terms of the linked issue, the node where the IDO feature is active (if enable_ha=true). |
|
Yes just got that :/, with the rc I have other weird errors. I'll install the latest snapshot version. |
|
I still see errors like: And in addition to that with the latest snaphot I have errors (api response) like: {"results":[{"code":500,"errors":["Error: Object 'kubernetes-volumes-tam' of type 'Host' re-defined: in /var/lib/icinga2/api/packages/_api/monitoring003-1466778742-1/conf.d/hosts/kubernetes-volumes-tam.conf: 1:0-1:35; previous definition: in /var/lib/icinga2/api/packages/_api/monitoring003-1466778742-1/conf.d/hosts/kubernetes-volumes-tam.conf: 1:0-1:35\nLocation: in /var/lib/icinga2/api/packages/_api/monitoring003-1466778742-1/conf.d/hosts/kubernetes-volumes-tam.conf: 1:0-1:35"],"status":"Object could not be created."}]}}The file /var/lib/icinga2/api/packages/_api/monitoring003-1466778742-1/conf.d/hosts/kubernetes-volumes-tam.conf doesn't exists, neither on the other two endpoints. Guess I'll rollback to v1.10. The api seems unusable with multiple endpoints in one zone. |
|
Ok, thanks for testing. Then it is still related to the sync order with #7362. I'm not sure when I will find the time to look into these issues, but definitely after 2.11 is released and highly likely after OSMC. Since the linked issue holds some pointers, you might want to look into the source code yourself. |
|
@raffis Please could you try v2.11.3? |
|
IMAO the lack of external feedback for a long time indicates that that feedback will never happen. Therefore closing this one. Feel free to re-open if the problem persists with the latest Icinga 2 version as long as you provide the desired information. |
Al2Klimov
removed
the
needs feedback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I experience various problems using the icinga2 api.
Using kube-icinga to distribute kubernetes objects to a three node icinga cluster.
I experience the following problems randomly:
Checking the icinga2 log, I have many errors like these:
This is only one such error, somehow icinga2 can't find the servicegroup, but the servicegroup does exist, at least this is what the rest endpoint tells me:
Even after restarting the icinga2 service I get those einconsistentrrors. I could cleanup all icinga2 nodes by removing everything created via the api but sooner or later I will get those errors again.
To me it looks like there is a fault in icinga somewhere because it can't find service groups which are clearly there.
What is also strange, If I query the icinga api on different nodes I may get different results but this could also be related to #6957
But overall it is quite inconsistent.
To Reproduce
Well not sure.
Expected behavior
No such error.
Your Environment
Include as many relevant details about the environment you experienced the problem in
icinga2 --version): r2.10.5-1icinga2 feature list):icinga2 daemon -C):zones.conffile (oricinga2 object list --type Endpointandicinga2 object list --type Zone) from all affected nodes.Additional context
Related to github.com/gyselroth/kube-icinga.
The text was updated successfully, but these errors were encountered: