Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ido*sqlConnection#InternalDeactivateObject(): mark object inactive also in memory #8626

Merged
merged 1 commit into from
Jul 2, 2021

Conversation

Al2Klimov
Copy link
Member

@Al2Klimov Al2Klimov commented Feb 8, 2021

Previously:
1. You delete an object from a config file
2. You reload Icinga
3. Icinga fetches all objects and whether they're active from the IDO
4. Icinga recognizes that the just deleted object doesn't exist anymore
5. Icinga marks it as inactive in the IDO, but not in memory
6. You re-create the just deleted object via API
7. Icinga still thinks it's active and doesn't activate it - it's invisible

fixes #8584

@icinga-probot icinga-probot bot added area/api REST API area/db-ido Database output bug Something isn't working ref/NC labels Feb 8, 2021
@icinga-probot icinga-probot bot added this to the 2.13.0 milestone Feb 8, 2021
@Al2Klimov
Copy link
Member Author

Before

Bildschirmfoto 2021-02-08 um 11 24 05
Bildschirmfoto 2021-02-08 um 11 25 24
Bildschirmfoto 2021-02-08 um 11 25 34
Bildschirmfoto 2021-02-08 um 11 26 42
Bildschirmfoto 2021-02-08 um 11 26 58
Bildschirmfoto 2021-02-08 um 11 31 02
Bildschirmfoto 2021-02-08 um 11 31 09

After

Bildschirmfoto 2021-02-08 um 16 38 17
Bildschirmfoto 2021-02-08 um 16 39 10
Bildschirmfoto 2021-02-08 um 16 39 26
Bildschirmfoto 2021-02-08 um 16 39 48
Bildschirmfoto 2021-02-08 um 16 39 55
Bildschirmfoto 2021-02-08 um 16 40 22
Bildschirmfoto 2021-02-08 um 16 40 27

…so in memory

Previously:
1. You delete an object from a config file
2. You reload Icinga
3. Icinga fetches all objects and whether they're active from the IDO
4. Icinga recognizes that the just deleted object doesn't exist anymore
5. Icinga marks it as inactive in the IDO, but not in memory
6. You re-create the just deleted object via API
7. Icinga still thinks it's active and doesn't activate it - it's invisible

refs #8584
@Al2Klimov Al2Klimov force-pushed the bugfix/recreate-object-invisible-ido-8584 branch from 63fa8ce to e0c1340 Compare February 8, 2021 16:09
@julianbrost
Copy link
Contributor

I've found how to reliably trigger this issue using only using the API (tested against 2.12.3). The key is that the host has to exist in the database when Icinga starts (or to be more precise: when Icinga (re)connects to the database).

  1. Create a host: curl -skSu root:icinga -H 'Accept: application/json' -X PUT https://127.0.0.1:5665/v1/objects/hosts/host-8584 -d '{"attrs":{"check_command":"dummy"}}'
  2. Restart Icinga 2: curl -skSu root:icinga -H 'Accept: application/json' -X POST https://localhost:5665/v1/actions/restart-process
  3. Delete the host: curl -skSu root:icinga -H 'Accept: application/json' -X DELETE https://localhost:5665/v1/objects/hosts/host-8584
  4. Recreate the same host: curl -skSu root:icinga -H 'Accept: application/json' -X PUT https://127.0.0.1:5665/v1/objects/hosts/host-8584 -d '{"attrs":{"check_command":"dummy"}}'

Other insights when investigating this: DbConnection::SetObjectActive() is only ever called in Ido{My,Pg}sqlConnection::Reconnect() (before this PR), so DbConnection::GetObjectActive() always reflects the state when establishing the DB connection, not considering any runtime updates.

This only results in a problem when activating objects, as there it is checked whether the object is considered to be active, while when deactivating an objects, this is already ignored:

bool dbActive = GetObjectActive(dbobj);
bool active = object->IsActive();
if (active) {
if (!dbActive)
ActivateObject(dbobj);
Dictionary::Ptr configFields = dbobj->GetConfigFields();
String configHash = dbobj->CalculateConfigHash(configFields);
ASSERT(configHash.GetLength() <= 64);
configFields->Set("config_hash", configHash);
String cachedHash = GetConfigHash(dbobj);
if (cachedHash != configHash) {
dbobj->SendConfigUpdateHeavy(configFields);
dbobj->SendStatusUpdate();
} else {
dbobj->SendConfigUpdateLight();
}
} else if (!active) {
/* This may happen on reload/restart actions too
* and is blocked above already.
*
* Deactivate the deleted object no matter
* which state it had in the database.
*/
DeactivateObject(dbobj);
}

So this PR fixes the issue but I think it can still be improved, either:

  1. fix the in-memory set of active objects to always reflect the state in the database (with the current PR, the set contains all that were active when connecting to the DB minus those that were deactivated after that, even though they could have been reactivated)
  2. get rid of this in-memory set altogether, it is only used to optimize away a single UPDATE icinga_objects SET is_active = 1 WHERE object_id = ... where this optimization isn't even done when deactivating objects (i.e. the same query with SET is_active = 0)

Copy link
Contributor

@julianbrost julianbrost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Al2Klimov
Copy link
Member Author

I'd leave this (change to an old-gen backend) as simple as possible as long as actually fixes the issue and doesn’t break anything.

@julianbrost
Copy link
Contributor

Would adding SetObjectActive(dbobj, true); in the appropriate places be really so much more complex that you want to avoid it? I think this should be fairly easy and then we have a consistent state here instead of some still half-broken set.

(Upon closer consideration, option 2 would probably need a bit more evaluation whether running a noop update query on each object on startup would hurt performance too much.)

Copy link
Contributor

@julianbrost julianbrost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current state was deemed good enough for IDO.

@Al2Klimov Al2Klimov merged commit b7efbd0 into master Jul 2, 2021
@icinga-probot icinga-probot bot deleted the bugfix/recreate-object-invisible-ido-8584 branch July 2, 2021 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api REST API area/db-ido Database output bug Something isn't working ref/NC
Projects
None yet
Development

Successfully merging this pull request may close these issues.

API created host not visible in UI until next reload
2 participants