Redesign proposition for the image mappings module, based on #207 #237
Replies: 3 comments 1 reply
-
Hi @ivan-zdravkov . First of all thanks for the amazing amount of work and detail for this proposal and the analysis you did! This is really some great stuff!!! I have a few questions, which we can take on the next meeting and proceed with finalising this design:
|
Beta Was this translation helpful? Give feedback.
-
After a discussion the following was decided:
Take a look at this example: images:
ignoredRegions:
- "us-west-1"
regions:
- "eu-east-1"
- "eu-east-2"
- "eu-east-3"
default: "eu-east-2" |
Beta Was this translation helpful? Give feedback.
-
Writing as a comment to respect the history of the proposition and comments. As @Michaelpalacce mentioned, we had a discussion and decided to keep using the JSON structure
single-redesign.json
dual-redesign.json
triple-redesign.json
In all of the cases, the The ExamplesPushing the
|
Beta Was this translation helpful? Give feedback.
-
Table of Contents
Bug Report #207
Background
Context
Build tools for VMware Aria has a functionality to pull and push image mappings and regions.
Regions
are the unique vCenterdata centers
, imported for the linked Ariacloud accounts
following the{cloudAccountName} / {dataCenterName}
naming convention.Images
are the available VM templates.Image Mappings
are named M:M relations between existingimages
andregions
Image mappings can be defined in Aria Automation.
![Image Mappings](https://private-user-images.githubusercontent.com/73837405/305884387-e7df1515-0fd6-42cd-b878-73efd236d24c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyODI0MzIsIm5iZiI6MTcyNDI4MjEzMiwicGF0aCI6Ii83MzgzNzQwNS8zMDU4ODQzODctZTdkZjE1MTUtMGZkNi00MmNkLWI4NzgtNzNlZmQyMzZkMjRjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIxVDIzMTUzMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEwOTRmNDhjZDY5ZWE5OGM4MzQ3YzFkMTQ4ZTdiMWU1ZWJmMzM2N2I4N2MzMDlkYWFhNGVmMzhjZmZkMTEyYTQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.VeOX9AstZuquwi36kavJ1eww7C-7NG71GkmJSAmD_1k)
And every image mapping can map to a specific image in each of its respective regions.
![Triple Image Mapping](https://private-user-images.githubusercontent.com/73837405/305884417-c69301fe-228f-45b1-9506-7427bb5a308c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyODI0MzIsIm5iZiI6MTcyNDI4MjEzMiwicGF0aCI6Ii83MzgzNzQwNS8zMDU4ODQ0MTctYzY5MzAxZmUtMjI4Zi00NWIxLTk1MDYtNzQyN2JiNWEzMDhjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIxVDIzMTUzMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJmMjYxZDAxMTRkMmU5NTIxNWFlNzAyNmI5ODM1M2QzYTM3ZjU0ZTlmNGUwZWM1ZGZmOWI3NjEyYWFjMjEzZjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Vp7hD3Wu1jmOyDQ5_PNvnYR9Th6GDZ2azgJnQGe21N0)
To do a pull, we have to specify the
image-mapping
names in Orchestrator'scontent.yaml
as well as theregion-mapping
tags.Pulling
Pulling generates a correct representation of the image mappings, grouped by the respective regions.
![pull-result](https://private-user-images.githubusercontent.com/73837405/305886313-f906f56f-c76e-49c4-805c-4e4552d026b6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyODI0MzIsIm5iZiI6MTcyNDI4MjEzMiwicGF0aCI6Ii83MzgzNzQwNS8zMDU4ODYzMTMtZjkwNmY1NmYtYzc2ZS00OWM0LTgwNWMtNGU0NTUyZDAyNmI2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIxVDIzMTUzMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTUzYzEzMDExNDBhZmVhMmFhNTI0Njc4MGUyMWE4ZDczZjg4OGQ4NjgyMTZjZTkwYzYwNWUzNTAzMjJmMDZmZGUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.PoIP3ocY9tVGoAdDqBLFz5tcfhIhw3Y32oFZ8YJsuoA)
Each regions's
src-region-profile.json
uniquely maps the region by the target'scloudAccountId
andregionId
itself.Wherever an image mapping has multiple regions, duplicate records for the image mapping are created in the respective region's directory, containing the specific region's image
externalId
.Pushing
Originally, and in line with the project's other resource elements, pushing is expected to populate Aria with the resources as described in code.
Issue
When importing the image mappings, it seems like the last one to be processed, would overwrite the data for the previous regions. Meaning that if you have image mappings that point to:
Test-1
,Test-2
,Test-3
images, then after import the one that was processed last will be the one that is in use everywhere.In the tested environment we had 3 data centers and one of the data center had 2 regions, if that matters
Steps to reproduce
Expected behavior
The region mappings to be mapped correctly per region.
Actual behavior
Last one overwrites all the rest.
Reproduces how often
Component:
Environment
Client
Server
Additional Context
This is when pulling and pushing to the SAME environment.
Reproduction
Steps
Single Region
, a mapping with a single region.Dual Region
, a mapping with two regions.Triple Region
, a mapping with three regions.Results
Executing the reproduction steps produces an incorrect environment state, just as described in the Issue section.
Execution changes all of the environment image mappings to have 3 regions and all of the regions to have the same image, regardless of what was previously there.
There isn't any apparent indication as to which of the code
externalId
of the original images gets populated after the push. It does not appear to be the file system's latest one, as indicated in the issue.Root Cause Analysis
As originally performed by Eman Elsayed and later confirmed by Ivan Zdravkov and Sugeesh Chandraweera, the cause of the bug is a missing check in the importContent method of the VraNgImageMappingStore.java file.
As a result the regions are not filtered properly, and every image mapping gets inserted every region resulting in the latest execution to override every change previously.
Proposed Solution
A simple nearly identical solution was proposed by each of the investigating members which adds a check to ensure each region correctly processes only their own records.
Solution Results
As a result of the implemented solution, pushing the regions now correctly populates them in Aria, resulting in the original state of the environment.
Solution Denial
Why
In a high level discussion with Stefan Genov, the proposed solution was denied.
It turns out that the alleged original intention of the module is very environmentally dependent. All of the entities in question (Cloud Account, Region, Image, etc.) are kept in code by their ids and not by their name, tag or another environmentally agnostic handler.
This means that pulling from a dev environment and trying to push into a client environment will never work, since no matching of ids will ever occur.
But it's been working fine... ish
What
accidentally
had been happening all along is that:develop
with asingle
dev region.client
usually hasmultiple
regions.all
of the regions on the client environment.Partial Conclusion
Fixing the issue as proposed will
restore
the original design, but will actuallyworsen
the dev and client experience.We discussed a number of ways of how to address this, so that it would both work as intended and continue doing meaningful work, and decided against implementing a
patch
on top of abug
on top of theoriginal
design,while simultaneously trying to both fix it and keep the accidental behavior of
Since the issue of how to map
dev
andclient
entity ids is both hard and requiring a lot of manual work post-implementation, we are currently looking into a solution redesign, which will try to solve the problems ofRedesign Proposition
The current solution relies on entity ids to define image mappings and regions, which makes it difficult to achieve environmental independence.
Having a full dev-client environment mapping as discussed earlier is both difficult and probably needless.
The following redesign proposition aims to
fully rework
the image mapping capabilities of the project in order to:JSON structure
Setup
Dev Environment Regions
vCenter / dc-01a
region.vCenter / dc-02a
region.vCenter / dc-03a
region.Client Environment Regions
vCenter / dc-01a
region.vCenter / dc-02a
region.vCenter / dc-03a
region.vCenter / dc-04a
region.vCenter / dc-05a
region.Dev Environment Image Mappings
where the
Cloud Configuration
value ofvCenter / dc-02a
ispassword:((sensitive:123456))
where the
Cloud Configuration
value ofvCenter / dc-02a
ispassword:((sensitive:123456))
vCenter / dc-03a
ispassword:((secret:v1:gdq7gdgeggfqu882yhf==))
Pulling
Pulling from an environment should generate a single
{imageMappingName}.json
file per a image mapping in theimage-mappings
directory.By default pulling from an environment pulls
all
image mappings specified in thecontent.yaml
file, with their first region set asdefault
, and all additional regions (if any), pulled into theregions
array.ignoredRegions
is never populated on a pull and is used to indicate which client regions won't be populated during a push.single-redesign.json
The
single-redesign.json
would be the most common scenario we are developing for. This is the scenario, where we have an image mapping with a single region, which is set asdefault
when pulling, expected to later be pushed to every client environment region.dual-redesign.json
triple-redesign.json
Pushing
As shown above, the
default
property is region agnostic and contains the properties that will be populated forall
client regions, unless specified otherwise in theregions
orignoredRegions
arrays.Algorithm
Same Environment
Pushing on the same or identical environment is expected to recreate exactly the described json.
Different Environment
Pushing on a different environment is expected to:
ignoredRegions
string array if any.regions
array if any.default
JSON properties to every other environment region.Examples
Default Scenario
Pushing as-is is the most common scenario where we are pulling from a single dev region and pushing to multiple client regions.
In this case, since the client has 3 entirely different regions
vCenter / dc-03a
vCenter / dc-04a
vCenter / dc-05a
than the original dev environment's region
vCenter / dc-01a
and since we don't have additional
regions
definitions orignoredRegions
the expectation is that thedefault
image mapping will be applied to all client regions.Ignoring a Region
Ignoring the
vCenter / dc-05a
client region in theignoredRedions
array will as expected not populate it on a push.Customizing a Region
Having definitions in the
regions
array will override thedefault
values for the listed regions, while all other regions will get thedefault
data.Backwards Compatibility
Functionality
Functionally wise, the proposed solution will behave similarly if not identically to what currently exists for the most common scenario of pulling from a single region and pushing to all client regions.
For multi-region scenarios, the proposed solution will work as described above, solving the original bug that triggered the whole redesign.
Most importantly pulling and pushing as-is will work mostly the same for both the same and different environments.
Structure
Structure wise, the proposed solution uses an entirely different file and JSON structure. As such the newly proposed solution is not structurally backwards compatible.
Release Planning
A release planning could help solve the backwards compatibility issues. We could implement both the original and proposed solutions and still have the original solution as default, until the proper time and version where we can switch the default version to the newly proposed one, while still keeping the availability of users to opt-in and use the legacy version.
Further down the line in the next major release, we might decommission the legacy version entirely.
Risks and Mitigation Strategies
The major risks identified with the proposed solution, besides the usual bugs and regressions, all revolve around actually implementing a solution to solve the correct problem.
One of the main purposes of this document is to clearly outline the original design and problem as well as the new solution proposition to make it easier for stakeholders to identify discrepancies or logic lapses.
Some of the mitigation strategies we can use to reduce the technical and business risks include:
Impact Analysis
Functional
Initially the impact of the new solution in terms of functionality does not seem huge, since the described bug is a bit of niche scenario. Additionally most user's day to day usage won't change much, besides some slimmer syntax.
Besides the obvious, implementing the solution will both simplify the JSON structure and bring it closer to the actual aria design.
Introducing environmental agnosticism, which coupled with the proposed ignore and default override functionality brings more flexibility for edge case development and a more wide range of capabilities for our deliverables.
Non-Functional
Major refactoring proposition process
to simplify future support and improve the knowledge base.Next Steps
Beta Was this translation helpful? Give feedback.
All reactions