Decouple configs from cluster #53

varl · 2019-05-23T15:37:03Z

Creates a new Cache for d2-cluster based on the global cache.

Would be nice to pair with https://github.com/dhis2/cli-helpers-engine/pull/15/files to remove the layered cache directory.

amcgee

This is cool! I'm not sure if we want to go all the way to implementing a generic key-value store, but if we don't (yet) this is a great solution.

I don't think the multi-tiered Caches are required, which would make this even simpler!

packages/cluster/src/common.js

varl · 2019-05-27T17:57:31Z

Rebased on master, variation on the multi-layered Cache.

Produces this structure:

tree ~/.cache/d2/ -L 4
/home/varl/.cache/d2/
└── cache
    ├── cluster
    │   ├── dev
    │   │   ├── config.json
    │   │   └── docker-compose
    │   └── master
    │       ├── config.json
    │       └── docker-compose
    ├── cluster-db-dev.sql.gz
    └── cluster-db-master.sql.gz

edit: updated with simpler paths.

Depends on dhis2/cli-helpers-engine#16

varl · 2019-05-27T18:49:01Z

varl@veronika:~/dev/dhis2/libs/cli$ ./packages/main/bin/d2 debug cache list
┌──────────────────────────┬──────────┬─────────────────────┐
│ Name                     │ Size     │ Modified            │
├──────────────────────────┼──────────┼─────────────────────┤
│ cluster                  │ 512      │ 2019-05-27 17:50:36 │
├──────────────────────────┼──────────┼─────────────────────┤
│ cluster-db-dev.sql.gz    │ 98904634 │ 2019-05-27 17:50:42 │
├──────────────────────────┼──────────┼─────────────────────┤
│ cluster-db-master.sql.gz │ 165532   │ 2019-05-27 17:50:20 │
└──────────────────────────┴──────────┴─────────────────────┘

varl@veronika:~/dev/dhis2/libs/cli$ ./packages/main/bin/d2 debug cache list cluster
┌───────────────────┬──────┬─────────────────────┐
│ Name              │ Size │ Modified            │
├───────────────────┼──────┼─────────────────────┤
│ dev               │ 512  │ 2019-05-27 17:50:37 │
├───────────────────┼──────┼─────────────────────┤
│ master            │ 512  │ 2019-05-27 17:44:02 │
└───────────────────┴──────┴─────────────────────┘

varl@veronika:~/dev/dhis2/libs/cli$ ./packages/main/bin/d2 debug cache list cluster/dev/docker-compose
┌────────────────────┬──────┬─────────────────────┐
│ Name               │ Size │ Modified            │
├────────────────────┼──────┼─────────────────────┤
│ config             │ 512  │ 2019-05-27 19:01:22 │
├────────────────────┼──────┼─────────────────────┤
│ docker-compose.yml │ 918  │ 2019-05-23 09:53:06 │
├────────────────────┼──────┼─────────────────────┤
│ README.md          │ 3641 │ 2019-05-23 09:53:06 │
├────────────────────┼──────┼─────────────────────┤
│ scripts            │ 512  │ 2019-05-27 19:01:22 │
└────────────────────┴──────┴─────────────────────┘

I think I'll rename the d2-cluster-dev to just dev since it's already subdired.

edit: updated.

packages/cluster/src/commands/down.js

packages/cluster/src/common.js

varl · 2019-05-30T10:15:13Z

Updated dhis2/cli-helpers-engine#16 with a read/write implementation for the cache, and could simplify this PR as a result.

The cached configuration reading/writing is handled after the runtime configuration has been resolved. Only a subset of the configuration object is cached for subsequent runs.

varl · 2019-06-05T14:05:05Z

@amcgee I think this is ready for review and testing without using links.

amcgee · 2019-06-12T14:13:43Z

@varl looks like there's been a bunch of activity since you said this was ready for review, are you still iterating on it?

varl · 2019-06-12T14:19:07Z

@amcgee No, I just rebased on master to bring it up to date. Looks like there's another conflict so I'll do it once more. 😇

varl · 2019-06-17T06:36:35Z

Rebased on master so incorporates #61.

varl · 2019-06-17T07:16:05Z

@amcgee This is ready for review.

amcgee

Looking good! Made a few comments

packages/cluster/src/commands/up.js

packages/cluster/src/common.js

packages/cluster/src/commands/status.js

varl · 2019-06-18T10:52:43Z

@amcgee changed the code with regards to your comments.

amcgee

@varl made some few minor comments, will test and approve

packages/cluster/src/commands/seed.js

packages/cluster/src/db.js

amcgee · 2019-06-26T09:40:01Z

packages/cluster/src/commands/logs.js

 const cacheLocation = await initDockerComposeCache({
+ composeProjectName: name,


Should this also be makeComposeProject(name) or the parameter just called name? Otherwise maybe initDockerComposeCache::composeProjectName is a bit misleading...

Yeah, I've considered that too.

The main reason I want name here is to make the cache a lot easier to browse: d2 debug cache list clusters/dev/docker-compose

If we use makeComposeProject: d2 debug cache list clusters/d2-cluster-dev/docker-compose

Since all the compose projects are already namespaced to clusters/{name} I don't think it makes sense to prefix it again with d2-cluster-{name}.

It's still the cache directory for the compose project, so the function name names sense (initDockerComposeCache), it's just the argument that is slightly misleading. I can rename it to composeCacheName if that makes it clearer.

@amcgee Alternatively initDockerComposeCache::cacheDirName?

We can rename this later but I think name might even be fine

amcgee

Testing and looks good but found one undefined variable!

packages/cluster/src/common.js

amcgee

Nice work @varl!! Tested and approved, sorry it took so much back-and-forth

varl · 2019-07-01T12:34:26Z

No worries, the PR is not that big but the implications of it are huge so good to be thorough!

# [1.3.0](v1.2.4...v1.3.0) (2019-07-01) ### Features * decouple configs from cluster ([#53](#53)) ([e5b40af](e5b40af))

dhis2-bot · 2019-07-01T12:37:44Z

🎉 This PR is included in version 1.3.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

varl mentioned this pull request May 25, 2019

feat: support official docker images #54

Merged

amcgee reviewed May 27, 2019

View reviewed changes

packages/cluster/src/common.js Outdated Show resolved Hide resolved

amcgee reviewed May 27, 2019

View reviewed changes

packages/cluster/src/common.js Outdated Show resolved Hide resolved

varl force-pushed the decouple-configs-from-cluster branch from e565467 to 91337d9 Compare May 27, 2019 17:55

varl mentioned this pull request May 27, 2019

feat: expose read and write commands for files in cache dhis2/cli-helpers-engine#16

Merged

varl marked this pull request as ready for review May 30, 2019 09:39

varl commented May 30, 2019

View reviewed changes

packages/cluster/src/commands/down.js Outdated Show resolved Hide resolved

varl commented May 30, 2019

View reviewed changes

packages/cluster/src/common.js Outdated Show resolved Hide resolved

varl commented May 30, 2019

View reviewed changes

packages/cluster/src/common.js Outdated Show resolved Hide resolved

varl commented May 30, 2019

View reviewed changes

packages/cluster/src/common.js Show resolved Hide resolved

varl force-pushed the decouple-configs-from-cluster branch 2 times, most recently from 046590e to 5e42b86 Compare June 6, 2019 13:02

varl force-pushed the decouple-configs-from-cluster branch from 601fb46 to 83d6504 Compare June 12, 2019 14:21

varl mentioned this pull request Jun 12, 2019

fix: use dhis2/docker-compose instead of amcgee/dhis2-backend as default #61

Merged

varl and others added 10 commits June 17, 2019 00:11

fix: each cluster has its own compose cache

e6067d2

fix: use multipe cluster caches

cdec88f

fix: store a cache obj

de99312

fix: use the dirname of the cache loc

ddbe035

fix: log cache obj

4544cb7

refactor: extra constant for cache config

0f9cc6a

fix: shorten the composeProject name after isolation

baef8a8

fix: simplify command cache handling

a0e0699

test: fix tests

0387b57

docs: fix desc for changed arg

a22dd5b

varl and others added 7 commits June 17, 2019 00:38

docs: update readme for cluster

e736081

docs: update readme

a55bbb4

docs: fix broken syntax

4935876

refactor: break up path and clean up imports

2d6209c

refactor: clean up imports

43cbcef

docs: add note about config order

18c1f6c

chore: add repo config files

6712b6e

varl force-pushed the decouple-configs-from-cluster branch from 3f849e8 to 2d6209c Compare June 17, 2019 06:35

amcgee requested changes Jun 18, 2019

View reviewed changes

packages/cluster/src/commands/up.js Outdated Show resolved Hide resolved

packages/cluster/src/common.js Outdated Show resolved Hide resolved

packages/cluster/src/common.js Outdated Show resolved Hide resolved

packages/cluster/src/commands/status.js Outdated Show resolved Hide resolved

varl added 2 commits June 18, 2019 12:33

refactor: address pr comments

4f7d7d2

refactor: prefix docker-compose projects

88e7702

Merge branch 'master' into decouple-configs-from-cluster

daa147b

amcgee reviewed Jun 26, 2019

View reviewed changes

varl and others added 5 commits June 26, 2019 13:11

refactor: rearrange noop change

eaccd9c

Merge branch 'master' into decouple-configs-from-cluster

b4d4362

Merge branch 'master' into decouple-configs-from-cluster

06be935

Merge branch 'master' into decouple-configs-from-cluster

53fca03

Merge branch 'master' into decouple-configs-from-cluster

82a19d0

amcgee requested changes Jul 1, 2019

View reviewed changes

packages/cluster/src/common.js Outdated Show resolved Hide resolved

chore: correct log variable

2a4ac2b

amcgee approved these changes Jul 1, 2019

View reviewed changes

varl merged commit e5b40af into master Jul 1, 2019

varl deleted the decouple-configs-from-cluster branch July 1, 2019 12:34

dhis2-bot pushed a commit that referenced this pull request Jul 1, 2019

chore(release): cut 1.3.0 [skip ci]

3b0eec5

# [1.3.0](v1.2.4...v1.3.0) (2019-07-01) ### Features * decouple configs from cluster ([#53](#53)) ([e5b40af](e5b40af))

dhis2-bot added the released label Jul 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decouple configs from cluster #53

Decouple configs from cluster #53

varl commented May 23, 2019 •

edited

Loading

amcgee left a comment

varl commented May 27, 2019 •

edited

Loading

varl commented May 27, 2019 •

edited

Loading

varl commented May 30, 2019 •

edited

Loading

varl commented Jun 5, 2019

amcgee commented Jun 12, 2019

varl commented Jun 12, 2019

varl commented Jun 17, 2019

varl commented Jun 17, 2019

amcgee left a comment

varl commented Jun 18, 2019

amcgee left a comment

amcgee Jun 26, 2019

varl Jun 26, 2019

varl Jun 26, 2019

amcgee Jul 1, 2019

amcgee left a comment

amcgee left a comment

varl commented Jul 1, 2019

dhis2-bot commented Jul 1, 2019

		const cacheLocation = await initDockerComposeCache({
		composeProjectName: name,

Decouple configs from cluster #53

Decouple configs from cluster #53

Conversation

varl commented May 23, 2019 • edited Loading

amcgee left a comment

Choose a reason for hiding this comment

varl commented May 27, 2019 • edited Loading

varl commented May 27, 2019 • edited Loading

varl commented May 30, 2019 • edited Loading

varl commented Jun 5, 2019

amcgee commented Jun 12, 2019

varl commented Jun 12, 2019

varl commented Jun 17, 2019

varl commented Jun 17, 2019

amcgee left a comment

Choose a reason for hiding this comment

varl commented Jun 18, 2019

amcgee left a comment

Choose a reason for hiding this comment

amcgee Jun 26, 2019

Choose a reason for hiding this comment

varl Jun 26, 2019

Choose a reason for hiding this comment

varl Jun 26, 2019

Choose a reason for hiding this comment

amcgee Jul 1, 2019

Choose a reason for hiding this comment

amcgee left a comment

Choose a reason for hiding this comment

amcgee left a comment

Choose a reason for hiding this comment

varl commented Jul 1, 2019

dhis2-bot commented Jul 1, 2019

varl commented May 23, 2019 •

edited

Loading

varl commented May 27, 2019 •

edited

Loading

varl commented May 27, 2019 •

edited

Loading

varl commented May 30, 2019 •

edited

Loading