-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #576 from owncloud/3.1_upgrade
ocis 3.1 upgrade
- Loading branch information
Showing
3 changed files
with
233 additions
and
106 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,9 +9,22 @@ | |
|
|
||
| NOTE: For any Infinite Scale version to upgrade to, see {ocis-downloadpage-url}?sort=time&order=desc[download.owncloud.com,window=_blank] or {docker-ocis-url}[Infinite Scale image] to get the right version. | ||
|
|
||
| IMPORTANT: Before starting any upgrade, make a xref:maintenance/b-r/backup.adoc[backup] first. | ||
|
|
||
| IMPORTANT: If not explicitly mentioned, any upgrade is forward only and a backstep is neither supported nor recommended and will most likely break the instance. | ||
|
|
||
| IMPORTANT: Before starting any upgrade, make a xref:maintenance/b-r/backup.adoc[backup] first. | ||
| IMPORTANT: When upgrading from an older release to the desired one, *ALL* upgrade steps from each release must be taken in order and none can be skipped. | ||
|
|
||
| == Version 3.0.0 to 3.1.0 | ||
|
|
||
| === Notable Changes Requiring Manual Intervention | ||
|
|
||
| * The space index of the decomposedFS will be automatically migrated from symlinks to messagepack. This migration needs preparation to run successfully. | ||
| * The xref:deployment/container/orchestration/orchestration.adoc#using-our-helm-charts-with-infinite-scale[Helm Chart] has been upgraded. | ||
|
|
||
| === Upgrade Steps | ||
|
|
||
| For a detailed description of the steps to upgrade see the xref:migration/upgrading_3.0.0_3.1.0.adoc[Upgrading from 3.0.0 to 3.1.0] documentation. | ||
|
|
||
| == Version 2.0.0 to 3.0.0 | ||
|
|
||
|
|
@@ -22,111 +35,8 @@ IMPORTANT: Before starting any upgrade, make a xref:maintenance/b-r/backup.adoc[ | |
| * A new `OCIS_LDAP_DISABLE_USER_MECHANISM` environment variable has been introduced that needs a settings review. | ||
| * The search index needs to be deleted as the layout has been changed. | ||
| * The xref:prerequisites/prerequisites.adoc#backend-for-metadata[metadata backend] has changed. | ||
| * The xref:deployment/container/orchestration/orchestration.adoc#using-helm-charts-with-infinite-scale[Helm Chart] has been upgraded. | ||
| * The xref:deployment/container/orchestration/orchestration.adoc#using-our-helm-charts-with-infinite-scale[Helm Chart] has been upgraded. | ||
|
|
||
| === Upgrade Steps | ||
|
|
||
| . Shut down the Infinite Scale instance. | ||
| . Update Infinite Scale via: | ||
| + | ||
| -- | ||
| * Docker | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| docker pull owncloud/ocis:3.0.0 | ||
| ---- | ||
|
|
||
| * or get the binary from the link above. | ||
| -- | ||
|
|
||
| . Manually add the xref:{s-path}/graph.adoc#environment-variables[GRAPH_APPLICATION_ID,window=_blank] to the config. The value can be any text string using characters defined by the https://en.wikipedia.org/wiki/Universally_unique_identifier[UUID definition] and is preferably a UUID4 string. | ||
| + | ||
| -- | ||
| - Either edit the xref:deployment/general/general-info.adoc#configuration-file-naming[ocis.yaml,window=_blank] (preferred and recommended) by adding: | ||
| + | ||
| [source,yaml] | ||
| ---- | ||
| graph: | ||
| application: | ||
| # required when upgrading ocis to v3.0.0 | ||
| id: <your UUID4 string> | ||
| ---- | ||
|
|
||
| - or by setting `GRAPH_APPLICATION_ID` as an xref:deployment/general/general-info.adoc#configuration-rules[ENV variable,window=_blank]. | ||
|
|
||
| NOTE: This environment variable will be defined automatically when installing a fresh instance and running xref:deployment/general/ocis-init.adoc[ocis init] to initialize it. | ||
| -- | ||
|
|
||
| . xref:deployment/services/s-list/proxy.adoc#automatic-role-assignments[Automatic Role Assignments,window=_blank] have been introduced that need a settings review. All users that do not have a role assigned at the time of their first login will get the role 'user' assigned if the default of the environment variable `PROXY_ROLE_ASSIGNMENT_DRIVER` is used. The assignment can be changed based to the values of an OpenID Connect Claim of that user using a different setting. See the referenced documentation for more details. | ||
|
|
||
| . The environment variable xref:deployment/services/env-vars-special-scope.adoc[OCIS_LDAP_DISABLE_USER_MECHANISM] is an option to control the behavior for disabling users. The default value is `attribute` and requires configuration on the LDAP server. Enabling and disabling users is LDAP implementation specific. | ||
| + | ||
| -- | ||
| - If you are using an external LDAP server you can either set `OCIS_LDAP_DISABLE_USER_MECHANISM` to `none` to disable it completely or to `attribute` in which case you need to set `OCIS_LDAP_USER_ENABLED_ATTRIBUTE` according to your external LDAP server's requirements. | ||
| - Additionally and due to a bug recently discovered in the xref:{s-path}/idp.adoc[IDP] service, you must set `OCIS_LDAP_USER_ENABLED_ATTRIBUTE=""` to overwrite the default setting when `OCIS_LDAP_DISABLE_USER_MECHANISM` is set to `none`. This bug will be fixed in a subsequent release. | ||
| -- | ||
|
|
||
| . Delete the full search index. For details about the used path see: xref:deployment/general/general-info.adoc#default-paths[OCIS_BASE_DATA_PATH,window=_blank]: | ||
| + | ||
| -- | ||
| [source,bash] | ||
| ---- | ||
| rm -rf <OCIS_BASE_DATA_PATH>/search/* | ||
| ---- | ||
|
|
||
| NOTE: The empty search index will be recreated space by space when something changes in a space. You can also use the Infinite Scale CLI to manually trigger a xref:maintenance/commands/commands.adoc#reindex-a-space-for-search[Reindex a Space for Search]. | ||
| -- | ||
|
|
||
| . Changes in the Helm Charts + | ||
| For any breaking changes that come along with Helm Charts, see the xref:deployment/container/orchestration/orchestration.adoc#breaking-changes[Breaking Changes,window=_blank] documentation. | ||
|
|
||
| . Messagepack Readme First + | ||
| If you have manually changed one of the following environment variables to a value other than the default or explicitly set one of the environment variables to the default which was `xattrs`: | ||
| + | ||
| -- | ||
|
|
||
| `OCIS_DECOMPOSEDFS_METADATA_BACKEND`, + | ||
| `STORAGE_SYSTEM_OCIS_METADATA_BACKEND`, + | ||
| `STORAGE_USERS_OCIS_METADATA_BACKEND`, | ||
|
|
||
| the instance will stay at the configured backend when passing the next step. | ||
|
|
||
| IMPORTANT: Though you can stay at the xattrs backend, it is highly recommended to migrate to messagepack as the xattrs backend is considered deprecated and will be removed soon. | ||
|
|
||
| To enable migration, remove any of the manually configured environment variables listed above. This can be also done in a later step but it is highly recommended to migrate now. | ||
|
|
||
| NOTE: If you want to prevent the migration for now, set `OCIS_DECOMPOSEDFS_METADATA_BACKEND=xattrs`. | ||
|
|
||
| IMPORTANT: Note that the migration of the metadata backend to messagepack is a one-way process and can not be reverted except via a full restore. | ||
| -- | ||
|
|
||
| . Migrate Metadata to Messagepack: + | ||
| It is important to disable clients and users logging in while the migration to `messagepack` is in progress. To do so, start Infinite Scale with only two necessary services. The example uses the binary as base: | ||
| + | ||
| -- | ||
| [source,bash] | ||
| ---- | ||
| OCIS_RUN_SERVICES="storage-users,nats" \ | ||
| ocis server | ||
| ---- | ||
|
|
||
| Because the proxy service is not running, no clients or users can login. During the restart of Infinite Scale, the metadata store in the DecomposedFS will be migrated to `messagepack` _automatically_ via the `storage-users` service. The start of the migration can be identified in the log with a message like: | ||
|
|
||
| [source,plaintext] | ||
| ---- | ||
| {"level":"info","root":"/var/lib/ocis/storage/users","time":"2023-06-16T12:21:49.441395153Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0003_switch_to_messagepack_metadata.go:45","message":"Migrating to messagepack metadata backend..."} | ||
| ---- | ||
|
|
||
| When migration has finished - which can take a while depending on the quantity of files - you can see a log message like below: | ||
|
|
||
| [source,plaintext] | ||
| ---- | ||
| {"level":"info","time":"2023-06-16T12:30:05.267986656Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0003_switch_to_messagepack_metadata.go:106","message":"done."}` | ||
| ---- | ||
|
|
||
| Finally start the Infinite Scale instance as usual. | ||
| -- | ||
|
|
||
| . Check your Instance + | ||
| When the instance has started successfully, check the logs for any unusual entries. | ||
| For a detailed description of the steps to upgrade see the xref:migration/upgrading_2.0.0_3.0.0.adoc[upgrading from 2.0.0 to 3.0.0] documentation. | ||
116 changes: 116 additions & 0 deletions
116
modules/ROOT/pages/migration/upgrading_2.0.0_3.0.0.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| = Upgrading from 2.0.0 to 3.0.0 | ||
| :toc: right | ||
| :description: This document describes the necessary steps when upgrading Infinite Scale from release 2.0.0 to 3.0.0. | ||
|
|
||
| == Introduction | ||
|
|
||
| {description} | ||
|
|
||
| IMPORTANT: Read the important notes in the xref:migration/upgrading-ocis.adoc#introduction[Upgrading Infinite Scale] documentation before you start. | ||
|
|
||
| == Upgrade Steps | ||
|
|
||
| . Shut down the Infinite Scale instance. | ||
| . Update Infinite Scale via: | ||
| + | ||
| -- | ||
| * Docker | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| docker pull owncloud/ocis:3.0.0 | ||
| ---- | ||
|
|
||
| * or get the binary from {ocis-downloadpage-url}?sort=time&order=desc[download.owncloud.com,window=_blank]. | ||
| -- | ||
|
|
||
| . Manually add the xref:{s-path}/graph.adoc#environment-variables[GRAPH_APPLICATION_ID,window=_blank] to the config. The value can be any text string using characters defined by the https://en.wikipedia.org/wiki/Universally_unique_identifier[UUID definition] and is preferably a UUID4 string. | ||
| + | ||
| -- | ||
| - Either edit the xref:deployment/general/general-info.adoc#configuration-file-naming[ocis.yaml,window=_blank] (preferred and recommended) by adding: | ||
| + | ||
| [source,yaml] | ||
| ---- | ||
| graph: | ||
| application: | ||
| # required when upgrading ocis to v3.0.0 | ||
| id: <your UUID4 string> | ||
| ---- | ||
|
|
||
| - or by setting `GRAPH_APPLICATION_ID` as an xref:deployment/general/general-info.adoc#configuration-rules[ENV variable,window=_blank]. | ||
|
|
||
| NOTE: This environment variable will be defined automatically when installing a fresh instance and running xref:deployment/general/ocis-init.adoc[ocis init] to initialize it. | ||
| -- | ||
|
|
||
| . xref:deployment/services/s-list/proxy.adoc#automatic-role-assignments[Automatic Role Assignments,window=_blank] have been introduced that need a settings review. All users that do not have a role assigned at the time of their first login will get the role 'user' assigned if the default of the environment variable `PROXY_ROLE_ASSIGNMENT_DRIVER` is used. The assignment can be changed based to the values of an OpenID Connect Claim of that user using a different setting. See the referenced documentation for more details. | ||
|
|
||
| . The environment variable xref:deployment/services/env-vars-special-scope.adoc[OCIS_LDAP_DISABLE_USER_MECHANISM] is an option to control the behavior for disabling users. The default value is `attribute` and requires configuration on the LDAP server. Enabling and disabling users is LDAP implementation specific. | ||
| + | ||
| -- | ||
| - If you are using an external LDAP server you can either set `OCIS_LDAP_DISABLE_USER_MECHANISM` to `none` to disable it completely or to `attribute` in which case you need to set `OCIS_LDAP_USER_ENABLED_ATTRIBUTE` according to your external LDAP server's requirements. | ||
| - Additionally and due to a bug recently discovered in the xref:{s-path}/idp.adoc[IDP] service, you must set `OCIS_LDAP_USER_ENABLED_ATTRIBUTE=""` to overwrite the default setting when `OCIS_LDAP_DISABLE_USER_MECHANISM` is set to `none`. This bug will be fixed in a subsequent release. | ||
| -- | ||
|
|
||
| . Delete the full search index. For details about the used path see: xref:deployment/general/general-info.adoc#default-paths[OCIS_BASE_DATA_PATH,window=_blank]: | ||
| + | ||
| -- | ||
| [source,bash] | ||
| ---- | ||
| rm -rf <OCIS_BASE_DATA_PATH>/search/* | ||
| ---- | ||
|
|
||
| NOTE: The empty search index will be recreated space by space when something changes in a space. You can also use the Infinite Scale CLI to manually trigger a xref:maintenance/commands/commands.adoc#reindex-a-space-for-search[Reindex a Space for Search]. | ||
| -- | ||
|
|
||
| . Changes in the Helm Charts + | ||
| For any breaking changes that come along with Helm Charts, see the xref:deployment/container/orchestration/orchestration.adoc#breaking-changes[Breaking Changes,window=_blank] documentation. | ||
|
|
||
| . Messagepack Readme First + | ||
| If you have manually changed one of the following environment variables to a value other than the default or explicitly set one of the environment variables to the default which was `xattrs`: | ||
| + | ||
| -- | ||
|
|
||
| `OCIS_DECOMPOSEDFS_METADATA_BACKEND`, + | ||
| `STORAGE_SYSTEM_OCIS_METADATA_BACKEND`, + | ||
| `STORAGE_USERS_OCIS_METADATA_BACKEND`, | ||
|
|
||
| the instance will stay at the configured backend when passing the next step. | ||
|
|
||
| IMPORTANT: Though you can stay at the xattrs backend, it is highly recommended to migrate to messagepack as the xattrs backend is considered deprecated and will be removed soon. | ||
|
|
||
| To enable migration, remove any of the manually configured environment variables listed above. This can also be done in a later step but it is highly recommended to migrate now. | ||
|
|
||
| NOTE: If you want to prevent the migration for now, set `OCIS_DECOMPOSEDFS_METADATA_BACKEND=xattrs`. | ||
|
|
||
| IMPORTANT: Note that the migration of the metadata backend to messagepack is a one-way process and can not be reverted except via a full restore. | ||
| -- | ||
|
|
||
| . Migrate Metadata to Messagepack: + | ||
| It is important to disable clients and users logging in while the migration to `messagepack` is in progress. To do so, start Infinite Scale with only two necessary services. The following example uses the binary as base: | ||
| + | ||
| -- | ||
| [source,bash] | ||
| ---- | ||
| OCIS_RUN_SERVICES="storage-users,nats" \ | ||
| ocis server | ||
| ---- | ||
|
|
||
| Because the proxy service is not running, no clients or users can login. During the restart of Infinite Scale, the metadata store in the DecomposedFS will be migrated to `messagepack` _automatically_ via the `storage-users` service. The start of the migration can be identified in the log with a message like: | ||
|
|
||
| [source,plaintext] | ||
| ---- | ||
| {"level":"info","root":"/var/lib/ocis/storage/users","time":"2023-06-16T12:21:49.441395153Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0003_switch_to_messagepack_metadata.go:45","message":"Migrating to messagepack metadata backend..."} | ||
| ---- | ||
|
|
||
| When migration has finished - which can take a while depending on the quantity of files - you can see a log message like the one below: | ||
|
|
||
| [source,plaintext] | ||
| ---- | ||
| {"level":"info","time":"2023-06-16T12:30:05.267986656Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0003_switch_to_messagepack_metadata.go:106","message":"done."}` | ||
| ---- | ||
|
|
||
| Finally, start the Infinite Scale instance as usual. | ||
| -- | ||
|
|
||
| . Check your Instance + | ||
| When the instance has started successfully, check the logs for any unusual entries. |
101 changes: 101 additions & 0 deletions
101
modules/ROOT/pages/migration/upgrading_3.0.0_3.1.0.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| = Upgrading from 3.0.0 to 3.1.0 | ||
| :toc: right | ||
| :description: This document describes the necessary steps when upgrading Infinite Scale from release 3.0.0 to 3.1.0. | ||
|
|
||
| == Introduction | ||
|
|
||
| {description} | ||
|
|
||
| IMPORTANT: Read the important notes in the xref:migration/upgrading-ocis.adoc#introduction[Upgrading Infinite Scale] documentation before you start. | ||
|
|
||
| == Upgrade Steps | ||
|
|
||
| . Shut down the Infinite Scale instance. | ||
| . Update Infinite Scale via: | ||
| + | ||
| -- | ||
| * Docker | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| docker pull owncloud/ocis:3.1.0 | ||
| ---- | ||
|
|
||
| * or get the binary from {ocis-downloadpage-url}?sort=time&order=desc[download.owncloud.com,window=_blank]. | ||
| -- | ||
|
|
||
| . Migrate the Space Index to Messagepack: + | ||
| It is important to disable clients and users logging in while the space index migration to `messagepack` is in progress. To do so, you have two choices: | ||
| + | ||
| -- | ||
| .. Start Infinite Scale with only two necessary services. The example uses the binary as base: | ||
| + | ||
| [source,bash] | ||
| ---- | ||
| OCIS_RUN_SERVICES="storage-users,nats" \ | ||
| ocis server | ||
| ---- | ||
|
|
||
| .. If configured, shut-down or disable the reverse proxy for Infinite Scale. | ||
|
|
||
| Because either the Infinite Scale proxy service is not running or the reverse proxy is inaccessible, no clients or users can login. | ||
| -- | ||
|
|
||
| . Migration process | ||
| + | ||
| -- | ||
| During the restart of Infinite Scale, the symlinks of spaces in the DecomposedFS will be migrated to `messagepack` _automatically_ via the `storage-users` service. | ||
| -- | ||
|
|
||
| .. Monitoring progress via logs | ||
| + | ||
| -- | ||
| The start of the migration can be identified in the log by messages like: | ||
|
|
||
| [source,plaintext] | ||
| ---- | ||
| {"level":"info","root":"/var/lib/ocis/storage/users","time":"2023-08-02T06:08:21.002825802Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0004_switch_to_messagepack_space_index.go:46","message":"Migrating /var/lib/ocis/storage/users/indexes/by-user-id/b7a2149b-3320-43a3-9bd2-c216340c212d.mpk to messagepack index format..."} | ||
| ... | ||
| {"level":"info","root":"/var/lib/ocis/storage/users","time":"2023-08-02T06:08:21.028153362Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0005_fix_messagepack_space_index_format.go:37","message":"Fixing index format of /var/lib/ocis/storage/users/indexes/by-user-id/b7a2149b-3320-43a3-9bd2-c216340c212d.mpk"} | ||
| ---- | ||
|
|
||
| When migration has finished - which can take a while depending on the quantity of spaces - you can see log messages like this: | ||
|
|
||
| [source,plaintext] | ||
| ---- | ||
| {"level":"info","time":"2023-08-02T06:08:21.026943689Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0004_switch_to_messagepack_space_index.go:92","message":"done."} | ||
| ... | ||
| {"level":"info","time":"2023-08-02T06:08:21.028422073Z","caller":"github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/migrator/0005_fix_messagepack_space_index_format.go:66","message":"done."} | ||
| ---- | ||
| -- | ||
|
|
||
| .. Monitoring progress via command | ||
| + | ||
| Use the following command to check the status of the migration steps. Replace `<path-to-decomposedFS>` with the physical path of the environment variable `OCIS_BASE_DATA_PATH`. For more details see the xref:deployment/general/general-info.adoc#base-data-directory[Base Data Directory] documentation. | ||
| + | ||
| -- | ||
| [source,bash] | ||
| ---- | ||
| ocis migrate decomposedfs -r <path-to-decomposedFS> list | ||
| ---- | ||
|
|
||
| Depending on the number of steps to process, you will see an output like the following: | ||
|
|
||
| [source,plaintext] | ||
| ---- | ||
| +-----------+-----------+---------+ | ||
| | Migration | State | Message | | ||
| +-----------+-----------+---------+ | ||
| | 0001 | succeeded | | | ||
| | 0002 | succeeded | | | ||
| | 0003 | succeeded | | | ||
| | 0004 | succeeded | | | ||
| | 0005 | succeeded | | | ||
| +-----------+-----------+---------+ | ||
| ---- | ||
| -- | ||
|
|
||
| . Finally, either start the Infinite Scale instance as usual or re-enable the reverse proxy. | ||
|
|
||
| . Check your Instance + | ||
| When the instance has started successfully, check the logs for any unusual entries. |