Merge db-tools:gdprify and db-tools:anonymize for a more GDPR-fri…

…endly workflow (#75) * Add GDPRify doc * Add GDPRify doc * Add GDPRify doc * Apply suggestions from code review apply suggestions Co-authored-by: Lonnytunes <[email protected]> * #76 - Update anonymization workflow * #76 - Update anonymization workflow + doc * #76 - Fix tests * #76 - Fix RestoreCommand * #76 - Docs fixes * #76 - typos * #76 - typos * #76 - typos * Apply suggestions from code review Co-authored-by: Lonnytunes <[email protected]> * Apply suggestion + add guardrails * Fix tests * fix test name * add test * coherence * Apply suggestions from code review Co-authored-by: Lonnytunes <[email protected]> * Apply suggestions from code review Co-authored-by: Lonnytunes <[email protected]> * Apply suggestions from code review Co-authored-by: Lonnytunes <[email protected]> --------- Co-authored-by: Lonnytunes <[email protected]>
makinacorpus · Dec 20, 2023 · 6fae8fc · 6fae8fc
1 parent 1793770
commit 6fae8fc
Show file tree

Hide file tree

Showing 14 changed files with 546 additions and 354 deletions.
diff --git a/config/services.yaml b/config/services.yaml
@@ -3,8 +3,11 @@ services:
  db_tools.command.anonymization.run:
  class: MakinaCorpus\DbToolsBundle\Command\Anonymization\AnonymizeCommand
  arguments:
- - '@db_tools.anonymization.anonymizator.factory'
  - '%doctrine.default_connection%'
+ - '@db_tools.restorer.factory'
+ - '@db_tools.backupper.factory'
+ - '@db_tools.anonymization.anonymizator.factory'
+ - '@db_tools.storage'
  tags: ['console.command']
  db_tools.command.anonymization.list:
  class: MakinaCorpus\DbToolsBundle\Command\Anonymization\AnonymizerListCommand
@@ -17,8 +20,8 @@ services:
  - '@db_tools.anonymization.anonymizator.factory'
  - '%doctrine.default_connection%'
  tags: ['console.command']
- db_tools.command.anonymization.check:
- class: MakinaCorpus\DbToolsBundle\Command\Anonymization\ConfigCheckCommand
+ db_tools.command.anonymization.dump:
+ class: MakinaCorpus\DbToolsBundle\Command\Anonymization\ConfigDumpCommand
  arguments:
  - '@db_tools.anonymization.anonymizator.factory'
  tags: ['console.command']
@@ -37,14 +40,6 @@ services:
  - '@db_tools.backupper.factory'
  - '@db_tools.restorer.factory'
  tags: ['console.command']
- db_tools.command.gdprify:
- class: MakinaCorpus\DbToolsBundle\Command\GdprifyCommand
- arguments:
- - '%doctrine.default_connection%'
- - '@db_tools.restorer.factory'
- - '@db_tools.backupper.factory'
- - '@db_tools.anonymization.anonymizator.factory'
- tags: ['console.command']
 
  db_tools.command.restore:
  class: MakinaCorpus\DbToolsBundle\Command\RestoreCommand

diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts
@@ -71,6 +71,7 @@ export default defineConfig({
  { text: 'Core Anonymizers', link: '/anonymization/core-anonymizers' },
  { text: 'Extra packs', link: '/anonymization/packs' },
  { text: 'Custom Anonymizers', link: '/anonymization/custom-anonymizers' },
+ { text: 'Anonymization command', link: '/anonymization/command' },
  { text: 'Internals', link: '/anonymization/internals' },
  ]
  },

diff --git a/docs/content/anonymization/command.md b/docs/content/anonymization/command.md
@@ -0,0 +1,127 @@
+---
+outline:
+ - 2
+ - 3
+---
+
+# Anonymization command
+
+Considering your anonymization has been configured, you can
+now anonymize a backup file by running:
+
+```sh
+console db-tools:anonymize path/to/your/backup/to/anonymized
+```
+
+This command will successively:
+
+1. Backup your local database,
+2. Restore the given backup file,
+3. Anonymize the data from the given backup file,
+4. Backup the newly anonymized database, **by overwritting the given backup file**,
+5. Restore your database to its original state from the backup produced at step 1.
+
+::: warning
+The `db-tools:anonymize` command alone is not enough to ensure you follow GDPR best practices.
+It depends on:
+
+* How you correctly configured your anonymization (obviously),
+* Where you run this command: anonymizing a backup file means it contains
+ sensitive data, hence, following GDPR recommendations, this **backup file
+ should never transit on an unsecured environment**.
+
+Read the next section to learn more about a proper workflow.
+:::
+
+## A GDPR-friendly workflow
+
+Here is an example of workflow - that follows GDPR recommendations - to retrieve anonymized production
+data on your local environment.
+
+### Prerequisites
+
+* You have a second secured environment besides your *production* (such as a preproduction)
+ and you can securely copy files from one to another,
+* You can shut down your service on this second environment,
+* Your anonymization is well configured: every sensitive data has been
+ mapped to an anonymizer that will erase/hash/randomize it.
+
+::: note
+Note that the second environment could be any environment, not only a preproduction. All it needs to work
+is the Symfony Console and a database. It doesn't need to be a complete working env.
+:::
+
+### Workflow
+
+Let's call *another_env* the environment we have besides the production one.
+
+1. Run `console db-tools:backup` on your production environment or
+ choose an existing backup with `console db-tools:restore --list`,
+2. Securely download your backup file from your *production* to your *another_env* environment,
+3. Stop services on your *another_env* to ensure no one is using it,
+4. Run `console db-tools:anonymize path/to/your/production/backup` to generate
+ a new backup cleaned from its sensitive data,
+5. Restart services on your *another_env*,
+6. Download the anonymized backup from *another_env* to your local machine
+7. Restore the backup with `console db-tools:restore --filename path/to/your/anonymized/backup`
+
+## Options
+
+You can specify the behaviour of the `db-tools:anonymize`command with some options detailed below.
+
+### Anonymizing local database
+
+The main purpose of this command is to provide a way to anonymize a backup file. But
+it could also be used to anonymize local database with `--local-database`.
+
+```sh
+console db-tools:anonymize --local-database
+```
+
+### Do not restore initial state after anonymization
+
+You can choose to not restore initial database with the `--no-restore` option.
+With this option, step 1 and 5 will be skipped during execution.
+
+```sh
+console db-tools:anonymize --no-restore
+```
+
+### Only anonymize specific targets
+
+Use this option if you want to anonymize only some specific targets during the process.
+
+```sh
+console db-tools:anonymize --target target_1 --taget target_2
+# or
+console db-tools:anonymize --t target_1 --t target_2
+```
+
+::: tip
+To know all your available targets, launch `db-tools:anonymization:dump-config`
+:::
+
+### Exclude targets from anonymization
+
+Use this option if you want to exclude some specific targets from anonymization.
+
+```sh
+console db-tools:anonymize --exclude target_1 --exclude target_2
+# or
+console db-tools:anonymize --x target_1 --x target_2
+```
+
+::: tip
+To know all your available targets, launch `db-tools:anonymization:dump-config`
+:::
+
+### Split update queries
+
+By default, the anonymization process use one update query per table.
+For debug purpose, it could be usefull to run not only one update query per table
+but one update query per target. To do so, use the `--split-per-column` option.
+
+::: info
+Learn more about how the anonymization process build these update queries reading
+the [Internals section](./internals).
+:::
diff --git a/docs/content/anonymization/core-anonymizers.md b/docs/content/anonymization/core-anonymizers.md
@@ -24,8 +24,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -40,7 +40,7 @@ class User
 ```yaml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  email_address: email
 
 #...
@@ -57,8 +57,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -73,7 +73,7 @@ class User
 ```yaml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  email_address:
  anonymizer: email
  options: {domain: 'custom-domain.com'}
@@ -105,8 +105,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -132,13 +132,13 @@ class User
 ```yaml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  password: password
 
 
 # Or, with options:
 
-user:
+customer:
  password:
  anonymizer: password
  options: {algorithm: 'sodium', password: '123456789'}
@@ -159,8 +159,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -175,7 +175,7 @@ class User
 ```yml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  age:
  anonymizer: integer
  options: {min: 10, max: 99}
@@ -196,8 +196,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -212,7 +212,7 @@ class User
 ```yaml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  size:
  anonymizer: float
  options: {min: 120, max: 300, precision: 4}
@@ -232,8 +232,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -248,7 +248,7 @@ class User
 ```yaml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  my_dirty_secret: md5
 #...
 ```
@@ -271,8 +271,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -286,7 +286,7 @@ class User
 
 ```yaml [YAML]
 # config/anonymization.yaml
-user:
+customer:
  level:
  anonymizer: string
  options: {sample: ['none', 'bad', 'good', 'expert']}
@@ -312,8 +312,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -328,7 +328,7 @@ class User
 ```yaml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  lastname: lastname
 #...
 ```
@@ -346,8 +346,8 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
-class User
+#[ORM\Table(name: 'customer')]
+class Customer
 {
  // ...
 
@@ -362,7 +362,7 @@ class User
 ```yaml [YAML]
 # config/anonymization.yaml
 
-user:
+customer:
  firstname: firstname
 #...
 ```
@@ -394,7 +394,7 @@ use Doctrine\ORM\Mapping as ORM;
 use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
 
 #[ORM\Entity()]
-#[ORM\Table(name: '`user`')]
+#[ORM\Table(name: 'customer')]
 #[Anonymize(type: 'address', options: [ // [!code ++]
  'street_address' => 'street', // [!code ++]
  'secondary_address': 'street_second_line' // [!code ++]
@@ -403,7 +403,7 @@ use MakinaCorpus\DbToolsBundle\Attribute\Anonymize;
  'region' => 'region' // [!code ++]
  'country' => 'country', // [!code ++]
 ])] // [!code ++]
-class User
+class Customer
 {
  // ...
 
@@ -431,7 +431,7 @@ class User
 
 ```yaml [YAML]
 # config/anonymization.yaml
-user:
+customer:
  address:
  target: table
  anonymizer: address