Skip to content

Latest commit

Β 

History

History
813 lines (602 loc) Β· 34.6 KB

RELEASE_NOTES.md

File metadata and controls

813 lines (602 loc) Β· 34.6 KB

CYBERTEC Migrator - Release Notes

The release notes of the last release may be found on README.md.

v3.19.2 - 2024-05-27

Resolved Bugs

  • Code editors occasionally loosing changes when saving
  • Dependency searchbar closing randomly

Regressions

  • Replacing code using the Search/Replace tab does not update the code in the editor without refreshing

v3.19.1 - 2024-05-21

Resolved Bugs

  • Format overly long migration effort estimates on the migration list
  • Certificate could not be obtained: no SSL error reported when using SSL for the PostgreSQL connection

v3.19.0 - 2024-05-07

  • Migration List:
    • Display migration status in list
    • Wrap long connection strings

  • Structure stage:
    • Detect Index Organized Tables and automatically exclude their overflow tables
  • Data stage:
    • Show total bytes and bytes per seconds for a finished transfer
    • Indicate whether a child process died from a signal, such as from the out-of-memory killer
    • Gracefully handle INT signals to hasten abort request
  • Migration Settings:
    • Changed data progress interval unit from milliseconds to seconds
    • Migration passwords are now encrypted within the database

Resolved Bugs

  • Code editors occasionally loose changes while typing
  • Renaming a table column does not display the new name in the constraint tab
  • PostgreSQL reserved identifiers are not read if the amount of CPU cores can not be retrieved
  • Replacing trigger code using the Search tab does not update the code in the editor without refreshing
  • ORA-01652: unable to grow temp segment in tablespace TEMP by 1MB during operation when reading indexes from some Oracle databases
  • Enabling the diff mode on a view-editor causes a page crash
  • DEFAULT is not allowed in this context when migrating list partitions with a non-uppercase default statement

v3.18.4 - 2023-09-27

  • Upon opening the Trial edition, the user must acknowledge the usage restrictions prior to using the application.

v3.18.3 - 2023-09-20

  • Package curl and psql with the core image for troubleshooting purposes

v3.18.2 - 2023-09-19

  • Package ping, telnet and traceroute with the core image for troubleshooting purposes

v3.18.1 - 2023-09-18

Resolved Bugs

  • unable to connect to internal database: Error: Entity metadata for Schema#packages was not found. Check if you specified a correct entity object and if it's connected in the connection options.
    Fresh installation fails during the database setup.

v3.18.0 - 2023-09-04

  • Automatic Code Transpilation: Receive suggestions from the CYBERTEC Migrator when rewriting functions, procedures, triggers and views.

    For more information, see Transpiler Features.

⚠️ Breaking changes

Beginning with this release, the Docker images include the edition in their name (e.g. cybertecpostgresql/cybertec_migrator-professional-core). Furthermore, the Standard edition has been renamed to Trial; its features have been adapted accordingly.

For users operating the installer: Execute ./migrator configure --edition <edition>.
Users operating on Kubernetes or OpenShift: Please modify your configurations accordingly.

v3.17.0 - 2023-06-19

  • Dependency Control: Manually manage object dependencies to precisely control the order of database object creation

  • Migration Creation: Enhanced error handling now allows you to identify and address issues, at the same time allowing the creation process to continue

  • Bulk Migration Creation: Provide expected JSON schema and downloadable template

v3.16.4 - 2023-05-22

Resolved Bugs

  • Error during data transfer: Failed to connect to database: could not translate host name "@localhost" to address: Name or service not known
    Data stage fails if the PostgreSQL username, password or database contain special characters such as @

v3.16.3 - 2023-05-11

Resolved Bugs

  • Creating a migration for Oracle 23 fails during the Analyze step

v3.16.2 - 2023-05-03

Changes

  • Drop Docker Compose V1 support (EOL June 2023) in favour of Compose V2

v3.16.1 - 2023-05-02

Resolved Bugs

  • Creating and cloning a migration may become unresponsive during schema selection

v3.16.0 - 2023-03-29

Features

  • Migration overview: View the assessed migration effort for each migration alongside performance improvements

  • Bulk migration creation: Rapidly create a multitude of migrations

    Example JSON:

    {
      "migrations": [
        {
          "name": "Migration",
          "description": "An optional description",
          "source": {
            "connection_string": "oracle:https://localhost:1521/pdb1",
            "username": "system",
            "password": "oracle"
          },
          "target": {
            "connection_string": "postgresql:https://localhost:5432/postgres",
            "username": "postgres",
            "password": "postgres"
          },
          "schema_selection": [
            "TOOLS", "ONLINE_STORE"
          ]
        }
      ]
    }
  • Data transfer prioritization: Customize the order in which tables are transferred

  • Custom SCN: Force the migrator to use a fixed SCN for the data stage

Other

  • Removed upper limit when generating table chunks
  • Removed deprecated C data migrator (and the correlating CORE_DATA_MIGRATOR_USE_RUST environment variable).

v3.15.1 - 2023-03-16

Resolved Bugs

  • Resuming data stage after chunking a table would truncate other chunked tables

v3.15.0 - 2023-02-21

Features

  • Table chunking: Speed up your data transfers through parallelization

    At present, chunking is limited to tables with

    • a primary/unique key or index
    • over exactly one column
    • of a numeric type

    Using a key/index, the migrator will create approximate chunks with it's min and max values.

  • View triggers: Show, edit and migrate view triggers

  • Improved migration creation: Enhanced keyboard navigation and introduced Go Back button

  • Sidebar: Show or hide excluded DBOs

3.14.1 - 2023-01-26

Resolved Bugs

  • Importing a migration archive on Windows causes File must be a .zip archive
  • Switching between procedure editors causes their contents to be overwritten

v3.14.0 - 2023-01-10

Features

  • Live Data stage overview: Determine the migration status of each table at a glance

  • Import migration: Create a new migration using a migration export, as an alternative to cloning

    Note: Only migration exports created with v3.14.0 and later are supported.

  • Log Panel Timezone: Choose a custom timezone for migration log timestamps

  • Sequence Editor: View and edit the details of sequences

  • Rename table columns:

    • Oracle columns named after PostgreSQL system columns (xmin, xmax, etc.) can not be migrated as is and must be renamed

  • Migration Settings:

    • Change migration name and description

    • Configure the interval between each data transfer report

v3.13.2 - 2022-12-02

Resolved Bugs

  • Regression: Can't create table and column comments containing single quotes

v3.13.1 - 2022-12-01

Resolved Bugs

  • Tables containing columns named after Oracle-only reserved identifiers (e.g. SIZE) fail during the data stage

v3.13.0 - 2022-11-30

Features

  • Migration Assessment:

    • Quickly determine how much effort an individual migration encompasses

    • Inspect the cost for discrete database objects and identify outliers

    Note: The figures given by the assessment are only a rough estimation.
    We will continue to tweak and improve these values as we collect experience and introduce new technologies, such as a PL/SQL parser.

  • Migration Overview: List empty and non-empty schemas separately

    Note: Empty schemas are now excluded per default

  • Migration Creation: Improve feedback for invalid connection strings

  • Migration Lifecycle:

    • Defer the creation of functions after those of tables to allow for the use of the %TYPE attribute
    • Enhance the migration log of the structure, integrity and logic stages to be on a par with those of the data stage
    • Configure the COPY FREEZE functionality introduced in v3.12.0 for each individual table instead of the whole system. The CORE_DATA_MIGRATOR_USE_COPY_FREEZE environment variable has subsequently been removed.
  • Miscellaneous: Rework the log output format and fidelity throughout the core container

Resolved Bugs

  • The Analyze step of the migration creation always shows Jobs as pending
  • LOBs exceeding the maximum size of 500MB do not fail with an appropriate error message
  • Setting the sidebar filter to Index does not include all indexes in the filter result
  • Usernames and passwords containing special characters cause the target connection check during migration creation to fail
  • Deleting a migration after executing a stage fails with an unexpected error

v3.12.0 - 2022-10-25

Features

  • Enhance migration log in the data stage:

    • Improve summary of the start and end logs

      resumed data stage from oracle:https://localhost:1521/pdb1 to postgresql:https://localhost:5432/postgres: using 8 workers
        excluded tables: 1
        successful transfers: 1 (out of 7)
        remaining transfers: 6
      
      ...
      
      failed executing data stage: stage run-time 00:01:534
        excluded tables: 1
        successful transfers: 6 (out of 7)
        remaining transfers: 1
        failed transfers: 1
      
    • Print a warning if the source and/or target connection are not encrypted

      Info     Data  using secure connection to read data from oracle:https://pdb_sec [oracle:https://10.0.0.127:2484/pdb?protocol=tcps]
      Warning  Data  using insecure connection to write data to postgresql:https://localhost:5432/postgres
      
    • Periodically log data-migrator transfer statistics for each table. This interval may be set using the CORE_DATA_MIGRATOR_PROGRESS_INTERVAL environment variable (default: 600000 milliseconds, 10 minutes)

      Verbose  Data  loading data for table:"HR"."EMPLOYEES" [hr.employees]: processed 982 rows in 01:20.000 - 45 rows/sec (total: 41 rows/sec in 05:00.000)
      Verbose  Data  finished loading data for table:"HR"."EMPLOYEES" [hr.employees]: processed 1012 rows in 00:02:01 - 40 rows/sec
      
  • Quote reserved identifiers when generating data queries for both Oracle and PostgreSQL

  • Add support for secured communication (TCPS) access to Oracle databases without using a net service name. For details, check out our FAQ - How do I configure TCPS for Oracle databases section

  • Configure the FREEZE parameter during the data transfer by setting the CORE_DATA_MIGRATOR_USE_COPY_FREEZE environment variable (default: false). Only use this setting if you are aware of its implications

Resolved Bugs

  • An error on the target connection during the structure, integrity or logic stage may cause the Migrator core to crash

v3.11.1 - 2022-10-05

Resolved Bugs

  • Data stage reported as successful for certain corner cases when transfer failed with No space left on device
  • Connectivity check to PostgreSQL with connection parameters fails when port is omitted (FATAL: database does not exist)
  • Failed to read data from Oracle for tables with lowercase identifiers
  • Regression: can not provide hostname with '-' in connection string

v3.11.0 - 2022-09-28

Features

  • Add support for Oracle 11 (closes #14).

  • We changed the default configuration of the reverse proxy to use HTTPS.
    This means you have to install a TLS/SSL certificate or create a self signed certificate, otherwise the web-server will refuse to start. To facilitate the installation of the certificate we added a new option --tls to the migrator configure command. Check out the FAQ for detailed information how to install a TLS/SSL certificate.

    migrator configure --tls self-signed-cert       Generate self-signed TLS/SSL certificate
    migrator configure --tls cert:<file-location>   Install TLS/SSL certificate
    migrator configure --tls key:<file-location>    Install private key of TLS/SSL certificate

    Attention
    If you upgrade from a previous Migrator version you have to create the TLS/SSL certificate after upgrading to the new version, before restarting the new Migrator.

    ./migrator update
    ./migrator upgrade
    # Don't forget the create or install a TLS/SSL certificate
    ./migrator configure --tls self-signed-cert
    ./migrator up
  • Add support for secured communication (TCPS) access to Oracle databases. For details check our FAQ - How do I configure TCPS for Oracle databases?.

  • Read meta-data of additional Oracle database object types: jobs and operators

  • Users may provide a custom Data Query, thus manipulating the SELECT statement executed on the source database to migrate the data.

    A few use cases where this feature may be useful:
    • Migrate the table partially by filtering rows with WHERE
    • Migrate only the table structure but no data (WHERE ROWID < 0)
    • You want to keep the table structure, but not the data of a specific column (see screenshot above)
    • Retrieve (and thus insert) the data in a specific ORDER BY
    • Convert the column data of an unsupported data type into a string representation which can be β€œcast” into a valid PostgreSQL type via the COPY statement
  • Improve output for erroneous SQL execution when reading Oracle meta-data.

Resolved Bugs

  • Columns starting with SYS are not included in primary keys or unique constraints
  • Data stage fails when column is named after an SQL reserved keywords (for example IN)

v3.10.1 - 2022-09-08

Resolved Bugs

  • Creating a migration for Oracle 11 and below fails on reading out sequences due to a non-existent table
  • The lower_bound of range subpartitions is not determined correctly during migration creation
  • Running the Structure stage with REFERENCE partitions results in a non-descriptive error
  • Regression in v3.10.0 causing the data of individual partitions to not be migrated correctly
  • Code editors mark their complete content as erroneous instead of only the faulty section
  • Revert style changes in the search and replace panel

v3.10.0 - 2022-08-31

Features

  • Improve Search Panel:
    • Search package specifications and bodies
    • Consolidate DBO type filter into a single input

    • Select multiple search results to replace at once
      • Shift-Click to add multiple rows
      • Ctrl-Click to add/remove a single row

    • Improve keyboard navigation when using <Tab>
  • Show information for Oracle Queues in the Sidebar

Resolved Bugs

  • DBO creation fails due to global lock_timeout setting on target database
  • Running ps inside the core container during the data stage reveals the database passwords
  • Abort of data stage may sometimes hang indefinitely

v3.9.0 - 2022-07-26

Features

  • Improve Sidebar:
    • Show meta-data as pop-up for constraints and indices

    • Extend the Object Type filter options with Partitions and provide granularity for Indexes (Unique, Functional, and Domain)

  • Migration Overview: Additional index types on Indices drill down

  • Switch Migrator core to the native libpq PostgreSQL driver to support additional authentication methods (e.g. GSSAPI)
  • Recurring security maintenance: upgrade package dependencies with known vulnerabilities to newest version

Resolved Bugs

  • PostgreSQL authentication with AuthenticationGSS causes a crash
  • Importing an Oracle View with a Null-Byte results in an unresponsive state
  • Unexpected shutdown of the target PostgreSQL connection may cause a crash

v3.8.0 - 2022-06-29

Features

  • Configure the maximal number of parallel data transfers and index creation workers for each migration.
    The Migrator picks reasonable default values when a new migration is created:

  • Enhance sidebar: show synonym meta-data as pop-up

  • Set/clear object type filter by selecting an Object Type in the overview tab. Use the CTRL key to filter for more than one database object type.

  • Add issue templates for asking questions.

Resolved Bugs

  • Empty connection check error message on 504 Gateway Time-Out
  • Broken GUI workflow in the Analyze step on 504 Gateway Time-Out
  • Error on migrating REVERSE indexes: PostgreSQL does not have/need an equivalent feature
  • Opening a non-existent migration endlessly shows the loading animation
  • Error on migrating sequences with negative INCREMENT - START value cannot be greater than MAXVALUE
  • Migration controls (resume, continue, etc.) occasionally do not behave as expected
  • Views depending on other views are not created in the correct order
  • Spacing between the migration log time and level is jumping
  • Incorrect name conflict detection for tables and triggers with the same identifier

v3.7.0 - 2022-05-18

Do you want to know if the Migrator can migrate your Oracle database to PostgreSQL?

Then get the Migrator Trial Edition, a free version (as in beer) of the CYBERTEC Migrator, follow the offline instructions provided in Getting Started section, and try it out.

Features

  • Improve migration job execution (which removed the Redis job queue as a dependency)
  • Provide help menu to reach out to CYBERTEC
  • Provide a Migrator demo database environment to facilitate a test-run of the Migrator Trial Edition
  • Added License information

Resolved Bugs

  • Error when attempting to edit a function, procedure, trigger or view containing a # in its name
  • Trigger Type and Level can be changed even if the trigger is excluded
  • Column data-types qualified with SYS are not translated properly

v3.6.0 - 2022-04-26

Features

  • Add support for stage post-hook SQL scripts to adapt the migration with functionalities not provided by the Migrator. The scripts are executed at the end of a stage. Typical use cases for such scripts are to create database objects which are not present in the source database, or to define ownership and access permissions.
    • Add a stage post-hook script in the Stages Tab.

    • Editor for stage hook scripts. It is possible to temporary disable a hook script.

    • The execution of the stage post-hook scripts are logged for audit purposes.

  • Improve Data stage: list and range partitions and sub-partitions of partitioned tables are migrated in parallel.
  • Support migration of GENERATED AS IDENTITY constraint.
  • Improve user experience:
    • Code editors verify DDL code to provide fast error feedback.

    • Add keyboard hotkey Ctrl-Enter to start migration job execution.

Resolved Bugs

  • Failed to created migration of an Oracle database containing a column that uses a data-type from the SYS schema
  • Starting a migration job with more than one stage and incorrect target connection bricks the migration
  • Sidebar filter showing schemas that contain none of the filter results
  • Misleading console log entries No metadata found. There is more than once class-validator version installed probably ...

v3.5.0 - 2022-03-29

Features

  • Several improvements in the execution of a migration:
    • Swap execution order of Integrity and Logic stage. Having indices in the Logic stage makes it easier to test performance of views, stored procedures and triggers.

    • Move creation of check constraints from Logic into Structure stage. Checking the data during the data bulk load is negligible compared to the time needed to re-read the table from disk in the Logic stage.
    • As a consequence, functions are created in the Structure stage prior to tables, since they may be used by check constraints. The Logic stage re-creates the functions once again still providing the means for fast change-test round-trips.
  • The Migration Overview was updated to reflect the changes in the migration execution:
    • It is visible in which order the database objects are created: first schemas, then user defined types, followed by sequences, etc.

    • Provide drill down on Indices entry to show the number of unique indices. We will enrich the Migration Overview with additional information in future releases.

  • Enhance migration configuration:
    • Constraint Renaming
      Rename constraints in case there are naming collisions with other database objects.

  • Improve handling of implicitly created indices via Unique Constraints or Primary Keys:
    • Show implicitly created indices in the sidebar and the Indices view. In our example, dept_id_pk is shown in the Constraints as well as the Indices section.
    • A hyperlink in the Indices view facilitates navigation to the constraint.

    • Rename implicitly created indices by renaming its constraint.
  • Log view shows detailed information about the started migration job:

  • Improve Sidebar:
    • Object Type Filter: add option for User Defined Types (UDT).

Resolved Bugs

  • Cloning a migration fails due to missing database object in the source database (dropped or renamed table, dropped column, etc.)
  • Integrity stage ERROR: timeout exceeded when trying to connect
  • Integrity stage keeps processing workers after an error even with enabled "Abort stage on first error"
  • Resume of an aborted job fails when the Migrator core was forcefully shut down
  • Core dump due to repeated, large error message (warn: Unable to process subpartition ... table not found)
  • Functional index with an expression containing a number was not migrated
  • Overview page does now show 0 database links, synonyms, packages

v3.4.3 - 2022-03-04

This is a bugfix release for v3.4.0.

Note: v3.4.1 and v3.4.2 were not released.

Resolved Bugs

  • Foreign keys are not created in parallel
  • Import fails for databases with a view on a view
  • Import fails for databases with a partition of a non-imported table (i.e. temporary, secondary, nested, or dropped table)

v3.4.2

Not released.

v3.4.1

Not released.

v3.4.0 - 2022-02-24

The main focus of this release is improving the performance for reading the source database meta-data (we tested databases with approximately 400.000 database objects). This includes improvements to the GUI to be responsive with a large number of database objects.

NOTE
After starting the migrator the dashboard may show an error until the Migrator converted existing migrations in its internal database. Depending on the number and size of existing migrations this may take some time.

Features

  • Reading and analyzing the source database meta-data is now faster
  • Overview page: drill down to show specific data types
  • Improve Sidebar:
    • Show number of database objects
    • Add new Constraints Object Type filter (Check, Foreign Keys, and Unique/Primary Keys)
    • Add Disable Partitioning for multiple tables
  • Improve Search and Replace
    • Include index names
    • Scope search with sidebar filter
  • Highlight element in the table editor when selected in the sidebar or when navigated from the log view
  • Log View:
    • Scope log entries based on job execution: ALL, CURRENT MIGRATION, and LAST EXECUTED JOB
    • Download content of log view

Resolved Bugs

  • Creating a migration of a database with a large number of DBOs fails
  • Unable to save large configuration changes
  • Partitions tab not responsive for a table with large number of partitions (>500)
  • Resuming data stage results in migrating the data from a different SCN
  • Data stage was not aborted properly by user request

v3.3.0 - 2021-12-21

Features

  • Show information of additional Oracle database object types
    • Database links
    • Packages (specification and body)
    • Synonyms
  • Show procedures and functions as separate types (until now both types were shown under Functions)
  • Improve data transfer for Oracle BLOB, CLOB, NCLOB and BFILE columns for rows with big LOBs. This may fix out-of-memory errors on the Oracle side, namely MSG: ORA-01062: unable to allocate memory for define buffer. It also significantly reduces the Migrator memory footprint.
  • The transfer of a row containing a LOB bigger than 500MB will abort with an informative error message.

Resolved Bugs

  • Creating a migration of a database with invisible columns causes a failure during meta-data import (null value in column "position" of relation "table_column" violates not-null constraint)

v3.2.0 - 2021-11-26

Features

  • Enhanced migration configuration:
    • Index Renaming
      Rename indices in case there are naming collisions with other database objects
    • Primary/Unique Key Modification
      Add, remove and change the order of table columns in a primary/unique key
  • Browser Tab Syncing
    Work on the same migration in multiple browser tabs, with them being synced in real-time
  • Search and Replace on check constraints and expression indexes
  • Usability Improvements
    • Improved sidebar functionalities:
      • Object Type filter: add options for indices, triggers, and sequences
      • Bulk exclude of schemata and tables
    • The tabs in the table editor (Columns, Constraints, etc.) display the identifiers used in the target database
    • Provide examples for source and target connection strings when creating new migrations
    • Performance improvements when running migration jobs
    • Provide more concise error details when writing database objects to the target database (e.g. miss-matching foreign key data-types)

Resolved Bugs

  • Creating a migration of a database with a view containing numerous columns causes a failure during meta-data import (ORA-40478: output value too large (maximum: 4000))
  • Creating a migration with an unreachable host shows a non-descriptive error
  • Replace All on different database object types sometimes doesn't replace all search results

v3.1.0 - 2021-10-20

Features

  • Search and Replace
    Perform bulk changes on column data types / default values, functions, views, and triggers (with support for even more object properties coming soon)

  • Significant Speed Improvements
    Feel the difference

  • Resume Failed Stages
    Spend less time fixing those last few migration errors

  • Realtime Syntax Checking
    Get immediate feedback about the syntactical correctness of code

  • Partition Renaming
    Rename partitions as part of a migration instead of altering the source database

  • Code Export
    Export code of function / views / triggers (with code import coming soon)

  • Migration Deletion
    Delete of completed migrations

  • Migration Cloning
    Prepare migrations against your staging environment and then execute them against production

Resolved Bugs

  • Keys for excluded columns are not excluded automatically
  • Column aliases in view code are not migrated
  • Migration status is sometimes displayed incorrectly
  • Excluding partition columns via the sidebar prevents the Structure stage from succeeding

v3.0.0 - 2022-09-16

With CYBERTEC Migrator v3 we've rebuilt the GUI from the ground up to simplify database migrations even further

  • Get a brief look at the structure of your source database by using the migration Overview
  • Quickly drill down into your data model with the help of the new Sidebar
  • Stages (natural synchronization points of a migration process) now guide you through the migration process
    • Structure stage: create database objects necessary to migrate the table data
    • Data stage: migrate table data in parallel
    • Logic stage: create functions, triggers, views
    • Integrity stage: parallel creation of primary keys, indices, foreign keys, and constraint checks
  • Enjoy quick round trips and gain confidence in the correctness of your migration thanks to the reworked Migration Controls
    • Start migration from the beginning
    • Rerun stage in case of an error
    • Continue to next stage
    • Abort migration and restart at any time
  • Explore the inner workings of your migration with the new, and easily accessible Log View
    • Log entries are inter-linked to the database objects configuration
    • Filter on log level (ERROR, WARNING, INFO, VERBOSE)
  • Extensive configuration
    • Exclude database objects (schemas, tables, columns, indexes, ...) from migration
    • Table editor to configure columns, constraints, indices, triggers, and partitions
    • Bulk change of data types (for example change all NUMBER(4) to int, instead of smallint)
    • Integrated code editor for functions, stored procedures, and views with diff feature