Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrations sentry.0640_index_together fails #2758

Closed
Tracked by #2776
pajowu opened this issue Feb 1, 2024 · 48 comments
Closed
Tracked by #2776

Migrations sentry.0640_index_together fails #2758

pajowu opened this issue Feb 1, 2024 · 48 comments

Comments

@pajowu
Copy link

pajowu commented Feb 1, 2024

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

  1. Checkout the nightly self-hosted repo into an existing project
  2. Run install.sh

Expected Result

Upgrade runs successfully

Actual Result

Upgrade fails in migration sentry.0640_index_together with the error below. This is caused by this following migration step which calls RenameIndex, which fails because two indices exist on these columns: one created by index_together as well as one created by unique_together.

This is a django bug that is fixed with this django commit. After applying this commit into our sentry installation, the migration ran successfully.

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/__init__.py", line 190, in main
    func(**kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/decorators.py", line 29, in inner
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/runner/commands/django.py", line 14, in django
    execute_from_command_line(argv=[ctx.command_path] + list(management_args))
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/new_migrations/monkey/executor.py", line 140, in apply_migration
    return super().apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/models.py", line 1042, in database_forwards
    raise ValueError(
ValueError: Found wrong number (2) of indexes for sentry_groupedmessage(project_id, id).

Product Area

Unknown

Link

No response

DSN

No response

Version

nightly

@getsantry
Copy link

getsantry bot commented Feb 1, 2024

Assigning to @getsentry/support for routing ⏲️

@Christophe31
Copy link

I got this issue with this event id
ee4e9e869ab6a19f443f9b338159211c

@Kobby-Bawuah Kobby-Bawuah transferred this issue from getsentry/sentry Feb 1, 2024
@SherinBloemendaal
Copy link

+1 got the same error. My sentry envelope id is: ee4e9e869ab6a19f443f9b338159211c

@codedmonkey
Copy link

Is it even possible to install Sentry at the moment?

Is there any workaround?

@Christophe31
Copy link

Christophe31 commented Feb 5, 2024

This is an index migration, it may impact performances but it looks like you can launch sentry without this migration as long as there are no structural migrations after.

Or even better, instead of running on master (nightly) branch, you may go to a tag and use a stable release: https://github.com/getsentry/self-hosted/tags

@hubertdeng123
Copy link
Member

This seems to be due to this PR here:
getsentry/sentry#64158

In SaaS, it looks like this was due to some dangerous migrations not being performed. These can be performed manually and you can get these migrations with
git log -G thetablename -- src/sentry/migrations

@hubertdeng123
Copy link
Member

Or, you can opt for a more hacky fix that I tried by going into your postgres database.

SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'sentry_groupedmessage';
DROP INDEX sentry_groupedmessage_project_id_id_515aaa7e_idx;
CREATE INDEX sentry_grou_project_41a5ce_idx ON public.sentry_groupedmessage USING btree (project_id, id);

@myonlylonely
Copy link

Or, you can opt for a more hacky fix that I tried by going into your postgres database.

SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'sentry_groupedmessage';
DROP INDEX sentry_groupedmessage_project_id_id_515aaa7e_idx;
CREATE INDEX sentry_grou_project_41a5ce_idx ON public.sentry_groupedmessage USING btree (project_id, id);

This hack does not work for me.

@codedmonkey
Copy link

@Christophe31 I feel so stupid, I did try to downgrade to a release tag but I forgot to update my .env.custom. Thank you!

@hubertdeng123 hubertdeng123 mentioned this issue Feb 6, 2024
18 tasks
@hubertdeng123
Copy link
Member

Hey folks, we've recently upgraded our nightly image to django 5. Could you do us a favor and try upgrading again to see if you get the same error?

We think that this may be resolved as django 5 should pick up django/django@7045661

@tankerkiller125
Copy link

@hubertdeng123 Pulled the latest master branch details and ran the install.sh command, no luck, same error message.

@razzeee
Copy link

razzeee commented Feb 7, 2024

I'm also still seeing this error with current master

@Christophe31
Copy link

Christophe31 commented Feb 8, 2024

+1, I had to comment DB upgrade line to finish install script.

@azaslavsky
Copy link
Contributor

We're working on a fix, which we will try to get out before the next release.

@hubertdeng123
Copy link
Member

@fibis Perhaps the script to drop indexes didn't quite work for you. Can you try dropping sentry_groupedmessage_project_id_id_515aaa7e_uniq?

@ZVilusinsky
Copy link

ZVilusinsky commented Feb 29, 2024

@hubertdeng123 Yes, I had wrong (2) as stated in #2805 . I've tried dropping both entry_groupedmessage_project_id_515aaa7e_idx and sentry_groupedmessage_project_id_id_515aaa7e_uniq like you asked.

As of this morning, I have successfully upgraded to 23.12.1 but will for now not try to migrate further to 24, given @fibis experience.
State of the indices after upgrade to latest 23 is

sentry=# SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'sentry_groupedmessage';
                           indexname                            |                                                                                 indexdef
----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 sentry_groupedmessage_pkey                                     | CREATE UNIQUE INDEX sentry_groupedmessage_pkey ON public.sentry_groupedmessage USING btree (id)
 sentry_groupedmessage_project_id_680bfe5607002523_uniq         | CREATE UNIQUE INDEX sentry_groupedmessage_project_id_680bfe5607002523_uniq ON public.sentry_groupedmessage USING btree (project_id, short_id)
 sentry_groupedmessage_active_at                                | CREATE INDEX sentry_groupedmessage_active_at ON public.sentry_groupedmessage USING btree (active_at)
 sentry_groupedmessage_first_release_id                         | CREATE INDEX sentry_groupedmessage_first_release_id ON public.sentry_groupedmessage USING btree (first_release_id)
 sentry_groupedmessage_first_seen                               | CREATE INDEX sentry_groupedmessage_first_seen ON public.sentry_groupedmessage USING btree (first_seen)
 sentry_groupedmessage_last_seen                                | CREATE INDEX sentry_groupedmessage_last_seen ON public.sentry_groupedmessage USING btree (last_seen)
 sentry_groupedmessage_level                                    | CREATE INDEX sentry_groupedmessage_level ON public.sentry_groupedmessage USING btree (level)
 sentry_groupedmessage_logger                                   | CREATE INDEX sentry_groupedmessage_logger ON public.sentry_groupedmessage USING btree (logger)
 sentry_groupedmessage_logger_like                              | CREATE INDEX sentry_groupedmessage_logger_like ON public.sentry_groupedmessage USING btree (logger varchar_pattern_ops)
 sentry_groupedmessage_project_id                               | CREATE INDEX sentry_groupedmessage_project_id ON public.sentry_groupedmessage USING btree (project_id)
 sentry_groupedmessage_project_id_31335ae34c8ef983              | CREATE INDEX sentry_groupedmessage_project_id_31335ae34c8ef983 ON public.sentry_groupedmessage USING btree (project_id, first_release_id)
 sentry_groupedmessage_project_id_515aaa7e_idx                  | CREATE INDEX sentry_groupedmessage_project_id_515aaa7e_idx ON public.sentry_groupedmessage USING btree (project_id, id)
 sentry_groupedmessage_project_id_id_515aaa7e_uniq              | CREATE UNIQUE INDEX sentry_groupedmessage_project_id_id_515aaa7e_uniq ON public.sentry_groupedmessage USING btree (project_id, id)
 sentry_groupedmessage_project_id_status_last_s_6b8195a7_idx    | CREATE INDEX sentry_groupedmessage_project_id_status_last_s_6b8195a7_idx ON public.sentry_groupedmessage USING btree (project_id, status, last_seen, id)
 sentry_groupedmessage_resolved_at                              | CREATE INDEX sentry_groupedmessage_resolved_at ON public.sentry_groupedmessage USING btree (resolved_at)
 sentry_groupedmessage_status                                   | CREATE INDEX sentry_groupedmessage_status ON public.sentry_groupedmessage USING btree (status)
 sentry_groupedmessage_times_seen                               | CREATE INDEX sentry_groupedmessage_times_seen ON public.sentry_groupedmessage USING btree (times_seen)
 sentry_groupedmessage_view                                     | CREATE INDEX sentry_groupedmessage_view ON public.sentry_groupedmessage USING btree (view)
 sentry_groupedmessage_view_like                                | CREATE INDEX sentry_groupedmessage_view_like ON public.sentry_groupedmessage USING btree (view varchar_pattern_ops)
 sentry_groupedmessage_project_id_type_status_l_074196b6_idx    | CREATE INDEX sentry_groupedmessage_project_id_type_status_l_074196b6_idx ON public.sentry_groupedmessage USING btree (project_id, status, type, last_seen, id)
 sentry_groupedmessage_type_de66b3a2                            | CREATE INDEX sentry_groupedmessage_type_de66b3a2 ON public.sentry_groupedmessage USING btree (type)
 sentry_groupedmessage_project_id_status_substa_fbe106b6_idx    | CREATE INDEX sentry_groupedmessage_project_id_status_substa_fbe106b6_idx ON public.sentry_groupedmessage USING btree (project_id, status, substatus, last_seen, id)
 sentry_groupedmessage_project_id_status_substa_c95bb62c_idx    | CREATE INDEX sentry_groupedmessage_project_id_status_substa_c95bb62c_idx ON public.sentry_groupedmessage USING btree (project_id, status, substatus, type, last_seen, id)
 sentry_groupedmessage_project_id_status_substa_78b9cc1c_idx    | CREATE INDEX sentry_groupedmessage_project_id_status_substa_78b9cc1c_idx ON public.sentry_groupedmessage USING btree (project_id, status, substatus, id)
 sentry_groupedmessage_status_substatus_id_e00e92ae_idx         | CREATE INDEX sentry_groupedmessage_status_substatus_id_e00e92ae_idx ON public.sentry_groupedmessage USING btree (status, substatus, id)
 sentry_groupedmessage_status_substatus_first_seen_99773cf6_idx | CREATE INDEX sentry_groupedmessage_status_substatus_first_seen_99773cf6_idx ON public.sentry_groupedmessage USING btree (status, substatus, first_seen)
(26 rows)

@pavelsshtanko
Copy link

I have some problem

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sentry/options/store.py", line 262, in set_cache
    self.cache.set(cache_key, value, self.ttl)
    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/connection.py", line 62, in __getitem__
    conn = self.create_connection(alias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/caching.py", line 108, in sentry_create_connection
    cache = original_create_connection(self, alias)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/cache/__init__.py", line 50, in create_connection
    raise InvalidCacheBackendError(
django.core.cache.backends.base.InvalidCacheBackendError: Could not find backend 'django.core.cache.backends.memcached.MemcachedCache': Module "django.core.cache.backends.memcached" does not define a "MemcachedCache" attribute/class
11:25:37 [WARNING] sentry: Unable to update option cache for transaction-events.force-disable-internal-project (key='transaction-events.force-disable-internal-project')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/new_migrations/migrations.py", line 22, in apply
    return super().apply(project_state, schema_editor, collect_sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/operations/models.py", line 1050, in database_forwards
    raise ValueError(
ValueError: Found wrong number (2) of indexes for sentry_groupedmessage(project_id, id).
Error in install/set-up-and-migrate-database.sh:12.
'$dcr web upgrade' exited with status 1
-> ./install.sh:main:34
--> install/set-up-and-migrate-database.sh:source:12

Looks like you've already sent this error to us, we're on it :)

@denys-kochetkov
Copy link

Dropping only the index sentry_groupedmessage_project_id_id_515aaa7e_uniq helped in our case.

@hubertdeng123
Copy link
Member

Yeah, I would assume that sentry_groupedmessage_project_id_id_515aaa7e_uniq is the one to drop. @ZVilusinsky I apologize I may have misremembered the fix we included in the install script, perhaps you should not have dropped sentry_groupedmessage_project_id_515aaa7e_idx.

@JustDoItSascha
Copy link

Will there be a automatic fix in an upcoming version? I tried to upgrade and are now facing the same issue...

@black3r
Copy link

black3r commented Mar 4, 2024

Faced the same issue here upgrading from 23.11.2 to 24.2.0 (also an archaic instance, was setup in August 2016 and slowly upgraded throughout since then).

The install script tried to fix the issue by deleting the sentry_groupedmessage_project_id_id_515aaa7e_uniq which was mentioned here, but it looked in the wrong database (postgres instead of sentry) which confused me as well so I had to figure out an alternate solution.

My alternate solution was faking the migration (INSERT INTO "django_migrations" (app, name, applied) VALUES ('sentry', '0640_index_together', current_timestamp);), running ./install.sh again and setting up the indices manually after the update successfully finished to match the table in #2758 (comment) (I assumed that's the current reference). Only at that point I realized what was wrong.

@mmorejon
Copy link

mmorejon commented Mar 8, 2024

I have the same error migrating from the helm chart 20.12.2 -> 21.5.0. (24.1.2)

ValueError: Found wrong number (2) of indexes for sentry_groupedmessage(project_id, id).
  Applying sentry.0640_index_together...% 

Do you know how I can fix it?

@xvilo
Copy link

xvilo commented Mar 8, 2024

Running drop index sentry_groupedmessage_project_id_id_515aaa7e_uniq in the postgres DB did indeed resolve this for us

@dedronek
Copy link

The same error while migrating from the chart version 20.6.0 -> 21.6.1 (sentry 23.9.1 -> 24.2.0).

It's fixed by running drop index sentry_groupedmessage_project_id_id_515aaa7e_uniq on the postgres DB.

@hubertdeng123
Copy link
Member

A general fix for this issue has been merged and now released, so I'm going to close this. For people who are still experiencing this issue, please attempt to try dropping this index in postgres sentry_groupedmessage_project_id_id_515aaa7e_uniq

@github-actions github-actions bot locked and limited conversation to collaborators Apr 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Archived in project
Development

No branches or pull requests