After migrate from 23.7.1 to 23.10.0 : Unable to fetch all projects stats

[meriturva]

Self-Hosted Version

23.10.0

CPU Architecture

x86:64

Docker Version

20.10.21

Docker Compose Version

2.20.2

Steps to Reproduce

Just open "projects" page and error occurs.

Expected Result

See all projects stats

Actual Result

I updated Sentry from version 23.7.1 to 23.10.0 (as usual) following official way.

Migration log has no errors but now when i go to https://sentry.xxx.com/organizations/xxx/projects/ i got error:

On web log i have:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sentry/api/base.py", line 257, in handle_exception
    response = super().handle_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/sentry/../rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/sentry/../rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.8/site-packages/sentry/api/base.py", line 381, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/drf_spectacular/drainage.py", line 193, in wrapped_method
    return method(self, request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/sentry/api/endpoints/organization_projects.py", line 172, in get
    return self.paginate(
  File "/usr/local/lib/python3.8/site-packages/sentry/api/base.py", line 491, in paginate
    results = on_results(cursor_result.results)
  File "/usr/local/lib/python3.8/site-packages/sentry/api/endpoints/organization_projects.py", line 170, in serialize_on_result
    return serialize(result, request.user, serializer)
  File "/usr/local/lib/python3.8/site-packages/sentry/api/serializers/base.py", line 69, in serialize
    attrs = serializer.get_attrs(
  File "/usr/local/lib/python3.8/site-packages/sentry/api/serializers/models/project.py", line 716, in get_attrs
    attrs = super().get_attrs(item_list, user)
  File "/usr/local/lib/python3.8/site-packages/sentry/api/serializers/models/project.py", line 603, in get_attrs
    attrs = super().get_attrs(item_list, user)
  File "/usr/local/lib/python3.8/site-packages/sentry/api/serializers/models/project.py", line 358, in get_attrs
    transaction_stats = self.get_stats(project_ids, "event.type:transaction")
  File "/usr/local/lib/python3.8/site-packages/sentry/api/serializers/models/project.py", line 443, in get_stats
    stats = self.dataset.top_events_timeseries(
  File "/usr/local/lib/python3.8/site-packages/sentry/snuba/metrics_enhanced_performance.py", line 222, in top_events_timeseries
    raise error
  File "/usr/local/lib/python3.8/site-packages/sentry/snuba/metrics_enhanced_performance.py", line 197, in top_events_timeseries
    return metrics_top_events_timeseries(
  File "/usr/local/lib/python3.8/site-packages/sentry/snuba/metrics_performance.py", line 321, in top_events_timeseries
    result = top_events_builder.run_query(referrer)
  File "/usr/local/lib/python3.8/site-packages/sentry/search/events/builder/metrics.py", line 1485, in run_query
    results = bulk_snql_query(queries, referrer, use_cache)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/snuba.py", line 827, in bulk_snql_query
    return _apply_cache_and_build_results(params, referrer=referrer, use_cache=use_cache)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/snuba.py", line 880, in _apply_cache_and_build_results
    query_results = _bulk_snuba_query([item[1] for item in to_query], headers)
  File "/usr/local/lib/python3.8/site-packages/sentry/utils/snuba.py", line 963, in _bulk_snuba_query
    raise clickhouse_error_codes_map.get(error["code"], QueryExecutionError)(
sentry.utils.snuba.QueryExecutionError: DB::Exception: Type mismatch in IN or VALUES section. Expected: UInt64. Got: String. Stack trace:
0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
2. ? @ 0xd5449a2 in /usr/bin/clickhouse
3. DB::convertFieldToType(DB::Field const&, DB::IDataType const&, DB::IDataType const*) @ 0xd541db7 in /usr/bin/clickhouse
4. ? @ 0xd30cd60 in /usr/bin/clickhouse
5. DB::makeExplicitSet(DB::ASTFunction const*, DB::Block const&, bool, DB::Context const&, DB::SizeLimits const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >&) @ 0xd30d41b in /usr/bin/clickhouse
6. DB::SelectQueryExpressionAnalyzer::makeSetsForIndex(std::__1::shared_ptr<DB::IAST> const&) @ 0xd2fcff5 in /usr/bin/clickhouse
7. DB::SelectQueryExpressionAnalyzer::makeSetsForIndex(std::__1::shared_ptr<DB::IAST> const&) @ 0xd2fce4b in /usr/bin/clickhouse
8. DB::SelectQueryExpressionAnalyzer::makeSetsForIndex(std::__1::shared_ptr<DB::IAST> const&) @ 0xd2fce4b in /usr/bin/clickhouse
9. DB::InterpreterSelectQuery::getSampleBlockImpl(bool) @ 0xd13891e in /usr/bin/clickhouse
10. ? @ 0xd13a3db in /usr/bin/clickhouse
11. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13b05e in /usr/bin/clickhouse
12. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13c619 in /usr/bin/clickhouse
13. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd341686 in /usr/bin/clickhouse
14. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum) @ 0xd0909b4 in /usr/bin/clickhouse
15. ? @ 0xd550655 in /usr/bin/clickhouse
16. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, bool) @ 0xd553441 in /usr/bin/clickhouse
17. DB::TCPHandler::runImpl() @ 0x9024489 in /usr/bin/clickhouse
18. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
19. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
20. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
21. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
22. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
23. ? @ 0x105c0abd in /usr/bin/clickhouse
24. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
25. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
07:43:27 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.organization_projects.OrganizationProjectsEndpoint' response=500 user_id='10' is_app='None' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/loccioni/projects/' caller_ip='172.18.81.223' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.1514575481414795 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
07:43:27 [ERROR] django.request: Internal Server Error: /api/0/organizations/loccioni/projects/ (status_code=500 request=<WSGIRequest: GET '/api/0/organizations/loccioni/projects/?dataset=metricsEnhanced&query=id%3A20%20id%3A23%20id%3A22%20id%3A21%20id%3A7%20id%3A27%20id%3A3%20id%3A14%20id%3A2&sessionStats=1&statsPeriod=24h&transactionStats=1'>)

Error : DB::Exception: Type mismatch in IN or VALUES section. Expected: UInt64. Got: String

Install log attached:
sentry_install_log-2023-10-17_07-20-04.txt

Event ID

{"detail":"Internal Error","errorId":"6588ce8d6abe4776b5354c1fd21fb90b"}

[hubertdeng123]

I'm also seeing this issue on our side. Thanks for reporting.

[SmoshySmosh]

I'm seeing this as well.

[hubertdeng123]

We're investigating. This seems to be an issue with a specific query that was added. The reason it's erroring is because we're using a different clickhouse version for self-hosted than we're using on SaaS.

[SmoshySmosh]

We're investigating. This seems to be an issue with a specific query that was added. The reason it's erroring is because we're using a different clickhouse version for self-hosted than we're using on SaaS.

Should we look at updating clickhouse to be inline with SaaS to prevent issues like this?

[hubertdeng123]

Should we look at updating clickhouse to be inline with SaaS to prevent issues like this?

Yes, and that's in talks right now.

[hubertdeng123]

In other news, I've verified the fix and will be looking to release a patch fix for this tomorrow.

PR that fixed this: getsentry/sentry#58481

[hubertdeng123]

23.10.1 is out!