Tags: cloudera/Impala
Tags
IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 hdfs:https:// URIs do not work on Impala's S3 tests. The fix is to drop the hdfs:https:// part of the URI. Our tests run against the default filesystem, so dropping the hdfs:https:// works on both HDFS test setups and S3 test setups. Testing: - Ran tests on S3 - Ran test_owner_privileges.py on HDFS locally Change-Id: Id7be961094f03c5fde3f8b4c340814d26470818a Reviewed-on: https://gerrit.cloudera.org:8080/14334 Reviewed-by: Michael Ho <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-on: https://gerrit.sjc.cloudera.com/c/cdh/impala/+/66469 API-Compat: Gabor Solya <[email protected]> CDH-Build: Gabor Solya <[email protected]> Reviewed-by: Gabor Solya <[email protected]> Quasar-L0: Gabor Solya <[email protected]> Tested-by: Gabor Solya <[email protected]>
CLOUDERA-BUILD. Preparing for 6.3.2 release. Commit performed on https://master-01.jenkins.cloudera.com/ Signed-off-by: cloudera <[email protected]>
CLOUDERA-BUILD. Preparing for 6.3.1 release. Commit performed on https://master-01.jenkins.cloudera.com/ Signed-off-by: cloudera <[email protected]>
CLOUDERA-BUILD. Preparing for 6.2.1 release. Commit performed on https://master-01.jenkins.cloudera.com/ Signed-off-by: cloudera <[email protected]>
CLOUDERA-BUILD. Preparing for 6.3.0 release. Commit performed on https://master-01.jenkins.cloudera.com/ Signed-off-by: cloudera <[email protected]>
IMPALA-8270: fix MemTracker teardown in FeSupport This patch tries to simplify and standardise the order in which control structures are torn down. As a consequence the bug is fixed. I've described the bug below. The changes are: * Make more control structures owned directly by QueryState::obj_pool_, so that they are all destroyed at the same time via ~QueryState. * Tear down local_query_state_ explicitly before other destructors run. Either change is sufficient to fix the bug, but I preferred to do both to reduce the chances of similar bugs in future. Description of bug: =================== In the normal query execution flow: - RuntimeState is in QueryState::obj_pool_ - RuntimeState owns RuntimeState::instance_mem_tracker_ via scoped_ptr - QueryState::query_mem_tracker_ is in QueryState::obj_pool_ - QueryState::query_mem_tracker_ has a reference to RuntimeState::instance_mem_tracker_ The tear-down works because ~QueryState unregisters query_mem_tracker_ from its parent, making the whole subtree unreachable before destroying QueryState::obj_pool_. It is thus safe to destroy instance_mem_tracker_ along with the rest of obj_pool_. FeSupport messes this up by having RuntimeState own the QueryState RuntimeState::local_query_state_ via a scoped_ptr, and the implied destructor order means that RuntimeState::instance_mem_tracker_ is destroyed before RuntimeState::local_query_state_, which breaks the above flow and the destroyed instance_mem_tracker_ is reachable from the process MemTracker via QueryState::query_mem_tracker_ for a small window until it is unregistered. Testing: Added a backend test that reproduced the ASAN use-after-free failure when run against unmodified RuntimeState code. I did not make it a unified backend test so that it would be easier to backport this fix to older versions that don't have unified tests. Change-Id: If815130cd4db00917746f10b28514f779ee254f0 Reviewed-on: https://gerrit.cloudera.org:8080/13057 Reviewed-by: Tim Armstrong <[email protected]> Reviewed-by: Joe McDonnell <[email protected]> Reviewed-by: Bikramjeet Vig <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-on: https://gerrit.sjc.cloudera.com/c/Impala/+/44586 Tested-by: Tim Armstrong <[email protected]> (cherry picked from commit ec0c0050b390c3e78ca0b9c7052e6e25d227509a) Reviewed-on: https://gerrit.sjc.cloudera.com/c/Impala/+/44664
IMPALA-7972 Detect self-events to avoid unnecessary invalidates This patch adds support to detect self-generated events from catalog. This is used to avoid unnecessary invalidates to the tables from such self-events. Currently, alter_table, alter_partition, add_partition and drop_partition event types can invalidate the table metadata. Originally, we planned to have a global version number support from metastore (see HIVE-21115). But since that is still not complete, we rely on a combination of other identifiers to determine if a event is self-generated or not. These self-event identifiers consists of values from the table/partition parameters. A catalog service uuid and the catalog version number. The uuid is generated for each catalogservice when it comes up and it adds it to the table/partition parameters with the key "impala.CatalogServiceId". The catalog version number is added with the key "impala.CatalogVersion". When catalog executes a DDL operation it appends the current catalog version to the list of version numbers for the in-flight events for the table. Events processor clears this version when the corresponding version number identified by serviceId is received in the event. This is needed since it is possible that a external non-Impala system which generates the event presents the same serviceId and version number later on. The algorithm to detect a self-event is as below. 1. Add the service id and expected catalog version to table/partition parameters when executing the DDL operation. When the HMS operation is successful, add the version number to the list of version for in-flight events at table level. 2. When the event is received, the first time you see the combination of serviceId and version number, event processor clears the version number from table's list and determines the event as self-generated (and hence ignored) 3. If the event data presents a unknown serviceId or if the version number is not present in the list of in-flight versions, event is not a self-event and needs to be processed. In order to limit the total memory footprint, only 10 version numbers are stored at the table. Since the event processor is expected to poll every few seconds this should be a reasonable bound which satisfies most use-cases. Otherwise, event processor may wrongly process a self-event to invalidate the table. In such a case, its a performance penalty not a correctness issue. In case of drop_partition event, the partition object is not available in the event. Hence we cannot determine if its a self-event. In such cases currently we always issue a invalidate command. This is a known limitation and will be improved in IMPALA-7973 Patch adds new tests to trigger alter table/partition DDLs from impala and makes sure that the table is not invalidated. Change-Id: I29e487b903ba7493f90575c26c44a0019ff0362e Reviewed-on: https://gerrit.sjc.cloudera.com/42671 Tested-by: Jenkins User <[email protected]> CDH-Build: Jenkins User <[email protected]> Quasar-L0: Jenkins User <[email protected]> Reviewed-by: Vihang Karajgaonkar <[email protected]>
IMPALA-7943: Bump the default client timeout set on impala-shell As part of IMPALA-7555, we added a default socket timeout of 5 seconds when connecting to an impalad. Under heavy load with kerberos and SSL enabled, we could hit this default timeout. This change bumps up the timeout to 60 secs to make the impala-shell more robust. Change-Id: Ifc40069e86cbf93634320804efba003fb5551afe Reviewed-on: https://gerrit.cloudera.org:8080/12051 Reviewed-by: Impala Public Jenkins <[email protected]> Reviewed-by: Lars Volker <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> (cherry picked from commit c2627ba) Reviewed-on: https://gerrit.sjc.cloudera.com/41169 Tested-by: Jenkins User <[email protected]> CDH-Build: Jenkins User <[email protected]> Quasar-L0: Laszlo Gaal <[email protected]> Reviewed-by: Laszlo Gaal <[email protected]> (cherry picked from commit 3794d20f1ef24435b575aba56bda2023cee60fa4) Reviewed-on: https://gerrit.sjc.cloudera.com/41201 Quasar-L0: Jenkins User <[email protected]>
Updated save-version.sh for CDH5.16.1 Change-Id: Iddcd9cddc6c3529ebb4442fd4cd7ff30ed512a83 Reviewed-on: https://gerrit.sjc.cloudera.com/39721 Reviewed-by: Bernadett Balogh <[email protected]> Tested-by: Bernadett Balogh <[email protected]>
PreviousNext