*: add InitChunkSize variable as chunk init capacity #8480

lysu · 2018-11-27T14:41:03Z

What problem does this PR solve?

enable chunk grow

What is changed and how it works?

Add init_chunk_size = 32
change max_chunk_size to 1024
global and session cross-fields validation

Check List

Tests

Unit test
Integration test https://github.com/pingcap/tidb-test/pull/711

Code changes

Add init_chunk_size = 32
change max_chunk_size to 1024

Side effects

Possible performance regression(distsql/join/agg/sort will use 1024 because need other pr to change them)

Related changes

Need to be included in the release note

This change is

lysu · 2018-11-27T14:46:21Z

/run-all-tests

shenli · 2018-11-27T14:48:04Z

What's the benefit of enable chunk grow?

lysu · 2018-11-28T02:58:10Z

@shenli enable chunk grow will real enable chunk grow(previous doesn't), the main benefit is it will alloc small chunk for small query, and accelerate the first row return time for big query result.

lysu · 2018-11-28T02:58:36Z

/run-all-tests

zz-jason · 2018-12-10T04:58:33Z

@lysu any update?

lysu · 2018-12-10T05:00:31Z

@zz-jason I will take time in this week, there are other high p thing under hands

lysu · 2018-12-10T05:02:01Z

@zz-jason and there are much test case rely on order that need address

zhouqiang-cl · 2018-12-10T05:18:00Z

/run-all-tests

executor/export_test.go

zhouqiang-cl · 2018-12-10T05:39:04Z

/run-all-tests

lysu · 2018-12-29T09:24:33Z

@zz-jason @jackysp PTAL

sessionctx/variable/session.go

util/mock/context.go

jackysp

LGTM

executor/distsql.go

distsql/select_result.go

distsql/stream.go

util/mock/context.go

zz-jason · 2019-01-02T05:37:29Z

@lysu How about split the code change for "refine distsql batch ctrl" to another PR? Make this PR focused on "enable chunk-grow" and the new session variable.

lysu · 2019-01-02T05:39:51Z

@zz-jason ok~

lysu · 2019-01-03T13:02:28Z

/run-all-tests tidb-test=pr/711

lysu · 2019-01-03T13:19:46Z

/run-common-test tidb-test=pr/711
/run-integration-common-test tidb-test=pr/711

lysu · 2019-01-03T13:22:14Z

/run-unit-test tidb-test=pr/711

lysu · 2019-01-03T13:30:14Z

/run-unit-test tidb-test=pr/711 gofail=etcd-io

executor/aggregate_test.go

sessionctx/variable/session.go

session/session.go

lysu · 2019-01-04T04:45:59Z

/run-all-tests tidb-test=pr/711

codecov-io · 2019-01-04T04:58:30Z

Codecov Report

❗ No coverage uploaded for pull request base (master@243d611). Click here to learn what that means.
The diff coverage is 57.14%.

@@            Coverage Diff            @@
##             master    #8480   +/-   ##
=========================================
  Coverage          ?   67.49%           
=========================================
  Files             ?      363           
  Lines             ?    75134           
  Branches          ?        0           
=========================================
  Hits              ?    50711           
  Misses            ?    19937           
  Partials          ?     4486

Impacted Files	Coverage Δ
sessionctx/variable/sysvar.go	`100% <ø> (ø)`
table/tables/tables.go	`52.81% <0%> (ø)`
statistics/handle.go	`83.43% <100%> (ø)`
session/session.go	`72.82% <100%> (ø)`
executor/executor.go	`66.76% <100%> (ø)`
util/mock/context.go	`21.11% <100%> (ø)`
sessionctx/variable/session.go	`30.84% <33.33%> (ø)`
sessionctx/variable/varsutil.go	`27.67% <50%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 243d611...a7dce62. Read the comment docs.

sessionctx/variable/varsutil.go

session/session_test.go

executor/join_test.go

executor/aggregate_test.go

executor/join_test.go

lysu · 2019-01-04T05:07:03Z

PTAL again if free @zz-jason @eurekaka @XuHuaiyu thx

and also tidb-test https://github.com/pingcap/tidb-test/pull/711

lysu · 2019-01-04T12:09:31Z

@XuHuaiyu @eurekaka PTAL thx

lysu · 2019-01-05T13:01:29Z

@XuHuaiyu @eurekaka PTAL thx

zz-jason

LGTM

zz-jason · 2019-01-05T13:13:38Z

/run-all-tests

lysu · 2019-01-05T13:15:11Z

/run-all-tests tidb-test=pr/711

lysu · 2019-01-05T13:23:31Z

/run-all-tests tidb-test=pr/711

eurekaka

LGTM

lysu added status/WIP sig/execution SIG execution labels Nov 27, 2018

lysu changed the title ~~distsql,executor: enable chunk-grow & refine distsql batch~~ distsql,executor: enable chunk-grow & refine distsql batch ctrl Nov 28, 2018

winoros reviewed Dec 10, 2018

View reviewed changes

executor/export_test.go Outdated Show resolved Hide resolved

lysu added status/all tests passed and removed status/WIP labels Dec 29, 2018

jackysp reviewed Dec 29, 2018

View reviewed changes

sessionctx/variable/session.go Outdated Show resolved Hide resolved

jackysp reviewed Dec 29, 2018

View reviewed changes

util/mock/context.go Outdated Show resolved Hide resolved

jackysp reviewed Jan 2, 2019

View reviewed changes

winoros reviewed Jan 2, 2019

View reviewed changes

executor/distsql.go Outdated Show resolved Hide resolved

zz-jason reviewed Jan 2, 2019

View reviewed changes

distsql/select_result.go Outdated Show resolved Hide resolved

distsql/select_result.go Outdated Show resolved Hide resolved

distsql/stream.go Outdated Show resolved Hide resolved

zz-jason requested review from zz-jason, XuHuaiyu and winoros January 2, 2019 05:10

zz-jason reviewed Jan 2, 2019

View reviewed changes

util/mock/context.go Show resolved Hide resolved

lysu changed the title ~~distsql,executor: enable chunk-grow & refine distsql batch ctrl~~ *: add InitChunkSize variable Jan 2, 2019

lysu changed the title *: add InitChunkSize variable config: add InitChunkSize variable Jan 2, 2019

lysu changed the title ~~config: add InitChunkSize variable~~ config: add InitChunkSize variable as chunk init capacity Jan 2, 2019

lysu removed the status/all tests passed label Jan 3, 2019

zz-jason reviewed Jan 3, 2019

View reviewed changes

executor/aggregate_test.go Outdated Show resolved Hide resolved

sessionctx/variable/session.go Outdated Show resolved Hide resolved

sessionctx/variable/session.go Outdated Show resolved Hide resolved

eurekaka reviewed Jan 4, 2019

View reviewed changes

session/session.go Outdated Show resolved Hide resolved

session/session.go Outdated Show resolved Hide resolved

session/session.go Outdated Show resolved Hide resolved

session/session.go Outdated Show resolved Hide resolved

lysu added the status/all tests passed label Jan 4, 2019

XuHuaiyu reviewed Jan 4, 2019

View reviewed changes

XuHuaiyu changed the title ~~config: add InitChunkSize variable as chunk init capacity~~ *: add InitChunkSize variable as chunk init capacity Jan 4, 2019

config: add initChunkSize config item, make chunk start with 32

2ea94f7

zz-jason reviewed Jan 5, 2019

View reviewed changes

zz-jason added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 5, 2019

zz-jason requested review from XuHuaiyu and eurekaka January 5, 2019 13:13

Merge branch 'master' into dev-chunk-grow-distsql

913f252

eurekaka approved these changes Jan 7, 2019

View reviewed changes

eurekaka added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 7, 2019

eurekaka and others added 2 commits January 7, 2019 10:26

Merge branch 'master' into dev-chunk-grow-distsql

e471959

Merge branch 'master' into dev-chunk-grow-distsql

a7dce62

jackysp merged commit 084aa5a into pingcap:master Jan 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add InitChunkSize variable as chunk init capacity #8480

*: add InitChunkSize variable as chunk init capacity #8480

lysu commented Nov 27, 2018 •

edited

Loading

lysu commented Nov 27, 2018

shenli commented Nov 27, 2018

lysu commented Nov 28, 2018

lysu commented Nov 28, 2018

zz-jason commented Dec 10, 2018

lysu commented Dec 10, 2018

lysu commented Dec 10, 2018

zhouqiang-cl commented Dec 10, 2018

zhouqiang-cl commented Dec 10, 2018

lysu commented Dec 29, 2018

jackysp left a comment

zz-jason commented Jan 2, 2019

lysu commented Jan 2, 2019

lysu commented Jan 3, 2019

lysu commented Jan 3, 2019

lysu commented Jan 3, 2019

lysu commented Jan 3, 2019

lysu commented Jan 4, 2019

codecov-io commented Jan 4, 2019 •

edited

Loading

lysu commented Jan 4, 2019

lysu commented Jan 4, 2019

lysu commented Jan 5, 2019

zz-jason left a comment

zz-jason commented Jan 5, 2019

lysu commented Jan 5, 2019

lysu commented Jan 5, 2019

eurekaka left a comment

*: add InitChunkSize variable as chunk init capacity #8480

*: add InitChunkSize variable as chunk init capacity #8480

Conversation

lysu commented Nov 27, 2018 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

lysu commented Nov 27, 2018

shenli commented Nov 27, 2018

lysu commented Nov 28, 2018

lysu commented Nov 28, 2018

zz-jason commented Dec 10, 2018

lysu commented Dec 10, 2018

lysu commented Dec 10, 2018

zhouqiang-cl commented Dec 10, 2018

zhouqiang-cl commented Dec 10, 2018

lysu commented Dec 29, 2018

jackysp left a comment

Choose a reason for hiding this comment

zz-jason commented Jan 2, 2019

lysu commented Jan 2, 2019

lysu commented Jan 3, 2019

lysu commented Jan 3, 2019

lysu commented Jan 3, 2019

lysu commented Jan 3, 2019

lysu commented Jan 4, 2019

codecov-io commented Jan 4, 2019 • edited Loading

Codecov Report

lysu commented Jan 4, 2019

lysu commented Jan 4, 2019

lysu commented Jan 5, 2019

zz-jason left a comment

Choose a reason for hiding this comment

zz-jason commented Jan 5, 2019

lysu commented Jan 5, 2019

lysu commented Jan 5, 2019

eurekaka left a comment

Choose a reason for hiding this comment

lysu commented Nov 27, 2018 •

edited

Loading

codecov-io commented Jan 4, 2019 •

edited

Loading