Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enchement](mc)opt maxcompute read arrow skip repeated check of isNull #45989

Merged
merged 1 commit into from
Dec 26, 2024
Merged

[enchement](mc)opt maxcompute read arrow skip repeated check of isNull #45989

merged 1 commit into from
Dec 26, 2024
+7 −1

Conversation

hubgeter
Copy link
Contributor

What problem does this PR solve?

Problem Summary:
Set MaxCompute Arrow NullCheckingForGet.NULL_CHECKING_ENABLED false.We will call isNull() before calling getXXX(), so we can set this parameter to skip the repeated check of isNull().

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32606 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 27990a8759c7e1fb53f54f2d31c8429422e9b9a9, data reload: false

------ Round 1 ----------------------------------
q1	17596	6142	6078	6078
q2	2056	303	168	168
q3	10506	1257	718	718
q4	10221	872	444	444
q5	7546	2206	1948	1948
q6	204	180	145	145
q7	894	742	608	608
q8	9246	1348	1145	1145
q9	5186	4912	4918	4912
q10	6771	2315	1849	1849
q11	476	275	277	275
q12	347	352	221	221
q13	17791	3565	2984	2984
q14	241	233	216	216
q15	566	508	491	491
q16	618	640	596	596
q17	576	846	332	332
q18	7133	6403	6426	6403
q19	2000	966	543	543
q20	306	306	183	183
q21	2752	2264	2026	2026
q22	361	358	321	321
Total cold run time: 103393 ms
Total hot run time: 32606 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6293	6221	6237	6221
q2	235	324	238	238
q3	2274	2603	2346	2346
q4	1434	1807	1367	1367
q5	4405	4739	4728	4728
q6	193	177	145	145
q7	2120	1946	1851	1851
q8	2610	2764	2630	2630
q9	7309	7214	7148	7148
q10	3060	3301	2815	2815
q11	589	517	505	505
q12	677	751	590	590
q13	3357	3727	3095	3095
q14	283	311	290	290
q15	564	531	507	507
q16	655	701	620	620
q17	1196	1728	1246	1246
q18	7607	7585	7137	7137
q19	791	925	1083	925
q20	1872	1973	1824	1824
q21	5351	5148	4787	4787
q22	607	627	627	627
Total cold run time: 53482 ms
Total hot run time: 51642 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191104 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 27990a8759c7e1fb53f54f2d31c8429422e9b9a9, data reload: false

query1	969	389	373	373
query2	6515	2355	2259	2259
query3	6715	212	213	212
query4	33935	23781	23422	23422
query5	4386	609	463	463
query6	286	203	182	182
query7	4624	488	306	306
query8	310	259	238	238
query9	9511	2763	2735	2735
query10	454	313	252	252
query11	18322	15379	15318	15318
query12	159	107	104	104
query13	1665	530	416	416
query14	9998	7331	7365	7331
query15	249	194	173	173
query16	8140	592	431	431
query17	1590	747	586	586
query18	2136	404	325	325
query19	216	178	157	157
query20	115	111	106	106
query21	208	122	99	99
query22	4141	4344	4231	4231
query23	34413	34270	33687	33687
query24	6317	2334	2272	2272
query25	502	444	379	379
query26	1042	271	151	151
query27	2046	462	328	328
query28	5273	2474	2468	2468
query29	734	550	414	414
query30	226	183	158	158
query31	968	891	812	812
query32	95	63	62	62
query33	490	344	289	289
query34	763	822	511	511
query35	804	835	742	742
query36	1009	1052	978	978
query37	118	105	79	79
query38	4060	4225	4247	4225
query39	1493	1436	1424	1424
query40	201	120	96	96
query41	45	43	45	43
query42	110	102	103	102
query43	514	507	484	484
query44	1351	813	801	801
query45	177	174	169	169
query46	858	1040	639	639
query47	1913	1891	1827	1827
query48	379	398	324	324
query49	752	492	405	405
query50	627	634	394	394
query51	7211	7142	7180	7142
query52	97	96	91	91
query53	226	246	185	185
query54	474	480	422	422
query55	78	81	81	81
query56	253	263	223	223
query57	1209	1195	1112	1112
query58	231	222	243	222
query59	3058	3032	2901	2901
query60	268	256	246	246
query61	106	105	110	105
query62	888	829	751	751
query63	226	187	183	183
query64	4417	1007	641	641
query65	3302	3214	3231	3214
query66	784	413	306	306
query67	15926	15722	15558	15558
query68	9759	750	522	522
query69	445	288	260	260
query70	1233	1132	1137	1132
query71	428	282	261	261
query72	5717	3883	3925	3883
query73	1163	755	365	365
query74	9807	9310	9080	9080
query75	4583	3140	2692	2692
query76	5565	1163	782	782
query77	1007	376	272	272
query78	10213	10211	9372	9372
query79	3598	888	573	573
query80	776	526	447	447
query81	488	270	232	232
query82	359	140	133	133
query83	193	168	149	149
query84	278	91	73	73
query85	759	363	301	301
query86	342	305	303	303
query87	4385	4557	4354	4354
query88	3065	2265	2225	2225
query89	428	339	291	291
query90	2090	192	187	187
query91	138	130	106	106
query92	69	58	52	52
query93	2180	879	533	533
query94	690	389	297	297
query95	326	259	256	256
query96	485	602	282	282
query97	2716	2771	2717	2717
query98	229	199	202	199
query99	1671	1574	1408	1408
Total cold run time: 295863 ms
Total hot run time: 191104 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 27990a8759c7e1fb53f54f2d31c8429422e9b9a9, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.41	0.41	0.41
query6	1.15	0.65	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.50
query10	0.55	0.60	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.72	2.70	2.75
query15	0.89	0.82	0.82
query16	0.38	0.38	0.39
query17	1.07	1.03	1.02
query18	0.23	0.22	0.20
query19	1.89	1.77	2.03
query20	0.01	0.01	0.01
query21	15.37	0.92	0.60
query22	0.76	0.79	0.79
query23	15.17	1.47	0.52
query24	2.95	0.85	1.31
query25	0.12	0.11	0.25
query26	0.31	0.15	0.14
query27	0.05	0.05	0.06
query28	14.07	1.54	1.05
query29	12.62	3.95	3.33
query30	0.26	0.10	0.07
query31	2.81	0.59	0.39
query32	3.22	0.54	0.46
query33	3.12	3.09	3.05
query34	16.67	5.10	4.45
query35	4.51	4.47	4.45
query36	0.67	0.49	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.13	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.02	0.03
Total cold run time: 105.98 s
Total hot run time: 31.21 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit e53f6a1 into apache:master Dec 26, 2024
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 26, 2024
@hubgeter
[enchement](mc)opt maxcompute read arrow skip repeated check of isNull (
166dc2d 
#45989)

### What problem does this PR solve?
Problem Summary:
Set MaxCompute Arrow `NullCheckingForGet.NULL_CHECKING_ENABLED` false.We
will call isNull() before calling getXXX(), so we can set this parameter
to skip the repeated check of isNull().
@github-actions github-actions bot mentioned this pull request Dec 26, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Dec 26, 2024
@hubgeter
[enchement](mc)opt maxcompute read arrow skip repeated check of isNull (
ec30b31 
#45989)

### What problem does this PR solve?
Problem Summary:
Set MaxCompute Arrow `NullCheckingForGet.NULL_CHECKING_ENABLED` false.We
will call isNull() before calling getXXX(), so we can set this parameter
to skip the repeated check of isNull().
@github-actions github-actions bot mentioned this pull request Dec 26, 2024
Merged
morningman pushed a commit that referenced this pull request Dec 26, 2024
@github-actions @hubgeter
branch-3.0: [enchement](mc)opt maxcompute read arrow skip repeated ch…
86369bb 
…eck of isNull #45989 (#46022)

Cherry-picked from #45989

Co-authored-by: daidai <changyuwei@selectdb.com>
morningman pushed a commit that referenced this pull request Dec 26, 2024
@github-actions @hubgeter
branch-2.1: [enchement](mc)opt maxcompute read arrow skip repeated ch…
6f8d2bf 
…eck of isNull #45989 (#46023)

Cherry-picked from #45989

Co-authored-by: daidai <changyuwei@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.x-conflict dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hubgeter @hello-stephen @doris-robot @morningman @CalvinKirs