标签：优化 ORDER block 排序 Expression order clickhouse Before
一、MergeTree order by执行计划
1）没有order by的执行计划：
+-------------------------------------------------------------------------------------------------+ | explain | +-------------------------------------------------------------------------------------------------+ | Expression ((Projection + Before ORDER BY)) | | Aggregating | | Expression ((Before GROUP BY + Projection)) | | SettingQuotaAndLimits (Set limits and quota after reading from storage) | | Union | | Expression ((Convert block structure for query from local replica + Before ORDER BY)) | | SettingQuotaAndLimits (Set limits and quota after reading from storage) | | ReadFromStorage (MergeTree) | | ReadFromPreparedSource (Read from remote replica) | +-------------------------------------------------------------------------------------------------+ 9 rows in set (0.05 sec)
流程：
第一步：从每个机器的硬盘上把数据读取出来
第二步：union一下每个机器读取的数据
第三步：求count
2）拥有order by的执行计划
+-------------------------------------------------------------------------------------------+
| explain |
+-------------------------------------------------------------------------------------------+
| Expression ((Projection + Before ORDER BY)) |
| Aggregating |
| Expression ((Before GROUP BY + Projection)) |
| MergingSorted (Merge sorted streams for ORDER BY) |
| SettingQuotaAndLimits (Set limits and quota after reading from storage) |
| Union |
| Expression (Convert block structure for query from local replica) |
| FinishSorting |
| Expression (Before ORDER BY) |
| SettingQuotaAndLimits (Set limits and quota after reading from storage) |
| Expression (Remove unused columns after reading from storage) |
| Union |
| MergingSorted (Merge sorting mark ranges) |
| Expression (Calculate sorting key prefix) |
| ReadFromStorage (MergeTree with order) |
| MergingSorted (Merge sorting mark ranges) |
| Expression (Calculate sorting key prefix) |
| ReadFromStorage (MergeTree with order) |
| MergingSorted (Merge sorting mark ranges) |
| Expression (Calculate sorting key prefix) |
| ReadFromStorage (MergeTree with order)
...................
--省略一摸一样的N行
...................
流程：
1、从硬盘读取需要的数据（部分，因为order by需要在内存里面快速排序，无法读取全部）
2、按照order by 的key进行排序
3、N多个order by排序完的数据，在做最终汇总，然后对汇总后的数据在做排序（这一步也会根据数据量分成多步完成）
4、最终做聚合求count
二、优化
1.切换引擎
采用最适合order by的引擎：*AggregatingMergeTree
1)没有order by的执行计划
+-------------------------------------------------------------------------------------------------+ | explain | +-------------------------------------------------------------------------------------------------+ | Expression ((Projection + Before ORDER BY)) | | Aggregating | | Expression ((Before GROUP BY + Projection)) | | SettingQuotaAndLimits (Set limits and quota after reading from storage) | | Union | | Expression ((Convert block structure for query from local replica + Before ORDER BY)) | | SettingQuotaAndLimits (Set limits and quota after reading from storage) | | ReadFromStorage (MergeTree) | | ReadFromPreparedSource (Read from remote replica) | +-------------------------------------------------------------------------------------------------+ 9 rows in set (0.05 sec)
不带order by的执行计划MergeTree的引擎一样。
2)拥有order by的执行计划
+-----------------------------------------------------------------------------------------------+ | explain | +-----------------------------------------------------------------------------------------------+ | Expression ((Projection + Before ORDER BY)) | | Aggregating | | Expression ((Before GROUP BY + Projection)) | | MergingSorted (Merge sorted streams for ORDER BY) | | SettingQuotaAndLimits (Set limits and quota after reading from storage) | | Union | | Expression (Convert block structure for query from local replica) | | MergingSorted (Merge sorted streams for ORDER BY) | | MergeSorting (Merge sorted blocks for ORDER BY) | | PartialSorting (Sort each block for ORDER BY) | | Expression (Before ORDER BY) | | SettingQuotaAndLimits (Set limits and quota after reading from storage) | | ReadFromStorage (MergeTree) | | ReadFromPreparedSource (Read from remote replica) | +-----------------------------------------------------------------------------------------------+ 14 rows in set (0.08 sec)
分机器/分区/分段的读取数据做排序，在汇总 ， 并且是按照block进行排序的
流程：
1、对每台机器，按照一定范围进行PartialSorting（不按照具体数据排序，而是按照block块排序）
2、对第一步排序好的block进行汇总排序（因为已经知道第一步block的顺序了,所以汇总的时候直接对比最大block和最小block就可以了）
3、本地的排序好了，在做分布式的排序进行最终的汇总排序
4、求count
上面的操作类似归并排序，适应大数据量分布式排序
2.进一步对sql做优化
1)进行分段sql拆分，然后结果合并
例如原来的sql是查询3个月的数据量，那么还可以进行分段查，比如每半月/一个月 做一次order by
原来的sql：
select count(1) from (select * from 表 where dt >= '2022-01-01' <= '2022-03-31' order by order_key排序条件)
优化后的sql：
select count(1) from (
select * from 表 where dt>='2021-01-01' and dt <= '2022-01-31' order by order_key排序条件
union all
select * from 表 where dt>='2021-02-01' and dt <= '2022-02-28' order by order_key排序条件
union all
select *from 表 where dt>='2021-03-01' and dt <= '2022-03-31' order by order_key排序条件
)
标签：优化,ORDER,block,排序,Expression,order,clickhouse,Before 来源： https://www.cnblogs.com/MrYang-11-GetKnow/p/16506174.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。