标签：tmp broadcast join 300 braodcast sql Spark spark files
BroadCast TimeOut 300.
org.apache.spark.SparkException: Could not execute broadcast in 300 secs.
You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting spark.sql.autoBroadcastJoinThreshold to -1
一个stt的job被Block之后，经查日志发现有两个表在广播的时候出现了broadcast TimeOut的问题。
Sql logic:
create or replace temporary view table_tmp
as
select
/** BROADCAST(A_tmp) */
A_tmp.c1,
A_tmp.c2,
A_tmp.c3,
...
from source_table A_tmp
join gdw_table.other_table1 B_tmp on ...
join gdw_table.other_table2 C_tmp on ...
group by 1,2,3,4,5,6,...11
检查了一下
A_tmp的数据量，有238 rows 数据。
B_tmp的数据量，有1834203873 rows 数据。
C_tmp的数据量，有289371375 rows 的数据。
| Property Name | Default | Meaning |
|---|---|---|
| spark.sql.files.maxPartitionBytes | 134217728 (128 MB) | The maximum number of bytes to pack into a single partition when reading files. |
| spark.sql.files.openCostInBytes | 4194304 (4 MB) | The estimated cost to open a file, measured by the number of bytes could be scanned in the same time. This is used when putting multiple files into a partition. It is better to over estimated, then the partitions with small files will be faster than partitions with bigger files (which is scheduled first). |
| spark.sql.broadcastTimeout | 300 | Timeout in seconds for the broadcast wait time in broadcast joins |
| spark.sql.autoBroadcastJoinThreshold | 10485760 (10 MB) | Configures the maximum size in bytes for a table that will be broadcast to all worker nodes when performing a join. By setting this value to -1 broadcasting can be disabled. Note that currently statistics are only supported for Hive Metastore tables where the command ANALYZE TABLE |
| spark.sql.shuffle.partitions | 200 | Configures the number of partitions to use when shuffling data for joins or aggregations. |
问题产生原因：
可能Job的执行时间是美国的凌晨一点，会有大量的资源被占用，资源紧缺，导致没有在规定时间内完成join并相应。
解决方案：
- 增大默认Timeout时间，set spark.sql.broadcastTimeout = 600.
- persist(), persist() 可以使broadcastHashjoin 变成 shuffleHashJoin.
- 关闭broadcast join set spark.sql.autoBroadcastJoinThreshold = -1
- reset job.
选择了一个其他的时间重新执行该job，成功执行。
标签：tmp,broadcast,join,300,braodcast,sql,Spark,spark,files 来源： https://www.cnblogs.com/yuexiuping/p/15043556.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。