标签：13793 elements returned 16 idea sparkstreaming kafka spark Poll
spark jira 用中文的同学看了我笑了，牛啊
Unexpected delay before fetch response transmission
Details
Type:Bug
Status:Open
Priority: Major
Resolution: Unresolved
Affects Version/s:
0.9.0.0, 0.9.0.1, 0.10.0.0, (3)
0.10.0.1, 0.10.1.0, 0.10.1.1
Fix Version/s:None
Component/s:None
Labels:None
Description
From the user list, Krzysztof Ciesielski reports the following:
Scenario description:
First, a producer writes 500000 elements into a topic
Then, a consumer starts to read, polling in a loop.
When "max.partition.fetch.bytes" is set to a relatively small value, each
"consumer.poll()" returns a batch of messages.
If this value is left as default, the output tends to look like this:
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 13793 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 0 elements
Poll returned 13793 elements
Poll returned 13793 elements
As we can see, there are weird "gaps" when poll returns 0 elements for some
time. What is the reason for that? Maybe there are some good practices
about setting "max.partition.fetch.bytes" which I don't follow?
The gist to reproduce this problem is here: https://gist.github.com/kciesielski/054bb4359a318aa17561.
After some initial investigation, the delay appears to be in the server’s networking layer. Basically I see a delay of 5 seconds from the time that Selector.send() is invoked in SocketServer.Processor with the fetch response to the time that the send is completed. Using netstat in the middle of the delay shows the following output:
tcp4 0 0 10.191.0.30.55455 10.191.0.30.9092 ESTABLISHED
tcp4 0 102400 10.191.0.30.9092 10.191.0.30.55454 ESTABLISHED
From this, it looks like the data reaches the send buffer, but needs to be flushed.
rangadi Raghu Angadi added a comment - 17/Mar/16 07:37
This shows server is ok. It is trying to send and already flushed/wrote on its end of the tcp connection. If you check netstat for the side of the connection (local port 55454) you should see data in receive buffers. Still don’t know why consumer is not reading.
I am also seeing similar weird behavior. with 64K receiver buffer reduces the intensity of the problem, but does not fix it.
Jason Gustafson added a comment - 28/Mar/16 22:12
I made a little progress understanding this problem. The 5 second pause that we’re seeing is caused by the TCP persist timer, which kicks in when the remote window size gets small enough. You can see this by watching the traffic through tcpdump and confirming the persist timeout event with ‘netstat -s’. Clearly there are some cases where the client is not reading the data off the socket fast enough, which is causing the window size to fall and the server to back off.
解决办法：
2021-03-17 11:16:56,711 WARN — [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding enable.auto.commit to false for executor
2021-03-17 11:16:56,714 WARN — [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding auto.offset.reset to none for executor
2021-03-17 11:16:56,714 WARN — [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding executor group.id to spark-executor-recommender
2021-03-17 11:16:56,715 WARN — [ main] org.apache.spark.streaming.kafka010.KafkaUtils (line: 66) : overriding receive.buffer.bytes to 65536
主要还是设置receive.buffer.bytes =65536
标签：13793,elements,returned,16,idea,sparkstreaming,kafka,spark,Poll 来源： https://blog.csdn.net/anddyhua/article/details/116354956
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。