服务器内核报错kernel: TCP: time wait bucket table overflow

wgkgood

今天线上一台Nginx web服务器，随着访问的人数越来越多，突然出现如下报错：Oh my god。。
Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow

1. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
2. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
3. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
4. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
5. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
6. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
7. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
8. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
9. Apr 15 10:03:25 PT-xxxx kernel: TCP: time wait bucket table overflow
   
10. Apr 15 10:03:40 PT-xxxx kernel: __ratelimit: 57 callbacks suppressed
    
11. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
12. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
13. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
14. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
15. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
16. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
17. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
18. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
19. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow
    
20. Apr 15 10:03:40 PT-xxxx kernel: TCP: time wait bucket table overflow

复制代码

这个问题怎么排查和解决呢？

wgkgood

原因是因为：内核tcp_max_tw_buckets的值太小，配置的为10000
解决方法：增大 tcp_max_tw_buckets的值，并不是这个值越小越好，中TIME_WAIT 大部是由Nginx产生的，是属于正常的现象。
需要修改vi /etc/sysctl.conf
net.ipv4.tcp_max_tw_buckets = 20000
sysctl -p 执行后，再查看tail -fn 100 /var/log/message 日志，错误消除。