Why My WebSocket Service Was Timing Out (And How I Fixed It)
I came back from lunch to find our chat service running slow. After some detective work, I discovered Cloudflare was silently blocking my WebSocket port!
We have several projects, and we moved all our domain names to Cloudflare, which uses proxying by default. As a result, the actual IP addresses of our servers are hidden. Everything was running smoothly until I came back from lunch one day and found a task assigned to me: the chat service was responding slowly, with noticeable latency.
I started investigating the problem:
- First, I checked the API endpoint that retrieves messages — it was indeed slow.
- I then checked the database query — it was fast.
- Next, I reviewed the code and noticed that it triggers an event in the WebSocket service.
- I had the idea to temporarily prevent this event from firing to see if the latency would improve — and it did. This confirmed that the WebSocket service was the source of the problem.
I then started debugging the WebSocket service:
- I checked the SSL certificate and verified that the service was running — everything looked fine.
- However, the service logs showed an ETIMEDOUT error, caused by a timeout. I had never seen this error before.
- I stopped and restarted the service multiple times, but the error persisted.
- I tried connecting to the WebSocket via Postman, but the connection still timed out.
- Finally, I used curl (curl -vk tls://domainname:9095) and noticed something important: the IP address I was connecting to was not the server’s actual IP. This made sense because Cloudflare hides the server’s real IP and does not allow all ports to be proxied.
Root cause: The WebSocket service was running on port 9095, which is not supported by Cloudflare. No wonder it kept timing out — it was like trying to sneak through a locked door.
Solution: I switched the WebSocket service to a port supported by Cloudflare, and just like magic, the problem was resolved. The chat service was back to its speedy self, and I could finally go back to thinking about lunch instead of timeouts.