Designing SYN Flood Mitigation for the Real Internet

Overview

Nexusguard’s mission is to protect our customers from disruptive DDoS attacks while preserving application performance. One of the most common threats we face is the TCP SYN flood. This attack targets the handshake stage of the Transmission Control Protocol (TCP), sending a barrage of SYN segments in an attempt to exhaust the server’s resources. Attackers often spoof the source addresses, so the targeted server invests resources into half‑open connections with hosts that will never respond. In extreme cases these SYNs can also be used to reflect traffic toward unsuspecting victims by exploiting the server’s willingness to answer every handshake.

‍

To counter these attacks our on‑premise Bastions deliberately drop the first SYN and only accept connections when a client proves its legitimacy by retransmitting. This approach eliminates the risk of the Bastions becoming a reflector and retains full TCP functionality for legitimate users. The following sections explain the nature of SYN floods, compare stateless SYN cookie mitigation with retransmission‑based validation, and discuss why we favour the latter.

How SYN Floods Work

The three‑way handshake begins when a client sends a SYN, progresses when the server replies with a SYN‑ACK, and completes when the client acknowledges with an ACK. Under normal conditions this handshake both synchronises initial sequence numbers and negotiates options like maximum segment size and window scaling. During a SYN flood the attacker sends a huge number of SYNs, often with forged addresses, to tie up the server’s half‑open connection queue. Since the server has no way of verifying the source before it responds, it dutifully sends SYN‑ACK packets and waits for acknowledgements that never arrive. The server will retransmit the SYN‑ACK if the expected ACK is not received, as required by the TCP specification. When the spoofed addresses belong to a third party, these repeated SYN‑ACKs become a reflection attack, amplifying the flood by forcing the server to relay traffic toward the victim. This amplification depends on how many times the server retransmits its SYN‑ACK.

‍

The diagram below illustrates a SYN‑ACK reflection: a botnet directs spoofed SYNs at a public server, causing that server to send repeated SYN‑ACKs to the unsuspecting victim. Because the victim never finishes the handshake, the server keeps retransmitting.

SYN Cookies: Advantages and Drawbacks

One widely adopted defence against SYN floods is the SYN cookie algorithm. When a server’s backlog is overwhelmed, instead of allocating a half‑open connection it encodes connection parameters into the sequence number of the SYN‑ACK, discards its state, and waits for the client’s ACK. The cookie is constructed from a limited 32‑bit field; only a few bits are available to represent parameters like the maximum segment size and window scaling. Because of this limitation the server cannot remember many optional TCP features and will negotiate conservative defaults. As a result, connections established via SYN cookies may have smaller windows, lack timestamps or ECN support, and cannot use newer options. Furthermore, when the SYN‑ACK itself is lost, the server using pure SYN cookies does not retransmit; the client must resend its SYN to reinitiate the handshake. Finally, since the server replies to all SYNs (including spoofed ones) with a SYN‑ACK, it can be abused as a reflector during a distributed attack.

‍

Despite these limitations, SYN cookies provide a stateless mechanism to maintain availability during a flood. Because the server does not allocate memory for half‑open connections, its resources cannot be exhausted. For some deployments this trade‑off is acceptable, especially where memory is scarce or the server is positioned at an upstream transit point. However, for Nexusguard’s Bastions, which operate at the customer’s edge with ample resources and the need to preserve full TCP performance, this compromise is undesirable.

Retransmission‑Based Validation

Our alternative is to intentionally drop the first SYN and only proceed with the handshake when the client retransmits. TCP clients are designed to retry when acknowledgements are not received; legitimate clients will therefore send a second SYN after a timeout, whereas spoofing tools cannot because they never see the server’s response. When the Bastions observes the retransmitted SYN, it forwards the request to the protected server and allows the SYN‑ACK to flow back to the client. The handshake then completes normally, preserving all negotiated options. This technique filters out forged SYNs without sending any traffic to the spoofed address, eliminating the Bastions’s role in reflection. Because the Bastion only forwards connections that have been retried, the probability of a false positive is extremely low and legitimate users experience only a small delay.

‍

We visualise the difference below. In the SYN cookie approach the server replies to every SYN immediately with an encoded SYN‑ACK, sacrificing some optional features. In the retransmission approach the Bastions drops the first SYN; only when the client retries does it pass the SYN and allow the handshake to continue.

Why Nexusguard Chooses Retransmission

Nexusguard’s design choices are guided by three principles: performance, reliability and safety.

• Preserving full TCP functionality. Many modern applications rely on TCP options such as selective acknowledgements, large windows and timestamps to achieve high throughput. By allowing the native handshake to proceed after the client proves itself, our Bastions ensure these features are negotiated. In contrast, SYN cookies approximate parameters using a handful of bits and cannot support larger MSS values or window scales.
• Avoiding reflection. Our mitigations must never become a weapon in someone else’s attack. Because the Bastion does not respond to initial SYNs, it sends no traffic back to spoofed addresses and cannot be leveraged for reflection.
• Leveraging inherent TCP behaviour. Retransmission is part of the TCP specification. Legitimate clients already implement retry logic; by relying on this mechanism we separate real users from automated floods without introducing proprietary protocols or requiring client cooperation.
• Operating at scale. Our Bastions are deployed at the network edge where bandwidth is available. Dropping large numbers of malicious SYNs is cheaper than computing cryptographic cookies for each request. Experience with large‑scale attacks shows that sending even stateless SYN‑ACKs at high packet rates can saturate upstream links; our approach avoids this by suppressing replies entirely.

Conclusion

TCP SYN floods remain a pervasive threat. Two primary mitigations exist: encoding state into the sequence number via SYN cookies, and retransmission‑based validation that filters spoofed SYNs by requiring clients to retry. While SYN cookies provide a stateless way to keep a server online, they prevent negotiation of advanced TCP options, do not retransmit lost SYN‑ACKs, and can unintentionally participate in reflection attacks. Nexusguard’s Bastions employ retransmission because it preserves full protocol functionality, leverages built‑in TCP behaviour, and ensures the device does not amplify attacks. This strategy allows us to deliver the highest level of protection and performance for our customers.