SYN Flood Asymmetric Routing: SYN Proxy in DDoS Tunnels

A SYN flood is hard to stop with a classic SYN proxy when routing is asymmetric, because the proxy has to see both directions of the TCP handshake and your return traffic usually bypasses it. In a GRE tunnel setup, clean traffic comes back through the tunnel but replies leave through your normal transit. The practical answer is to use stateless or inbound-only validation at the scrubbing edge, and to run SYN proxy or SYN cookies on the hosts or firewall that see both directions.

This article explains why the problem exists, how SYN and SYN-ACK floods differ, and what you can configure today.

SYN flood vs. SYN-ACK flood

The two attacks look similar in graphs but need different handling.

In a SYN-ACK flood your servers never sent a SYN, so there is no matching connection. Dropping these packets is easy in principle, but only if some device can tell that no SYN went out. That is where asymmetric routing hurts.

Why asymmetric routing breaks SYN proxy

A SYN proxy answers the client's SYN itself with a cookie-based SYN-ACK. If the client returns a valid ACK, the proxy opens a real connection to the backend. After that it must translate sequence numbers for the rest of the connection, because the client agreed on the proxy's initial sequence number and the backend picked its own.

This requires the proxy to sit in the path of both directions. With a tunnel-based design:

1. Your prefixes are announced from the scrubbing edge, so inbound traffic hits the scrubber first.
2. Clean traffic is delivered to your router over GRE.
3. Your servers reply through your own transit or IX, not back through the tunnel.

The scrubber therefore sees SYN, ACK and data from the client but never the server's SYN-ACK or data. Three things go wrong:

• Sequence translation fails. The server's replies leave directly with sequence numbers the client does not expect, so the connection resets or stalls.
• State cannot be tracked. The scrubber cannot tell a legitimate SYN-ACK from a reflected one, since it never saw the outbound SYN.
• TCP options mismatch. The proxy picks MSS, window scale, SACK and timestamps without knowing what the backend negotiates. Add 24 bytes of GRE overhead and you get blackholed large packets.

Some scrubbing designs work around this by forcing return traffic through the tunnel, which makes the path symmetric. That is a design choice, so confirm with your provider whether it is supported.

Mitigation options that work with asymmetry

1. Stateless validation at the edge

Instead of full proxying, the edge validates the client and then steps aside:

• Drop-first-SYN / retransmit check. The edge drops the first SYN from an unknown source. A real client retransmits (typically after about one second); a spoofed source never does. After the retransmit, the source goes onto a temporary allow list.
• Cookie challenge with whitelist. The edge replies with a cookie SYN-ACK and, on a valid ACK, whitelists the source IP. The client then retries and the connection goes straight to your server with no sequence translation.

Both need only inbound state, so they survive asymmetric return paths. The cost is added latency on the first connection from each new source, and some odd clients may fail the check. Test with your actual traffic mix.

2. Run SYN cookies or SYNPROXY where state is symmetric

Your own server or Linux firewall box sees both directions by definition. Enable SYN cookies and, if needed, the netfilter SYNPROXY target:

# Kernel basics
sysctl -w net.ipv4.tcp_syncookies=1
sysctl -w net.ipv4.tcp_synack_retries=2
sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sysctl -w net.netfilter.nf_conntrack_tcp_loose=0
sysctl -w net.ipv4.tcp_timestamps=1

# SYNPROXY for a protected web port (192.0.2.10:443)
iptables -t raw -A PREROUTING -p tcp -d 192.0.2.10 --dport 443 --syn -j CT --notrack
iptables -A INPUT -p tcp -d 192.0.2.10 --dport 443 \
  -m conntrack --ctstate INVALID,UNTRACKED \
  -j SYNPROXY --sack-perm --timestamp --wscale 7 --mss 1436
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

Note the --mss 1436: it is set for a GRE tunnel path (see below). SYNPROXY protects the host from half-open state, but it still burns CPU, so it does not replace upstream filtering for volumetric attacks.

3. Fix MSS and MTU on the tunnel

With IPv4 GRE the outer header adds 24 bytes. On a 1500-byte underlay, the tunnel MTU is 1476 and the TCP MSS should be at most 1436. Clamp it on the tunnel interface:

! Cisco IOS / IOS-XE
interface Tunnel0
 ip mtu 1476
 ip tcp adjust-mss 1436

# Juniper
set interfaces gr-0/0/0 unit 0 family inet mtu 1476
set interfaces gr-0/0/0 unit 0 family inet tcp-mss 1436

# MikroTik
/ip firewall mangle add chain=forward protocol=tcp tcp-flags=syn \
  out-interface=gre-tunnel1 action=change-mss new-mss=1436 passthrough=yes

Without this, SYN handling can look fine while real users still stall on large responses.

4. Rate-limit SYN-ACKs to servers that rarely make outbound connections

For a block of pure inbound servers, legitimate SYN-ACKs are rare. They only arrive when those servers connect outward, for example to APIs or DNS over TCP. A policer is safer than a hard drop.

Juniper edge filter:

set firewall policer SYNACK-LIMIT if-exceeding bandwidth-limit 10m burst-size-limit 150k
set firewall policer SYNACK-LIMIT then discard

set firewall family inet filter PROTECT-SERVERS term SYNACK from destination-address 192.0.2.0/24
set firewall family inet filter PROTECT-SERVERS term SYNACK from protocol tcp
set firewall family inet filter PROTECT-SERVERS term SYNACK from tcp-flags "syn&ack"
set firewall family inet filter PROTECT-SERVERS term SYNACK then policer SYNACK-LIMIT
set firewall family inet filter PROTECT-SERVERS term SYNACK then accept
set firewall family inet filter PROTECT-SERVERS term REST then accept

MikroTik (raw table, before conntrack):

/ip firewall raw
add chain=prerouting protocol=tcp tcp-flags=syn,ack,!fin,!rst \
    dst-address=192.0.2.0/24 limit=2000,100:packet action=accept
add chain=prerouting protocol=tcp tcp-flags=syn,ack,!fin,!rst \
    dst-address=192.0.2.0/24 action=drop

Adjust the numbers to your normal baseline. The same match can be sent upstream as a BGP FlowSpec rule (protocol TCP, TCP flags SYN+ACK, destination prefix, rate-limit action), which keeps the junk off your transit links. See our FlowSpec guide for safety practices.

5. Make the return path symmetric where you can

If your tunnel provider supports return traffic over the tunnel, use source-based routing for the protected prefix. On Linux:

ip rule add from 192.0.2.0/24 table 100
ip route add default dev gre1 table 100

On routers the equivalent is policy-based routing or a VRF/routing instance for the protected space. The tradeoff: all outbound traffic from that prefix now crosses the tunnel, which may add latency and consume tunnel capacity. Many networks do this only for specific customers or only during attacks.

Detecting SYN and SYN-ACK floods from flow data

Flow telemetry (NetFlow v5/v9, IPFIX, sFlow) gives you a fast first signal:

• Per-destination-IP packets per second spikes with small average packet size (40–60 bytes).
• A high share of flows with only the SYN flag, or SYN+ACK flags, to a single IP.
• For reflection, many unique source IPs with the same source port and no matching outbound flows.

Remember that NetFlow TCP flags are cumulative (OR-ed) per flow. Single-packet flows from floods are accurate, but long-lived flows mix flags. With packet sampling, small floods may be under-counted, so tune pps thresholds per IP and per prefix, not just per link. Our posts on NetFlow detection thresholds and carpet bombing cover baselines and prefix-wide attacks.

Common mistakes

• Enabling SYN proxy on a scrubber for traffic with asymmetric return. It may pass tests with synthetic traffic and then reset real sessions.
• Forgetting MSS clamping after moving to a GRE tunnel.
• Hard-dropping all SYN-ACKs to networks that include outbound-heavy hosts (mail servers, crawlers, NAT pools).
• Blackholing the victim IP for a reflection flood when a FlowSpec or filter rule would keep the service up. RTBH takes the target offline by design.
• Leaving nf_conntrack_tcp_loose at its default when using SYNPROXY.
• No testing of the failover path, where routing symmetry can differ from normal operation.

Where NixMon fits

NixMon detects floods from NetFlow, IPFIX and sFlow with per-IP and per-prefix bps/pps thresholds, then can trigger BGP FlowSpec, RTBH, or on-prem NixGuard actions. For networks with their own ASN and at least one /24, NixGuard Edge announces your prefixes from the scrubbing edge and delivers clean traffic over a GRE tunnel with a BGP session inside, in always-on or on-demand mode. If you are unsure how asymmetric your return path is, test it before you rely on any proxy-style mitigation. You can try the platform with the 15-day free trial, no credit card required.