Exercise 20: Simulating SYN Flood Attacks

Objective

Understand the impact of a SYN flood attack on a server and explore mitigation techniques such as SYN cookies and firewall rate limiting.

Scenario

As a network security engineer, you are tasked with testing the resilience of your organization’s web server against denial-of-service (DoS) attacks. One of the most common DoS techniques is the SYN flood attack, where an attacker overwhelms a server with TCP connection requests, consuming resources and potentially causing downtime. In this exercise, you’ll simulate a SYN flood attack using hping3, monitor its impact, and implement mitigation strategies to protect the server.

⚠️ Important: Perform this exercise only in a legal and controlled lab environment. Unauthorized DoS attacks are illegal and unethical.

Lab Instructions

Step 1: Set Up the Lab Environment

• Target Server: Linux machine running a web server (e.g., Apache or Nginx).
• Attacker Machine: Linux system with hping3 installed.
• Monitoring Tools: Installed on the target server (e.g., netstat, top).

Step 2: Install hping3

On the Attacker Machine, install hping3:

sudo apt update 
sudo apt install hping3 -y

Step 3: Launch a SYN Flood Attack

On the Attacker Machine, run the following command to flood the server with SYN packets:

sudo hping3 -S --flood -V -p 80 <target-ip>

-S: Sends SYN packets.

--flood: Sends packets as fast as possible.

-V: Verbose mode.

-p 80: Targets port 80 (HTTP).

Step 4: Monitor Server Performance

On the Target Server, monitor the SYN backlog:

sudo netstat -nat | grep SYN_RECV

Check CPU usage:

top

Observe the system load and how the server responds to the attack.

Step 5: Mitigate the Attack

a. Enable SYN Cookies

SYN cookies help mitigate SYN floods by preventing the server from allocating resources until the handshake completes.

Enable SYN cookies:

sudo sysctl -w net.ipv4.tcp_syncookies=1

Make it permanent:

echo "net.ipv4.tcp_syncookies = 1" | sudo tee -a /etc/sysctl.conf 
sudo sysctl -p

b. Apply Firewall Rate Limiting (iptables)

Limit the rate of incoming SYN packets:

sudo iptables -A INPUT -p tcp --syn -m limit --limit 10/s --limit-burst 20 -j ACCEPT 
sudo iptables -A INPUT -p tcp --syn -j DROP

Allows up to 10 SYN packets per second and drops excessive ones.

Step 6: Verify Mitigation Effectiveness

Re-run the SYN flood attack:

sudo hping3 -S --flood -V -p 80 <target-ip>

On the Target Server, monitor the backlog:

sudo netstat -nat | grep SYN_RECV

Check CPU usage and server responsiveness.

If the mitigation is effective, the server should remain stable.

Step 7: Persist Firewall Rules

Save the firewall rules to persist after reboot:

sudo apt install iptables-persistent -y 
sudo netfilter-persistent save 
sudo netfilter-persistent reload

Solution & Explanation

How SYN Flood Attacks Work

• TCP Handshake: A standard connection requires a SYN, SYN-ACK, and ACK packet exchange.
• Flooding SYNs: The attacker sends numerous SYN requests without completing the handshake, filling the server’s connection backlog.
• Resource Exhaustion: The server uses resources for half-open connections, potentially leading to denial of service.

Impact on the Server

• Without Mitigation: High CPU usage, large SYN backlog, and potential service unavailability.
• With Mitigation: Server resources are preserved, and excessive SYN packets are dropped.

Mitigation Techniques

• SYN Cookies: Prevent the server from allocating resources until the handshake is completed.
• Rate Limiting: Limits the rate of incoming SYN packets to prevent overload.
• Firewall Rules: Drop or reject malicious packets.

Testing & Verification

• Verify that SYN cookies and rate limiting reduce SYN backlog during the attack.
• Confirm that CPU usage remains stable under attack.
• Ensure legitimate traffic is still served correctly.

Security Best Practices

1. Enable SYN Cookies: Reduces the impact of SYN floods.
2. Use Rate Limiting: Controls excessive traffic at the firewall level.
3. Deploy Web Application Firewalls (WAF): Provides additional protection.
4. Monitor Logs: Regularly check system logs for unusual traffic.
5. Network Segmentation: Isolate critical systems from public networks.

Additional Script (Optional)

Automate SYN cookie and firewall setup:

#!/bin/bash
# Enable SYN cookies
sudo sysctl -w net.ipv4.tcp_syncookies=1

# Apply SYN rate limiting
sudo iptables -A INPUT -p tcp --syn -m limit --limit 10/s --limit-burst 20 -j ACCEPT
sudo iptables -A INPUT -p tcp --syn -j DROP

# Save firewall rules
sudo netfilter-persistent save

Run the script:

chmod +x mitigate_syn_flood.sh
sudo ./mitigate_syn_flood.sh

Conclusion

In this exercise, you simulated a SYN flood attack using hping3 and observed its impact on server performance. You implemented mitigation strategies such as SYN cookies and firewall rate limiting to protect the server. Understanding how to defend against SYN flood attacks is critical for maintaining network security and ensuring service availability.