ISP Network Design Checklist: Transit, IX, Edge Routers & DDoS

An ISP network design checklist should cover five things: a sensible transit and IX peering mix, an edge router that fits your routing table and traffic profile, flow telemetry from day one, a tested DDoS response plan, and capacity headroom that survives a link failure. Skip any of them and you will find out during an outage.

This checklist is written for ISPs, data centers and hosting networks that run their own ASN. Examples use documentation ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, AS64496-AS64511).

1. Transit vs IX peering: pick the mix deliberately

You almost always need both. Transit gives you reachability to the whole internet. IX peering gives you cheaper, lower-latency paths to specific networks.

Checklist items:

• At least two transit providers, on different physical paths and ideally different upstream networks. One provider is a single point of failure for both routing and filtering.
• Ask every provider about RTBH. Get the blackhole community in writing (many follow RFC 7999, 65535:666), the accepted prefix length (usually /32 for IPv4, /128 for IPv6) and how fast they act.
• Join an IX only if you have traffic to move. Look at your flow data first: if a handful of ASNs make up a large share of your traffic, peering with them pays off.
• Do not treat route servers as trusted. Filter what you receive from them the same way you filter transit.

BGP session hygiene (non-negotiable)

• Prefix filters per peer, built from IRR data (AS-SETs) where possible.
• Max-prefix limits on every session.
• RPKI origin validation: reject invalid routes.
• Drop bogons and your own prefixes learned from outside.
• Only announce your own prefixes and customer prefixes you have verified.
• Set up the blackhole community handling before you need it.

2. Edge router choice: Juniper MX, MikroTik, VyOS

There is no single best router. The right choice depends on port speed, number of BGP sessions, the full-table requirement and the budget for support.

Questions to answer before you buy:

1. How many full tables will it hold (2 transit + IX = 3 or more), IPv4 and IPv6 together, with headroom for growth?
2. How long does BGP convergence take after a session reset on that platform?
3. Can it export flow data at your line rate without hurting the control plane?
4. Does it support the mitigation tools you need: communities for RTBH, and FlowSpec if you plan to use it? Verify this against your exact software version.
5. What is the failure model: redundant routers, redundant uplinks, graceful restart?

3. Flow export: configure it at build time

You cannot detect or investigate a DDoS you cannot see. Turn on flow export on every edge-facing interface before production traffic arrives. A collector at 192.0.2.50 UDP/2055 is used below.

Juniper MX (inline IPFIX)

set services flow-monitoring version-ipfix template ipv4 flow-active-timeout 10
set services flow-monitoring version-ipfix template ipv4 flow-inactive-timeout 10
set services flow-monitoring version-ipfix template ipv4 template-refresh-rate seconds 30
set services flow-monitoring version-ipfix template ipv4 ipv4-template
set chassis fpc 0 sampling-instance SAMPLE-1
set forwarding-options sampling instance SAMPLE-1 input rate 1000
set forwarding-options sampling instance SAMPLE-1 family inet output flow-server 192.0.2.50 port 2055
set forwarding-options sampling instance SAMPLE-1 family inet output flow-server 192.0.2.50 version-ipfix template ipv4
set forwarding-options sampling instance SAMPLE-1 family inet output inline-jflow source-address 192.0.2.1
set interfaces xe-0/0/0 unit 0 family inet sampling input

Exact hierarchy and flow table sizing vary by Junos version and line card, so check the documentation for your release.

MikroTik RouterOS

/ip traffic-flow set enabled=yes interfaces=sfp-sfpplus1 active-flow-timeout=1m inactive-flow-timeout=15s
/ip traffic-flow target add dst-address=192.0.2.50 port=2055 version=9

RouterOS traffic-flow exports flows without a configurable packet sampling rate, so watch CPU on busy routers and limit it to the interfaces you need.

VyOS (sFlow, 1.4 syntax)

set system sflow agent-address 192.0.2.3
set system sflow interface eth0
set system sflow server 192.0.2.50 port 6343
set system sflow sample-rate 1000

Verify the collector actually receives data

tcpdump -ni eth0 udp port 2055 or udp port 6343

Common mistake: exporting flows from only the transit ports. Export from IX ports and customer-facing ports too, otherwise you miss attacks that arrive through peering and traffic leaving toward a victim's neighbors.

4. DDoS readiness before the first attack

Design the response while things are calm. A practical layered plan:

1. Detection. Per-IP and per-prefix thresholds in bps and pps from flow data. Prefer thresholds per destination IP; a total-link threshold fires too late.
2. RTBH as a last resort. Blackholing a victim /32 protects the rest of the network but completes the attack for that IP. Prepare the policy now.
3. FlowSpec for targeted filtering. If your routers and upstreams support it, you can drop or rate-limit specific protocols and ports (for example UDP/53 reflection traffic) without blackholing the whole host.
4. Scrubbing for volumetric attacks. When an attack exceeds your uplink capacity, local filtering cannot help because the link is already full upstream of you. You need the traffic cleaned before it reaches your port.

A minimal Junos RTBH setup on the customer-triggered side:

set routing-options static route 192.0.2.254/32 discard
set policy-options community BLACKHOLE members 65535:666
set policy-options policy-statement RTBH-IN term bh from community BLACKHOLE
set policy-options policy-statement RTBH-IN term bh from route-filter 0.0.0.0/0 prefix-length-range /32-/32
set policy-options policy-statement RTBH-IN term bh then next-hop 192.0.2.254
set policy-options policy-statement RTBH-IN term bh then accept

Place this term before your normal import filters, and make sure only authorized sessions can trigger it and only for prefixes the sender actually owns. An open RTBH trigger is an easy way to take your own customers offline.

Scrubbing and your prefix size

Cloud scrubbing over a BGP tunnel needs routable prefixes. Because most networks filter anything longer than /24 in IPv4, you generally need at least a /24 and your own ASN. Plan your address space with that in mind: a /24 you can divert is more useful than several scattered smaller blocks. A service like NixGuard Edge works this way: your prefixes are announced from the scrubbing edge and clean traffic returns over a GRE tunnel with a BGP session inside, either always-on or on-demand. Make sure your edge router can terminate GRE and that your MTU plan accounts for the tunnel overhead (clamp TCP MSS to avoid fragmentation).

Test it

• Run a drill: announce a test /32 with the blackhole community to a lab or a test peer and confirm it is dropped.
• Time the full loop from attack start to detection to mitigation.
• Document who can trigger what, at 3 a.m.

5. Capacity planning

• Measure 95th percentile and peak separately. Billing uses one; outages are caused by the other.
• Plan for N-1. If one of two uplinks fails, the remaining one must carry peak traffic. A common practice is to start upgrading when peak utilization on a link regularly passes roughly 50-60% in a dual-homed design.
• Track growth with flow data, not just interface counters. Flow data shows which customers and destinations are driving growth.
• Separate attack headroom from normal headroom. Spare capacity will not stop a volumetric attack bigger than your uplinks; that is what scrubbing or upstream filtering is for.
• Watch pps, not only bps. Small-packet floods can overload router forwarding or firewalls while link utilization looks normal.
• Check control-plane limits: FIB size, number of BGP sessions, flow cache entries.

Common mistakes

• Single transit provider or two providers sharing one upstream and one fiber path.
• No documented blackhole community, discovered during the incident.
• Flow export enabled on only some interfaces.
• Collector timestamps and router clocks not in sync (use NTP).
• Thresholds set once and never reviewed after traffic grew.
• Announcing less-specifics only, so you cannot divert a single /24 to scrubbing.

Where NixMon fits

NixMon receives NetFlow v5/v9, IPFIX and sFlow, analyzes traffic with sub-second analytics, applies per-IP and per-prefix bps/pps thresholds, and can trigger BGP blackhole, BGP FlowSpec, an on-prem NixGuard firewall or NixGuard Edge. You can test it against your own edge routers with the 15-day free trial, no credit card needed. If your design needs a second opinion first, a network consulting review of BGP and ISP design is also available.