IPIP Tunnelling
1. Networking Documentation
With the some serious caveats re out-of-date doc of IPIP firmly in mind:
- Guide to IP Layer Network Administration with Linux, Martin A. Brown, Version 0.4.5, March 2007.
- Linux Advanced Routing & Traffic Control HOWTO, 1.0.0, 2004-03-31.
2. Tunnelling Options
3. IP over IP — IPIP
3.1. What Not to Read
Googling for documentation on IPIP seems to lead to much out-of-date doc, stuff written for 2.0 Kernels. A clue is that the doc uses ifconfig to dig tunnels rather than ip (from IPRoute2).
3.2. What to read
Read Tunnels over IP in Linux-2.2, by Alexey N. Kuznetsov.
3.3. Required Software Installation
You need the IPRoute2 suite of tools, not the venerable arp, ifconfig and route:
apt-get install iprouteor
yum install iproute
3.4. Troubleshooting: it doesn't [insert adverb of choice] work!
My tunnel does not work:
- ifconfig tunl<n> reports errors and collisions
- Did you use ifconfig, perhaps ifconfig ... pointopoint ... to set up your tunnel? Shut it down; delete it; start again with ip.
- It all looks good, but nothing happens!
- Is your IPTables config denying, rejecting or failing to forward traffic?
4. A Noddy, Artificial, Working Example
4.1. Set IPTables
Being to lazy to set the appropriate IPTables configuration:
doolittle> /etc/init.d/iptables stop calculon> /etc/init.d/iptables stopDon't forget to restart the firewalls after playing. . .
4.2. Enable Forwarding in the Kernel
doolittle> echo 1 > /proc/sys/net/ipv4/ip_forward calculon> echo 1 > /proc/sys/net/ipv4/ip_forward
4.3. Required Config on Doolittle
root@doolittle> ip tunnel add tunl1 mode ipip remote 130.88.196.129 # ...130.88.196.129 = calculon.cs.man.ac.uk... root@doolittle> ifconfig eth0:0 10.0.1.1 netmask 255.255.255.0 root@doolittle> ifconfig tunl1 10.0.1.2 netmask 255.255.255.0 # ...alias 10.0.1.1, an "unrouted" address, onto eth0 and give the # tunnel an address on the same segment... root@doolittle> route add -net 10.0.2.0 netmask 255.255.255.0 device tunl1 # ...ensure traffic destined for 10.0.2.0/24 goes through the tunnel... route del -net 10.0.1.0 netmask 255.255.255.0 device tunl1 route del -net 10.0.1.0 netmask 255.255.255.0 device eth0 # ......
4.4. So, what do we have on Doolittle?
doolittle> ip tunnel show sit0: ipv6/ip remote any local any ttl 64 nopmtudisc tunl0: ip/ip remote any local any ttl inherit nopmtudisc tunl1: ip/ip remote 130.88.196.129 local any ttl inherit
doolittle:~# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 tunl1 130.88.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 130.88.198.250 0.0.0.0 UG 0 0 0 eth0
doolittle> ifconfig -a eth0 Link encap:Ethernet HWaddr 00:12:3F:32:AD:DB inet addr:130.88.198.58 Bcast:130.88.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:376040228 errors:0 dropped:0 overruns:0 frame:0 TX packets:65501597 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:864719757 (824.6 MiB) TX bytes:2127554590 (1.9 GiB) Interrupt:169 eth0:0 Link encap:Ethernet HWaddr 00:12:3F:32:AD:DB inet addr:10.0.1.1 Bcast:10.0.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:169 tunl0 Link encap:IPIP Tunnel HWaddr NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:38 dropped:0 overruns:0 carrier:0 collisions:38 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) tunl1 Link encap:IPIP Tunnel HWaddr inet addr:10.0.1.2 P-t-P:10.0.1.2 Mask:255.255.255.0 UP POINTOPOINT RUNNING NOARP MTU:1480 Metric:1 RX packets:87 errors:0 dropped:0 overruns:0 frame:0 TX packets:109 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:13311 (12.9 KiB) TX bytes:15524 (15.1 KiB)
4.5. Required Config on Calculon
root@calculon> ip tunnel add tunl1 mode ipip remote 130.88.198.58 # ...... root@calculon> ifconfig eth0:0 10.0.2.1 netmask 255.255.255.0 root@calculon> ifconfig tunl1 10.0.2.2 netmask 255.255.255.0 # ...... root@calculon> route add -net 10.0.1.0 netmask 255.255.255.0 device tunl1 # ...... root@calculon> route del -net 10.0.2.0 netmask 255.255.255.0 device tunl1 root@calculon> route del -net 10.0.2.0 netmask 255.255.255.0 device eth0 # ......
4.6. So, what do we have on Calculon?
calculon> ip tunnel show sit0: ipv6/ip remote any local any ttl 64 nopmtudisc tunl0: ip/ip remote any local any ttl inherit nopmtudisc tunl1: ip/ip remote 130.88.198.58 local any ttl inherit
root@calculon> netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.0.1.0 0.0.0.0 255.255.255.0 U 0 0 0 tunl1 130.88.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 130.88.197.250 0.0.0.0 UG 0 0 0 eth0
calculon> ifconfig -a eth0 Link encap:Ethernet HWaddr 00:11:43:A7:E9:0A inet addr:130.88.196.129 Bcast:130.88.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:168719919 errors:0 dropped:0 overruns:0 frame:0 TX packets:3635887 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1110314006 (1.0 GiB) TX bytes:750141929 (715.3 MiB) Interrupt:169 eth0:0 Link encap:Ethernet HWaddr 00:11:43:A7:E9:0A inet addr:10.0.2.1 Bcast:10.0.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:169 tunl0 Link encap:IPIP Tunnel HWaddr NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:5 dropped:0 overruns:0 carrier:0 collisions:5 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) tunl1 Link encap:IPIP Tunnel HWaddr inet addr:10.0.2.2 P-t-P:10.0.2.2 Mask:255.255.255.0 UP POINTOPOINT RUNNING NOARP MTU:1480 Metric:1 RX packets:98 errors:0 dropped:0 overruns:0 frame:0 TX packets:87 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:12724 (12.4 KiB) TX bytes:15051 (14.6 KiB)
4.7. It works, look:
simonh@doolittle> ssh 10.0.2.1 simonh@10.0.2.1's password: simonh@calculon>then
root@doolittle> ifconfig tunl1 down root@doolittle> ssh 10.0.2.1 ssh: connect to host 10.0.2.1 port 22: Connection timed outthen
root@doolittle> ifconfig tunl1 up root@doolittle> route add -net 10.0.2.0 netmask 255.255.255.0 device tunl1 root@doolittle> ssh 10.0.2.1 -l simonh simonh@10.0.2.1's password:
5. A Real, Working Example
Can we connect directly from a machine on the public network to a compute node in a HPC cluster (Beowulf), which lives on a private network (only)?
5.1. The Hosts in Question
doolittle.cs.man.ac.uk, desktop computer on public network:
130.88.198.58 netmask 255.255.0.0 netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 130.88.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 130.88.198.250 0.0.0.0 UG 0 0 0 eth0
jill, a compute node on a HPC cluster, which has head node ps3:
root@ps3> cat /etc/hosts 130.88.203.56 ps3.mc.manchester.ac.uk ps3 10.10.10.54 jill 10.10.10.60 jackand
root@ps3> netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 130.88.203.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 0.0.0.0 130.88.203.250 0.0.0.0 UG 0 0 0 eth0
5.2. Setup on Doolittle
root@doolittle> modprobe ipip # ...ensure the kernel module is loaded... root@doolittle> ip tunnel add tunl1 mode ipip remote 130.88.203.56 # ...add the tunnel... root@doolittle> ifconfig tunl1 10.200.100.100 netmask 255.255.255.0 # ...config the tunnel... root@doolittle> route del -net 10.200.100.0 netmask 255.255.255.0 device tunl1 # ...delete the automatic route added... root@doolittle> route add -net 10.200.200.0 netmask 255.255.255.0 device tunl1 # ...add the route to send traffic intended for the PS3 head node via the # tunnel... root@doolittle> route add -net 10.10.10.0 netmask 255.255.255.0 device tunl1 # ...add the route to send traffic intended for the PS3 compute nodes via the # tunnel...
5.3. Setup on PS3 Head Node
root@ps3> echo 1 > /proc/sys/net/ipv4/ip_forward # ...... root@ps3> /etc/init.d/iptables stop # ...since we are too lazy to properly config forwarding here for # now; DON'T FORGET TO TURN THE FIREWALL BACK ON AFTERWARDS... root@ps3> ip tunnel add tunl1 mode ipip remote 130.88.198.58 # ...... root@ps3 ip tunnel show sit0: ipv6/ip remote any local any ttl 64 nopmtudisc tunl0: ip/ip remote any local any ttl inherit nopmtudisc tunl1: ip/ip remote 130.88.198.58 local any ttl inherit root@ps3> ifconfig tunl1 10.200.200.200 netmask 255.255.255.0 # ...configure the tunnel device... root@ps3> route del -net 10.200.200.0 netmask 255.255.255.0 device tunl1 # ...delete the automatically-added route... root@ps3> route add -net 10.200.100.0 netmask 255.255.255.0 device tunl1 # ...ensure traffic intended for doolittle goes through the tunnel... root@ps3> netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.200.100.0 0.0.0.0 255.255.255.0 U 0 0 0 tunl1 130.88.203.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 # ...PS3's internal network... 0.0.0.0 130.88.203.250 0.0.0.0 UG 0 0 0 eth0 root@ps3> # ......
5.4. It works!
doolittle:~# ssh 10.10.10.54 -l simonh simonh@10.10.10.54's password: Last login: Fri Jul 25 13:12:57 2008 from 10.200.100.100
5.5. Last But Not Lease
- SWITCH THAT FIREWALL ON PS3 BACK ON!
- Configure the firewall on PS3 properly!
6. A Complete, Working Example
-- can't use rfe as router for desktop machines as gateway must be directly connected [[is this true?]] -- security issues --- iptables allows traffic to tunl1 on rfe from 10.200.200.100 but someone else could do this... -- require all desktop machines to be on same network segment in this scenario, else can't use doolittle as their gateway to the 'wulf -- could cope with small number of groups of desktop machines each on own segment --- n groups <--> n desktop gateways and n tunnels required
6.1. Compute Nodes
route del -net 169.254.0.0 netmask 255.255.0.0 dev eth0 # ...optional... route add default gw 192.168.1.253 route del default gw 192.168.1.254 # ...set RFE to be default gateway and delete old default (Man2 login # node)...
comp000> netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.0.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 10.0.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3 192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 192.168.1.253 0.0.0.0 UG 0 0 0 eth0
6.2. RFE
echo 1 > /proc/sys/net/ipv4/ip_forward
route del -net 10.200.100.0 netmask 255.255.255.0 device tunl1 route add -net 10.200.200.0 netmask 255.255.255.0 device tunl1 # # ...del auto-added (by ifconfig) route and add route to desktop # machines' (plural!) gateway, doolittle...
rfe> netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 130.88.200.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3 10.0.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 10.200.200.0 0.0.0.0 255.255.255.0 U 0 0 0 tunl1 192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 130.88.200.250 0.0.0.0 UG 0 0 0 eth3
$IPT -t filter -i $EXTIF -A INPUT -p tcp -s 130.88.198.58 -j ACCEPT $IPT -t filter -o $EXTIF -A OUTPUT -p tcp -d 130.88.198.58 -j ACCEPT $IPT -t filter -i $EXTIF -A INPUT -p 4 -s 130.88.198.58 -j ACCEPT $IPT -t filter -o $EXTIF -A OUTPUT -p 4 -d 130.88.198.58 -j ACCEPT # # ...Proto 4 is IPIP... $IPT -t filter -i $EXTIF -A INPUT -s 10.200.100.0/24 -j ACCEPT $IPT -t filter -o $EXTIF -A OUTPUT -d 10.200.100.0/24 -j ACCEPT
$IPT -A FORWARD -p tcp -i eth0 -o eth3 -s 192.168.104.0/24 -d 130.88.0.0/16 -j ACCEPT $IPT -A FORWARD -p udp -i eth0 -o eth3 -s 192.168.104.0/24 -d 130.88.0.0/16 -j ACCEPT # # ...forward traffic from 'wulf compute/private nodes to public nodes... # $IPT -A FORWARD -p tcp -i tunl1 -o eth0 -s 10.200.200.100 -d 192.168.104.0/24 -j ACCEPT $IPT -A FORWARD -p udp -i tunl1 -o eth0 -s 10.200.200.100 -d 192.168.104.0/24 -j ACCEPT # $IPT -A FORWARD -p tcp -i tunl1 -o eth0 -s calculon.cs.man.ac.uk -d 192.168.104.0/24 -j ACCEPT $IPT -A FORWARD -p udp -i tunl1 -o eth0 -s calculon.cs.man.ac.uk -d 192.168.104.0/24 -j ACCEPT # # ...forward traffic from approved public nodes to 'wulf compute/private nodes... # $IPT -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT # -- we're going to a ball : # $IPT -t nat -A POSTROUTING -s 192.168.104.0/24 -j MASQUERADE
6.3. Desktop Gateway: Doolittle
echo 1 > /proc/sys/net/ipv4/ip_forward
route del -net 10.200.200.0 netmask 255.255.255.0 device tunl1 route add -net 10.200.100.0 netmask 255.255.255.0 device tunl1 # ...delete auto-added (by ifconfig) route and add route to # remote end of tunnel... route add -net 192.168.104.0 netmask 255.255.255.0 device tunl1 # ...add route to compute nodes of Man2...
netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.200.100.0 0.0.0.0 255.255.255.0 U 0 0 0 tunl1 192.168.104.0 0.0.0.0 255.255.255.0 U 0 0 0 tunl1 130.88.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 130.88.198.250 0.0.0.0 UG 0 0 0 eth0
for host in ... ... 10.200.100.0/24 10.200.200.0/24 calculon.cs.man.ac.uk do $IPT -t filter -A SSH_SERVICE -s $host -m state --state NEW,ESTABLISHED -j ACCEPT $IPT -t filter -A SSH_SERVICE -d $host -m state --state ESTABLISHED -j ACCEPT done
$IPT -A FORWARD -i eth0 -o tunl1 -s calculon.cs.man.ac.uk -d 192.168.104.0/24 -j ACCEPT # # ...forward traffic from approved public nodes to 'wulf compute/private nodes... # $IPT -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT ## ## N.B. No masquerading required or wanted. ## N.B. No masquerading required or wanted. ##
6.4. Desktop Pleb: Calculon
route add -net 192.168.104.0 netmask 255.255.255.0 gw doolittle.cs.man.ac.uk
netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.104.0 130.88.198.58 255.255.255.0 UG 0 0 0 eth0 130.88.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 130.88.197.250 0.0.0.0 UG 0 0 0 eth0
6.5. It works!
Masqerading:
comp000:~ # ssh calculon.cs.man.ac.uk The authenticity of host 'calculon.cs.man.ac.uk (130.88.196.129)' can't be established. RSA key fingerprint is 7d:d8:99:c5:5f:01:bc:be:f8:19:34:7b:54:9a:b5:61. Are you sure you want to continue connecting (yes/no)? comp000:~ # ssh calculon.cs.man.ac.uk The authenticity of host 'calculon.cs.man.ac.uk (130.88.196.129)' can't be established. RSA key fingerprint is 7d:d8:99:c5:5f:01:bc:be:f8:19:34:7b:54:9a:b5:61. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'calculon.cs.man.ac.uk,130.88.196.129' (RSA) to the list of known hosts. root@calculon.cs.man.ac.uk's password: comp000:~ # ssh doolittle.cs.man.ac.uk root@doolittle.cs.man.ac.uk's password:
Tunnelling:
doolittle:~# ssh 192.168.104.100 -l simonh Password: doolittle:~# ssh 192.168.104.100 -l simonh Password: Last login: Tue Jul 29 11:51:26 2008 from calculon.cs.man.ac.uk Have a lot of fun... simonh@comp000:~>and, even better
[root@calculon ~]# ssh 192.168.104.100 -l simonh Password: Last login: Tue Jul 29 11:18:28 2008 from calculon.cs.man.ac.uk Have a lot of fun... simonh@comp000:~>