Other Stuff

NW-Grid

RCS

Simonh

UoM::RCS::Talby

Page Contents:

Page Group:

. . .  

. . .  

Related Pages:



IPIP Tunnelling

1. Networking Documentation

With the some serious caveats re out-of-date doc of IPIP firmly in mind:

2. Tunnelling Options

3. IP over IP — IPIP

3.1. What Not to Read

Googling for documentation on IPIP seems to lead to much out-of-date doc, stuff written for 2.0 Kernels. A clue is that the doc uses ifconfig to dig tunnels rather than ip (from IPRoute2).

3.2. What to read

Read Tunnels over IP in Linux-2.2, by Alexey N. Kuznetsov.

3.3. Required Software Installation

You need the IPRoute2 suite of tools, not the venerable arp, ifconfig and route: 

    apt-get install iproute
or 
    yum install iproute

3.4. Troubleshooting: it doesn't [insert adverb of choice] work!

My tunnel does not work:

ifconfig tunl<n> reports errors and collisions
Did you use ifconfig, perhaps ifconfig ... pointopoint ... to set up your tunnel? Shut it down; delete it; start again with ip.
It all looks good, but nothing happens!
Is your IPTables config denying, rejecting or failing to forward traffic?

4. A Noddy, Artificial, Working Example

4.1. Set IPTables

Being to lazy to set the appropriate IPTables configuration: 

  doolittle> /etc/init.d/iptables stop

  calculon> /etc/init.d/iptables stop
Don't forget to restart the firewalls after playing. . .  

4.2. Enable Forwarding in the Kernel

  doolittle> echo 1 > /proc/sys/net/ipv4/ip_forward

  calculon> echo 1 > /proc/sys/net/ipv4/ip_forward

4.3. Required Config on Doolittle

    root@doolittle> ip tunnel add tunl1 mode ipip remote 130.88.196.129
        # ...130.88.196.129 = calculon.cs.man.ac.uk...

    root@doolittle> ifconfig eth0:0 10.0.1.1 netmask 255.255.255.0
    root@doolittle> ifconfig tunl1 10.0.1.2 netmask 255.255.255.0
        # ...alias 10.0.1.1, an "unrouted" address, onto eth0 and give the 
        #    tunnel an address on the same segment...

    root@doolittle> route add -net 10.0.2.0 netmask 255.255.255.0 device tunl1
        # ...ensure traffic destined for 10.0.2.0/24 goes through the tunnel...

    route del -net 10.0.1.0 netmask 255.255.255.0 device tunl1
    route del -net 10.0.1.0 netmask 255.255.255.0 device eth0
        # ......

4.4. So, what do we have on Doolittle?

 
  doolittle> ip tunnel show

  sit0: ipv6/ip  remote any  local any  ttl 64  nopmtudisc
  tunl0: ip/ip  remote any  local any  ttl inherit  nopmtudisc
  tunl1: ip/ip  remote 130.88.196.129  local any  ttl inherit


    doolittle:~# netstat -rn

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 tunl1
    130.88.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
    0.0.0.0         130.88.198.250  0.0.0.0         UG        0 0          0 eth0


  doolittle> ifconfig -a

  eth0      Link encap:Ethernet  HWaddr 00:12:3F:32:AD:DB
            inet addr:130.88.198.58  Bcast:130.88.255.255  Mask:255.255.0.0
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:376040228 errors:0 dropped:0 overruns:0 frame:0
            TX packets:65501597 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:864719757 (824.6 MiB)  TX bytes:2127554590 (1.9 GiB)
            Interrupt:169

  eth0:0    Link encap:Ethernet  HWaddr 00:12:3F:32:AD:DB
            inet addr:10.0.1.1  Bcast:10.0.1.255  Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            Interrupt:169

  tunl0     Link encap:IPIP Tunnel  HWaddr
            NOARP  MTU:1480  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:0 errors:38 dropped:0 overruns:0 carrier:0
            collisions:38 txqueuelen:0
            RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

  tunl1     Link encap:IPIP Tunnel  HWaddr
            inet addr:10.0.1.2  P-t-P:10.0.1.2  Mask:255.255.255.0
            UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
            RX packets:87 errors:0 dropped:0 overruns:0 frame:0
            TX packets:109 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:13311 (12.9 KiB)  TX bytes:15524 (15.1 KiB)

4.5. Required Config on Calculon

    root@calculon> ip tunnel add tunl1 mode ipip remote 130.88.198.58
        # ......

    root@calculon> ifconfig eth0:0 10.0.2.1 netmask 255.255.255.0
    root@calculon> ifconfig tunl1 10.0.2.2 netmask 255.255.255.0
        # ......

    root@calculon> route add -net 10.0.1.0 netmask 255.255.255.0 device tunl1
        # ......

    root@calculon> route del -net 10.0.2.0 netmask 255.255.255.0 device tunl1
    root@calculon> route del -net 10.0.2.0 netmask 255.255.255.0 device eth0
        # ......

4.6. So, what do we have on Calculon?

  calculon> ip tunnel show

  sit0: ipv6/ip  remote any  local any  ttl 64  nopmtudisc
  tunl0: ip/ip  remote any  local any  ttl inherit  nopmtudisc
  tunl1: ip/ip  remote 130.88.198.58  local any  ttl inherit


    root@calculon> netstat -rn

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    10.0.1.0        0.0.0.0         255.255.255.0   U         0 0          0 tunl1
    130.88.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
    0.0.0.0         130.88.197.250  0.0.0.0         UG        0 0          0 eth0


  calculon> ifconfig -a

  eth0      Link encap:Ethernet  HWaddr 00:11:43:A7:E9:0A
            inet addr:130.88.196.129  Bcast:130.88.255.255  Mask:255.255.0.0
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:168719919 errors:0 dropped:0 overruns:0 frame:0
            TX packets:3635887 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:1110314006 (1.0 GiB)  TX bytes:750141929 (715.3 MiB)
            Interrupt:169

  eth0:0    Link encap:Ethernet  HWaddr 00:11:43:A7:E9:0A
            inet addr:10.0.2.1  Bcast:10.0.2.255  Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            Interrupt:169

  tunl0     Link encap:IPIP Tunnel  HWaddr
            NOARP  MTU:1480  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:0 errors:5 dropped:0 overruns:0 carrier:0
            collisions:5 txqueuelen:0
            RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

  tunl1     Link encap:IPIP Tunnel  HWaddr
            inet addr:10.0.2.2  P-t-P:10.0.2.2  Mask:255.255.255.0
            UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
            RX packets:98 errors:0 dropped:0 overruns:0 frame:0
            TX packets:87 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:12724 (12.4 KiB)  TX bytes:15051 (14.6 KiB)

4.7. It works, look:

  simonh@doolittle> ssh 10.0.2.1
  simonh@10.0.2.1's password:
  simonh@calculon>
then 
  root@doolittle> ifconfig tunl1 down
  root@doolittle> ssh 10.0.2.1
  ssh: connect to host 10.0.2.1 port 22: Connection timed out
then 
  root@doolittle>
  ifconfig tunl1 up
  root@doolittle> route add -net 10.0.2.0 netmask 255.255.255.0 device tunl1
  root@doolittle> ssh 10.0.2.1 -l simonh
  simonh@10.0.2.1's password:

5. A Real, Working Example

Can we connect directly from a machine on the public network to a compute node in a HPC cluster (Beowulf), which lives on a private network (only)?

5.1. The Hosts in Question

doolittle.cs.man.ac.uk, desktop computer on public network: 

    130.88.198.58 netmask 255.255.0.0


    netstat -rn

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    130.88.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
    0.0.0.0         130.88.198.250  0.0.0.0         UG        0 0          0 eth0

jill, a compute node on a HPC cluster, which has head node ps3: 

    root@ps3> cat /etc/hosts

    130.88.203.56    ps3.mc.manchester.ac.uk    ps3

    10.10.10.54      jill
    10.10.10.60      jack
and 
    root@ps3> netstat -rn

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    130.88.203.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
    10.10.10.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
    0.0.0.0         130.88.203.250  0.0.0.0         UG        0 0          0 eth0

5.2. Setup on Doolittle

  root@doolittle> modprobe ipip
      # ...ensure the kernel module is loaded...

  root@doolittle> ip tunnel add tunl1 mode ipip remote 130.88.203.56
      # ...add the tunnel...

  root@doolittle> ifconfig tunl1 10.200.100.100 netmask 255.255.255.0
      # ...config the tunnel...

  root@doolittle> route del -net 10.200.100.0 netmask 255.255.255.0 device tunl1
      # ...delete the automatic route added...

  root@doolittle> route add -net 10.200.200.0 netmask 255.255.255.0 device tunl1
      # ...add the route to send traffic intended for the PS3 head node via the
      #    tunnel...

  root@doolittle> route add -net 10.10.10.0 netmask 255.255.255.0 device tunl1
      # ...add the route to send traffic intended for the PS3 compute nodes via the
      #    tunnel...

5.3. Setup on PS3 Head Node

  root@ps3> echo 1 > /proc/sys/net/ipv4/ip_forward
      # ......

  root@ps3> /etc/init.d/iptables stop
      # ...since we are too lazy to properly config forwarding here for 
      #    now;  DON'T FORGET TO TURN THE FIREWALL BACK ON AFTERWARDS...

  root@ps3> ip tunnel add tunl1 mode ipip remote 130.88.198.58
      # ......

  root@ps3  ip tunnel show

    sit0: ipv6/ip  remote any  local any  ttl 64  nopmtudisc
    tunl0: ip/ip  remote any  local any  ttl inherit  nopmtudisc
    tunl1: ip/ip  remote 130.88.198.58  local any  ttl inherit

  root@ps3> ifconfig tunl1 10.200.200.200 netmask 255.255.255.0
      # ...configure the tunnel device...

  root@ps3> route del -net 10.200.200.0 netmask 255.255.255.0 device tunl1
      # ...delete the automatically-added route...

  root@ps3> route add -net 10.200.100.0 netmask 255.255.255.0 device tunl1
      # ...ensure traffic intended for doolittle goes through the tunnel...

  root@ps3> netstat -rn

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    10.200.100.0    0.0.0.0         255.255.255.0   U         0 0          0 tunl1
    130.88.203.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
    10.10.10.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
        # ...PS3's internal network...
    0.0.0.0         130.88.203.250  0.0.0.0         UG        0 0          0 eth0

  root@ps3>
      # ......

5.4. It works!

  doolittle:~# ssh 10.10.10.54 -l simonh
  simonh@10.10.10.54's password: 
  Last login: Fri Jul 25 13:12:57 2008 from 10.200.100.100

5.5. Last But Not Lease

  1. SWITCH THAT FIREWALL ON PS3 BACK ON!
  2. Configure the firewall on PS3 properly!

6. A Complete, Working Example

 

 -- can't use rfe as router for desktop machines as gateway must be directly
    connected [[is this true?]]
 
 -- security issues --- iptables allows traffic to tunl1 on rfe from 
    10.200.200.100 but someone else could do this...

 -- require all desktop machines to be on same network segment in this
    scenario, else can't use doolittle as their gateway to the 'wulf

     -- could cope with small number of groups of desktop machines each
        on own segment --- n groups <--> n desktop gateways and n tunnels
        required

6.1. Compute Nodes

  route del -net 169.254.0.0 netmask 255.255.0.0 dev eth0
    # ...optional...
 
  route add default gw 192.168.1.253
  route del default gw 192.168.1.254
      # ...set RFE to be default gateway and delete old default (Man2 login 
      #    node)...


  comp000> netstat -rn

  Kernel IP routing table
  Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
  10.0.1.0        0.0.0.0         255.255.255.0   U         0 0          0 eth1
  10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 eth2
  10.0.3.0        0.0.0.0         255.255.255.0   U         0 0          0 eth3
  192.168.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
  127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
  0.0.0.0         192.168.1.253   0.0.0.0         UG        0 0          0 eth0

6.2. RFE

  echo 1 > /proc/sys/net/ipv4/ip_forward


 
  route del -net 10.200.100.0 netmask 255.255.255.0 device tunl1
  route add -net 10.200.200.0 netmask 255.255.255.0 device tunl1
      #
      # ...del auto-added (by ifconfig) route and add route to desktop
      #    machines' (plural!) gateway, doolittle...


 
  rfe> netstat -rn

  Kernel IP routing table
  Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
  130.88.200.0    0.0.0.0         255.255.255.0   U         0 0          0 eth3
  10.0.1.0        0.0.0.0         255.255.255.0   U         0 0          0 eth1
  10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 eth2
  10.200.200.0    0.0.0.0         255.255.255.0   U         0 0          0 tunl1
  192.168.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
  127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
  0.0.0.0         130.88.200.250  0.0.0.0         UG        0 0          0 eth3


  $IPT -t filter -i $EXTIF  -A INPUT  -p tcp -s 130.88.198.58 -j ACCEPT
  $IPT -t filter -o $EXTIF  -A OUTPUT -p tcp -d 130.88.198.58 -j ACCEPT

  $IPT -t filter -i $EXTIF  -A INPUT  -p 4 -s 130.88.198.58 -j ACCEPT
  $IPT -t filter -o $EXTIF  -A OUTPUT -p 4 -d 130.88.198.58 -j ACCEPT
      #
      # ...Proto 4 is IPIP...

  $IPT -t filter -i $EXTIF  -A INPUT   -s 10.200.100.0/24 -j ACCEPT
  $IPT -t filter -o $EXTIF  -A OUTPUT  -d 10.200.100.0/24 -j ACCEPT


  $IPT -A FORWARD -p tcp  -i eth0 -o eth3  -s 192.168.104.0/24 -d 130.88.0.0/16  -j ACCEPT
  $IPT -A FORWARD -p udp  -i eth0 -o eth3  -s 192.168.104.0/24 -d 130.88.0.0/16  -j ACCEPT
      #
      # ...forward traffic from 'wulf compute/private nodes to public nodes...
      #

  $IPT -A FORWARD -p tcp  -i tunl1 -o eth0  -s 10.200.200.100  -d 192.168.104.0/24  -j ACCEPT
  $IPT -A FORWARD -p udp  -i tunl1 -o eth0  -s 10.200.200.100  -d 192.168.104.0/24  -j ACCEPT
  #
  $IPT -A FORWARD -p tcp  -i tunl1 -o eth0  -s calculon.cs.man.ac.uk  -d 192.168.104.0/24  -j ACCEPT
  $IPT -A FORWARD -p udp  -i tunl1 -o eth0  -s calculon.cs.man.ac.uk  -d 192.168.104.0/24  -j ACCEPT
      #
      # ...forward traffic from approved public nodes to 'wulf compute/private nodes...
      #


  $IPT -A FORWARD  -m state --state ESTABLISHED,RELATED  -j ACCEPT


  # -- we're going to a ball :
  #
  $IPT -t nat -A POSTROUTING -s 192.168.104.0/24 -j MASQUERADE

6.3. Desktop Gateway: Doolittle

  echo 1 > /proc/sys/net/ipv4/ip_forward


   
  route del -net 10.200.200.0 netmask 255.255.255.0 device tunl1
  route add -net 10.200.100.0 netmask 255.255.255.0 device tunl1
      # ...delete auto-added (by ifconfig) route and add route to 
      #    remote end of tunnel...

  route add -net 192.168.104.0 netmask 255.255.255.0 device tunl1
      # ...add route to compute nodes of Man2...


  netstat -rn

  Kernel IP routing table
  Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
  10.200.100.0    0.0.0.0         255.255.255.0   U         0 0          0 tunl1
  192.168.104.0   0.0.0.0         255.255.255.0   U         0 0          0 tunl1
  130.88.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
  0.0.0.0         130.88.198.250  0.0.0.0         UG        0 0          0 eth0


  for host in  ... ... 10.200.100.0/24  10.200.200.0/24  calculon.cs.man.ac.uk
  do
      $IPT -t filter -A SSH_SERVICE  -s $host  -m state --state NEW,ESTABLISHED  -j ACCEPT
      $IPT -t filter -A SSH_SERVICE  -d $host  -m state --state ESTABLISHED      -j ACCEPT
  done


$IPT -A FORWARD  -i eth0 -o tunl1  -s calculon.cs.man.ac.uk  -d 192.168.104.0/24  -j ACCEPT
    #
    # ...forward traffic from approved public nodes to 'wulf compute/private nodes...
    #

$IPT -A FORWARD  -m state --state ESTABLISHED,RELATED  -j ACCEPT


    ##
    ##  N.B. No masquerading required or wanted.
    ##  N.B. No masquerading required or wanted.
    ##

6.4. Desktop Pleb: Calculon

  route add -net 192.168.104.0 netmask 255.255.255.0 gw doolittle.cs.man.ac.uk


  netstat -rn

  Kernel IP routing table
  Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
  192.168.104.0   130.88.198.58   255.255.255.0   UG        0 0          0 eth0
  130.88.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
  0.0.0.0         130.88.197.250  0.0.0.0         UG        0 0          0 eth0

6.5. It works!

Masqerading: 

comp000:~ # ssh calculon.cs.man.ac.uk
The authenticity of host 'calculon.cs.man.ac.uk (130.88.196.129)' can't be established.
RSA key fingerprint is 7d:d8:99:c5:5f:01:bc:be:f8:19:34:7b:54:9a:b5:61.
Are you sure you want to continue connecting (yes/no)?
comp000:~ # ssh calculon.cs.man.ac.uk
The authenticity of host 'calculon.cs.man.ac.uk (130.88.196.129)' can't be established.
RSA key fingerprint is 7d:d8:99:c5:5f:01:bc:be:f8:19:34:7b:54:9a:b5:61.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'calculon.cs.man.ac.uk,130.88.196.129' (RSA) to the list of known hosts.
root@calculon.cs.man.ac.uk's password:

comp000:~ # ssh doolittle.cs.man.ac.uk
root@doolittle.cs.man.ac.uk's password:

Tunnelling: 

doolittle:~# ssh 192.168.104.100 -l simonh
Password:

doolittle:~# ssh 192.168.104.100 -l simonh
Password:
Last login: Tue Jul 29 11:51:26 2008 from calculon.cs.man.ac.uk
Have a lot of fun...
simonh@comp000:~>
and, even better 
[root@calculon ~]# ssh 192.168.104.100 -l simonh
Password:
Last login: Tue Jul 29 11:18:28 2008 from calculon.cs.man.ac.uk
Have a lot of fun...
simonh@comp000:~>