Experimental network

How to leverage minimega's tools for augmenting your network topology

The minimega miniclass series

Sandia National Laboratories

VM Network for experiments

In module 05 we looked at how to configure network information like VLANs for your VMs. We recommend reviewing that module before working through this one.

In this module, we'll look at:

The first step is assigning IPs to your VMs, and there are numerous ways we can do that.

Of course, you can always manually specify the IPs for each and every VM, but that would quickly become impractical.

Fortunately, minimega offers several options for automating this process.

Automated IP assignment - DHCP with DNSMASQ

DHCP is a quick way of assigning IPs to your VMs. The minimega toolset includes DNSMASQ which can serve as a DHCP server for your experimental network. Let's look at how we can set that up.

DNSMASQ will run on the same host as the VMs, not on a VM itself. Thus, we need a way to allow the host to access the experiment network, and we can do that using a tap:

tap create 100 ip 10.0.0.1/24

Let's unpack that command a bit:

We covered taps in the previous module. Head back to Module 05 if you need a review.

In addition to creating the tap, we also need to define an environment to network together.
To reiterate, be sure to have the VMs exist on the same VLAN as the tap, to ensure communication with DHCP.

vm config net 100
vm config disk foo.qc2
vm launch kvm foo[1-10]

With the tap in place and the VMs defined and launched, let's launch DNSMASQ

dnsmasq start 10.0.0.1 10.0.0.2 10.0.0.254

This command tells DNSMASQ to listen on 10.0.0.1, distributing IPs in range 10.0.0.2 through 10.0.0.254

Putting it all together

From start to finish, here are the commands needed to accomplish automating the assignment of IPs using DNSMASQ as a DHCP server running on the host.

tap create 100 ip 10.0.0.1/24
vm config disk foo.qc2
vm config memory 128
vm config net 100
dnsmasq start 10.0.0.1 10.0.0.2 10.0.0.254
vm launch kvm linux[1-10]
vm start all

Taking it Further

You can run multiple DHCP servers on multiple VLANs.

tap create 100 ip 10.0.0.1/24
tap create 200 ip 20.0.0.1/24
dnsmasq start 10.0.0.1 10.0.0.2 10.0.0.254
dnsmasq start 20.0.0.1 20.0.0.2 20.0.0.254

Print all running DNSMASQ DHCP servers with the command by itself:

minimega$ dnsmasq
host   | ID | Listening Address | Min      | Max        | Path                            | PID
ubuntu | 1  | 20.0.0.1          | 20.0.0.2 | 20.0.0.254 | /tmp/minimega/dnsmasq_264204396 | 3610

Stop with the kill command

minimega$ dnsmasq kill 1
ubuntu | 0  | 10.0.0.1          | 10.0.0.2 | 10.0.0.254 | /tmp/minimega/dnsmasq_826235649 | 3216

other commands

See the minimega API for more details on dnsmasq, or help in minimega:

minimega$ help dnsmasq

minirouter

minirouter is a simple tool, run in a VM, that orchestrates various router functions such as DHCP, DNS, IPv4/IPv6 assignments, and, of course, routing. The minirouter tool is interfaced by minimega's router API and the minimega distribution provides a prebuilt minirouter container image.

minirouter currently supports several protocols and capabilities including DHCP, DNS, router advertisements, OSPF, and static routes. It can route in excess of 40 gigabits per second when running as a container.

minirouter can run on bare metal, as a container, or a KVM image.

vmbetter can be used to create and deploy a minirouter image (kernel/initrd pair or container filesystem). For more information on using vmbetter, see module 2.5: Better vmbetter

Running minirouter without an image

minirouter is simply a Linux binary that can run on any Linux system. You do not specifically need to build an image to run it, although it is more convenient.

To use minirouter, you must have the miniccc agent running, and minirouter must be able to access the miniccc tool and files directory (see minirouter -h for default paths).

minirouter uses iptool, dnsmasq, dhclient, and bird, all of which must be installed but not already running. minirouter must run as root.

Beyond these few requirements, minirouter should run on most linux systems.

In this module, we will use minirouter to act as DHCP to assign IPs and also to assign static IPs to VMs.

VMs running the minirouter tool must have miniccc running as well (this is already configured in the prebuilt minirouter image).

for more information on miniccc, minimega's command and control tool, see module 07

Starting minirouter

The router API requires a VM name or ID when configuring a router. For example, to set a static IP on a running minirouter VM named 'foo':

router foo interface 0 10.0.0.1/24
router foo commit

While the first command above sets the configuration for the router image, the second line actually commits the configuration by sending commands to minirouter over the command and control layer in minimega. Multiple configuration commands can be issued and then later committed with a single commit command.

Interfaces

Routers often have statically assigned IP addresses and minirouter supports both IPv4 and IPv6 address specification using the interface API. For example, to add the IP 10.0.0.1/24 to the second interface on a minirouter VM:

vm config net a b

# add an ip to interface b (index 1)
router foo interface 1 10.0.0.1/24

Multiple addresses can be added to the same interface as well:

router foo interface 0 10.0.0.1/24
router foo interface 0 2001:1::1/64

DHCP with minirouter

minirouter supports DHCP assignment of connected clients and supports both IP range and static IP assignment. minirouter also supports several DHCP options such as setting the default gateway and nameserver.

For example, to serve the IP range 10.0.0.2 - 10.0.0.254 on a 10.0.0.0/24 network, specify the network prefix and DHCP range:

router foo dhcp 10.0.0.0 range 10.0.0.2 10.0.0.254
# You can also specify static IP assignments with a MAC/IP address pair:
router foo dhcp 10.0.0.0 static 00:11:22:33:44:55 10.0.0.100
# Additionally, you can specify the default gateway and nameserver:
router foo dhcp 10.0.0.0 router 10.0.0.254
router foo dhcp 10.0.0.0 dns 8.8.8.8

All of these DHCP options can be used together in a single DHCP specification, and multiple DHCP servers can be specified on a single minirouter instance (for serving DHCP on multiple interfaces/networks).

minirouter - other features

minirouter supports IPv6 router advertisements using the Neighbor Discovery Protocol to enable SLAAC addressing. To enable route advertisements simply provide the subnet. Only the subnet prefix is required as SLAAC addressing requires a /64 and is implied.

router foo ra 2001:1:2:3::

minirouter provides a simple mechanism to add A or AAAA records for any host/IP (including IPv6) pair. Simply specify the host and IP address of the record:

router foo dns 1.2.3.4 foo.com

Routing

minirouter uses the bird routing daemon to provide routing using a variety of protocols. Currently, minirouter only supports static and OSPF routes. Support for BGP and others are planned.

Bird is a lightweight routing daemon that scales well. In our tests we were able to scale minirouter with bird to at least 40 gigabit

Routing - Static Routes

minirouter makes possible adding IPv4 or IPv6 static routes by simply specifying the destination network and net-hop IP. For example, to add a static IPv4 route for the 1.2.3.0/24 network via 1.2.3.254:

router foo route static 1.2.3.0/24 1.2.3.254

* Or to specify a default route:
router foo route static 0.0.0.0/0 1.2.3.254

* IPv6 routes are added in the same way:
router foo route static 2001:1:2:3::/64 2001:1:2:3::1

Routing - OSPF

minirouter provides basic support for OSPF and OSPFv3 (IPv6 enabled OSPF) by specifying the OSPF area and interface to include in the area. OSPF generally supports specifying networks and many other options, which minirouter may add in the future. For now, specifying an interface (and all of the networks on that interface) is provided. Both OSPF and OSPFv3 are enabled by minirouter.

Interfaces are identified by the index in which they were added by the vm config net API. For example, to add the first and third network of the router VM to area 0 in an OSPF route:

vm config net a b c

# add interface 'a', index 0
router foo route ospf 0 0

# add interface 'c', index 2
router foo route ospf 0 2

Connecting to the internet

It is sometimes useful to connect the experiment network to the Internet to install software or access external resources. This article describes the simple case of connecting a single VM to the Internet. With minor changes, this technique can also be used to connect an entire experiment to the Internet if the single VM acts as a router.

The VM must be configured with at least one network interface.

To connect this interface to the Internet, we setup a NAT on the host machine by creating a tap. Also, we need to enable IP forwarding on the host machine. And finally, configuring iptables to enable the NAT on the host machine. You may also need to configure DNS on the VM.

vm config net LAN

# Set up a NAT on the host machine by creating a tap:
tap create LAN ip 10.0.0.1/24 nat0

# Enabling IP forwarding on the host machine via the shell command:
shell sysctl -w net.ipv4.ip_forward=1

# And configuring iptables to enable the NAT on the host machine:
shell iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
shell iptables -A INPUT -i nat0 -j ACCEPT
shell iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
shell iptables -A OUTPUT -j ACCEPT
# NOTE: You may need to change eth0 to match the interface on the host machine with Internet access.

# On the VM, we would then configure a static IP of 10.0.0.2/24, using 10.0.0.1 as the default gateway. On Linux this can be achieved with the following:
ip addr add 10.0.0.2/24 dev eth0
ip route add default via 10.0.0.1

Troubleshooting

When setting up an experiment, numerous issues can prevent VMs from being able to connect to one another.

For more information on network troubleshooting, visit the network troubleshooting article at minimega.org

Next up…

Module 07: Command and control

Thank you

The minimega miniclass series

Sandia National Laboratories