This tutorial illustrates the use of each feature present in the command and
control (cc) API, including executing commands, file i/o, and TCP tunnels. This
is an interactive tutorial, and can be run against a running minimega instance
when minidoc is run with
In order to run the content in this tutorial you will need a running minimega instance and at least one virtual machine. We use a standard debian linux VM in the examples given here. To use the virtio service described below, your VM and host both need to have virtio support available.
The Command and Control (cc) API in minimega provides a mechanism to programmatically execute programs and send and receive files to VMs launched by minimega. In addition, the cc API allows creating TCP tunnels (as well as reverse tunnels) from the host machine to/from any VM.
The cc API supports two modes of communication with VMs. The first is a simple
TCP connection from the VM to the host via some routable network. In this mode
the VMs must be able to communicate directly with the host they are launched
on. This can be accomplished with the
tap API, or through several other
The second connection mode is over either virtio-serial (for kvm-based VMs), or over a UNIX domain socket (for container-based VMs), which provides a simple i/o layer, similar to a UNIX pipe between the host and VM. In order to use the virtio-serial mode, both host and VM must have virtio-serial support. Most modern linux distributions have this enabled by default. Windows VMs must install virtio-serial drivers, available from here. Domain sockets for container-based VMs are created by default when launching containers.
Both connection modes provide the same features, although the TCP mode can be significantly faster, depending on the underlying network.
Commands issued with the cc API (including file I/O) are always executed in order on the client. This means you can chain commands, such as "send a file, execute that file, then receive the output from that file".
Commands can also contain filters, which allow you to select which VMs execute that command. Filters are stacked, meaning you can apply filters such as "execute this command on windows VMs with an IP in the network 10.0.0.0/24."
The miniccc client uses several command line switches to control how to connect
to minimega, as well as where to store files received for the client. To
connect over TCP, provide the
-parent flag with the host/ip of host running
miniccc -parent 10.0.0.1
To use virtio-serial, instead use the
-serial flag, with the path to the
miniccc -serial /dev/virtio-ports/cc
NOTE: In linux, the default virtio-serial cc port is
In Windows, the path is
By default, miniccc will create the directory
/tmp/miniccc to store state and
files in. Files sent to the client will be stored in
can change this directory with the
Typically, miniccc is baked into images and set to start automatically when the
VM or container starts. Adding miniccc can be done in several ways.
VMs/containers built with
vmbetter can make use of overlays to add miniccc
and the necessary files to start miniccc at boot. Existing VMs/containers can
be modified to include miniccc by starting a single VM/container with `vm
config snapshot false`, adding the miniccc binary, configuring it to run
automatically, shutting down the VM, and saving the disk image or containerfs.
There are several examples of integrating miniccc into vmbetter-built VMs/containers in the repo.
To start miniccc automatically with systemd, add the following to
[Unit] Description=miniccc [Service] ExecStart=/miniccc -v=false -serial /dev/virtio-ports/cc -logfile /var/log/miniccc.log [Install] WantedBy=multi-user.target
You may need to adjust the ExecStart command if you copied miniccc elsewhere. And then enable the service:
systemctl enable miniccc.service
This will start miniccc whenever the VM starts. If you are adding to a VM with snapshot=false, you should now shutdown the VM and save the disk image.
Clients report their UUID, hostname, OS, architecture, IP and MAC addresses to minimega. This information is updated periodically, so if an IP changes, minimega will see the change.
To list clients, use the `cc clients` API.
# list the connected clients cc clients
Client information is stored by UUID in minimega. When a client responds to a command, the response is logged by minimega in a subdirectory named after the UUID for that client. We'll discuss responses later.
Executing commands is simple - just issue
cc exec with the command you want
to execute. You may need to wrap your command in quotes or escape special
characters. You can inspect current in-flight commands with
which shows the contents of the command, any applied filters (more on that
later), and how many clients have responded.
# get a directory listing of the client's root cc exec ls / # list all commands cc commands
There are two things to note at this point. First, commands don't go away until
you delete them with
cc delete command. This means that if you were to reset
a VM or start new VMs, they would all see and execute this command. Second, the
response from the client isn't printed to screen. Instead, responses are logged
in a special directory structure in minimega's base path. You can browse to the
responses yourself, or use the
cc responses command to view responses from
clients (more on that later).
For now, we'll simply wait for the client to respond by checking
and then ask minimega to print the response.
# print the response of the previous command cc responses 1
When the client responds to a
cc exec command, standard out and error are
stored in the files
Sometimes you want to execute a command on the client that doesn't return, such
as a daemon or other agent. To tell cc not to wait on a response for a given
cc background instead of
cc exec. When using background mode,
miniccc will execute the command and immediately move on to the next command
queued to run.
Processes started in the background can later be listed and killed using the
cc process API. For example:
# start some background process cc background sshd # list all processes started in the background cc process list all # you can kill a process by PID # cc process kill <pid>
The cc API supports sending and receiving files to and from the client. Files can be specified by name or glob (wildcard). When sending files, they must be rooted in a specific directory provided by minimega.
In order to send files to a client, the files must be rooted in the
subdirectory in the minimega base directory. By default minimega uses
/tmp/minimega/files. Let's send a simple bash script to the client by placing
/tmp/minimega/files (or whatever base directory your
minimega uses). Have
foo.bash do something simple, like
#!/bin/bash echo "hello cc!"
Now we'll send it to the client by using the
cc send command.
# send `foo.bash` to all clients cc send foo.bash cc commands
Clients will fetch the files from minimega before moving on to any other
commands. You can inspect file send commands with
cc commands, just like we
cc exec before.
We can also send globs (wildcard) of files, using the * operator. For example:
minimega$ cc send somedirectory/*
Files will appear in the
files subdirectory in the client's base directory.
By default, this is
At this point we can both send and execute a file on the client. Let's execute the script we sent a moment ago:
# execute foo.bash cc exec bash /tmp/miniccc/files/foo.bash
After the client responds to the
cc exec command, we can check the output.
cc responses 3
Receiving files is just like sending files, except that you can specify any path on the client to receive files from. Globs (wildcards) work with receiving files too, so you can receive entire directories of files.
Let's modify our example above and send a bash script that creates a file,
execute it, and then get that file back. We'll have our bash script create the
/foo/bar.out, and name is
#!/bin/bash mkdir /foo echo "hello cc!" >> /foo/bar.out
cc send bar.bash cc exec bash /tmp/miniccc/files/bar.bash cc recv /foo/bar.out
cc responses to check the file received after it completes. You'll notice
in the output that the file
bar.out was received and stored in the response
/foo. This is because the
/foo subdirectory was specified with
the receive command.
# check on our received file cc responses 6
We've seen the use of
cc responses several times so far. The
command simply outputs all files of the specified command, indexed by id as
cc commands, with the name of the file. If you don't want the
filename shown, you can suffix the response command with
# display our response without the client and file header cc responses 6 raw
You can also print all responses by using the
all keyword instead of a
# show all responses cc responses all
It's useful to group similar commands into named groups. The
command allows creating groups of commands by associating all issued commands
cc prefix command with that prefix. You can then use the prefix name
instead of a command id when using
cc responses, or when deleting commands.
# create a prefix cc prefix foo # and issue some commands cc exec ls / cc exec echo "foo" # we'll clear the prefix moving forward clear cc prefix cc commands
After the commands complete, we'll look at the responses.
# and look at what we've done cc responses foo
In all of the examples so far, each command is run by every connecting client.
Sometimes you want to only send commands to specific clients, say, by hostname
or IP range. The
cc filter command allows you to set a client filter that
will be applied to every command issued while the filter is assigned. For
example to send a command only to clients that have an IP in the network
# set a client filter on the 10.0.0.0/24 network cc filter ip=10.0.0.0/24 # and show the current filter cc filter
Any commands issued from now on will be executed only by clients that meet all filter fields. You can set multiple filter arguments, such as restricting commands to only windows machines in a specific IP range.
You can filter on any of UUID, hostname, architecture, OS, IP (including CIDR notation), and MAC address.
To clear the current filter, use
clear cc filter.
cc interface allows creating forward and reverse TCP tunnels over the
cc connection, including over virtio-serial connections. This is similar to
the forward and reverse tunnel support in
ssh. To create a forward tunnel,
that is, a listening port on the minimega host that is tunneled to a
destination host and port from the perspective of the client, use
When creating a forward tunnel the UUID of the client must be specified. The
destination host can be
localhost or any other host reachable from the
Similarly, a reverse tunnel, a listening port on the client tunneled to a host
and port reachable from the minimega host, can be created by using
cc rtunnel. Reverse tunnels do not require a UUID to be specified, and
instead use the current client filter to restrict which clients create the
tunnel. That means you can tunnel a port for every client to a resource outside
of the experiment.
For example, to tunnel local port 4444 on each client to a web server reachable from the minimega host:
minimega$ cc rtunnel 4444 myserver 80
miniccc client can add tags to the info for the VM the client is running
on. This enables third party tools to upstream information about a VM to
minimega via miniccc. Tags are key/value pairs, and are added simply by using
-tag switch on a running
./miniccc -tag foo bar
You may adjust the log level of clients at runtime from minimega:
minimega$ cc log level debug
This will apply to clients matching the current filter.