namespaces allows users to say they want VMs of a certain configurations and minimega will do the heavy lifting underneath to schedule those VMs across a cluster of nodes. This article describes some of that heavy lifting. It assumes that you have already read the article describing namespaces for users.
One of the major design goals of namespaces was to make as few changes to the
existing API as possible. Ideally, scripts that ran in minimega 2.2 should run
the exact same on 2.3. We achieved this for the most part with only a minor
modification to the
vm launch command.
The active namespace is stored in the
namespace global string. The empty
string means that there is no active namespace.
VMs, VLANs, and taps all belong to the active namespace when they were created.
Commands that list these resources (e.g.
minimega's internal data structures to limit the results to just those that are
part of the active namespace. When there is no active namespace, these commands
list resources across all namespaces.
Nodes may belong to one or more namespaces and are listed as part of the
namespace command. For brevity, we will refer to the nodes that belong to the
active namespace as active nodes.
API handlers may behave differently depending on whether a namespace is active
or not. For example,
vm kill all should kill all VMs that belong to the
namespace, regardless of which active nodes the VMs are running on while
vm kill all without an active namespace should apply to all locally running
VMs, regardless of namespace. The
host command should return information about
just the host executing the command but when a namespace is active, it should
collect the information from all the active nodes. API handlers must be able to
do both -- the namespace-active behavior, and the local-instance behavior.
Fortunately, when looking at the API, we were able to divide the
namespace-active behaviors into three categories of commands:
* `local`: always do `local` behavior * `broadcast`: all active nodes do `local` behavior * `vm`target`: all active nodes with a targeted VM do `local` behavior
When a namespace is not active, the local-instance behavior is always applied.
In the pre-namespaces code, minimega API handlers typically used the
wrapSimpleCLI function to wrap a handler that returns a single
minicli.Response into one that sends a slice of responses over the provided
channel. In the post-namespaces code, using
wrapSimpleCLI is equivalent to the
local behavior. Two new functions:
vm target behaviors, respectively.
wrapVMTargetCLI check whether there is a namespace
active, and, if there is, perform their respective fan out phases. For
broadcast, this is simply calling
mesh send all with the original command
embedded in a
namespace <namespace> (command) command and collecting the
responses. Note that
all resolves to the nodes that belong in the active
namespace and that the embedding ensures that the correct namespace is active on
the remote node when it executes the command. For
vm target, we perform the
same action, however, we do post-filtering of the responses in order to filter
out the `vm not found` errors unless there are no successful responses.
One complication with the above approach is that how does the remote node know
that it should perform the local-instance behavior rather than trying to fan out
again? Without some mechanism to resolve this, we would fan out again and cause
a deadlock. To prevent this, we tag the outgoing
minicli.Command using the
Source field. Specifically, we set the
Source field to the active namespace.
wrapVMTargetCLI handlers check the
against the active namespace and, if it matches, perform the local-instance
behavior. Otherwise, they will `fan out`.
vm launch is special -- when a namespace is active, we want to queue the
configured VM to launch when the user calls
vm launch with no arguments.
vm launch performs the same check of the
Source field as
wrapVMTargetCLI to determine whether to execute the local-instance or
namespace-active behavior. When VMs are queued, the current
parameters are saved in the
namespaces data structure.
The scheduler is pretty simple at this point. It applies a simple round-robin
approach to schedule an even number of VMs across the nodes. Specifically, it
takes a queued VM (which may be one or more VMs of the same type) and blasts the
vm config commands to the selected remote node. If all of those
succeeded, it then instructs the remote node to launch the correct number of VMs
of that type.