namespaces allows users to say they want VMs of a certain configurations and minimega will do the heavy lifting underneath to schedule those VMs across a cluster of nodes. This article describes some of that heavy lifting. It assumes that you have already read the article describing namespaces for users.


One of the major design goals of namespaces was to make as few changes to the existing API as possible. Ideally, scripts that ran in minimega 2.2 should run the exact same on 2.3. We achieved this for the most part with only a minor modification to the vm launch command.

Storing the namespace

The active namespace is stored in the namespace global string. The empty string means that there is no active namespace.

namespace resources

VMs, VLANs, and taps all belong to the active namespace when they were created. Commands that list these resources (e.g. vm info, vlans, and taps) filter minimega's internal data structures to limit the results to just those that are part of the active namespace. When there is no active namespace, these commands list resources across all namespaces.

Nodes may belong to one or more namespaces and are listed as part of the namespace command. For brevity, we will refer to the nodes that belong to the active namespace as active nodes.

API handler duality

API handlers may behave differently depending on whether a namespace is active or not. For example, vm kill all should kill all VMs that belong to the namespace, regardless of which active nodes the VMs are running on while vm kill all without an active namespace should apply to all locally running VMs, regardless of namespace. The host command should return information about just the host executing the command but when a namespace is active, it should collect the information from all the active nodes. API handlers must be able to do both -- the namespace-active behavior, and the local-instance behavior. Fortunately, when looking at the API, we were able to divide the namespace-active behaviors into three categories of commands:

* `local`: always do `local` behavior
* `broadcast`: all active nodes do `local` behavior
* `vm`target`: all active nodes with a targeted VM do `local` behavior

When a namespace is not active, the local-instance behavior is always applied. In the pre-namespaces code, minimega API handlers typically used the wrapSimpleCLI function to wrap a handler that returns a single minicli.Response into one that sends a slice of responses over the provided channel. In the post-namespaces code, using wrapSimpleCLI is equivalent to the local behavior. Two new functions: wrapBroadcastCLI and wrapVMTargetCLI provide the broadcast and vm target behaviors, respectively.

wrapBroadcastCLI and wrapVMTargetCLI check whether there is a namespace active, and, if there is, perform their respective fan out phases. For broadcast, this is simply calling mesh send all with the original command embedded in a namespace <namespace> (command) command and collecting the responses. Note that all resolves to the nodes that belong in the active namespace and that the embedding ensures that the correct namespace is active on the remote node when it executes the command. For vm target, we perform the same action, however, we do post-filtering of the responses in order to filter out the `vm not found` errors unless there are no successful responses.

One complication with the above approach is that how does the remote node know that it should perform the local-instance behavior rather than trying to fan out again? Without some mechanism to resolve this, we would fan out again and cause a deadlock. To prevent this, we tag the outgoing minicli.Command using the Source field. Specifically, we set the Source field to the active namespace. The wrapBroadcastCLI and wrapVMTargetCLI handlers check the Source field against the active namespace and, if it matches, perform the local-instance behavior. Otherwise, they will `fan out`.


vm launch is special -- when a namespace is active, we want to queue the configured VM to launch when the user calls vm launch with no arguments. vm launch performs the same check of the Source field as wrapBroadcastCLI and wrapVMTargetCLI to determine whether to execute the local-instance or namespace-active behavior. When VMs are queued, the current vm config parameters are saved in the namespaces data structure.


The scheduler is pretty simple at this point. It applies a simple round-robin approach to schedule an even number of VMs across the nodes. Specifically, it takes a queued VM (which may be one or more VMs of the same type) and blasts the necessary vm config commands to the selected remote node. If all of those succeeded, it then instructs the remote node to launch the correct number of VMs of that type.


The minimega authors

22 Apr 2016