One of the early feature requests for minimega was a scheduler that would launch
VMs across a cluster of machines as easily as VMs are launched on a single
machine. In minimega 2.3, we introduced the concept of
attempts to provide this functionality.
namespaces are a way to automatically pool resources across a cluster. Specifically, namespaces allow you to configure and launch VMs without worrying too much about which host that they actually run on. namespaces also provide a logical separation between experiments, allowing for multitenancy among cooperating users.
One of the design goals for namespaces was to minimize changes to the existing API. Specifically, we wanted to allow users to create the same scripts to run experiments on a single host and on a cluster of hundreds of hosts. To support this, there are minimal changes to the existing APIs (except behind the scenes, of course) and a few new namespace-specific APIs.
namespaces are managed by the
namespace API. For example, to create a new
foo and set it as the active namespace:
minimega$ namespace foo minimega[foo]$
Now that the namespace
foo is active, commands will apply only to resources,
such as VMs, that belong to the namespace. In a clustered environment, a
newly-created namespace includes all nodes in the mesh except the local node,
which is treated as the head node. When there are not any nodes in the mesh, the
namespace includes just the local node.
To deactivate a namespace, use:
minimega[foo]$ clear namespace minimega$
When run without arguments,
namespace displays the available namespaces. If a
namespace is active,
namespace displays information about the active namespace
minimega$ namespace Namespaces: [foo] minimega$ namespace foo minimega[foo]$ namespace Namespace: "foo" Hosts: map Number of queuedVMs: 0 Schedules: start end state launched failures total hosts
The displayed information includes the hosts that are part of the namespace, how many VM configurations have been queued so far (explained below), and the status of any schedules that have been started.
To make it easier to run commands that target a namespace, users may prefix
commands with the namespace they with to use. For example, to display
information about VMs running inside the
foo namespace, any of the following
minimega$ namespace foo minimega[foo]$ .columns name,state,namespace vm info name | state | namespace vm-foo-0 | BUILDING | foo minimega$ namespace foo .columns name,state,namespace vm info name | state | namespace vm-foo-0 | BUILDING | foo minimega$ .columns name,state,namespace namespace foo vm info name | state | namespace vm-foo-0 | BUILDING | foo
Finally, to delete a namespace, again use the
clear namespace API:
minimega$ clear namespace foo
nsmod API allows users to configure parameters of the active namespace
such as which hosts belong to the namespace.
To add hosts to the namespace, use
minimega$ nsmod add-host ccc[1-10]
To remove hosts, use
minimega$ nsmod del-host ccc[1,3,5,7,9]
minimega only adds hosts that are already part of the mesh.
nsmod API also allows you to control parameters of the scheduler such as
how the scheduler determines which host is the least loaded. This is done via
nsmod load API:
minimega$ nsmod load cpucommit
See the Scheduler section below for a description of the different ways the scheduler can compute load.
Launching VMs in a namespace is similar to launching VMs without namespaces. VMs
are configured exactly the same as before -- with the
vm config APIs. The only
difference for users is queued launching. Specifically, when the user calls
vm launch the specified VMs are not created immediately -- they are instead
added to a queue. This queue allows the scheduler to make smarter decisions
about where it launches VMs. For example, the scheduler could schedule VMs with
the same VLANs or disk image on the same host.
Each call to
vm launch queues a new VM:
minimega$ namespace foo minimega[foo]$ vm launch kvm a minimega[foo]$ vm launch kvm b minimega[foo]$ vm info minimega[foo]$ namespace Namespace: "foo" Hosts: map Number of queuedVMs: 2 Schedules: start end state launched failures total hosts
vm launch with no additional arguments flushes the queue and invokes
minimega[foo]$ namespace shepherd: Namespace: "foo" Hosts: ccc[1-15] Number of queuedVMs: 0 Schedules: start end state launched failures total hosts 2016-03-22 18:04:50.217220992 -0700 PDT 2016-03-22 18:04:50.236229396 -0700 PDT completed 2 0 2 1
The scheduler, described below, distributes the queued VMs to nodes in the
namespace and starts them. Once the queue is flushed, the VMs become visible in
The scheduler for namespaces is fairly simple -- for each VM, it finds the least loaded node and schedules the VM on it. Load is calculated in one of the following ways:
* CPU commit : Sum of the Virtual CPUs across all launched VMs. * Network commit : Sum of the count of network interfaces across all launched VMs. * Memory load : Sum of the total memory minus the total memory reserved for all launched VMs.
These values are summed across all VMs running on the host, regardless of namespace. This means that the scheduler will avoid launching new VMs on already busy nodes if there are multiple namespaces are using the same nodes or there are VMs running outside of a namespace.
In order to allow users to statically schedule some portions of their experiment
(such as when there is hardware or people in the loop), we have added two new
vm config APIs:
* vm config schedule : schedule these VMs on a particular node. * vm config coschedule : limit the number of coscheduled VMs
These two APIs can be used together or separately:
minimega$ vm config schedule ccc50 minimega$ vm config coschedule 0 minimega$ vm launch kvm solo
Instructs the scheduler to launch a VM called
solo on ccc50 and not to
schedule any other VMs on ccc50.
minimega$ vm config coschedule 0 minimega$ vm launch kvm solo
Instructs the scheduler to launch a VM called
solo on any node and not to
schedule any other VMs on that node.
minimega$ vm config coschedule 3 minimega$ vm launch kvm quad[0-3]
Instructs the scheduler to launch four VMs called quad[0-3] on any node and not to schedule at most four other VMs on those nodes. Note: because of the way the least loaded scheduler works, quad[0-3] will most likely not be scheduled on the same node.
Besides the changes noted above to
vm launch, many of the
vm APIs are now
* `vm`info` * `vm`flush`
These commands are broadcast out to all hosts in the namespace and the responses
are collected on the issuing node. These are roughly equivalent to doing
mesh send <hosts> vm info where
<hosts> is the list of nodes in the
APIs that target one or more VMs now apply to VMs across the namespace on any host. This includes:
* `vm`start` * `vm`pause` * `vm`kill` * `vm`hotplug` * `vm`qmp` * `vm`screenshot` * `vm`tag` * `vm`cdrom`
Note: because of the above changes, minimega now enforces globally unique VM
names within a namespace. VMs of the same name can exist in different
namespaces. However, minimega does not reuse VM IDs which means that
vm kill 0
will kill all VMs with ID 0 on hosts in the active namespace regardless of
whether those VMs belong to the namespace or not.
mesh API has a minor tweak -- when a namespace is active,
mesh send all
all to hosts in the namespace rather than all hosts in the
cc API adds an implicit filter for VMs running in the active namespace.
host API broadcasts the
host command to all hosts in the namespace and
collects the responses on the issuing host when a namespace is active.
Otherwise, it only reports information for the issuing node.