由于KVM的架构为
Libvirt –> qemu –> KVM
所以对于live migration有两种方式，一种是qemu + KVM自己的方式，一种是libvirt的方式，当然libvirt也是基于qemu+kvm的方式
qemu + KVM自己的方式为使用monitor
KVM Migration
KVM currently supports savevm/loadvm and offline or live migration Migration commands are given when in qemu-monitor (Alt-Ctrl-2). Upon successful completion, the migrated VM continues to run on the destination host.
Requirements
- The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs).
- It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...")
- The src and dst hosts must be on the same subnet (keeping guest's network when tap is used).
- Do not use -snapshot qemu command line option.
- For tcp: migration protocol
the guest on the destination must be started the same way it was started on the source.
The live migration process has the following steps:
-
The virtual machine instance is running on the source host.
-
The virtual machine is started on the destination host in the frozen listening mode. The parameters used are the same as on the source host plus the
-incoming tcp:parameter, whereip:portipspecifies the IP address andportspecifies the port for listening to the incoming migration. If 0 is set as IP address, the virtual machine listens on all interfaces. -
On the source host, switch to the monitor console and use the migrate -d tcp:
destination_ip:portcommand to initiate the migration. -
To determine the state of the migration, use the info migrate command in the monitor console on the source host.
-
To cancel the migration, use the migrate_cancel command in the monitor console on the source host.
-
To set the maximum tolerable downtime for migration in seconds, use the migrate_set_downtime
number_of_secondscommand. To set the maximum speed for migration in bytes per second, use the migrate_set_speed
bytes_per_secondcommand.
要进行live migration首先要存储共享，我们这里用nfs
在一台nfs server上安装
apt-get install nfs-kernel-server
export如下的文件夹
# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
#
/home/cliu8/nfs *(rw,sync,no_root_squash)
在source机器和destination机器上，都mount这个文件夹
mount 16.158.166.150:/home/cliu8/nfs /home/cliu8/migrate
在文件夹里面放入两个image
root@escto-bj-hp-z620:/home/cliu8/migrate# ls -l
total 3842908
-rwxr-xr-x 1 root root 1717567488 Jul 18 00:01 ubuntu-14.04.img
-rwxr-xr-x 1 root root 2217869312 Jul 17 22:13 ubuntutest.img
在source机器上启动
qemu-system-x86_64 -enable-kvm -name ubuntutest -m 2048 -hda /home/cliu8/migrate/ubuntu-14.04.img -vnc :19 -net nic -net user
在dest机器上同样启动，但是多了listen的参数
qemu-system-x86_64 -enable-kvm -name ubuntutest -m 2048 -hda /home/cliu8/migrate/ubuntu-14.04.img -vnc :19 -net nic -net user -incoming tcp:0:4444
打开source机器的monitor
运行migrate -d tcp:16.158.166.150:4444
这个时候info migrate，显示Migration status: active
等变成complete，则migration结束
这个时候，另一面的机器已经起来了。
Libvirt Migration
Network data transports
migration的时候的数据传输有两种方式：
Hypervisor native transport
所谓native的transport，就是依赖于hypervisor，也即KVM的自有的机制去做网络传输，不支持加密，可能对于某些hypervisor来讲，网络还需要特殊的配置。
libvirt tunnelled transport
依赖于libvirt的RPC的网络通路进行网络传输，支持加密。
只要下面的命令能够执行，就能够传输
virsh -c qemu+ssh://cliu8@16.158.166.150/system list --all
virsh -c qemu+tcp://popsuper1982/system list --all
virsh -c qemu+tls://popsuper1982/system list --all
缺点是有很多额外的性能损耗
Communication control paths/flows
看完了数据通路，我们再来看控制通路
Migration一般涉及三个较色，admin, source, destination
Managed direct migration
admin节点控制整个migration的全过程，admin既控制source, 也控制destination，然而source和destination之间并不交互，因而migration过程中，如果admin挂了，就失败了。
Managed peer to peer migration
admin节点仅仅和source交互，告诉source，你要migrate到destination，然后source控制整个过程，如果admin在这个过程中挂了，不影响接着进行migration
注意的是，admin登录source的credential信息和source登录destination的credential信息是不一样的。
Unmanaged direct migration
admin和source都不控制migration的过程，而是admin的libvirt直接调用hypervisor的控制器，让hypervisor自己进行migration。
Configuration file handling
A transient guest only exists while it is running, and has no configuration file stored on disk.
A persistent guest maintains a configuration file on disk even when it is not running.
The virsh command has two flags to influence this behaviour.
The --undefine-source flag will cause the configuration file to be removed on the source host after a successful migration.
The --persist flag will cause a configuration file to be created on the destination host after a successful migration.
我们首先需要使得source和destination之间的libvirt是相互通的。
在source机器上，16.158.166.197
root@escto-bj-hp-z620:/home/cliu8/certtool# ls -l
total 60
-r--r--r-- 1 root root 1204 Jul 17 20:17 certificate_authority_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:17 certificate_authority_key.pem
-r--r--r-- 1 root root 37 Jul 17 20:17 certificate_authority_template.info
-r--r--r-- 1 root root 1379 Jul 17 20:17 escto-bj-hp-z620_client_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 escto-bj-hp-z620_client_key.pem
-r--r--r-- 1 root root 139 Jul 17 20:17 escto-bj-hp-z620_client_template.info
-r--r--r-- 1 root root 1310 Jul 17 20:17 escto-bj-hp-z620_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 escto-bj-hp-z620_server_key.pem
-r--r--r-- 1 root root 91 Jul 17 20:17 escto-bj-hp-z620_server_template.info
-r--r--r-- 1 root root 1371 Jul 17 20:17 popsuper1982_client_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:17 popsuper1982_client_key.pem
-r--r--r-- 1 root root 135 Jul 17 20:17 popsuper1982_client_template.info
-r--r--r-- 1 root root 1306 Jul 17 20:17 popsuper1982_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:17 popsuper1982_server_key.pem
-r--r--r-- 1 root root 87 Jul 17 20:17 popsuper1982_server_template.info
root@escto-bj-hp-z620:/home/cliu8/certtool# tree --charset ASCII /etc/pki/
/etc/pki/
|-- CA
| `-- cacert.pem -> /home/cliu8/certtool/certificate_authority_certificate.pem
|-- libvirt
| |-- clientcert.pem -> /home/cliu8/certtool/escto-bj-hp-z620_client_certificate.pem
| |-- private
| | |-- clientkey.pem -> /home/cliu8/certtool/escto-bj-hp-z620_client_key.pem
| | `-- serverkey.pem -> /home/cliu8/certtool/escto-bj-hp-z620_server_key.pem
| `-- servercert.pem -> /home/cliu8/certtool/escto-bj-hp-z620_server_certificate.pem
`-- nssdb -> /var/lib/nssdb
配置/etc/libvirt/libvirtd.conf
为了方便测试，我们对tcp, tls不进行密码设置
listen_tls = 1
listen_tcp = 1
tls_port = "16514"
tcp_port = "16509"
unix_sock_group = "libvirtd"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none"
auth_tls = "none"
测试下面的命令都能通过
virsh -c qemu+ssh://cliu8@16.158.166.150/system list --all
virsh -c qemu+tcp://popsuper1982/system list --all
virsh -c qemu+tls://popsuper1982/system list --all
在destination机器上，16.158.166.150
root@popsuper1982:/home/cliu8/certtool# ls -l
total 60
-r--r--r-- 1 root root 1204 Jul 15 22:31 certificate_authority_certificate.pem
-r--r--r-- 1 root root 1972 Jul 15 22:28 certificate_authority_key.pem
-r--r--r-- 1 root root 37 Jul 15 22:26 certificate_authority_template.info
-r--r--r-- 1 root root 1379 Jul 16 00:27 escto-bj-hp-z620_client_certificate.pem
-r--r--r-- 1 root root 1968 Jul 16 00:25 escto-bj-hp-z620_client_key.pem
-r--r--r-- 1 root root 139 Jul 16 00:24 escto-bj-hp-z620_client_template.info
-r--r--r-- 1 root root 1310 Jul 17 20:09 escto-bj-hp-z620_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 17 20:07 escto-bj-hp-z620_server_key.pem
-r--r--r-- 1 root root 91 Jul 17 20:06 escto-bj-hp-z620_server_template.info
-r--r--r-- 1 root root 1371 Jul 17 20:14 popsuper1982_client_certificate.pem
-r--r--r-- 1 root root 1972 Jul 17 20:13 popsuper1982_client_key.pem
-r--r--r-- 1 root root 135 Jul 17 20:12 popsuper1982_client_template.info
-r--r--r-- 1 root root 1306 Jul 16 00:09 popsuper1982_server_certificate.pem
-r--r--r-- 1 root root 1968 Jul 16 00:06 popsuper1982_server_key.pem
-r--r--r-- 1 root root 87 Jul 16 00:05 popsuper1982_server_template.info
root@popsuper1982:/home/cliu8/certtool# tree --charset ASCII /etc/pki/
/etc/pki/
|-- CA
| `-- cacert.pem -> /home/cliu8/certtool/certificate_authority_certificate.pem
|-- libvirt
| |-- clientcert.pem -> /home/cliu8/certtool/popsuper1982_client_certificate.pem
| |-- private
| | |-- clientkey.pem -> /home/cliu8/certtool/popsuper1982_client_key.pem
| | `-- serverkey.pem -> /home/cliu8/certtool/popsuper1982_server_key.pem
| `-- servercert.pem -> /home/cliu8/certtool/popsuper1982_server_certificate.pem
`-- nssdb -> /var/lib/nssdb
virsh -c qemu+ssh://cliu8@16.158.166.197/system list --all
virsh -c qemu+tcp://escto-bj-hp-z620/system list --all
virsh -c qemu+tls://escto-bj-hp-z620/system list –all
在source机器上，启动一个虚拟机virsh start ubuntu-14.04
# virsh dumpxml ubuntu-14.04
<domain type='kvm' id='55'>
<name>ubuntu-14.04</name>
<uuid>0f0806ab-531d-6134-5def-c5b495529284</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/home/cliu8/migrate/ubuntu-14.04.img'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:11:9b:d5:11'/>
<source bridge='ubuntu_br'/>
<virtualport type='openvswitch'>
<parameters interfaceid='18a45d7e-d96b-4b9e-9d92-dc9ff3ea77e0'/>
</virtualport>
<target dev='vnet8'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/16'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/16'>
<source path='/dev/pts/16'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5908' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</memballoon>
</devices>
<seclabel type='none'/>
</domain>
其中cdrom和usb的都应该去掉，才能migration成功。
# virsh migrate --verbose --live --persistent ubuntu-14.04 qemu+tcp://popsuper1982/system
Migration: [100 %]