文章大纲
KubeVirt 支持虚拟机快照和克隆，但是都依赖 Kubernetes 中 CSI 提供的功能。
快照
在做快照之前需要创建 VolumeSnapshot 资源，参考以下连接：
安装好 snapshot controller 和创建好 VolumeSnapshotClass 后，可以通过以下命令进行检查：
vagrant@master01:~$ kubectl get volumesnapshotclass
NAME DRIVER DELETIONPOLICY AGE
csi-rbdplugin-snapclass rook-ceph.rbd.csi.ceph.com Delete 10s启用快照的特性
同样的， KubeVirt 的快照功能也是需要手动开启的，可以直接编辑 kubevirt 进行开启：
vagrant@master01:~$ kubectl edit kubevirt kubevirt -n kubevirt
spec:
certificateRotateStrategy: {}
configuration:
developerConfiguration:
featureGates:
- HotplugVolumes
- Snapshot创建快照
KubeVirt 支持对在线和离线的虚拟机创建快照。
对于在线的虚拟机创建快照需要 qemu agent guest 处于运行的状态，确保在拍摄快照的时候冻结文件系统，拍摄完成后解除冻结，如果虚拟机没有运行 qemu agent guest 将无法拍摄运行时的快照，只能关机拍摄快照。
查看虚拟机中的 qemu agent guest 的状态，只需查看 VM 中的 AgentConnected ：
vagrant@master01:~$ k describe vm fedora-vm -n cdi-test | grep -A 1 Agent
Type: AgentConnected
Created: true使用以下资源定义为 cdi-test 命令空间中的 fedora-vm 创建快照：
apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
name: snap-fedora-vm
spec:
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: fedora-vm拍摄快照：
vagrant@master01:~$ kubectl get vmi -n cdi-test
NAME AGE PHASE IP NODENAME READY
fedora-vm 40m Running 10.244.241.93 master01 True
vagrant@master01:~$ kubectl -n cdi-test apply -f snap-fedora-vm.yaml
virtualmachinesnapshot.snapshot.kubevirt.io/snap-fedora-vm created
vagrant@master01:~$ kubectl wait vmsnapshot snap-fedora-vm -n cdi-test --for condition=Ready
virtualmachinesnapshot.snapshot.kubevirt.io/snap-fedora-vm condition met
vagrant@master01:~$ kubectl get volumesnapshot -n cdi-test
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
vmsnapshot-3331182f-bd76-4f9b-9e93-c253322c1f49-volume-disk0 true fedora 5Gi csi-rbdplugin-snapclass snapcontent-77ea2f79-0411-4702-b343-1c93112a6142 17s 18s
vmsnapshot-3331182f-bd76-4f9b-9e93-c253322c1f49-volume-example-volume-hotplug true example-volume-hotplug 10Gi csi-rbdplugin-snapclass snapcontent-62f50a95-6987-44bb-b248-74988a0c7b70 17s 18s可以看到针对 fedora-vm 创建了两个快照，因为前面附加了一块磁盘。
恢复快照
在恢复快照之前，对 fedora-vm 进行内容的更改，方便恢复快照后进行对比：
vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa fedora@fedora-vm.cdi-test
Last login: Sun Apr 28 10:41:35 2024 from 10.244.30.110
[fedora@fedora-vm ~]$ sudo -i
[root@fedora-vm ~]# echo test > /etc/test.txt
[root@fedora-vm ~]# parted /dev/sda mklabel msdos
Information: You may need to update /etc/fstab.
[root@fedora-vm ~]# parted /dev/sda mkpart primary xfs 2048s 100%
Information: You may need to update /etc/fstab.
[root@fedora-vm ~]# udevadm settle
[root@fedora-vm ~]# lsblk /dev/sda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part 首先需要将 VM 停止：
vagrant@master01:~$ virtctl stop fedora-vm -n cdi-test
VM fedora-vm was scheduled to stop
vagrant@master01:~$ kubectl get vm -n cdi-test
NAME AGE STATUS READY
fedora-vm 24h Stopped False然后使用以下资源定义进行还原快照：
apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineRestore
metadata:
name: restore-fedora-vm
spec:
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: fedora-vm
virtualMachineSnapshotName: snap-fedora-vm还原快照：
vagrant@master01:~$ kubectl -n cdi-test apply -f restore-fedora-vm.yaml
virtualmachinerestore.snapshot.kubevirt.io/restore-fedora-vm created
vagrant@master01:~$ kubectl get virtualmachinerestore -n cdi-test
NAME TARGETKIND TARGETNAME COMPLETE RESTORETIME ERROR
restore-fedora-vm VirtualMachine fedora-vm true 41s启动 VM 进行检查磁盘是否被还原：
vagrant@master01:~$ virtctl start fedora-vm -n cdi-test
VM fedora-vm was scheduled to start
vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa fedora@fedora-vm.cdi-test
Last login: Sun Apr 28 10:41:35 2024 from 10.244.30.110
[fedora@fedora-vm ~]$ cat /etc/test.txt
cat: /etc/test.txt: No such file or directory
[fedora@fedora-vm ~]$ lsblk /dev/sda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 10G 0 disk可以看到两块的磁盘的内容已经还原到了拍摄快照时候的内容。
克隆
在满足 Snapshot & Restore 后，就可以对虚拟机实现克隆了，如果不满足，参考前面的链接首先创建 VolumeSnapshotClass 。
通过创建 VirtualMachineClone CDR 资源来对虚拟机进行克隆：
kind: VirtualMachineClone
apiVersion: "clone.kubevirt.io/v1alpha1"
metadata:
name: testclone
spec:
# source & target definitions
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: fedora-vm
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: fedora-vm-clone
# labels & annotations definitions
labelFilters:
- "*"
- "!someKey/*"
annotationFilters:
- "anotherKey/*"
# template labels & annotations definitions
template:
labelFilters:
- "*"
- "!someKey/*"
annotationFilters:
- "anotherKey/*"
# other identity stripping specs:
newMacAddresses:
interfaceName: "00-11-22"
newSMBiosSerial: "new-serial"其中 labelFilters 和 annotationFilters 是筛选要 copy 的标签和注释。template 一些网络 CNI 会将网络信息注入到 template.label 或 template.annotation 中，此处也是筛选要 copy 的标签和注释。newMacAddresses 是在克隆时为特定的网卡设置新的 Mac Addresses。newSMBiosSerial 也是一样克隆时生成新的 SMBios Serial。
克隆虚拟机：
vagrant@master01:~$ cat testclone.yaml
kind: VirtualMachineClone
apiVersion: "clone.kubevirt.io/v1alpha1"
metadata:
name: testclone
spec:
# source & target definitions
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: fedora-vm
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: fedora-vm-clone
# labels & annotations definitions
labelFilters:
- "*"
annotationFilters:
- "*"
# template labels & annotations definitions
template:
labelFilters:
- "*"
annotationFilters:
- "*"
vagrant@master01:~$ kubectl -n cdi-test apply -f testclone.yaml
virtualmachineclone.clone.kubevirt.io/testclone created验证克隆的虚拟机：
vagrant@master01:~$ kubectl get virtualmachineclone -n cdi-test
NAME PHASE SOURCEVIRTUALMACHINE TARGETVIRTUALMACHINE
testclone Succeeded fedora-vm fedora-vm-clone
vagrant@master01:~$ kubectl get vm -n cdi-test
NAME AGE STATUS READY
fedora-vm 25h Running True
fedora-vm-clone 51s Stopped False
vagrant@master01:~$ virtctl start fedora-vm-clone -n cdi-test
VM fedora-vm-clone was scheduled to start
vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa fedora@fedora-vm-clone.cdi-test
The authenticity of host 'vmi/fedora-vm-clone.cdi-test:22 (192.168.121.1:7890)' can't be established.
ECDSA key fingerprint is SHA256:xyti2s597SmKOZ+m4Jalsgi5CdW65CvlzjgvasU/v6s.
Are you sure you want to continue connecting (yes/no)? yes
Last login: Mon Apr 29 06:43:51 2024 from 10.244.30.66
[fedora@fedora-vm-clone ~]$
[fedora@fedora-vm-clone ~]$ hostnamectl status
Static hostname: fedora-vm-clone
Icon name: computer-vm
Chassis: vm 🖴
Machine ID: bb0bebea15a455d18a61c5e10c294b1d
Boot ID: 68efaf1d0f2149a8811ed34e7e6909df
Virtualization: qemu
Operating System: Fedora Linux 37 (Cloud Edition)
CPE OS Name: cpe:/o:fedoraproject:fedora:37
Kernel: Linux 6.0.7-301.fc37.x86_64
Architecture: x86-64
Hardware Vendor: KubeVirt
Hardware Model: None
Firmware Version: 1.16.1-1.el9虚拟机封装
当从官网下载构建好的虚拟机后，在企业中针对每个虚拟机还要做好安全基线，监控等任务，如果每个虚拟机都手动一遍一遍从头操作的话，效率很低下，比较好的方法是制作虚拟机的模板，然后基于模板创建虚拟机。
基于现有的虚拟机做模板首先需要对虚拟机进行封装（sysprep），封装的过程会清楚主机特有的信息。
先假设对默认的虚拟机做设置：
vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa root@centos9s-vm.vm-clone
The authenticity of host 'vmi/centos9s-vm.vm-clone:22 (192.168.121.168:6443)' can't be established.
ECDSA key fingerprint is SHA256:rMimCVGTHp59dJFFJRfEs8ItM67IsmuR4CCOyq1zCFg.
Are you sure you want to continue connecting (yes/no)? yes
Activate the web console with: systemctl enable --now cockpit.socket
[root@centos9s-vm ~]# echo "CentOS Stream 9 Template" > /etc/motd在对虚拟机封装之前，首先要停止虚拟机：
vagrant@master01:~$ virtctl stop centos9s-vm -n vm-clone
VM centos9s-vm was scheduled to stop
vagrant@master01:~$ kubectl get vm -n vm-clone
NAME AGE STATUS READY
centos9s-vm 9m14s Stopped False使用 virtctl guestfs 对虚拟机使用的 PVC 进行挂载：
vagrant@master01:~$ virtctl guestfs centos9s -n vm-clone
Use image: quay.io/kubevirt/libguestfs-tools:v1.2.0
The PVC has been mounted at /dev/vda
Waiting for container libguestfs still in pending, reason: ContainerCreating, message:
If you don't see a command prompt, try pressing enter.
bash-5.1$此时 PVC 已经挂载到了 /dev/vda ，接下来对 vda 进行封装：
bash-5.1$ virt-sysprep -a /dev/vda
[ 0.0] Examining the guest ...
[ 6.5] Performing "abrt-data" ...
[ 6.6] Performing "backup-files" ...
[ 7.6] Performing "bash-history" ...
[ 7.6] Performing "blkid-tab" ...
[ 7.7] Performing "crash-data" ...
[ 7.7] Performing "cron-spool" ...
[ 7.7] Performing "dhcp-client-state" ...
[ 7.7] Performing "dhcp-server-state" ...
[ 7.7] Performing "dovecot-data" ...
[ 7.8] Performing "ipa-client" ...
[ 7.8] Performing "kerberos-hostkeytab" ...
[ 7.8] Performing "logfiles" ...
[ 7.9] Performing "lvm-system-devices" ...
[ 7.9] Performing "machine-id" ...
[ 8.0] Performing "mail-spool" ...
[ 8.0] Performing "net-hostname" ...
[ 8.0] Performing "net-hwaddr" ...
[ 8.1] Performing "net-nmconn" ...
[ 8.1] Performing "pacct-log" ...
[ 8.1] Performing "package-manager-cache" ...
[ 8.2] Performing "pam-data" ...
[ 8.2] Performing "passwd-backups" ...
[ 8.2] Performing "puppet-data-log" ...
[ 8.2] Performing "rh-subscription-manager" ...
[ 8.2] Performing "rhn-systemid" ...
[ 8.3] Performing "rpm-db" ...
[ 8.3] Performing "samba-db-log" ...
[ 8.3] Performing "script" ...
[ 8.3] Performing "smolt-uuid" ...
[ 8.3] Performing "ssh-hostkeys" ...
[ 8.3] Performing "ssh-userdir" ...
[ 8.4] Performing "sssd-db-log" ...
[ 8.4] Performing "tmp-files" ...
[ 8.4] Performing "udev-persistent-net" ...
[ 8.4] Performing "utmp" ...
[ 8.4] Performing "yum-uuid" ...
[ 8.4] Performing "customize" ...
[ 8.4] Setting a random seed
[ 8.5] Setting the machine ID in /etc/machine-id
[ 8.5] SELinux relabelling
[ 27.8] Performing "lvm-uuids" ...执行完毕后 exit 退出即可，此时不要启动 VM，否则又会生成主机特有信息。
如果虚拟机使用的文件系统是 Btrfs ，将会遇到错误（所以我使用了 CentOS9Stream 作为测试）：
bash-5.1$ virt-sysprep -a /dev/vda
[ 0.0] Examining the guest ...
virt-sysprep: error: libguestfs error: inspect_os: mount exited with status
32: mount: /tmp/btrfse339EF: unknown filesystem type 'btrfs'.
If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:
virt-sysprep -v -x [...]通过使用 DataVolume 对虚拟机的根盘的克隆：
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: "centos9s-template"
spec:
storage:
resources:
requests:
storage: 30Gi
source:
pvc:
name: centos9s
namespace: vm-clone应用：
vagrant@master01:~$ kubectl -n vm-clone apply -f dv_template.yaml
datavolume.cdi.kubevirt.io/centos9s-template created
vagrant@master01:~$
vagrant@master01:~$ kubectl get dv -n vm-clone
NAME PHASE PROGRESS RESTARTS AGE
centos9s Succeeded 100.0% 3h34m
centos9s-template Succeeded 100.0% 9s后续可以基于 centos9s-template 再做克隆卷，然后用于虚拟机。
vagrant@master01:~$ kubectl get dv -n vm-clone
NAME PHASE PROGRESS RESTARTS AGE
centos9s Succeeded 100.0% 3h37m
centos9s-template Succeeded 100.0% 3m18s
centos9s-template-vm1 Succeeded 100.0% 23s基于 centos9s-template 创建了 centos9s-template-vm1 ，然后将 centos-template-vm1 用于 centos9s-template-vm1 虚拟机（此处仅仅是快速演示，最佳实践是创建 Virtual Machine Template 这个需要基于 OpenShift 的 Template 资源）：
vagrant@master01:~$ kubectl get vm -n vm-clone
NAME AGE STATUS READY
centos9s-template-vm1 13s Running True
centos9s-vm 22m Stopped False
vagrant@master01:~$ virtctl ssh -i .ssh/id_rsa root@centos9s-template-vm1.vm-clone
The authenticity of host 'vmi/centos9s-template-vm1.vm-clone:22 (192.168.121.168:6443)' can't be established.
ECDSA key fingerprint is SHA256:l+naPMcmcDr+zslu4Uqakpuxq9Mbfa1XwzbmObYkPRs.
Are you sure you want to continue connecting (yes/no)? yes
CentOS Stream 9 Template
Activate the web console with: systemctl enable --now cockpit.socket可以看到之前写入 motd 的内容了。