文章大纲
默认情况下，Kubernetes 管理 Pod 网络和 Service 网络，Pod 网络为每个 Pod 提供网络接口，用于 Pod 之间的网络通信。Service 网络是为 Pod 提供稳定的寻址。
Multus CNI 插件可以将 Pod 附加到自定义的网络，来提供特定流量的性能豁满足安全需要。
Multus 网络示意图：
在 Pod 中默认的网络接口名称为 eth0 ，通过 Multus 添加的网络接口名称为 netX 。
安装 Multus
如果使用现有的自定义网络，首先必须确保在集群节点上使网络可用。
安装 Multus 的难易程度取决于所使用的 K8S 发行版，例如 OpenShift 可以方便的从 Operator Hub 里面安装，Rancher 可以在创建集群的时候选择使用 Muluts 网络。
在这里使用的方式是通过 Rancher 部署 RKE2 集群时，选择了 mulus,calico 网络。
验证相关 Pod 是否运行正常：
[vagrant@master01 ~]$ kubectl get pods --all-namespaces | grep -i multus
kube-system helm-install-rke2-multus-c5hbb 0/1 Completed 0 2d12h
kube-system rke2-multus-bn9kn 1/1 Running 5 (53m ago) 2d12h
kube-system rke2-multus-h54z4 1/1 Running 7 (52m ago) 2d4h
kube-system rke2-multus-mklpk 1/1 Running 13 (52m ago) 2d4h在每个节点上都自动生成了 /etc/cni/net.d/00-multus.conf 配置文件：
[vagrant@master01 ~]$ sudo cat /etc/cni/net.d/00-multus.conf
{
"cniVersion": "0.3.1",
"name": "multus-cni-network",
"type": "multus",
"capabilities": {"bandwidth":true,"portMappings":true},
"kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",
"delegates": [
{"cniVersion":"0.3.1","name":"k8s-pod-network","plugins":[{"container_settings":{"allow_ip_forwarding":false},"datastore_type":"kubernetes","ipam":{"assign_ipv4":"true","assign_ipv6":"false","type":"calico-ipam"},"kubernetes":{"k8s_api_root":"https://10.43.0.1:443","kubeconfig":"/etc/cni/net.d/calico-kubeconfig"},"log_file_max_age":30,"log_file_max_count":10,"log_file_max_size":100,"log_file_path":"/var/log/calico/cni/cni.log","log_level":"Info","mtu":0,"nodename_file_optional":false,"policy":{"type":"k8s"},"type":"calico"},{"capabilities":{"bandwidth":true},"type":"bandwidth"},{"capabilities":{"portMappings":true},"snat":true,"type":"portmap"}]}
]
}创建 NetworkAttachmenDefinition
要创建附加接口，需要创建自定义资源用于定义针对接口的 CNI 配置。
这种资源称为 NetworkAttachmentDefinition 。
创建一个名为 example-nad 的 NetworkAttachmentDefinition 资源：
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: example-nad
namespace: example-multus
spec:
config: |-
{
"cniVersion": "0.3.1",
"type": "ipvlan",
"master": "eth1",
"ipam": {
"type": "static",
"routes": [
{ "dst": "192.168.122.0/24", "gw": "192.168.122.1" }
]
}
}中创建该资源并查看：
kubectl apply -f example-nad.yml
networkattachmentdefinition.k8s.cni.cncf.io/example-nad created
[vagrant@master01 ~]$ kubectl describe net-attach-def example-nad -n example-multus
Name: example-nad
Namespace: example-multus
Labels: <none>
Annotations: <none>
API Version: k8s.cni.cncf.io/v1
Kind: NetworkAttachmentDefinition
Metadata:
Creation Timestamp: 2024-05-05T11:45:32Z
Generation: 1
Resource Version: 481874
UID: 413fcfb7-fbc9-44b0-83fe-730fb72c1e31
Spec:
Config: {
"cniVersion": "0.3.1",
"type": "ipvlan",
"master": "eth1",
"ipam": {
"type": "static",
"routes": [
{ "dst": "192.168.122.0/24", "gw": "192.168.122.1" }
]
}
}
Events: <none>创建一个测试 Pod 来使用 example-nad 这个附加网络：
apiVersion: v1
kind: Pod
metadata:
name: example-nad-pod
namespace: example-multus
annotations:
k8s.v1.cni.cncf.io/networks: |
[{
"name": "example-nad",
"namespace": "example-multus",
"ips": ["192.168.122.188/24"]
}]
spec:
containers:
- name: samplepod
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine创建并查看状态：
[vagrant@master01 ~]$ kubectl create -f example-nad-pod.yml
pod/example-nad-pod created
[vagrant@master01 ~]$ k get pod -n example-multus
NAME READY STATUS RESTARTS AGE
example-nad-pod 1/1 Running 0 10s
[vagrant@master01 ~]$ kubectl describe pod example-nad-pod -n example-multus
Name: example-nad-pod
Namespace: example-multus
Priority: 0
Service Account: default
Node: worker01.example.com/192.168.121.27
Start Time: Sun, 05 May 2024 13:05:38 +0000
Labels: <none>
Annotations: cni.projectcalico.org/containerID: 81dcd5467f0d8be6275f61c0b22329086c5bc18a7fa595031696bd200b6708a0
cni.projectcalico.org/podIP: 10.42.95.152/32
cni.projectcalico.org/podIPs: 10.42.95.152/32
k8s.v1.cni.cncf.io/network-status:
[{
"name": "k8s-pod-network",
"ips": [
"10.42.95.152"
],
"default": true,
"dns": {}
},{
"name": "example-multus/example-nad",
"interface": "net1",
"ips": [
"192.168.122.188"
],
"mac": "52:54:00:26:5a:9f",
"dns": {}
}]
k8s.v1.cni.cncf.io/networks:
[{
"name": "example-nad",
"namespace": "example-multus",
"ips": ["192.168.122.188/24"]
}]
Status: Running
IP: 10.42.95.152
IPs:
IP: 10.42.95.152
Containers:
samplepod:
Container ID: containerd://ff28ad8ac418342ca212763184600e313b287bc8e92b386ef3e81de52f6f953c
Image: alpine
Image ID: docker.io/library/alpine@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
Port: <none>
Host Port: <none>
Command:
/bin/ash
-c
trap : TERM INT; sleep infinity & wait
State: Running
Started: Sun, 05 May 2024 13:05:41 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d44h6 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-d44h6:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 76s default-scheduler Successfully assigned example-multus/example-nad-pod to worker01.example.com
Normal AddedInterface 75s multus Add eth0 [10.42.95.152/32] from k8s-pod-network
Normal AddedInterface 75s multus Add net1 [192.168.122.188/24] from example-multus/example-nad
Normal Pulling 75s kubelet Pulling image "alpine"
Normal Pulled 73s kubelet Successfully pulled image "alpine" in 2.218s (2.218s including waiting)
Normal Created 73s kubelet Created container samplepod
Normal Started 73s kubelet Started container samplepod在 Events 中可以看到已经将 192.168.122.188 分配给 Pod 。
在 Pod 中查看网络信息：
[vagrant@master01 ~]$ kubectl -n example-multus exec example-nad-pod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
link/ether 96:fa:00:7d:38:8a brd ff:ff:ff:ff:ff:ff
inet 10.42.95.152/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::94fa:ff:fe7d:388a/64 scope link
valid_lft forever preferred_lft forever
3: net1@net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 52:54:00:26:5a:9f brd ff:ff:ff:ff:ff:ff
inet 192.168.122.188/24 brd 192.168.122.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::5254:0:126:5a9f/64 scope link
valid_lft forever preferred_lft forever其中 eth0 是默认的 Pod 网络，net1 是通过 Multus 添加的附加网络。
此时可以从集群外部与 Pod 进行通信：
hcai@P1-Gen4:~$ ip a s virbr0
5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:e8:39:b6 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
hcai@P1-Gen4:~$ ping -c 4 192.168.122.188
PING 192.168.122.188 (192.168.122.188) 56(84) bytes of data.
64 bytes from 192.168.122.188: icmp_seq=1 ttl=64 time=0.244 ms
64 bytes from 192.168.122.188: icmp_seq=2 ttl=64 time=0.242 ms
64 bytes from 192.168.122.188: icmp_seq=3 ttl=64 time=0.175 ms
64 bytes from 192.168.122.188: icmp_seq=4 ttl=64 time=0.144 ms
--- 192.168.122.188 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3049ms
rtt min/avg/max/mdev = 0.144/0.201/0.244/0.043 msNetworkAttachDefiniton 类型
可以创建以下 NAD 类型：
host-device：将网络接口附加到单个 Podbridge：在节点上使用现有的网络接口，或者配置一个新的网桥，连接到该网络的 Pod 可以通过 bridge 相互通信，也可以与连接到该 bridge 的任何其它网络通信IPVLAN：创建基于 IPVLAN 并附加到网络接口的网络MACVLAN： 创建基于 MACVLAN 并附加网络接口的网络
其中 bridge 非常适合虚拟化环境，IPVLAN 和 MACVLAN 是专为容器环境设计的 Linux 网络驱动程序。
IPVLAN 和 MACVLAN 是两种不同的虚拟网络技术它们的主要区别在于 MAC 地址和 IP 地址的分配以及数据包的路由方式
- IPVLAN允许一个物理网卡拥有多个IP地址，而所有的虚拟接口使用同一个MAC地址，IPVLAN支持两种模式，L2 模式和 L3 模式，在 L2 模式下，每个端点获得相同的MAC地址但不同的IP地址，而在 L3 模式下，数据包在端点之间路由，提供更好的可伸缩性。
- MACVLAN 允许同一个网卡拥有多个 MAC 地址，而虚拟出的网卡可以没有IP地址，在 MACVLAN 中，通过MAC地址查找设备。
此外，IPVLAN 通过IP查找设备，而 MACVLAN 通过MAC地址查找设备。