Flannel安装部署

Kubernetes设计了网络模型,但且将它的实现交给了网络插件,CNI网络插件最主要的功能就是实现POD资源能够跨宿主机就行通信
  • Flannel
  • Calico
  • Canal
  • Contiv
  • OpenContrail
  • NSX-T
  • Kube-router

主机环境

主机名 角色 IP
HDSS7-21.host.com flannel 10.4.7.21
HDSS7-22.host.com flannel 10.4.7.22
注意:这里部署文档以HDSS7-21.host.com主机为例,另外一台运算节点安装部署方法类似

下载软件,解压

[root@hdss7-21 src]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@hdss7-21 src]# mkdir /opt/flannel-v0.11.0
[root@hdss7-21 src]# tar xf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0
[root@hdss7-21 src]# cd /opt/flannel

拷贝证书

[root@hdss7-21 flannel]# mkdir cert
[root@hdss7-21 flannel]# cd cert/
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/ca.pem .
root@hdss7-200's password:
ca.pem                               100% 1354   109.5KB/s   00:00    
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client.pem .
root@hdss7-200's password:
client.pem                           100% 1371    64.0KB/s   00:00    
[root@hdss7-21 cert]# scp hdss7-200:/opt/certs/client-key.pem .
root@hdss7-200's password:
client-key.pem                       100% 1675    59.7KB/s   00:00
[root@hdss7-21 cert]# ll
total 12
-rw-r--r-- 1 root root 1354 Nov 29 10:09 ca.pem
-rw------- 1 root root 1675 Nov 29 10:10 client-key.pem
-rw-r--r-- 1 root root 1371 Nov 29 10:10 client.pem

创建配置文件

[root@hdss7-21 cert]# cd ../
[root@hdss7-21 flannel]# vim subnet.env
FLANNEL_NETWORK=172.7.0.0/16      # pod的ip地址范围
FLANNEL_SUBNET=172.7.21.1/24        # 本机的ip地址范围
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
注意:flannel集群各主机的配置略有不同,部署其他节点时注意修改

创建flanneld启动脚本

[root@hdss7-21 flannel]# vim flanneld.sh
#!/bin/sh
./flanneld \
  --public-ip=10.4.7.21 \
  --etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \
  --etcd-keyfile=./cert/client-key.pem \
  --etcd-certfile=./cert/client.pem \
  --etcd-cafile=./cert/ca.pem \
  --iface=eth0 \
  --subnet-file=./subnet.env \
  --healthz-port=2401
[root@hdss7-21 flannel]# chmod +x flanneld.sh
注意:flannel集群各主机的配置略有不同,部署其他节点时注意修改

操作etcd,增加host-gw

启动flannel之前,需要在etcd中添加网络配置记录
[root@hdss7-21 flannel]# cd /opt/etcd
[root@hdss7-21 etcd]# ./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}
[root@hdss7-21 etcd]# ./etcdctl get /coreos.com/network/config
{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw”}}
host-gw:直接路由的方式,将容器网络的路由信息直接更新到主机的路由表中,仅适用于二层直接可达的网络

创建supervisor配置

[root@hdss7-21 etcd]# mkdir /data/logs/flanneld
[root@hdss7-21 etcd]# vim /etc/supervisord.d/flannel.ini
[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh            ; the program (relative uses PATH, can take args)
numprocs=1                                  ; number of processes copies to start (def 1)
directory=/opt/flannel                      ; directory to cwd to before exec (def no cwd)
autostart=true                              ; start at supervisord start (default: true)
autorestart=true                            ; retstart at unexpected quit (default: true)
startsecs=30                                ; number of secs prog must stay running (def. 1)
startretries=3                              ; max # of serial start failures (default 3)
exitcodes=0,2                               ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                             ; signal used to kill process (default TERM)
stopwaitsecs=10                             ; max num secs to wait b4 SIGKILL (default 10)
user=root                                   ; setuid to this UNIX account to run the program
redirect_stderr=true                        ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log       ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                    ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                 ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                 ; emit events on stdout writes (default false)
stopasgroup=true                            ;默认为false,进程被杀死时,是否向这个进程组发送stop信号,包括子进程
killasgroup=true                            ;默认为false,向进程组发送kill信号,包括子进程

启动服务并检查

[root@hdss7-21 etcd]# supervisorctl update
flanneld-7-21: added process group
[root@hdss7-21 etcd]# supervisorctl status
etcd-server-7-21                 RUNNING   pid 4044, uptime 6 days, 12:02:50
flanneld-7-21                    STARTING  
kube-apiserver-7-21              RUNNING   pid 4048, uptime 6 days, 12:02:50
kube-controller-manager-7-21     RUNNING   pid 4050, uptime 6 days, 12:02:50
kube-kubelet-7-21                RUNNING   pid 4035, uptime 6 days, 12:02:50
kube-proxy-7-21                  RUNNING   pid 4055, uptime 6 days, 12:02:50
kube-scheduler-7-21              RUNNING   pid 4057, uptime 6 days, 12:02:50

测试两个node中容器跨网络通信

[root@hdss7-21 flannel]# ping 172.7.21.2
PING 172.7.21.2 (172.7.21.2) 56(84) bytes of data.
64 bytes from 172.7.21.2: icmp_seq=1 ttl=64 time=0.146 ms
64 bytes from 172.7.21.2: icmp_seq=2 ttl=64 time=0.126 ms

[root@hdss7-21 flannel]# ping 172.7.22.2
PING 172.7.22.2 (172.7.22.2) 56(84) bytes of data.
64 bytes from 172.7.22.2: icmp_seq=1 ttl=63 time=1.95 ms
64 bytes from 172.7.22.2: icmp_seq=2 ttl=63 time=13.0 ms

在各运算节点上增加iptables规则

注意:iptables规则各主机的略有不同,其他运算节点上执行时注意修改。
因为centos7,需要安装iptables
[root@hdss7-21 ~]# yum install -y iptables-services
[root@hdss7-21 ~]# systemctl start iptables
[root@hdss7-21 ~]# systemctl enable iptables
优化SNAT规则,各运算节点之间的各POD之间的网络通信不再出网
[root@hdss7-21 ~]# iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE
[root@hdss7-21 ~]# iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
10.4.7.21主机上的,来源是172.7.21.0/24段的docker的ip,目标ip不是172.7.0.0/16段,网络发包不从docker0桥设备出站的,才进行SNAT转换
删除拒绝所有的规则
[root@hdss7-21 ~]# iptables-save|grep -i reject
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited
[root@hdss7-21 ~]# iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited
测试ping
[root@hdss7-21 ~]# ping 172.7.22.2
PING 172.7.22.2 (172.7.22.2) 56(84) bytes of data.
64 bytes from 172.7.22.2: icmp_seq=1 ttl=63 time=4.98 ms
64 bytes from 172.7.22.2: icmp_seq=2 ttl=63 time=4.93 ms
各运算节点保存iptables规则
[root@hdss7-21 ~]# iptables-save >/etc/sysconfig/iptables
[root@hdss7-21 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
创建nginx-daemonset资源
[root@hdss7-21 ~]# vim nginx-ds.yml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ds
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: harbor.od.com/public/nginx:curl   # 带有curl的nginx自建镜像
        ports:
        - containerPort: 80
[root@hdss7-21 ~]# kubectl apply -f nginx-ds.ym
[root@hdss7-21 ~]# kubectl get pod -o wide
NAME             READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
nginx-ds-wkf9w   1/1     Running   0          41m   172.7.21.2   hdss7-21.host.com   <none>           <none>
nginx-ds-x5lrc   1/1     Running   0          41m   172.7.22.2   hdss7-22.host.com   <none>           <none>
curl nginx查看访问ip是那个
容器网络里坦诚相待,不需要遮掩
hdss7-21
[root@hdss7-21 ~]# kubectl exec -it nginx-ds-wkf9w sh
# curl 172.7.22.2
# curl 172.7.22.2
hdss7-22
[root@hdss7-22 ~]# kubectl logs -f nginx-ds-x5lrc
10.4.7.21 - - [29/Nov/2019:07:33:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
iptables优化后的日志
172.7.21.2 - - [29/Nov/2019:07:41:02 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"
点赞

发表评论

电子邮件地址不会被公开。

10 + 19 =