通过Rancher初步创建K8s集群过程记录

Posted on | In

目录

集群架构参考图：

1551952435046

集群资源列表：

IP role description
192.168.101.71 rancher server
k8s-master (etcd, control)		 master节点
rancher server
192.168.101.72 k8s-master (etcd, control) master节点
192.168.101.73 k8s-master (etcd, control) master节点
192.168.101.74 k8s-node (worker) worker节点（可被调度）
192.168.101.75 k8s-node (worker) worker节点
192.168.101.76 k8s-node (worker) worker节点
192.168.101.70 VIP 虚拟 IP

安装步骤：

  • 初始准备
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.101.71  b71-k8s-m01
192.168.101.72  b72-k8s-m02
192.168.101.73  b73-k8s-m03
192.168.101.74  b74-k8s-c01
192.168.101.75  b75-k8s-c02
192.168.101.76  b76-k8s-c03

# 关闭firewalld
systemctl stop firewalld.service
systemctl disable firewalld.service

# 无swap分区，不涉及swapoff操作

# 关闭selinux
setenforce 0
sed -i '/^SELINUX=/s/=.*$/=disabled/' /etc/selinux/config

# 安装docker-ce 18.06
wget -P /etc/yum.repos.d https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce-18.06.1.ce -y
systemctl start docker.service
systemctl enable docker.service

haproxy + keepalived 配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
### 对3个master节点：添加到文件/etc/sysctl.conf
cat <<EOF >>/etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
EOF

### 修改keepalived.conf
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
	script "killall -0 haproxy"
	interval 3
	weight -2
	fall 10
	rise 2
}

vrrp_instance VI_1 {
	state MASTER # 另2个分别是BACKUP、BACKUP
	interface ens192
	virtual_router_id 51
	priority 120 # 优先级，另2个可以设置为100
	advert_int 1
	authentication {
		auth_type PASS
		auth_pass 59144e4504f2b953e7b9
	}
	virtual_ipaddress {
		192.168.101.70
	}
	track_script {
		check_haproxy
	}
}
EOF

# 安装killall
yum install psmisc -y

systemctl enable keepalived.service
systemctl start keepalived.service
[root@B72-k8s-m02 haproxy]# ip address show ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:85:8a:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.72/24 brd 192.168.101.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 192.168.101.70/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe85:8a00/64 scope link
       valid_lft forever preferred_lft forever

### 修改haproxy.cfg
...
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver

backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server  b71-k8s-m01 192.168.101.71:6443 check weight 1
    server  b72-k8s-m02 192.168.101.72:6443 check weight 2
    server  b73-k8s-m03 192.168.101.73:6443 check weight 2

listen stats
    bind                 *:1080
    stats auth           admin:password
    stats refresh        5s
    stats realm          HAProxy\ Statistics
    stats uri            /admin?stats

# 启动haproxy
systemctl enable haproxy.service
systemctl start haproxy.service

需开放的端口见 https://rancher.com/docs/rancher/v2.x/en/installation/references/。

为了简便，关闭了防火墙。通过Rancher安装后，查看一下iptables规则：

1
2
3
4
5
6
7
8
9
10
11
[root@B71-k8s-m01 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-EXTERNAL-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW 
KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy DROP)
target     prot opt source               destination         
KUBE-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            
DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0           
...

  • 安装rancher server

    在节点101.71上执行：

1
docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher:stable

1551953142393

​ 添加master节点时，是按如下设置的：

1551840677589

1551840771418

1551840932234

​ 复制命令在待添加的节点机器上执行，依次添加完3个master节点后如下，其中101.71是第二个添加的：

1551841031094

​ 在机器数量足够的情况下，可将rancher server单独部署，此处是与K8s_master共用。经反复试验，添加master节点时，先添加101.72或101.73，不要首先添加rancher server所在节点101.71。否则，后续添加master时会出问题：

1551839742271

上图显示101.72一直处于Registering状态，不能添加进集群。

在添加节点异常时，先通过Web管理界面删除问题节点，然后在重新执行添加前，通过以下脚本清理一下待添加节点的环境：

1
2
3
4
5
6
# 清理安装文件目录
DLIST="/var/lib/etcd /etc/kubernetes /etc/cni /opt/cni /var/lib/cni /var/run/calico /opt/rke"
for dir in $DLIST; do
  echo "Removing $dir"
  rm -rf $dir
done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 清理docker
CLIST=$(docker ps -qa)
if [ "x"$CLIST == "x" ]; then
  echo "No containers exist - skipping container cleanup"
else
  docker stop -f $CLIST && docker rm -f $CLIST
fi

ILIST=$(docker images -a -q)
if [ "x"$ILIST == "x" ]; then
  echo "No images exist - skipping image cleanup"
else
  docker rmi $ILIST
fi

VLIST=$(docker volume ls -q)
if [ "x"$VLIST == "x" ]; then
  echo "No volumes exist - skipping volume cleanup"
else
  docker volume rm -f $VLIST
fi

systemctl restart docker.service

下图是添加完3个master和3个node后的主机界面：

1551844009517

  • 部署服务

1551844649598

测试部署5个jenkins实例：

1551856322207

1551844947277

1551845031997

1551845141705

访问 Jenkins：

1551845268617

  • 执行kubectl命令行

通过Rancher Web界面来执行kubectl命令行：

1551845918415

1551845411567

如果直接登录主机，找不到kubectl命令，需进入相关的容器

1551845747894

  • 添加Dashboard

1551846399730

1551846438286

找到kubernetes-dashboard，并部署（启动）

1551846551235

1551846635456

1551953637260

0%