摘要:
- ipvsadm的用法
- lvs-nat拓扑结构实验
- lvs-dr拓扑结构实验
- FWM
- ldirector的用法简介
ipvsadm
- 程序包:ipvsadm,用于管理集群
Unit File: ipvsadm.service
主程序:/usr/sbin/ipvsadm
规则保存工具:/usr/sbin/ipvsadm-save
规则重载工具:/usr/sbin/ipvsadm-restore
配置文件:/etc/sysconfig/ipvsadm-config
- ipvsadm 的基本用法
ipvsadm -A|E -t|u|f service-address [-s scheduler] [-p [timeout]] [-M netmask] [--pe
persistence_engine] [-b sched-flags]
ipvsadm -D -t|u|f service-address 删除
ipvsadm –C 清空
ipvsadm –R 重载
ipvsadm -S [-n] 保存
ipvsadm -a|e -t|u|f service-address -r server-address [options]
ipvsadm -d -t|u|f service-address -r server-address
ipvsadm -L|l [options]
ipvsadm -Z [-t|u|f service-address]
- 管理集群服务:增、改、删
ipvsadm -A|E -t|u|f service-address [-s scheduler][-p [timeout]]
ipvsadm -D -t|u|f service-address
- service-address:
-t|u|f:
-t: TCP协议的端口,VIP:TCP_PORT
-u: UDP协议的端口,VIP:UDP_PORT
-f:firewall MARK,标记,一个数字
- [-s scheduler]:指定集群的调度算法,默认为wlc
管理集群上的RS:增、改、删
增、改:ipvsadm -a|e -t|u|f service-address -r server-address [-g|i|m] [-w weight]
删:ipvsadm -d -t|u|f service-address -r server-address
server-address:
rip[:port] 如省略port,不作端口映射
- 选项:
-g: gateway, dr类型,默认
-i: ipip, tun类型
-m: masquerade, nat类型
-w weight:权重
集群内容的查看
ipvsadm –C 清空定义的所有内容
ipvsadm -Z [-t|u|f service-address] 清空计数器
ipvsadm -L|l [options]
--numeric, -n:以数字形式输出地址和端口号
--exact:扩展信息,精确值
--connection,-c:当前IPVS连接输出
--stats:统计信息
--rate :输出速率信息
ipvs规则:/proc/net/ip_vs ipvs连接:/proc/net/ip_vs_conn
- 保存:建议保存至/etc/sysconfig/ipvsadm
ipvsadm-save > /PATH/TO/IPVSADM_FILE
ipvsadm -S > /PATH/TO/IPVSADM_FILE
systemctl stop ipvsadm.service
- 重载:
ipvsadm-restore < /PATH/FROM/IPVSADM_FILE
ipvsadm -R < /PATH/FROM/IPVSADM_FILE
systemctl restart ipvsadm.service
集群NAT服务试验
LVS-nat
director的设置
前提网关已经全部搭载好
vip:172.20.0.24 dip:192.168.1.1/24
rip1:192.168.1.16 ;rip2:192.168.1.8
$ ifconfig eth0 172.20.0.16/16 up
$ ifconfig eth0:0 172.20.0.24/16 up
$ ifconfig eth1 192.168.1.1/24 up
$ ipvsadm -A -t 172.20.0.24:80 -s rr
$ ipvsadm -L -n
$ ipvsadm -a -t 172.20.0.24:80 -r 192.168.1.16 -m
$ ipvsadm -a -t 172.20.0.24:80 -r 192.168.1.8 -m
$ ipvsadm -L -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 172.20.0.24:80 15 47 0 2540 0
-> 192.168.1.8:80 8 24 0 1304 0
-> 192.168.1.16:80 7 23 0 1236 0
##查看状态
$ ipvsadm -Z
$ ipvsadm -L -n --stats
$ ipvsadm -L -n --rate
##切换wrr模式
$ ipvsadm -E -t 172.20.0.24:80 -s wrr
$ ipvsadm -e -t 172.20.0.24:80 -r 192.168.1.16 -m -w 3
$ ipvsadm -e -t 172.20.0.24:80 -r 192.168.1.8 -m
$ ipvsadm -L -n --stats
$ ipvsadm -L -c
$ ipvsadm -L --timeout
ps: keepalive
下载: wget http://www.keepalived.org/software/keepalived-1.2.5.tar.gz
lvs-dr
实现如下拓扑结构的实验
- router的相关设置
$ vim /etc/sysctl.conf
$ net.ipv4.ip_forward = 1
$ ifconfig eth2 192.168.1.254/24 up
$ ifconfig eth1 172.20.0.1/16 up
$ ifconfig eth0 192.168.0.254/24 up
- director的相关设置
dip:192.168.1.7/24
vip:172.20.0.24/16
$ iptables -t filter -F
$ ifconfig eth0 192.168.1.7/24 up
$ ifconfig eth0:0 172.20.0.24/16 up
$ route add -host 172.20.0.24 dev eth0:0 #目标是172.20.0.24的,必须经过eth0:0
$ ipvsadm -A -t 172.20.0.24:80 -s rr
$ ipvsadm -a -t 172.20.0.24:80 -r 192.168.1.8 -g
$ ipvsadm -a -t 172.20.0.24:80 -r 192.168.1.16 -g
- RS1的相关设置
rip:192.168.1.16/24
vip:172.20.0.24
$ ifconfig eth0 192.168.1.16/24 up
$ route add default gw 192.168.1.254
$ echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
$ echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
$ echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
$ echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
$ ifconfig lo0:0 172.20.0.24 netmask 255.255.255.255 broadcast 172.20.0.24 up
$ route add -host 172.20.0.24 dev lo:0
$ yum install -y httpd
$ echo RS1 > /var/www/html/index.html
$ systemctl start httpd
- RS2的相关设置
rip:192.168.1.8/24
vip:172.20.0.24
$ ifconfig eth0 192.168.1.8/24 up
$ route add default gw 192.168.1.254
$ echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
$ echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
$ echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
$ echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
$ ifconfig lo0:0 172.20.0.24 netmask 255.255.255.255 broadcast 172.20.0.24 up
$ route add -host 172.20.0.24 dev lo:0
$ yum install -y httpd
$ echo RS2 > /var/www/html/index.html
$ systemctl start httpd
- client的设置及效果检验
$ ifconfig eth0 192.168.0.10.24 up
$ route add default 192.168.0.254 dev eth0
$ for i in {1..10};do curl 172.20.0.24 ;done
RS1
RS2
RS1
RS2
RS1
RS2
RS1
RS2
RS1
RS2
FireWall Mark
FWM:FireWall Mark
MARK target 可用于给特定的报文打标记–set-mark value
其中:value 可为0xffff格式,表示十六进制数字
借助于防火墙标记来分类报文,而后基于标记定义集群服务;可将多个不同的应用使用同一个集群服务进行调度
实现方法:
- 在Director主机打标记:
$ iptables -t mangle -A PREROUTING -d $vip -p $proto –m multiport \
--dports $port1,$port2,… -j MARK --set-mark NUMBER
- 在Director主机基于标记定义集群服务:
ipvsadm -A -f NUMBER [options]
ipvsadm -a -f NUMBER -r 192.168.1.8 -g
ipvsadm -a -f NUMBER -r 192.168.1.16 -g
LVS Persistence
- 持久连接( lvs persistence )模板:实现无论使用任何调度算法,在一段时间内(默认300s ),能够实现将来自同一个地址的请求始终发往同一个RS
PCC:将来自于同一个客户端发往VIP的所有请求统统定向至一个RS;
PPC:将来与一个客户端发往某VIP的某端口的所有请求统统定向至统一个RS;
PFWMC: 端口绑定,基于防火墙标记,将两个或两个以上的端口绑定为同一个服务,即port affinity.
ipvsadm -A|E -t|u|f service-address -s scheduler]
ldirctord
监控和控制LVS守护进程,可管理LVS规则;
监控和管理实际服务器守护进程在LVS集群负载均衡的虚拟服务器 。
- 包名:
ldirectord-3.9.6-0rc1.1.1.x86_64.rpm
- 下载:官网
$ wget http://opensuse.ucom.am/repositories/network:/ha-clustering:/Stable/RedHat_RHEL-6/x86_64/ldirectord-3.9.6-0rc1.1.1.x86_64.rpm
$ yum install -y ldirectord-3.9.6-0rc1.1.1.x86_64.rpm
$ rpm -ql ldirectord
/etc/ha.d
/etc/ha.d/resource.d
/etc/ha.d/resource.d/ldirectord
/etc/init.d/ldirectord
/etc/logrotate.d/ldirectord
/usr/lib/ocf/resource.d/heartbeat/ldirectord
/usr/sbin/ldirectord
/usr/share/doc/ldirectord-3.9.6
/usr/share/doc/ldirectord-3.9.6/COPYING
/usr/share/doc/ldirectord-3.9.6/ldirectord.cf
/usr/share/man/man8/ldirectord.8.gz
- 相关文件的介绍:
/etc/ha.d/ldirectord.cf 主配置文件,按需修改,可以通过模板复制过来。
/usr/share/doc/ldirectord-3.9.6/ldirectord.cf 配置模版文件
/usr/lib/systemd/system/ldirectord.service 服务
/usr/sbin/ldirectord 主程序
/var/log/ldirectord.log 日志
/var/run/ldirectord.ldirectord.pid pid文件
- 配置文件说明
$ iptables -t mangle -A PREROUTING -d $vip -p $proto –m multiport \
--dports $port1,$port2,… -j MARK --set-mark 5
$ vim /etc/ha.d/ldirectord.cf
checktimeout=3
checkinterval=1
autoreload=yes #自动加载已经更改的配置
logfile="/var/log/ldirectord.log" #日志文件
quiescent=no #down时yes权重为0,no为删除
virtual=172.20.0.24:80 #指定VS的FWM或IP:port
real=172.16.0.7 gate 2 #gate(DR模型) 1(权重)
real=172.16.0.8 gate 1
fallback=127.0.0.1 gate #sorry server
service=http
scheduler=wrr
protocol=fwm
checktype=negotiate #测试的类型(健康性检查的方式)
checkport=80
request="index.html" #要探测的页面(准备一个测试页比较好)
receive="Test Ldirectord"
实现ldirecrtor
搭建DR模型
在rs服务器运行脚本
$ cat lvs_dr_rs.sh
#!/bin/bash
vip=172.20.0.24
mask='255.255.255.255'
dev=lo:1
rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null
service httpd start &> /dev/null && echo "The httpd Server is Ready!"
echo "`hostname`" > /var/www/html/index.html
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
$ bash lvs_dr_rs.sh
$ yum install -y httpd
$ echo `hostname` > /var/www/html/index.html
$ systemctl start httpd
在lvs服务器上运行
$ cat lvs_dr_vs.sh
#!/bin/bash
vip='172.20.0.24'
iface='ens33:1'
mask='255.255.255.255'
port='80'
rs1='192.168.1.16'
rs2='192.168.1.8'
scheduler='wrr'
type='-g'
rpm -q ipvsadm &> /dev/null || yum -y install ipvsadm &> /dev/null
case $1 in
start)
ifconfig $iface $vip netmask $mask #broadcast $vip up
iptables -F
ipvsadm -A -t ${vip}:${port} -s $scheduler
ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
echo "The VS Server is Ready!"
;;
stop)
ipvsadm -C
ifconfig $iface down
echo "The VS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
在lvs上配置ldiretord
$ vim /etc/ha.d/ldirectord.cf
# Sample for an http virtual service
virtual=172.20.0.24:80
real=192.168.1.16 gate 1 不用写端口号
real=192.168.1.8 gate 3 不用写端口号
# fallback=127.0.0.1:80 gate
service=http
scheduler=wrr
#persistent=600 持久连接,启用后就会一直往一个服务器上调度了
#netmask=255.255.255.255
protocol=fwm #这个加不加都可以
checktype=negotiate
checkport=80
request="index.html"
receive="test"
# virtualhost=www.x.y.z
启动ldirecrord
没有手工加ipvsadm策略,启动服务的时候会自动根据配置文件,生成的ipvsadm策略
$ systemctl start ldirectord
$ ipvsadm -L -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 172.20.0.24:80 15 47 0 2540 0
-> 192.168.1.8:80 8 24 0 1304 0
-> 192.168.1.16:80 7 23 0 1236 0
ldirectord实现的检测功能,一旦RS宕机,会立即从ipvsadm删除宕机的服务器;
模拟宕机停止rs上的httpd服务即可实验。
观察ldirectord日志
$ tail -f /var/log/ldirectord.log
结语
ldirectord实现的是RS的服务器高可用,一旦ldirectord所在服务器宕机,存在单点失败的问题;
keepalived负责负载均衡器之间的failover,haproxy或者lvs(ipvsadm)负责健康检查和失败切换;
可以使用三者结合的方法,实现架构的可用性。这个后续有待验证。