iptables and Traffic between 2 Host-Only interfaces on KVM using forward mode='open'
Thet networks configured on the Hypervisor
virsh net-list
Name State Autostart Persistent
----------------------------------------------------------
nat active yes yes
net20 active yes yes
net21 active yes yes
Details of each network
The NAT network
[root@win2k12r2 ~]# virsh net-dumpxml nat
<network connections='7'>
<name>nat</name>
<uuid>04640e45-0095-4877-a9bc-31970f8aa9d6</uuid>
<forward dev='eno49' mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
<interface dev='eno49'/>
</forward>
<bridge name='virbr2' stp='on' delay='0'/>
<mac address='52:54:00:0a:89:16'/>
<domain name='nat'/>
<ip address='172.16.0.1' netmask='255.255.0.0'>
</ip>
</network>
The remaining 2 networks being used here as 'nat20' and 'nat21' are Host-only network but have the forward mode as open.
nat20 has the subnet as 172.20.0.0/16
nat21 has the subnet as 172.21.0.0/16
The XML files for these networks are as
net20
[root@win2k12r2 ~]# virsh net-dumpxml net20
<network connections='2'>
<name>net20</name>
<uuid>e9579155-77cf-4823-a2a1-b7605b8d332c</uuid>
<forward mode='open'/>
<bridge name='br20' stp='on' delay='0'/>
<mac address='52:54:00:0d:8c:95'/>
<ip address='172.20.0.1' netmask='255.255.0.0'>
<dhcp>
<range start='172.20.0.2' end='172.20.255.222'/>
</dhcp>
</ip>
</network>
[root@win2k12r2 ~]#
net21
[root@win2k12r2 ~]# virsh net-dumpxml net21
<network connections='2'>
<name>net21</name>
<uuid>0dc5556c-1082-40da-b746-034c0307b351</uuid>
<forward mode='open'/>
<bridge name='br21' stp='on' delay='0'/>
<mac address='52:54:00:f4:f8:f5'/>
<ip address='172.21.0.1' netmask='255.255.0.0'>
<dhcp>
<range start='172.21.0.2' end='172.21.255.254'/>
</dhcp>
</ip>
</network>
[root@win2k12r2 ~]#
Deploy threee Virtual Macines, server1, server2 and server3 such that there connectivity is as
server1 - only to network 'net20' and IP of 172.20.0.11/16
server2 - only to network 'net21' and IP of 172.21.0.21/16
server3 - 3 Networks - nat, net20 and net21 such that on server3
eth0 - nat Network - 172.16.254.222/16
eth1- 'net20' network - 172.20.0.31/16
eth2- 'net21' network - 172.21.0.31/16
[root@win2k12r2 ]# for virtualmachine in server{1..3}; do echo "Virtual machine $virtualmachine"; virsh domiflist $virtualmachine ; done
Virtual machine server1
Interface Type Source Model MAC
-------------------------------------------------------
eth0 network net20 virtio 52:54:00:41:22:e0
Virtual machine server2
Interface Type Source Model MAC
-------------------------------------------------------
vnet5 network net21 rtl8139 52:54:00:08:d7:b3
Virtual machine server3
Interface Type Source Model MAC
-------------------------------------------------------
vnet4 network nat virtio 52:54:00:c5:ad:e5
eth1 network net20 virtio 52:54:00:fd:dd:88
eth2 network net21 virtio 52:54:00:b5:75:6c
Note: In this example iptables and firewalld are disabled on server1 and server2. On the server3 that will be doing the routing and natting we have iptables installed. 'systemctl' is used on the server3 so as start and enable 'iptables.service' and disabled and stop 'firewalld' if already found running.
This is how the setup is
On the server3
IP address configuration of the interfaces
[root@server3 ~]# ip -4 a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 172.16.254.222/16 brd 172.16.255.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
[root@server3 ~]#
[root@server3 ~]# ip -4 a s eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 172.20.0.31/16 brd 172.20.255.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
[root@server3 ~]#
[root@server3 ~]# ip -4 a s eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 172.21.0.31/16 brd 172.21.255.255 scope global noprefixroute eth2
valid_lft forever preferred_lft forever
[root@server3 ~]#
See the routing tables on the server server3
[root@server3 ~]# ip r s
default via 172.16.0.1 dev eth0 proto static metric 103
172.16.0.0/16 dev eth0 proto kernel scope link src 172.16.254.222 metric 103
172.20.0.0/16 dev eth1 proto kernel scope link src 172.20.0.31 metric 101
172.21.0.0/16 dev eth2 proto kernel scope link src 172.21.0.31 metric 102
The Routing tables are self-explnatory
Enable ipv4 forwarding on server3
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
sysctl -w
Confirm forwarding is enabled
sysctl -a | grep net.ipv4.ip_forward
net.ipv4.ip_forward = 1
Install and initially configure iptables with no-rules
post this install 'iptables'.
Post that start and enable iptables and stop and disable firewalld if already running.
yum -y install iptables
systemctl start iptables.service
systemctl enable iptables.service
Flush the iptables to start with to ensure all the connections as needed are working
iptables -F
iptables-save > /etc/sysconfig/iptables
systemctl restart iptables
Here on server 2 ensure that there are no iptables and firewalld running.
To make things simplere, we do not install or enable iptables and firewalld on server1 and server2
on server1 and server2: yum -y remove iptables firewalld
Set the default GWs on The server1 and server2
These are just adding the GWs temporarily on the servers. To make these persist across reboots ensure that the /etc/sysconfig/network-scripts/ifcfg-ethX file on the servers as the GATEWAY=X.X.X.X entry
On Server1 add the default GW as the 172.20.0.31 (this is the IP of the eth1 interface on the server3)
ip route add 0.0.0.0/0 via 172.20.0.31 dev eth0
On Server2 add the default GW as 172.21.0.31 (this is the IP of the eth2 interface on the server3)
ip route add 0.0.0.0/0 via 172.21.0.31 dev eth0
After these settings all the outgoing traffics from these servers will reach to the corresponding subnet interface on server3.
IPTABLES Rules on the server3
We will finally put the deny rules, first we are putting the allow rules
Allow the incoming SSH, HTTP, HTTPS, MYSQL and the corresponding outgoing rules on the server3 from 172.16.0.0/16, 172.20.0.0/16 and 172.21.0.0/16 subnets. (Please note that 172.16.0.0/16 is the NATTED Subnet)
These 3 Rules allow incoming SSH,HTTP,HTTPS,MYSQL and port 8080 on the interfaces of server3 from the corresponding subnet
iptables -A INPUT -p tcp -s 172.16.0.0/16 -i eth0 --match multiport --destination-port 22,80,443,8080,3306 -j ACCEPT
iptables -A INPUT -p tcp -s 172.20.0.0/16 -i eth1 --match multiport --destination-port 22,80,443,8080,3306 -j ACCEPT
iptables -A INPUT -p tcp -s 172.21.0.0/16 -i eth2 --match multiport --destination-port 22,80,443,8080,3306 -j ACCEPT
Allow all outgoing traffics that originate at the Server3 itself
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
Allow the incoming PING traffic on all the interfaces on server3 and the corresponding reply for the same
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT
Now the IPTABLES rules for the FORWARDING TRAFFIC between the eth1 and eth2 on the server3.
Here is how this looks schematically.
Here we allow server3 to foward the traffic from the 172.20.0.0/16 subnet (eth1 interface 172.20.0.31) to 172.21.0.0/16 subnet (eth2 interface on server3 has IP of 172.21.0.31) and vice-versa.
To Forward the HTTP and HTTPS traffic from server1 subnet (172.20.0.0/16) to the subnet of the server2 (172.21.0.0/16 where web service HTTPD is running) also SSH port 22 TCP.
iptables -A FORWARD -p tcp --match mulltiport --destination-port 22,80,443 -i eth1 -s 172.20.0.0/16 -d 172.21.0.0/16 -j ACCEPT
Forward the packets from server2 subnet (172.21.0.0/16) to reach the SSH Port 22 and MYSQL port 3306/TCP to server1 subnet (172.20.0.0/16)
iptables -A FORWARD -p tcp --match mulltiport --destination-port 22,3306 -i eth1 -s 172.20.0.0/16 -d 172.21.0.0/16 -j ACCEPT
Now we put the block rules in the OUTPUT as well as FORWARD chains
iptables -A OUTPUT -s 0.0.0.0/0 -j REJECT
iptables -A FORWARD -j REJECT
This is how the IPTABLES rules look on server3
[root@server3 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT icmp -- anywhere anywhere icmp echo-reply
ACCEPT tcp -- 172.16.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
ACCEPT tcp -- 172.20.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
ACCEPT tcp -- 172.21.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
ACCEPT udp -- 172.20.0.0/16 anywhere udp dpt:domain
ACCEPT udp -- 172.21.0.0/16 anywhere udp dpt:domain
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 172.20.0.0/16 172.21.0.0/16 multiport dports ssh,https,http
ACCEPT tcp -- 172.21.0.0/16 172.20.0.0/16 multiport dports ssh,mysql
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere ctstate NEW,RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere icmp echo-request
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain L (0 references)
target prot opt source destination
[root@server3 ~]#
Internet and DNS Traffic from server1 and server2 via server3
For the Servers server1 and server2 to reach to the internet and get the YUM packages from there.
1) server1 and server2 to first to be able to reach to the DNS server for name queries reply for the CentOS mirrors. The DNS server in this case is 172.16.0.1
2) Once the name resolution is fine, then the servers server1 and server2 to be able to reach to these servers. The internet access for all the networks is via the internet proxy server that is working at the port 8080
3) The Server server3 has to perform the masqurading for the requests for the outgoing DNS and YUM Fetch traffic for the servers server1 and server2.
Here is how the above are achieved by setting the rules on the server3.
This is how this looks
for the onward traffic set the MASQUERADING
iptables -A FORWARD -o eth0 -m state --state NEW,ESTABLISHED,RELATED
iptables -A FORWARD -i eth0 -m state --state ESTABLISHED,RELATED
Set the Masquerading of the intended traffic to be going out from eth0 of server3
iptables -A -t nat -A POSTROUTING -s 172.20.0.0/16 -o eth0 -j MASQUERADE
iptables -A -t nat -A POSTROUTING -s 172.21.0.0/16 -o eth0 -j MASQUERADE
This is how the iptables look like
The Filter TABLE
Here is the NAT Table
[root@server3 ~]# iptables -t nat -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 MASQUERADE all -- 172.20.0.0/16 anywhere
2 MASQUERADE all -- 172.21.0.0/16 anywhere
[root@server3 ~]#
Install the services on the servers now
On server1 install mariadb
Mariadb : Configured to run on the default port 3306 and bind to address 172.20.0.11
yum -y install mariadb
This is how the /etc/my.cnf looks
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
port=3306
bind-address=172.20.0.11
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
Start and Enable mariadb service
systemctl restart mariadb
systemctl enable mariadb
Ensure that mariadb is up and running and bound to the IP address of the server1
[root@server1 network-scripts]# systemctl status mariadb
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-07-25 01:11:41 EDT; 2 days ago
Main PID: 4071 (mysqld_safe)
CGroup: /system.slice/mariadb.service
├─4071 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
└─4260 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/my...
Jul 25 01:11:39 server1.sujitnet11.net systemd[1]: Starting MariaDB database server...
Jul 25 01:11:39 server1.sujitnet11.net mariadb-prepare-db-dir[4041]: Database MariaDB is probably initi....
Jul 25 01:11:39 server1.sujitnet11.net mariadb-prepare-db-dir[4041]: If this is not the case, make sure....
Jul 25 01:11:39 server1.sujitnet11.net mysqld_safe[4071]: 180725 01:11:39 mysqld_safe Logging to '/var...'.
Jul 25 01:11:39 server1.sujitnet11.net mysqld_safe[4071]: 180725 01:11:39 mysqld_safe Starting mysqld ...ql
Jul 25 01:11:41 server1.sujitnet11.net systemd[1]: Started MariaDB database server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@server1 network-scripts]#
[root@server1 network-scripts]# ss -tunlp | grep mysql
tcp LISTEN 0 50 172.20.0.11:3306 *:* users:(("mysqld",pid=4260,fd=14))
[root@server1 network-scripts]#
Give the root user all privileges
MariaDB [mysql]> grant all privileges on *.* to 'root'@'localhost' identified by '<PASSWORD HERE>';
MariaDB [mysql]> grant all privileges on *.* to 'root'@'%' identified by <PASSWORD_HERE>';
MariaDB [mysql]> flush privileges;
MariaDB [mysql]> select Host,User from user;
+------------------------+------+
| Host | User |
+------------------------+------+
| % | root |
| 127.0.0.1 | root |
| ::1 | root |
| localhost | root |
| server1.sujitnet11.net | root |
+------------------------+------+
On server2 install Apache (httpd for CentOS)
yum -y install httpd
in the /etc/httpd/conf/http.conf
replace
Listen 80
with
Listen 172.21.0.21:80
Start and Enable httpd
systemctl start httpd
systemctl enable httpd
Ensure HTTPD service is up and running and the httpd service listening on default port 80 on the interface of the server
[root@server2 ~]# ss -tunlp | grep 80
tcp LISTEN 0 128 172.21.0.21:80 *:* users:(("httpd",pid=2953,fd=3),("httpd",pid=2952,fd=3),("httpd",pid=2951,fd=3),("httpd",pid=2858,fd=3),("httpd",pid=2857,fd=3),("httpd",pid=2856,fd=3),("httpd",pid=2855,fd=3),("httpd",pid=2854,fd=3),("httpd",pid=2853,fd=3))
[root@server2 ~]#
[root@server2 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-07-24 05:55:00 EDT; 2 days ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 2853 (httpd)
Status: "Total requests: 9; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─2853 /usr/sbin/httpd -DFOREGROUND
├─2854 /usr/sbin/httpd -DFOREGROUND
├─2855 /usr/sbin/httpd -DFOREGROUND
├─2856 /usr/sbin/httpd -DFOREGROUND
├─2857 /usr/sbin/httpd -DFOREGROUND
├─2858 /usr/sbin/httpd -DFOREGROUND
├─2951 /usr/sbin/httpd -DFOREGROUND
├─2952 /usr/sbin/httpd -DFOREGROUND
└─2953 /usr/sbin/httpd -DFOREGROUND
Jul 24 05:55:00 server2.sujitnet11.net systemd[1]: Starting The Apache HTTP Server...
Jul 24 05:55:00 server2.sujitnet11.net systemd[1]: Started The Apache HTTP Server.
[root@server2 ~]#
Try accessing the services from the each other server.
from server1: elinks http://172.21.0.21
and
from server2: mysql -h 172.20.0.11 -u root -p -P 3306
Note: you might need to install the 'elinks' and 'mariadb-clients' are installed on the servers where from you try these tests.
LIbvirt allows for the forward mode of 'nat' and 'route' to be used while defining the networks. In this example the virtual network named 'nat' is having a forward mode of the 'nat' and is the network using which the VMs reach to the internet.
As libvirt creates the iptables firewall rules when the networks are defined, if there are 2 normal host-only networks, they work fine, in the sense that these allow the Virtual machines to communicate among themselves within the host-only network.
Again the host-only network wont be able to talk to the other host-only networks and vice-versa.
In addition to the forward modes as above, there is one more that is provided by libvirt that is 'mode=open'. When a network is defined using the forward mode as 'open' as in the examples below, then libvirt does not set the iptables blocking rules for these networks and you can have your own set of iptables rules control these networks.
Thet networks configured on the Hypervisor
virsh net-list
Name State Autostart Persistent
----------------------------------------------------------
nat active yes yes
net20 active yes yes
net21 active yes yes
Details of each network
The NAT network
NOTE: this network is connected to 'eno49' network of the KVM Hypervisor host. 'eno49' network interface is the only physical network on the Hypervisor and that has the access to the internet via a PROXY.
KVM ensures that when this network is attached to a VM, and as an IP has been set to the subnet (172.16.0.0/16 subnet) with GW as 172.16.0.1, the natting of the outgoing traffic to internet from the VM is natted.
[root@win2k12r2 ~]# virsh net-dumpxml nat
<network connections='7'>
<name>nat</name>
<uuid>04640e45-0095-4877-a9bc-31970f8aa9d6</uuid>
<forward dev='eno49' mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
<interface dev='eno49'/>
</forward>
<bridge name='virbr2' stp='on' delay='0'/>
<mac address='52:54:00:0a:89:16'/>
<domain name='nat'/>
<ip address='172.16.0.1' netmask='255.255.0.0'>
</ip>
</network>
The remaining 2 networks being used here as 'nat20' and 'nat21' are Host-only network but have the forward mode as open.
nat20 has the subnet as 172.20.0.0/16
nat21 has the subnet as 172.21.0.0/16
The XML files for these networks are as
net20
[root@win2k12r2 ~]# virsh net-dumpxml net20
<network connections='2'>
<name>net20</name>
<uuid>e9579155-77cf-4823-a2a1-b7605b8d332c</uuid>
<forward mode='open'/>
<bridge name='br20' stp='on' delay='0'/>
<mac address='52:54:00:0d:8c:95'/>
<ip address='172.20.0.1' netmask='255.255.0.0'>
<dhcp>
<range start='172.20.0.2' end='172.20.255.222'/>
</dhcp>
</ip>
</network>
[root@win2k12r2 ~]#
net21
[root@win2k12r2 ~]# virsh net-dumpxml net21
<network connections='2'>
<name>net21</name>
<uuid>0dc5556c-1082-40da-b746-034c0307b351</uuid>
<forward mode='open'/>
<bridge name='br21' stp='on' delay='0'/>
<mac address='52:54:00:f4:f8:f5'/>
<ip address='172.21.0.1' netmask='255.255.0.0'>
<dhcp>
<range start='172.21.0.2' end='172.21.255.254'/>
</dhcp>
</ip>
</network>
[root@win2k12r2 ~]#
Deploy threee Virtual Macines, server1, server2 and server3 such that there connectivity is as
server1 - only to network 'net20' and IP of 172.20.0.11/16
server2 - only to network 'net21' and IP of 172.21.0.21/16
server3 - 3 Networks - nat, net20 and net21 such that on server3
eth0 - nat Network - 172.16.254.222/16
eth1- 'net20' network - 172.20.0.31/16
eth2- 'net21' network - 172.21.0.31/16
Very important to note that for the server3, the 'nat' network of the KVM is attached. Also server3 has the default gateway set as the 172.16.0.1 which is the NAT network interface on the KVM. This allow default access to the internet and Namelookups for server3.
As the 'nat' network functions on the KVM, any VM having that interface and having the NAT network interface IP on the KVM gets the default access to the internet.
Here the server1 and server2 are not connected to the KVM 'nat' interface hence they do not have default access to the internet. These are connected to KVM networks 'nat20' and 'nat21' interfaces respectively, that are host-only networks.
server3 iptables will be used to allow traffic flow between server1 and server2 and also access to the internet to these virtual machines via server3.
In my case all these Virtual Machines use the CentOS7.x
This is how the network associated to the Virtual Machines look from the KVM Hypervisor.
[root@win2k12r2 ]# for virtualmachine in server{1..3}; do echo "Virtual machine $virtualmachine"; virsh domiflist $virtualmachine ; done
Virtual machine server1
Interface Type Source Model MAC
-------------------------------------------------------
eth0 network net20 virtio 52:54:00:41:22:e0
Virtual machine server2
Interface Type Source Model MAC
-------------------------------------------------------
vnet5 network net21 rtl8139 52:54:00:08:d7:b3
Virtual machine server3
Interface Type Source Model MAC
-------------------------------------------------------
vnet4 network nat virtio 52:54:00:c5:ad:e5
eth1 network net20 virtio 52:54:00:fd:dd:88
eth2 network net21 virtio 52:54:00:b5:75:6c
Note: In this example iptables and firewalld are disabled on server1 and server2. On the server3 that will be doing the routing and natting we have iptables installed. 'systemctl' is used on the server3 so as start and enable 'iptables.service' and disabled and stop 'firewalld' if already found running.
This is how the setup is
IP address configuration of the interfaces
[root@server3 ~]# ip -4 a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 172.16.254.222/16 brd 172.16.255.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
[root@server3 ~]#
[root@server3 ~]# ip -4 a s eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 172.20.0.31/16 brd 172.20.255.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
[root@server3 ~]#
[root@server3 ~]# ip -4 a s eth2
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 172.21.0.31/16 brd 172.21.255.255 scope global noprefixroute eth2
valid_lft forever preferred_lft forever
[root@server3 ~]#
See the routing tables on the server server3
[root@server3 ~]# ip r s
default via 172.16.0.1 dev eth0 proto static metric 103
172.16.0.0/16 dev eth0 proto kernel scope link src 172.16.254.222 metric 103
172.20.0.0/16 dev eth1 proto kernel scope link src 172.20.0.31 metric 101
172.21.0.0/16 dev eth2 proto kernel scope link src 172.21.0.31 metric 102
The Routing tables are self-explnatory
Enable ipv4 forwarding on server3
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
sysctl -w
Confirm forwarding is enabled
sysctl -a | grep net.ipv4.ip_forward
net.ipv4.ip_forward = 1
Install and initially configure iptables with no-rules
post this install 'iptables'.
Post that start and enable iptables and stop and disable firewalld if already running.
yum -y install iptables
systemctl start iptables.service
systemctl enable iptables.service
Flush the iptables to start with to ensure all the connections as needed are working
iptables -F
iptables-save > /etc/sysconfig/iptables
systemctl restart iptables
Here on server 2 ensure that there are no iptables and firewalld running.
To make things simplere, we do not install or enable iptables and firewalld on server1 and server2
on server1 and server2: yum -y remove iptables firewalld
Set the default GWs on The server1 and server2
These are just adding the GWs temporarily on the servers. To make these persist across reboots ensure that the /etc/sysconfig/network-scripts/ifcfg-ethX file on the servers as the GATEWAY=X.X.X.X entry
On Server1 add the default GW as the 172.20.0.31 (this is the IP of the eth1 interface on the server3)
ip route add 0.0.0.0/0 via 172.20.0.31 dev eth0
On Server2 add the default GW as 172.21.0.31 (this is the IP of the eth2 interface on the server3)
ip route add 0.0.0.0/0 via 172.21.0.31 dev eth0
After these settings all the outgoing traffics from these servers will reach to the corresponding subnet interface on server3.
IPTABLES Rules on the server3
We will finally put the deny rules, first we are putting the allow rules
Allow the incoming SSH, HTTP, HTTPS, MYSQL and the corresponding outgoing rules on the server3 from 172.16.0.0/16, 172.20.0.0/16 and 172.21.0.0/16 subnets. (Please note that 172.16.0.0/16 is the NATTED Subnet)
These 3 Rules allow incoming SSH,HTTP,HTTPS,MYSQL and port 8080 on the interfaces of server3 from the corresponding subnet
iptables -A INPUT -p tcp -s 172.16.0.0/16 -i eth0 --match multiport --destination-port 22,80,443,8080,3306 -j ACCEPT
iptables -A INPUT -p tcp -s 172.20.0.0/16 -i eth1 --match multiport --destination-port 22,80,443,8080,3306 -j ACCEPT
iptables -A INPUT -p tcp -s 172.21.0.0/16 -i eth2 --match multiport --destination-port 22,80,443,8080,3306 -j ACCEPT
Allow all outgoing traffics that originate at the Server3 itself
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
Allow the incoming PING traffic on all the interfaces on server3 and the corresponding reply for the same
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT
Now the IPTABLES rules for the FORWARDING TRAFFIC between the eth1 and eth2 on the server3.
Here is how this looks schematically.
Here we allow server3 to foward the traffic from the 172.20.0.0/16 subnet (eth1 interface 172.20.0.31) to 172.21.0.0/16 subnet (eth2 interface on server3 has IP of 172.21.0.31) and vice-versa.
To Forward the HTTP and HTTPS traffic from server1 subnet (172.20.0.0/16) to the subnet of the server2 (172.21.0.0/16 where web service HTTPD is running) also SSH port 22 TCP.
iptables -A FORWARD -p tcp --match mulltiport --destination-port 22,80,443 -i eth1 -s 172.20.0.0/16 -d 172.21.0.0/16 -j ACCEPT
Forward the packets from server2 subnet (172.21.0.0/16) to reach the SSH Port 22 and MYSQL port 3306/TCP to server1 subnet (172.20.0.0/16)
iptables -A FORWARD -p tcp --match mulltiport --destination-port 22,3306 -i eth1 -s 172.20.0.0/16 -d 172.21.0.0/16 -j ACCEPT
Now we put the block rules in the OUTPUT as well as FORWARD chains
iptables -A OUTPUT -s 0.0.0.0/0 -j REJECT
iptables -A FORWARD -j REJECT
This is how the IPTABLES rules look on server3
[root@server3 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT icmp -- anywhere anywhere icmp echo-reply
ACCEPT tcp -- 172.16.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
ACCEPT tcp -- 172.20.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
ACCEPT tcp -- 172.21.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
ACCEPT udp -- 172.20.0.0/16 anywhere udp dpt:domain
ACCEPT udp -- 172.21.0.0/16 anywhere udp dpt:domain
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 172.20.0.0/16 172.21.0.0/16 multiport dports ssh,https,http
ACCEPT tcp -- 172.21.0.0/16 172.20.0.0/16 multiport dports ssh,mysql
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere ctstate NEW,RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere icmp echo-request
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain L (0 references)
target prot opt source destination
[root@server3 ~]#
Internet and DNS Traffic from server1 and server2 via server3
For the Servers server1 and server2 to reach to the internet and get the YUM packages from there.
1) server1 and server2 to first to be able to reach to the DNS server for name queries reply for the CentOS mirrors. The DNS server in this case is 172.16.0.1
2) Once the name resolution is fine, then the servers server1 and server2 to be able to reach to these servers. The internet access for all the networks is via the internet proxy server that is working at the port 8080
3) The Server server3 has to perform the masqurading for the requests for the outgoing DNS and YUM Fetch traffic for the servers server1 and server2.
Here is how the above are achieved by setting the rules on the server3.
This is how this looks
This is actually done by masquerading on the port eth0 of the server3 that is the interface of the server3 that is attached to the 'nat' network of the KVM. This means that all requests to the internet from the server2 (coming on interface eth2 of server3) and server1 (request coming on interface eth1 of the server3) will be sent out from server3 off the interface eth0. These traffic once out of the server3 will seem to be coming from the IP address of eth0 interface of the server3.
for the onward traffic set the MASQUERADING
Prior to masquerading the traffic internally allowed to reach eth0 of server3. The NEW connections from eth0 to outgo has to be forwarded, and the return incoming of the same that is ESTABLISHED and RELATED are to be allowed.
iptables -A FORWARD -o eth0 -m state --state NEW,ESTABLISHED,RELATED
iptables -A FORWARD -i eth0 -m state --state ESTABLISHED,RELATED
Set the Masquerading of the intended traffic to be going out from eth0 of server3
iptables -A -t nat -A POSTROUTING -s 172.20.0.0/16 -o eth0 -j MASQUERADE
iptables -A -t nat -A POSTROUTING -s 172.21.0.0/16 -o eth0 -j MASQUERADE
This is how the iptables look like
The Filter TABLE
[root@server3 ~]# iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT icmp -- anywhere anywhere icmp echo-reply
2 ACCEPT tcp -- 172.16.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
3 ACCEPT tcp -- 172.20.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
4 ACCEPT tcp -- 172.21.0.0/16 anywhere multiport dports ssh,https,webcache,mysql,http
5 ACCEPT udp -- 172.20.0.0/16 anywhere udp dpt:domain
6 ACCEPT udp -- 172.21.0.0/16 anywhere udp dpt:domain
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
1 ACCEPT tcp -- 172.20.0.0/16 172.21.0.0/16 multiport dports ssh,https,http
2 ACCEPT tcp -- 172.21.0.0/16 172.20.0.0/16 multiport dports ssh,mysql
3 ACCEPT all -- anywhere anywhere state NEW,RELATED,ESTABLISHED
4 ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
2 ACCEPT all -- anywhere anywhere ctstate NEW,RELATED,ESTABLISHED
3 ACCEPT icmp -- anywhere anywhere icmp echo-request
4 REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain L (0 references)
num target prot opt source destination
Here is the NAT Table
[root@server3 ~]# iptables -t nat -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 MASQUERADE all -- 172.20.0.0/16 anywhere
2 MASQUERADE all -- 172.21.0.0/16 anywhere
[root@server3 ~]#
Install the services on the servers now
On server1 install mariadb
Mariadb : Configured to run on the default port 3306 and bind to address 172.20.0.11
yum -y install mariadb
This is how the /etc/my.cnf looks
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
port=3306
bind-address=172.20.0.11
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
Start and Enable mariadb service
systemctl restart mariadb
systemctl enable mariadb
Ensure that mariadb is up and running and bound to the IP address of the server1
[root@server1 network-scripts]# systemctl status mariadb
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-07-25 01:11:41 EDT; 2 days ago
Main PID: 4071 (mysqld_safe)
CGroup: /system.slice/mariadb.service
├─4071 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
└─4260 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/my...
Jul 25 01:11:39 server1.sujitnet11.net systemd[1]: Starting MariaDB database server...
Jul 25 01:11:39 server1.sujitnet11.net mariadb-prepare-db-dir[4041]: Database MariaDB is probably initi....
Jul 25 01:11:39 server1.sujitnet11.net mariadb-prepare-db-dir[4041]: If this is not the case, make sure....
Jul 25 01:11:39 server1.sujitnet11.net mysqld_safe[4071]: 180725 01:11:39 mysqld_safe Logging to '/var...'.
Jul 25 01:11:39 server1.sujitnet11.net mysqld_safe[4071]: 180725 01:11:39 mysqld_safe Starting mysqld ...ql
Jul 25 01:11:41 server1.sujitnet11.net systemd[1]: Started MariaDB database server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@server1 network-scripts]#
[root@server1 network-scripts]# ss -tunlp | grep mysql
tcp LISTEN 0 50 172.20.0.11:3306 *:* users:(("mysqld",pid=4260,fd=14))
[root@server1 network-scripts]#
Additionally you can further fo 'mysql_secure_installation' and ensure that password based login is allowed to the users to access the DB.
Post that we create a user called 'root' in mysql and give that permission to local login as well as login from any server.
Give the root user all privileges
MariaDB [mysql]> grant all privileges on *.* to 'root'@'localhost' identified by '<PASSWORD HERE>';
MariaDB [mysql]> grant all privileges on *.* to 'root'@'%' identified by <PASSWORD_HERE>';
MariaDB [mysql]> flush privileges;
MariaDB [mysql]> select Host,User from user;
+------------------------+------+
| Host | User |
+------------------------+------+
| % | root |
| 127.0.0.1 | root |
| ::1 | root |
| localhost | root |
| server1.sujitnet11.net | root |
+------------------------+------+
On server2 install Apache (httpd for CentOS)
yum -y install httpd
in the /etc/httpd/conf/http.conf
replace
Listen 80
with
Listen 172.21.0.21:80
Start and Enable httpd
systemctl start httpd
systemctl enable httpd
Ensure HTTPD service is up and running and the httpd service listening on default port 80 on the interface of the server
[root@server2 ~]# ss -tunlp | grep 80
tcp LISTEN 0 128 172.21.0.21:80 *:* users:(("httpd",pid=2953,fd=3),("httpd",pid=2952,fd=3),("httpd",pid=2951,fd=3),("httpd",pid=2858,fd=3),("httpd",pid=2857,fd=3),("httpd",pid=2856,fd=3),("httpd",pid=2855,fd=3),("httpd",pid=2854,fd=3),("httpd",pid=2853,fd=3))
[root@server2 ~]#
[root@server2 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-07-24 05:55:00 EDT; 2 days ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 2853 (httpd)
Status: "Total requests: 9; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─2853 /usr/sbin/httpd -DFOREGROUND
├─2854 /usr/sbin/httpd -DFOREGROUND
├─2855 /usr/sbin/httpd -DFOREGROUND
├─2856 /usr/sbin/httpd -DFOREGROUND
├─2857 /usr/sbin/httpd -DFOREGROUND
├─2858 /usr/sbin/httpd -DFOREGROUND
├─2951 /usr/sbin/httpd -DFOREGROUND
├─2952 /usr/sbin/httpd -DFOREGROUND
└─2953 /usr/sbin/httpd -DFOREGROUND
Jul 24 05:55:00 server2.sujitnet11.net systemd[1]: Starting The Apache HTTP Server...
Jul 24 05:55:00 server2.sujitnet11.net systemd[1]: Started The Apache HTTP Server.
[root@server2 ~]#
Try accessing the services from the each other server.
from server1: elinks http://172.21.0.21
and
from server2: mysql -h 172.20.0.11 -u root -p -P 3306
Note: you might need to install the 'elinks' and 'mariadb-clients' are installed on the servers where from you try these tests.
