Firewall Early Routing

First, a quick test can be performed to demonstrate a problem. For example:


The machine that will be running the “DHCP/IPv4 server” software is intended to use the firewall as a “default gateway”.

Understanding gateways

Being a “gateway” refers to the idea that the firewall will relay traffic, and therefore provides a method for the other computer to communicate to a remote subnet.

If multiple gateways are used, the most specific gateway is possible. So, if there is a fast Internet connection to a neighboring building, and another gateway that communicates more slowly but has access to the entire Internet, then the fast Internet connection will be used for traffic that can be sent to that direction. The term “default gateway” means that the firewall acts like a “gateway” that relays the communications to most computers on the planet, with the key exception being any computers that can be reached by a better route.

The term “gateway” is related to relaying traffic to multiple subnets, which is done using the process that is called “routing”. That is why it makes sense to see that the DHCP Option for the “default gateway” is called the “Router” Option (in RFC 2132: “DHCP Options and BOOTP Vendor Extensions”, section 3.5: “Router Option”, and also in the dhcpd.conf configuration which uses the term “option routers”.

There are multiple problems with the routing. Therefore, getting communication to be fully functional is going to require multiple steps. Here is a list of some key problems:

Physical Firewall

The “physical machine” (which runs the “virtual machine” software) may be running “firewall” software, which could be cause some problems. Here are some steps to help eliminate that.

PF: The OpenBSD Packet Filter

This firewall software has been known to cause issues with the TUN/TAP devices that send information to a bridge. The feature of recognizing the “state” of a packet can help to optimize speed, but also results in breaking communication (perhaps because the packet enters the firewall twice, which is likely not a situation that the firewall software was optimized for).

cpytobak /etc/pf.conf
echo ${VISUAL}
sudoedit /etc/pf.conf

A rule must be made which affects every TUN/TAP device that communications on the vmSvrs subnet. All such traffic ends up being part of the bridge that connects the devices on this subnet.

Actually, the rule probably makes sense for any traffic that is on an interface that is part of a bridge, so this is the rule that this guide recommends:

pass on usebrdge no state
pass on usebrdge proto tcp flags any no state

The reason for the second line is a recommendation by OpenBSD PF (Packet Filter) FAQ: Rule Syntax. (Alternatively, the lines could be combined so just set “flags any” on all traffic. However, the way this is written will focus the special handling on only the traffic where that handling was recommended.)

Then, save and apply the file:

sudo pfctl -n -f /etc/pf.conf && sudo pfctl -f /etc/pf.conf

Note: If someone is using ping, then a problematic state may have already been created. In that case, the new rules may not take effect until that state is cleared. It is possible to “flush” such states, although doing so may also break some useful existing conversations (TCP connections, or a similar series of UDP or ICMP communications). Therefore, routine flushing is not recommended if unnecessary. However, if it is necessary, then:

sudo pfctl -n -f /etc/pf.conf && sudo pfctl -F states -f /etc/pf.conf

(It might also be possible to “kill” off only specific states, by appropriate usage of the -k option. man page for pfctl discusses options. The “state ID” can be seen using “ sudo pfctl -ss -vv ”.)

Relaxing the firewall
OpenBSD Firewall
cpytobak /etc/pf.conf
cpytobak /etc/pf/pfnics.cnf
echo vmSvrs_if=em1| sudo -n tee -a /etc/pf/pfnics.cnf
echo | sudo -n tee -a /etc/pf.conf
echo \# Permit ICMP from vmSvrs| sudo -n tee -a /etc/pf.conf
echo pass in quick on \$vmSvrs_if proto icmp from \$vmSvrs_if:network keep state| sudo -n tee -a /etc/pf.conf
sudo pfctl -nf /etc/pf.conf
echo ${?}
echo ${VISUAL}
sudoedit /etc/pf.conf


sudo pfctl -nf /etc/pf.conf&&sudo pfctl -ef /etc/pf.conf
actually that might not have been quite loose enough.
The following is probably closer to what wouldbe nice to have in /etc/pf.conf
(or maybe some of this would go in another file which is /etc/pf/* )
NOTE: THIS HAS < and > which PRE might not be showing... check HTML source...

# Permit ICMP from vmSvrs

pass in quick on $vmSvrs_if inet proto icmp from $vmSvrs_if:network \
        keep state
pass out quick proto icmp from $vmSvrs_if:network keep state

pass in quick on $vmSvrs_if inet proto icmp6 from $vmSvrs_if:network \
        keep state
pass out quick proto icmp6 from $vmSvrs_if:network keep state

table  { $vmSvrs_if:network \
        $ext_if:network }

pass in quick inet proto icmp from  to $vmSvrs_if:network keep state
pass in quick inet6 proto icmp6 from  to $vmSvrs_if:network keep state
pass out on $vmSvrs_if proto icmp from  to any keep state
pass out on $vmSvrs_if proto icmp6 from  to any keep state

#Allow SSH from trusted servers to whereever they want to go

table  { $vmSvrs_if:network \
        $ext_if:network }

pass in quick proto tcp from  to \
        $vmSvrs_if:network port 49222 keep state
pass out quick on $vmSvrs_if proto tcp from  to \
        $vmSvrs_if:network port 49222 keep state

Unfortunately, you might not notice an immediate impact of things working better, because there may be some additional problems that also need to be taken care of.

Gateway needed
Removing unwanted IP addresses

For the next part, look at the subnet(s) of the second NIC of the firewall “virtual machine”. (There may be an IPv6 subnet, and an IPv4 subnet.) Make sure that none of the TUN/TAP devices on the physical machine have an address on that subnet. If they do, then traffic could be routed to the physical machine without going through one of the firewall's other NICs, which is the intended design.

By the “second NIC”, this refers to the NIC that the virtual machine uses to communicate to the bridge that connects to the TUN/TAP devices of other virtual machines. That is the subnet that needs to appear on none of the TUN/TAP devices.

This needs to be taken care of now, because otherwise some communication might work, though the method of that communication is improper (by taking a route that isn't really intended). The TUN/TAP devices should not have need to have an IP address on the same subnet as the “virtual machine” firewall's second NIC. If any of the TUN/TAP devices do have such an IP address, that causes an automatically-generated route to affect how traffic flows. That is undesirable, and so there needs to be no addresses from the subnets that are used by the second NIC of the firewall “virtual machine” is using.

If this guide recommended adding such addresses to the TUN/TAP devices earlier, that was just meant as a temporary step to help with troubleshooting. Those addresses need to be removed at this time. Removing those addresses may break something that is working improperly. Upcoming instructions will help describe how to make that communication work properly (using the intended routing).

Note: These instructions are meant to apply to ALL TUN/TAP devices that are connected to the “virtual machine” firewall's second NIC, whether directly or via a bridge. The other TUN/TAP devices, which are directly connected to the other NICs of the firewall “virtual machine” should keep their addresses.

In other words, using the names of the subnets described by the demonstration subnets (used by this guide), and the numbers related to those subnets:

  • remove the IP addresses from the TUN/TAP devices related to the vmSvrs subnet (example documentation addresses from IPv6 2001:db8:1::/125 and IPv4 198.51.100/29).
    • except, do not (even bother trying to) remove the IPv6 link-local addresses (from the fe80::/64 range). Attempting to remove those addresses will be unlikely to work if IPv6 continues to work. Fortunately, these addresses won't be affecting routing.
  • do not remove the IP addresses from the TUN/TAP devices related to the Outside VM NIC subnet (example documentation addresses from IPv6 2001:db8:8::8/126 and IPv4
  • do not remove the IP addresses from the TUN/TAP devices related to the Inside VM NIC subnet (example documentation addresses from IPv6 2001:db8:c::c/126 and IPv4

If you have a bunch of TUN/TAP NICs, you may wish to focus your attention on the TUN/TAP NICs that use the bridge.

e.g., to remove subnets in OpenBSD:

ifconfig tun244
sudo ifconfig tun244 inet6 delete
sudo ifconfig tun244 inet delete
ifconfig tun244

Also, if the IP addresses get assigned by a system-wide NIC configuration script, or a Qemu NIC configuration script, remove the lines that set those addresses.

ls -l /etc/hostname.tun244
echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
ls -l ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/*2*
echo ${VISUAL}
Routing issues: Physical box cannot reach virtual machine subnet
Public communication rejecting private addresses