Enable Network Communications/Traffic Handling for Virtual Machines

There are various possible reasons why communication may not work with other systems. This guide now reviews some of the most likely problems, and how they can be handled.

Cannot communicate with farther NICs

Note: This section is currently redundant with: Network Traffic Forwarding, which was separated so that a person can easily reference just this problem. (Only the first part (intro) and last parts (summary) may differ slightly; the middle parts (instructions) are the same.) However, this page also contains the information, so that there is an easy way to check multiple problems at once.

Example Scenario

If using a TUN/TAP device, then the layout of network communications may look something like this:

virtual machine
192.0.2.2
< - > internal NIC
192.0.2.1
< - > external NIC
198.51.100.2
< - > Modem-or-ISP
198.51.100.1
Identifying the pieces

In this diagram, the term “virtual machine” refers to a NIC which is part of the virtual machine. If you log into the virtual machine, and look at the list of NICs, then this NIC would show up.

In this diagram, the “internal NIC” may be some software running on the “physical” machine, such as a “TUN/TAP” device that communicates with the virtual machine. This NIC might have a name like tun144 or tap2, depending on things like what operating system is being used.

The “external NIC” may be another network interface that is part of the “physical” machine that is running the “virtual machine” software.

The “Modem-or-ISP” is typically a different piece of hardware, such as a “DSL modem” or a “cable modem”. So it is not typically part of the same physical device as the computer that runs the “virtual machine” software.

Now, in this layout, the following scenario may frequently be encountered.

Findings described Verifying Test
  • the virtual machine may be able to communicate with the IP address on the “internal NIC” on the physical machine,
  On the virtual machine:
ping 192.0.2.1
works.
  • but may be unable to communicate with the IP address on the “external NIC”,
  However, if the virtual machine tries
ping 198.51.100.2
then that fails.
    • nor (can the virtual machine communicate with) any network that involves going through the external NIC.
  And, given that finding, it is less surprising that
ping 198.51.100.2
also fails from the virtual machine.
  • However, the physical machine can seemingly communicate with the internal NIC and the modem/ISP.
 

However, the physical machine can reach all of these addresses, without problem.

virtual machine
192.0.2.2
< - > internal NIC
192.0.2.1
< - > external NIC
198.51.100.2
< - > Modem-or-ISP
198.51.100.1

These findings can often be verified by performing various ping tests.

So, the question being looked at is: why won't the virtual machine communicate with the external NIC?

Solution

Enable forwarding.

OpenBSD guide
Back up data

First, on the machine that isn't forwarding the traffic (which is the “physical” machine that is running the “virtual machine” sofwtare), let's create a snapshot/backup of the /etc/sysctl.conf file.

Note: This should be done on the machine that needs to route traffic between the interfaces. (With the sample diagram that was just shown, this means that the steps need to be done on the “physical” machine, not the “virtual machine”. It is the system that has the “internal NIC”, which may be a TUN/TAP interface.)

export FILETOBK=/etc/sysctl.conf

Then, follow these instructions to create a snapshot/backup of the file.

Then, the remaining steps are currently part of another set of documented instructions. (You can skip the part of those instructions that back up a file, becausae the prior instructions just backed up the /etc/sysctl.conf file.)

  • Follow the Implementations (how to forward traffic). (Note that those instructions are not part of this tutorial, so remember to return to this tutorial after implementing the action of forwarding traffic.)

This should resolve that problem. (The virtual machine might not be able to have bidirectional conversations with the modem/ISP, but it can now reach the external device.)

Cannot communicate with farther devices/computers

Note: This section is currently redundant with: Network Traffic Routing, which was separated so that a person can easily reference just this problem. (Only the first part (intro) and last parts (summary) may differ slightly; the middle parts (instructions) are the same.) However, this page also contains the information, so that there is an easy way to check multiple problems at once.

Example Scenario

If using a TUN/TAP device, then the layout of network communications may look something like this:

virtual machine
192.0.2.2
< - > internal NIC
192.0.2.1
< - > external NIC
198.51.100.2
< - > Modem-or-ISP
198.51.100.1

Now, in this layout, the virtual machine may be able to communicate with the IP address on the “external NIC” on the physical machine, but may be unable to communicate beyond that. So, the virtual machine cannot communicate with the modem/ISP. So, the question being looked at is: why won't the virtual machine communicate with the modem/ISP?

Recap

On the virtual machine:

ping 198.51.100.2

works. However, no response is obtained when trying to reach:

ping 198.51.100.1
Solution

Handle routing.

If you can sufficiently perform network traffic packet sniffing, the findings may reveal with the traffic is, in fact, reaching the further system (the modem/ISP at 198.51.100.1). However, that further system is not successfully sending replies back to the virtual machine.

There are two ways to handle this.

One way is to modify the routing process of the modem/ISP. This is often less convenient, but then avoids NAT. That can simplify certain situations.

However, NAT is often required for IPv4 traffic anyway. When that is the case, often the easiest option is to just have the firewall perform the NAT, at least for IPv4 traffic.

The changes (desribed in the upcoming section) need to be getting made to the “physical” system (which is running the “virtual machine” software).

OpenBSD guide

This guide was created based on an existing example, and might be largely identical/similar to firewall implementations

WARNING: at this time, these instructions have not been thoroughly tested yet... These instructions are currently rather specific to using pf (and perhaps, more specifically, pf within OpenBSD).

Back up data

Newer versions of OpenBSD store the default configuration file at /etc/examples/pf.conf, but older versions did not. (The change might have happened about version 5.6). For older versions of OpenBSD:

sudo mkdir /etc/examples/
sudo cp -i /etc/pf.conf /etc/examples/

If you have done any other customizing of the firewall beforehand, then you may wish to back up that file. If you've installed cpytobak then you could use that. Otherwise, as a multi-step process, first...

export FILETOBK=/etc/pf.conf

... and then follow the remaining steps described by backing up user data.

(There is about to be an instruction that potentially backs up more data.)

Creating a file for NIC identification

The PF: The OpenBSD Packet Filter: section on “Macros” shows how to create a “macro”. As a quick note to programmers: we're not using the term “macro” to refer to instruction code. Rather, a “macro” in the /etc/pf.conf is how a variable gets created.

The /etc/pf.conf file format supports an include statement. This guide recommends using that to identify NICs. This is not a required approach, and is not the approach used by the default sample /etc/pf.conf which defines the NIC variables within the main file. By using a separate file, any changes regarding the name of a NIC can be done without needing to alter the main file that contains firewall rules.

Back up data

First, let's verify that this hasn't been done yet.

ls -ld /etc/pf*

e.g.:

$ ls -ld /etc/pf*
-rw-------  1 root  wheel    #### Mon DD hh:mm/YYYY /etc/pf.conf
-rw-r--r--  1 root  wheel    #### Mon DD hh:mm/YYYY /etc/pf.os
$

Why we did that: to see if, just before those lines, there is also:

drwxr-xr-x  2 root  wheel    #### Mon DD hh:mm/YYYY /etc/pf

This guide largely presumes not. However, if so, it may be worthwhile to view the NICs file, and then follow the standard process of backing up a file before changing it.

$ ls -l /etc/pf/*
$ cat /etc/pf/pfnics.cnf

Then complete the process of making a backup. (Generally, this guide would recommend using the cpytobak program, although it probably isn't installed yet. So, to complete the process of backing up this desired data...) Follow the remaining steps described by backing up user data.

Make a subdirectory for configuration files related to pf.

Note that this process requires knowledge of the names of the NICs.

sudo mkdir /etc/pf/

One last tidbit of knowledge to gather is the name of the NICs. This may vary between different systems, so this is good to look up before trying to make this file. See: available NICs.

ifconfig

Then, if you want to create the file in a text editor, you can start that:

echo ${VISUAL}
sudoedit /etc/pf/pfnics.cnf

This guide will show the process of creating the file from the command line. If you're going to make the file directly within a text editor, just make sure to remove the backslashes from the examples shown.

For this part of the process, the only thing that is really needed is to type in the name of the NIC that will be used for external communications. From the PF: The OpenBSD Packet Filter: section on “Macros”, a name for the external interface can be “$_extif”. This guide uses that name, which is a name that has typically been seen by many people who use the pf software.

echo ext_if = \"nic#\"| sudo tee -a /etc/pf/pfnics.cnf

(Normally, a # character on the command line would need to be escaped for it to go into a text file. However, in this case, the # character should not be typed. Instead, it should be customized, using the name of the actual NIC. Make sure to customize the names of the physical NICs shown in these examples.)

It is unnecessary to define all of the NICs for the process of enabling forwarding. However, why not? While in the mindset of looking at NIC names and editing this file, go ahead and do that for all of the NICs that will be used on the system. People building this file from the command line may benefit from using “command line” history/recall features, to speed things up.

Now may as well be the time to do this, if that information is known and conveniently available. If any of this information seems less available, like if there are doubts about what interfaces will be used later, then this process can just be skipped for now, and handled later when needed. As an example, here is the configuration for a system that may use three more physical NICs and two TUN/TAP interfaces.

echo intlan_if = \"nic#\"| sudo tee -a /etc/pf/pfnics.cnf
echo intwifi_if = \"nic#\"| sudo tee -a /etc/pf/pfnics.cnf
echo dmz_if = \"nic#\"| sudo tee -a /etc/pf/pfnics.cnf
echo intfwvmext_if = \"nic#\"| sudo tee -a /etc/pf/pfnics.cnf
echo intfwvmint_if = \"nic#\"| sudo tee -a /etc/pf/pfnics.cnf

Okay. Check the results.

cat /etc/pf/pfnics.cnf
echo ${VISUAL}
sudoedit /etc/pf/pfnics.cnf
Enabling NAT for IPv4

We're presuming that the configuration file is /etc/pf.conf since that is the default in OpenBSD. In OpenBSD, we can see this filename by running:

grep -i ^pf /etc/rc.conf

(That value may be overridden. In OpenBSD, the file to do that would be a /etc/rc.conf.local file, although that file might not exist. Other BSD operating systems may use other locations.)

In case that file doesn't exist yet, let's grab it. This example will, with useless redundancy, use two methods to verify that the file isn't overwritten if the file does happen to pre-exist.

[ -f /etc/pf.conf ] || echo n | sudo -n cp -ip /etc/examples/pf.conf /etc/

So, let's edit that /etc/pf.conf file.

echo ${VISUAL}
sudoedit /etc/pf.conf

First, make sure the macros are included. With very old versions of pf, there might be an expectation that this is done at some earlier point in the file. With modern versions, this can just be done after any default rules, and before any custom code. So, place this at the end of the file:

include "/etc/pf/pfnics.cnf"

# IETF BCP5 represents "private" IPv4 addresses
table <ietfbcp5> persist { 10/8 172.16/12 192.168/16 }

Then, we'll go ahead and just perform NAT on address coming from IPv4 private addresses. That will likely work just fine in the current scenario, and won't perform NAT if we ever complicate things by having public IPv4 addresses on the internal network.

Note: the method of using NAT has been known to change over the years. (Further info could be nicer... version numbers, details about older variations.) Here is the syntax for the latest versions of PF.

# We never want to pass out traffic from a private address.
# If traffic is good, we want to NAT so it is no longer from a private address.
# (This is optional if another device, outside this network, addresses this.)
pass out on $ext_if inet from { <ietfbcp5> } to any nat-to ($ext_if:0)
pass out on $ext_if inet6 from fd00::/8 to any nat-to ($ext_if:0)

We can also NAT all traffic. (Technically, this can be done in addition to the above rule. However, that does uselessly create an additional rule. If the next process is done, it makes sense to also comment out the previous rule, by inserting the comment character (a hash mark, #, and then an optional space for easier reading) before the final non-blank line in the last example.)

# We are NATting all traffic to assist with routing return traffic.
pass out on $ext_if from any to any nat-to ($ext_if:0)

(If an address family is not specified, then both supported address families, inet and inet6, are implied.)

[#obsdpfgo]: Enabling the new changes

Now, verify that the configuration file is working:

sudo cat /etc/pf.conf | pfctl -n -f -
echo -

As is the case of many Unix programs, using a specified filename of “-” results in the program using “standard input”.

The following example shows a firewall that was disabled, but then becomes enabled.

$ sudo pfctl -s info | head -1
Status: Disabled for # days hh:mm:ss             Debug: err
$ sudo pfctl -e -f /etc/pf.conf
pf enabled
$ echo ${?}
0
$ sudo pfctl -s info | head -1
Status: Enabled for 0 days 00:00:0#              Debug: err
$

Without the output, the input/commands that were run were:

sudo pfctl -s info | head -1
sudo pfctl -e -f /etc/pf.conf
echo ${?}
sudo pfctl -s info | head -1

At that point, things will generally start working. If they don't, sometimes the following command may be needed. Just know, before running it, that this command will frequently break current communications sessions, so existing SSH sessions may be lost. That may be unpleasant, but sometimes this really is the quickest way to fix things (other than waiting/hoping for a timeout)...

sudo pfctl -F states

or, more rarely, flushing the rules may help. This can make the firewall software become basically useless until they are re-loaded. (If flushing and then reloading are done as two seperate commands (which might be required?), then this can also result in disrupting existing networking connections.)

sudo pfctl -F rules
sudo pfctl -f /etc/pf.conf

(or perhaps this instead...)

sudo pfctl -F rules -f /etc/pf.conf

(You can flush both the states and the rules at the same time. One way is to use “-F all”, which also flushes some other things like temporary data that can be used for reporting statistics. Flushing multiple things at once might be faster than flushing a smaller thing at a time; it just feels like a bit of overkill, with the result feeling less precise and less clear about just which piece of the fix was the cause of success.)

Making this work post-reboot

The configuration files have been successfully modified. The other piece to a successful post-reboot experience is to make sure that the firewall gets automatically started. Systems might commonly be pre-configured to do that, so this might not need to be done at all. However, it is probably worth checking.

Here is a sample of what things might commonly look like if this isn't working. Note that the filenames shown are locations in OpenBSD. Other operating systems that use pf may use different files (perhaps even with the same name, but at a different location).

$ grep -i ^pf /etc/rc.conf*
/etc/rc.conf:pf=YES # Packet filter / NAT
/etc/rc.conf:pf_rules=/etc/pf.conf # Packet filter rules file
/etc/rc.conf:pflogd_flags= # add more flags, e.g. "-s 256"
$

What those example lines show is that the pf command will automatically be started (because pf=YES). However, there is another file that may override those settings. If the lines showed like this:

$ grep -i ^pf /etc/rc.conf*
/etc/rc.conf:pf=YES # Packet filter / NAT
/etc/rc.conf:pf_rules=/etc/pf.conf # Packet filter rules file
/etc/rc.conf:pflogd_flags= # add more flags, e.g. "-s 256"
/etc/rc.conf.local:pf=""
$

Then that file has overridden the default, and is causing pf to not load. In that case, it's time to use a text editor to remove the unwanted line that is preventing the packet filter from starting.

As a side note, the previous grep command shows a syntax that works to search both files from the same directory. However, FreeBSD/NetBSD/BOTH may place their equivilent files into other locations. So, the sample commands shown may require some customization so that those commands look in the right locations.