Networking with virtualization
- Resources Available Here
There is a bunch of information available, which has been categorized so that less related information is separated out. Unfortunately, this organization does mean that multiple categories may be needed to effectively find the best information quickly. For someone seeking a walkthrough, the most specific information can be found by checking the following categories, in order.
Be sure to check for related emulation software in each category before moving onto the next category. For instance, if using the Linux Kernel Virtual Machine (Linux KVM) software, check for information related to Qemu and an available walkthrough will be found in the first category.
For some virtual machine software, walkthroughs are available. See walkthroughs for setting up networking devices in virtual machine software
- Specific program information
If a walkthrough is not available, the best resource being offered here may be virtual machine documentation.
- Specific guide(s)
There are some details available for Specific Implementations. That will probably just refer to the prior categories, but a quick check might be worthwhile. These guides may be specific to certain virtual machine software. This may seem like it duplicates knowledge with the details provided in the more generalized section. This is true. However, for those wanting a quick solution, the implementation-specific guides may provide the fewest details about a single specific solution, resulting in a more streamlined process. So, use a specific guide, if one is readily available.
- Generic information on this page
- See the upcoming information.
- Generalized info on setting up networking in virtual machines
Following is information from an older guide that had been written on the subject of virtual machines. It is currently remaining for review, as it may still have some useful details worth keeping. However, some of it may have been effectively replaced by othe resources such as those used by TOOGAM's tutorial for making a virtual machine.
(This section is largely about the setup/creation/configuration of networking options. There will often be no strong differences between physical machines and virtual machines, other than the details in this section about setting up the configuration.
- [#vmnictyp]: Types of (virtual) NIC “hardware” used by virtual machines
- There are a few options, covered in some detail, in the section on types of (virtual) NIC “hardware” used by virtual machines.
- [#hndlhsnc]: How the traffic (involving the NIC on the host machine) gets processed
Here is some new information. Consider using it.
The precise options available, and which of those may be preferred, may vary based on how the (virtual) networking hardware has been implemented in the virtual machine software. (An overview of some various approaches to implementing virtual networking hardware is covered in the section about types of virtual machine NICs).
- Host machine's handling of traffic that indicates the virtual machine's subnet
If the network traffic seen by the host machine shows that it is using a subnet that is used by the virtual machine, and this subnet differs from the subnet of the host computer, then simple traffic handling, like routing or forwarding, may be used.
- Overviews to decide which method to use
The section about network traffic routing: section comparing bridging to traffic forwarding provides a contrast between these two techniques, and then provides recommendations.
So implement one or more of the following options:
- Using (layer 3) traffic forwarding
There's three pieces to this method. Two-way communications likely won't be working until all three pieces are implemented. Therefore, after completing just one section (but before the other two sections are addressed), don't expect positive results when testing two-way communication.
In theory, one way communication, such as sending packets on one machine and listening to network traffic on another machine, may be useful to perform some testing (using traffic sniffing. However, in practice, implementing all three segments may not take very long. The speed at which all three segments can be done may be fast enough testing just part of the solution may not be worthwhile. (Setting up network sniffing can take a while if extra software needs to be installed, which is often not the case with Unix machines that very often have
installed by default. However, even on those machines, fiddling with traffic handling knobs, like firewalls, may take some time.)
- Enabling forwarding on the device that will do forwarding
- On the host machine, implement traffic forwarding so that traffic from the virtual machine will reach the default gateway of the host computer.
Once that is done, there's a couple of routing issues that probably need to be taken care of.
- Routing in one of the directions
For the first routing issue, to have the host machine forward the virtual machine's traffic, the virtual machine should route the traffic to the host machine directly. In many cases, the simple way to do this will be the following: make sure that an IP address on the host machine's (virtual/“TUN/TAP”) NIC is the address used as the virtual machine's “default gateway”. For IPv4, such as route is often handled automatically as part of DHCP, as described in the section about Automatic IPv4 network addressing section.
- Routing reverse traffic
Secondly, the default gateway of the host machine will need to know how to reply to the traffic. There's a couple of ways to pull this off. Do one of the following:
- Adjust routing on the host machine's relevant gateway
This would generally be the preferred option, when available. If access is available to the routing tables of the host machine, then make sure that it knows how to route to the subnet of the machines that communicate to the host machine's TUN/TAP device. (Having the host machine's default gateway communicate directly with the TUN/TAP device is likely a sufficient test.)
Such a route may be added manually or automatically. (Choose one of the following approaches.)
- [#rtpmodgw]: Automatic adjustment of device's routing tables
(text, one version)
For a device's routing tables to be automatically adjusted, the device will need to be running a routing protocol. Perhaps because these protocols have some cost in overhead, they are typically disabled by default. If the device supports multiple routing protocols, then select which one is desired, and then enable it.(text, another version)
One method is to use automatic routing protocols on the default gateway. If the host machine has a way to automatically run some sort of network configuration, then the host machine can start sharing the routing information whenever the virtual machine is started.
As an example, Qemu supports using a network configuration script that the virtual machine software runs when it creates the NIC, as specified by the (optional)
,script=option which can be part of a command line parameter used to create the NIC.
If the virtual machine software allows a different script configuration file to be run when the virtual machine software is shutting down, then the virtual machine software can easily cause changes so the host machine stops advertising the subnet that is used by the virtual machine. As an example, Qemu (versions that are new enough) can support this by using a
,downscript=option as part of the command line parameter that creates the NIC.
(Further details are needed here: Details about how to set up a routing protocol. Clearly such information should be on another page about routing, and so this page should hyperlink to such information. For now, this method is not completely described.)
- Manual efforts specific to adding a single route
- [#mangwtov]: Example of adding a route (to the default gateway device) manually (for immediate effect / temporary use)
The precise method can certainly depend on what type of device the default gateway actually is.
- If the device is running Unix or a modern version of Microsoft Windows
If the route is being added manually, the
parameter may help. First up is an example of how to do this in Unix, followed by an example command line in Windows. Both example command lines will need further customization, which is described further after the examples.
An example of a command to run, for IPv4 in Unix, may be:
An example of a command to run, for Microsoft Windows, may be the following (which will need to be run as an Administrator):
Is this old info, that should now be just simply referencing adding a networking route?
The above addresses are prefixes from RFC 5737: IPv4 Address Blocks Reserved for Documentation, and so should be customized. The first network range shown in the example describes the traffic that needs to be routed. It probably should describe a whole subnet, although it could be a single address. (The way to determine if this is a whole subnet or a single address has to do with the prefix-length/subnet-mask that is used.)
The next network address on the example command line is the destination where such traffic should be getting routed to. (That would likely be an address on the NIC on the device that is performing traffic forwarding.) For the routing to work usefully, this destination address needs to already be reachable by the device that the route is being added to. In this example, the destination address here is on a remote NIC that is on the same subnet as the device that has the route being added. (So, use an address on the traffic-forwarding device which is an address in the same subnet as what is being used by the machine that is having the route being added.)
To possibly help clarify (or perhaps further confuse) the above decisions of what destination address to use: Presumably, in a simple case, the destination address is in a subnet that the device (which is having the route be added) already knows how to route to. The common reason that the device (which is having the route be added) will know how to route to that destination address is because the destination IP address is on the same subnet as one of the NICs on the device (which is having the route be added).
- Automating the route additions (for long term use, after a reboot)
At this point, routes may or may not have been added manually. Whether or not routes have been added manually, do remember that using Automatic adjustment of device's routing tables is one option. The alternative, for a setup that will work in the long term (including after the device is reset, perhaps due to losing power at some time), any manual routes need to be re-added when the device starts.
Following are details of that approach.
- Manual routing setup (repeatedly implemented for long term use)
The default gateway device could just be modified to support the manually added route long term. In Unix, this can be done by modifying the configuration scripts that run when the NIC is initiated during the operating system's startup sequence. (This method may not be as preferable, due to requiring more work to maintain any changes. If the subnet of the virtual machine changed, then the default gateway's manual configuration would need to change. In contrast, with automatic routing protocols, the default gateway could still just use the same automatic routing protocol, and so changes would be made on the host machine, which is a bit more sensible as the changes are on a machine that is “closer” (in the network topology) to the subnet being changed.)
Details of how to implement this are (at least currently) left as an excercise for the reader. (Hint: In Unix, see the configuration files that are accessed when the NIC is started, and determine how to run a command from those configuration files. Then run the same sort of command that is used to temporarily add the route.)Is this old info, that should now be just simply referencing adding a networking route?
- Use NAT on the host machine
If access to the default gateway's configuration is not available, this might be able to be worked around by using NAT on the host machine. Then the default gateway will reply to the NATted address, sending the data to the host machine. The default gateway will then reply to the NATted address, and then the host machine will process the reply and know how to get the data to the correct subnet.
Many network designers will not like the sounds of this idea, and generally for some good reasons. This may introduce some additional overhead, and cause some logs to be less clear (and perhaps less detailed). This is not recommended as a preferred practice. However, if the default gateway is unconfigurable (perhaps because there is no permission by a third party who owns the default gateway), then this method may be the best option just because it may feasibly work technically. (Some network designers would prefer to try harder to make other options available. For instance, they may try to contact the owner of the device and get cooperation to have the routing be modified, either by gaining permission to modify the device or by requesting the owner to just make the desired modifications. Other options may involve replacing the device, or abandoning this approach of traffic forwarding and to instead use briding.)
Presumably it will be desired to have the host machine's NIC that communicates with the virtual machine (which would be a NIC that is a TUN/TAP device) communicate with the NIC that the host machine uses to communicate to the default gateway. Set up that bridging. Typically, the devices that communicate with the TUN/TAP device (such as the virtual machine) will want to to be in the same subnet as the NIC that communicates with the host machine's default gateway. (The TUN/TAP device itself may also be set to have an IP address in that same range, or it might be possible to have an IP address on that subnet, and even to not have an IP address at all.)
If the virtual machine can then communicate with the host machine's NIC that communicates with the host machine's default gateway, or if the virtual machine can communicate with the host machine's default gateway, then the bridging is working. In that case, be sure to set the virtual machine's default gateway. The desirable default gateway for the virtual machine will probably be the same as the default gateway of the host machine.
- Traffic using a host machine's IP address (but meant for a virtual machine)
An example of this sort of situation is network traffic that comes in on the host machine's physical network interface, and that network traffic specifies an IP address that indicates the traffic is meant for the host computer. However, the traffic may really be meant for the virtual machine.
The host machine will receive the traffic, and then send it to the virtual machine. (The way that the traffic gets sent to the virtual machine may depend on what type of virtual hardware is used. For instance, if this is virtual machine software getting treated like a standard network application, then the network stack on the host operating system will simply send the traffic to a network-capable application. Otherwise, if there is virtual machine software using a network device visible to the host machine, then the network traffic may be forwarded/routed out the interface.
NAT may be getting used.
- NAT packets from the virtual machine's address to the host machine's address, and then send traffic from the host machine (similar to how any other software on the host machine may send traffic)
The virtual machine software may support using NAT to forward traffic to and from the virtual machine. Virtual machine software that is doing this may have the host machine treat the virtual machine like a standard networking application. In such a case, the virtual machine software may interact with the host operating system, similar to any other application that uses network traffic.
The network traffic translation may involve PAT (TCP and/or UDP port address translation), changing TCP and/or UDP port addresses as needed while possibly leaving IP addresses unchanged.
When NAT is being performed, the network addresses on each side of the NAT do not have to match, nor do they need to be unique from each other. This might allow the virtual machine to have its own network configuration without much/any need to care about what the network settings are on the host, and vice versa. For example, Qemu implements a local DHCP server that, while configurable with command line options, usually doesn't need to be configured to reach the Internet, and (perhaps less often) usually won't need to be configured to be able to communicate with the host server.
NAT may involve changing addresses. Using the virtual machine's software's internal NAT can be the simplest of methods to set up.Following is some old information that was written about this topic. Is it better in some ways than the new information that was written above this text?
First, the NIC (on the host machine) will need to receive, and pay attention, to the data being sent over the network. This either means that the frames need to have a destination address which is the NIC, or the NIC needs to be in “promiscuous mode”, which basically means that the NIC will ignore traffic that has a destinationFollowing is some old information that was written about this topic. Is it better in some ways than the new information that was written above this text?
Once the NIC on the host machine is able to communicate with the virtual machine, here are some common ways that the host machine may interact with the traffic on the virtual machine.
Bridging and NATting are two options that may work fairly well, although it does complicate the networking code of the virtual machine. (In contrast, the scenario of communicating with a virtual NIC on the host operating system environment may take a little bit more effort to set up, but then details about bridging and using NAT and so forth can be implemented simply using standard software that handles networking, instead of needing to be software code that is implemented as part of the virtual machine software.
- [#vnetbrdg] Have the host machine bridge the NICs
In this mode, the host machine listens to all traffic of some type on one NIC or more NICs on the virtual machine, and will serve as a bridge so all such traffic gets sent to one or more other (real or virtual) NIC(s). Bridging is typically set up to allow all sorts of Layer 2 traffic, whether IPv6 or IPv4 or IPX or any other layer 2 traffic. DHCP traffic might reach a DHCP on another link, and if that happens then this NIC will end up getting its own separate IP address just like any other machine reacing that DHCP server. (The virtual machine appears to the DHCP server to be a machine requesting an IP address, not fundamentally different in nature to any other physical device or virtual device that requests an IP address.)
This may require a bit of setup on the host machine. Additionally, this often allows unnecessary network traffic to reach the virtual machine, and protecting that machine from some types of layer 2 traffic from other machines may require additional processing beyond standard layer 3 routing and layer 3 firewalling.
A simple setup could involve bridging the traffic between two NICs where one NIC is on the host machine, and may be a virtual NIC, and which communicates with the virtual machines, and the other NIC on the bridge is a different NIC on the host machine which communicates directly with other physical devices (using wired connections or attennas).
See bridging network traffic for implementation details about implementing such a setup.
- [#vnetfprc]: Have the host machine process the traffic (without bridging), handling such traffic similar to how other NICs are handled
This is typically an excellent setup, not resulting in the limitations of the other setups (although sometimes bridging may be intentionally desirable, and so bridging the virtual machine's NICs may be preferred). However, this setup may also require the most configuration on the host machine (setting up a virtual NIC, providing addressing, and forwarding traffic).
The first piece of setup complexity may be simply to get the virtual machine to have the desired IP address(es).
For instance, one approach to getting some functional networking may instance, this may involve checking what IP address and subnet are being used by the (virtual) NIC on the host machine. Then the IP address on the virtual machine may need to be set manually. manually on the virtual machine so that the NIC on the virtual machine can communicate with the virtual NIC on the host machine.
An alternative setup, which certainly can work well, is to have the host machine listen for automatic addressing requests on the (virtual) NIC on the host server. The host machine may then fulfill the automatic addressing requests, possibly by using a custom DHCP server on the host machine, or by using DHCRelay. However, this also can certainly take a bit more time to set up.
Unlike treating the virtual machine like a standard networking application using networking, using a combination of
IPv4generally works very well for testing communication, including the host machine.
However, once communication with the host machine is tested to be working, there may be some additional required setup to have the host machine forward traffic to other machines. This may often involve having the virtual NIC on the host machine be on a subnet (which the virtual machine is a part of) which is a different subnet than other NICs on the host machine. In this case, the host machine may need to perform some degree of network traffic routing. (Additionally, automatic addressing for
IPv4may require that the host machine listen for
IPv4traffic, and provide a response by using either a DHCP server running on the host machine or DHCP Relay software running on the host machine.)
- (The information which had previously been here, has been moved.)
- [#vnettovn]: Having virtual machines communicate directly with each other
Qemu supports a “socket” type of connection. Qemu Wiki documentation on networking says “In general, if you want to have multiple guests communicate, tap is a better choice unless you do not have root access to the host environment.” (If you do not have root access to the host environment, then tap may be a better choice because the socket type connection may not be available, since the tap type of connection may require root access to the host environment.)
This may require less configuration on the host machine than some other options, and allows virtual machines to communicate with each other. However, this method does not provide a method of having one or more virtual machines communicate with the host machine or any other physical hardware.
A virtual NIC using this method of networking could communicate with physical hardware indirectly, if the gap is bridged by successfully communicating with another NIC (which may be on the same machine, or on a different machine that can be communicated with). However, a NIC using this method of networking does not bridge that gap. To be able to communicate with physical hardware, such as a remote networking computers/equipment on the Internet, another networking method is needed. (The other networking method may be implemented on a different virtual machine that handles routing of network traffic.) So, a whole bunch of virtual machines may use this networking method, but, to allow communication with physical devices, at least one of those virtual machines will need to use another networking method.
- [#vnetspim]: Specific implementations of virtual machine networking
Details are available about the specific implementations of virtual machine networking used by the various different software programs that implement virtual machines.
- [#vnetaddr]: Providing addresses for the virtual network
Choose addresses, if not done yet. Some details on doing this may be available in Tutorial on making (and setting up) a virtual machine: Planning Network Addresses. (Such information may be moved to this page, or another location, in the future.)
If the host machine has a virtual NIC that is going to be used, make sure it has a network address (see the network addressing page if needed), and says it is
Assign addresses to the NIC on the virtual machine. (Feel free to use automatic network addressing if it is actually functional. However, automatic addressing might not be set up yet, in which case manual network addressing will probably be faster to implement.) The ability to see the virtual machine's screen is likely to be able to help.
- [#vnetvrfy]: Verifying the networking
Here is some information on verifying that communication between the virtual machine and the (virtual) NIC on the host machine is successfully working communication.
Before running the virtual machine, consider what will happen when the virtual machine is run. If the answer is that the operating system (and/or the installation program for the operating system) that is going to get run is something that will not be likely to allow quick access to networking hardware, it may be (or, perhaps, may not be) worthwhile to boot off of some different bootable media that does allow fairly quick access to the network hardware.
An assumption is that directions available in the section about “specific implementations of virtual machine networking” have already been performed.
That being said, go ahead and run the virtual machine if it is not currently running.
” works, that's a great sign that network hardware works. If not, then don't despair.
- Some basic troubleshooting steps
The ability to see the virtual machine's screen is likely to be able to help.
If communication(s) with host name(s) fail, try using numeric addresses that don't require working name resolution. (This means to try using IP addresses instead of using DNS names.) If communications without name resolution is not working, plan to keep working without name resolution until basic communications without name resolution do work. If communications without name resolution works, one may try to get name resolution working. However, be sure not to extend the length of communications to different subnets. For example, if communications to internal machines work without name resolution, and communications to a remote system using name resolution fails, the ongoing problems may have to do with network traffic routing, and be related to the distance of the remote system, and not be caused by name resolution.
If the virtual machine software is known to be performing some NAT (like what frequently happens when using a standard-looking virtual NIC on the host machine to communicate with the virtual machine's NIC), realize that some network tests may not work. The virtual machine documentation may provide some services on the “inside” of the virtual machine's internal firewall.
If communication is enabled with a specific network interface device visible to the host machine, then attempt to communicate with just that device. That may be the most likely device to be reachable. If that works, but if Internet access is undesirably not working yet, then the next step will be to deal with traffic forwarding. (The way to do that may be substantially different than trying to troubleshoot whether the network card on the virtual machine is working.)
Is the remote end actually up? A classic example may be when using the virtual machine's NIC to communicate with a NIC on the host machine. In such a case, make sure the virtual NIC on the host machine exists, has a network address assigned, and is considered to be UP/connected. If a remote machine is being used to help test things, can some other tests (from another machine) successfully verify that the remote machine is reachable?
If communication isn't working, check firewalls. One way that is often fast and effective is to try communication in the opposite direction: if outgoing communications from the virtual machine aren't working, but incoming communications to the virtual machine do work, or vice versa, then that is often a case of hardware (whether virtual or physical) working, and traffic routing (including firewalling) being a culprit.
- Examples from specific implementations
- Example in OpenBSD (OpenBSD running in a virtual machine (running in Qemu running under OpenBSD))
This section assumes that there is command line access to an OpenBSD machine. If installation media for OpenBSD is used, get to a shell prompt by entering
at the “
” prompt, or by entering
(I)nstall, (U)pgrade or (S)hell?
at many other prompts that are shown during the installation process (perhaps not including when the installation process is running another program, like
Check the flags that are visible with “
”. On the host machine, the specific tun* interface that is specified on qemu's command line should show the LINK0 flag as well as the flag indicating that the device is UP. However, chances are that if tun2 is being used, “
” will actually show a result such as “
”. If this device does not exist or does not have the desired flags, then run something like “
tun2: no such interface
”. (However, be sure to specify the correct tun interface. e.g., maybe use “
tun0link0 up description
tun2link0 up description
Once this is done, the flags (for the network interface on the virtual NIC on the host machine) may show up as “
MULTICAST. ”. Make a temporary note of the IPv6 address, as one should have appeared (from within the
/10address range). If IPv4 is desired, go ahead and assign an IPv4 address (e.g. “
The following details are meant to be for the virtual machine. If booting off of the OpenBSD install media, select
Sto get to a shell. Type “
”. The NIC will probably show up as
ne3(unless there are other NICs). (The reason for this is not confirmed but may have to do with the names ne0 through ne2 being reserved for specific I/O port addresses (0x240 and 0x300 and 0x280) which are identified by some documentation such as OpenBSD FAQ 12.7.1: ISA NICs and the “hardware” section of the INSTALL.i386 file.) This interface is probably not up. Use “
”. After that is done, the IPv6
fe80::address should appear. If IPv4 is desired, go ahead and also assign an IPv4 address in the same subnet as the host machine's tun interface.
Presumably pinging will now work. Details how to perform the pings are about to be provided. If a ping test does not work, then check the results of standard troubleshooting techniques such as verifying if the network interface is saying that it is UP and whether the right IP address is claimed to be used by the card. A common thing to stop traffic from working as expected may be firewall software running on one host or the other: If the virtual machine is using bootable media designed to install the operating system then the source of firewall-related troubles, if that is what is causing troubles, is likely to be found in the configuration file of the host system's filewall.
IPv4 pings should now work. e.g. “
IPv6 pings should also work. This will require looking up the IPv6 addresses: The NIC is expected to have an address within the fe80::/16 range, but remaining bits will be rather random. (RFC 4193 section 3.2 discusses the requirements such as compliance to BCP 106 (currently RFC 4193). The sample code in RFC 4193 section 3.2.2 uses NTP and a MAC address to achieve enough pseudo-randomness that it is likely easier to look up the address than to try to guess what it is.) However, if randomness isn't being implemented very thoroughly, perhaps a MAC address of the virtual machine is 52:54:00:12:02:11 would end up with an IPv6 address such as fe80::5054:ff:fe12:211. Using that example, on the host machine, use “
” or use “
”. This may seem a bit trickly, because the command line is based on information coming from both systems: the IPv6 address to specify on the command line is the remote system, but the interface name to use is the one on the local system.
Similarly, on the host machine, use something like “
” followed by the rest of the IPv6 address on the
tuninterface, or use something like “
”. This concept of specifying the interface name may seem weird to those used to IPv4, but this is simply required when using IPv6 addresses in the fe80: range. This is because if there are multiple NICs that are up with IPv6 addresses, each NIC will have a subnet within the
fe80:range (according to the rules of RFC 4291 section 2.1). (The requirement for specifying the interface would be true of all addresses starting within the fe80::/10 range, and presumably also for addresses that start with the byte ff or fd, or perhaps fc or other addresses starting with fe). (This requirement does exist, even if the system seems to just have one non-loopback NIC. Although perhaps software could intelligently figure out which interface to use if there is just one non-loopback NIC, the software doesn't so the NIC needs to be specified.)
- This guide does not currently have information about other setups in this list of examples.
- Other notes about verifying networking
- Next steps