See requirements: requirements for the tutorial on making a virtual machine as it may be similar to the requirements needed for this guide. That info should possibly be merged.

New Release

This guide is being re-made from scratch. The older guide is available under a different header. It may have information which is now largely covered by making a virtual machine and setting up an operating system.

At this time, this guide is being written under the assumption that a base image was created using making a virtual machine and setting up an operating system. If not, please go through those steps first.

Go ahead and decide where the information for the new machine will be kept. This includes where the disk image(s) will be, where the configuration files will be, and where the documentation will be.

Learning new processes can be difficult enough without also trying to remember lots of individual details about how machines differ from each other. Hopefully the previous guides led to some simple documentation being made, for practice. It will start to really get essential here.

Disk space design

Before we go about customizing each machine much further, let's consider an idea about how we will design the disks. Some people may find this theory to be a stroke of genius: others may scoff at some aspect or another. Still, it is presented simply as an idea for consideration, so that people may take away (i.e. implement) whatever ideas seem good.

The layout

Consider breaking data up into four different categories. As an example, we will consider using one hard drive for each of these categories.

The boot image

This contains data that is similar among multiple different machines. For example, the installation of the operating system.

This hard drive takes advantage of child image technology. Every time it boots up, the virtual machine software uses a child image. The parent image checks for the presense of the other hard drives described in this design. It copies over any files that contain customizations. It also checks for a batch/script file to automatically run. These changes get stored in the child image.

If an operating system needs to be patched/updated/upgraded, then a new child image may be made. If the update does not go very well, then this image may be rolled back to a previous state, fixing the problems that were introduced by updating the operating system. None of the other, more custom data, ends up getting lost by the rollback.

Parent images may be read-only. Also, if an IT company were to support multiple different organizations, each organization may be able to use the exact same initial configuration files. This may simplify deployment and maintenance.

In Unix, the /usr/ directory might commonly be part of this hard drive image.

Initial setup customizations

This data is specific to a machine, and controls the initial setup customizations.

In Unix, that would likely be as easy as copying a directory that contains a bunch of text files. For Microsoft Windows, that may or may not be quite so easy: it could involve copying files, and importing registry settings, but may also involve copying DLL files, ideally before the machine is booted up. We will focus on Unix as an example, as this will be pretty painless to implement.

So, when the operating system is booted from the first hard drive, it may find data on this hard drive that says that the DHCP server and the DNS server need to be active. For a different virtual machine, the data on this hard drive may say that firewall software should be active.

Once this data is set up, this drive may be read-only.

Also, this drive could be completely unaffected by an operating system upgrade (such as from an old version of Debian to a new major version of Debian). A person might even be able to switch among relatively compatible operating systems, such as from an old version of Ubuntu to a brand new version of OpenBSD, without needing to change this hard drive at all. Such a change would involve substantial changes to the drive that is used for the base operating system. However, once that change is made, all of the different virtual machines could be started and this data may not need to be changed at all, and then all the virtual machines can be performing their own unique functions on the new operating system. Like any other change (especially a large change that affects multiple things, like a service pack), there might be some problems that arise from differences where the operating systems aren't entire compatible. So, some testing may be needed. However, depending on things like complexity and just how similar the operating systems are, it is actually feasibly possible that no changes at all would need to be made to this hard drive.

Also, if an IT company is rolling out similar configurations for multiple different companies, it is possible that these drives might not need many customizations per company. Perhaps there could be a series of hard drive images that are parent images, and those may be identical even for different organizations. Then, there may be just one or two child images to contain the key critical differences, like domain names and, if different, IP addresses. (Such IP addresses might not even need to be different, in the case of “private” (NATted) IPv4 addresses. Or, perhaps such organization-specific data may go onto the third hard drive.

Organization-specific data

This third type of data contains additional stuff that is written, and which should be kept. For example: logs. Another example: user data (like data that stores a player's progress in a game, or perhaps an office document).

When a machine gets started up, the parent image may check for this sort of data, and either copy it over or perhaps just mount the data from this location. For instance, perhaps all of Unix's /var/ directory, which contains this sort of variable data, may go into this sort of directory.

When the machine gets shut down, if data was copied to a child image on the first hard drive, then updated data may be copied/synchronized back onto this hard drive. Even slicker, any data which was on a partition that just got mounted directly from this hard drive won't need any such copying.

Temporary data

Data which typically does not need to be kept between reboots. e.g. in Unix this would be a swap partition, and a partition that will be used as a mount point for /tmp/. For Microsoft Windows, the equivilent for the first part would be a partition that is dedicated to a swap file. The latter (user temporary data) may be a bit more difficult to implement in modern versions of Microsoft Windows, since user temporary data is typically stored in the heirarchy in the same parent folders that store other user data which may not be meant to be quite so temporary.

The temp data could go onto its own dedicated virtual hard drive. All temporary data could be written to a child image which is routinely destroyed every time the virtual machine is started. Then, if an attacker did get access to the virtual machine, the attacker could not steal confidential data by performing data recovery of deleted data that was previously in a swap file, because that hard drive was destroyed the last time the virtual machine was shut down. Instead, a new virtual hard drive was used, and the new image may have never had that confidential data written.

Advantages of this design

If this design was implemented by using four virtual hard drives, then the data is essentially sorted by how the data needs to be handled. Large amounts of data may be read-only (stored in parent images), and could even be identical between different organizations, without any breach of confidentiality where one organization has access to a different organization's data. Data which is classified as “read only” (never being written to) over a long period of time may not need to get backed up quite as frequently, as even old backups will still be valid. Temporary data doesn't need to be saved, so the only data that needs to be routinely backed up and kept highly confidential is the third hard drive image, and perhaps the child images for the second hard drive. Operating system upgrades could only involve careful updating of the first hard drive, with no changes made to any of the other drives.

Creating a working network infrastructure could be as easy as using standardized copies of the first and second and fourth hard drives, as well as standardized (empty) copies of the third drive. That means people could very quickly have access to critical network infrastructure services (automatic addressing and DNS) and perhaps even outgoing E-Mail, all by using entirely standardized images that contain no data that is specific or confidential to an organization. That could be highly automatable. It may still take some time to restore (from backup) data that needs to go onto the third hard drive, but meanwhile people are partially functional. They can surf the web. This may be better than a scenario that, at least historically, had been typical, where one physical machine handles everything and so nobody in the organization can do anything until the technician spends time getting everything restored.

Patching could involve using a copy of the read-only hard drive image that the boot hard drive is using. Then, if that patched smoothly, the other machines could be “patched” by being shut down, altering the reference to look at the updated version of the boot drive, and then starting up. If the actual patching process takes a long time to “apply” patches (meaning to change all of the individual files), that can be done on just one machine that isn't running any other services, during which time there is not any downtime being experienced for any of the other services. (The other machines may still need other patching to be performed. For instance, if an SQL service was updated, the SQL service could be updated on the boot drive, which would affect all child machines, or just on the machine that runs SQL. In the latter case, the SQL server may still need to be patched. However, any required downtime may be reduced compared to a situation where operating system updates also needed to be applied. An alternative idea could be to patch a copy of the SQL server, similar to what is done with the operating system drive.)

Implementation
Using fewer drives

Implementing the above design may work for a computer (even a physical machine, but more likely a virtual machine) that has four hard drives. However, most physical machines don't have four hard drives, and even at least some virtual machine software may typically be limited to just four main storage devices. If one of those storage devices is going to be a virtual optical drive that reads CD images, that leaves just three hard drives.

There may be various ways to deal with using fewer hard drives. One may be to install the operating system without using separate drives for organization-specific data and temporary data. (The organization-specific data and temporary data could be, at least temporarily, combined onto one drive. Actually, both organization-specific data and swap data might be completely unneeded at this early part of the process. Then, once the operating system is installed, remove the CD drive and use the other hard drives as intended by the design. Note that this may cause some complications if an optical drive is later deemed to be useful. (Hopefully the temporary data won't be needed then.)

The drive for temporary data could be combined with the drive for organization-specific data (and then disregard the comments about not keeping a copy of the data on the drive that stores temporary data). The organization-specific data (including the temporary data) could be implemented as a child image of a hard drive that contains the initial setup customizations. If that is done, then changes would need to not be casually made to the parent image if such changes would wreck the ability for the child image (with the organization-specific data) from being able to be changed. So, any subsequent changes would need to go onto the child image that stores the organization-specific data. By making these two changes to the design, the design would then only be requiring two hard drives, which may be more feasible.

Storing temporary data on another drive

As nice as it sounds to have a hard drive without any real concern about its contents, it is a good idea to have some space reserved on the main operating system drive to be able to use temporary data as needed. In an environment using a copy-on-write (“COW”) hard drive image, this reserved space may not really take up any actual space on the computer that runs the virtual machine software. However, at least having enough space for such partitions can be useful if there is ever a time when it is desirable to boot the hard drive by itself. For instance, if creating a new virtual machine from scratch, perhaps it will be faster to use an existing hard drive, rather than needing to tell the virtual machine about a second hard drive (and perhaps then needing to spend time configuring the second hard drive, such as laying out partitions). The task of configuring the second hard drive may be something that is easier to do when the system is nicely booted up, so life may be less convenient if the first hard drive requires another drive just in order to boot the system.

So, in Unix, make /tmp/ and have some space for a swap partition, even if they aren't heavily used. These areas on the disk might not need to be very big. (Having a large /tmp/ can be useful for everyday use. For instance, lynx may write files there by default, so downloading a large file could fill up a small /tmp/ directory.) However, for not-everyday use, smaller drives may be tolerable (and allow the space to instead be available for use by some other sort of more “everyday” purpose.

The following is likely some old text that needs to be deleted; perhaps there is something useful to merge with the above.

However, this drive storing the customized settings for each virtual machine might actually require zero changes throughout that entire process. Then, once the new operating system is all ready and configured, existing virtual machine configurations may be modified to use the new hard drive image for the operating system. The references to the hard drive images that store individual customizations won't need to change.

This data may be read-only.

  • Data which is similar among multiple different machines. For example, the operating system
  • Data which is specific to a machine, and controls the initial setup customizations.
  • Data which contains additional stuff that is written, and which should be kept. For example: logs. Another example: user data (like data that stores a player's progress in a game, or perhaps an office document)
  • Data which typically does not need to be kept between reboots. e.g. in Unix this would be a swap partition, and a partition that will be used as a mount point for /tmp/. For Microsoft Windows, the equivilent for the first part would be a partition that is dedicated to a swap file, and the latter... may be a bit more difficult to implement.

Now, consider if these were created as four different hard drive image files. They could be treated in the following manner:

The temp data could go onto its own dedicated virtual hard drive. That hard drive could be routinely destroyed every time the virtual machine is stopped.

The data which is similar among multiple machines could be shared as a parent image.

When the parent image boots up, it could simply copy over the initial setup customizations. In Unix, that would likely be as easy as copying a directory that contains a bunch of text files. (For Microsoft Windows, that may or may not be quite so easy: it could involve copying files, and importing registry settings, but may also involve copying DLL files, ideally before the machine is booted up.)

Then, the machine copies over additional stuff that has been written, such as old logs.

When the machine is shut down, it copies the new logs.

now, er, consider a design: each hard drive consists of three parts: parent image, child image with initial setup customizations, additional image for more stuff that is written, and which needs to be kept. e.g., like logs additional data for stuff that gets written, but doesn't need to get saved. e.g. a swap partition, or a folder for /tmp in Unix.
Making documentation

If a new network is about to be created, then make a new location (such as a single file, or even a directory that will store multiple files) to store details about this network.

[#mltvmknw]: Making a new virtual machine

If the process of making a virtual machine has been completed once, makinga nother virtual machine may go faster than creating a first virtual machine. One reason is increased experience working with the software. Another reason may be an ability to copy details about how the first machine was configured.

Creating a disk image

Create a image for the primary storage disk. This may mean creating a brand new hard drive image (as discussed in the section about creating disk image(s) for a virtual machine), or copying an existing image file (including all of its contents), or making a snapshot.

How to make a snapshot

See the warning about permanent path references, in the section about image files of primary storage devices: section about child images. Then find the details that are specific to the type of image being used, covered in the section about child images and/or the individual sections about the various formats that support this feature.

Now, there is no need to install the operating system, because it is already installed using the data from the parent/base/“backing” image file.

Determine configuration details
  • Where is the file to start the virtual machine going to be? (Note: If there are going to be multiple virtual machines, it may be nice to have a rather standard type of location for such files. Having that type of location standardized may help automation later, such as if one wishes to start all machines by running “ for $x in ./*/sysstart/* ; do $x ; done ” or some similar craziness.
  • Where is the documentation going to be?
  • For virtual machines, make sure the EUI-48 MAC address will be unique (at least for the LAN). One way to do that may be to choose what address will be assigned. (Note for virtual machines: a simple option may be to make the ninth and tenth hex digits of the EUI-48 MAC simply match a custom machine number. Choose such a number.) What is the IP address going to be?

Creating the new configuration
Copy an existing virtual machine

One method may be to copy an existing virtual machine.

Note that if this is done, there may be a disaster of some sort if the newly created machine starts running before additional customizations are made. So, go ahead and start making/copying the configuration, but know that the machine may not be ready to be started. (Even if the software would be able to attempt, without complaining, to start the machine, be careful to not start this unprepared virtual machine too soon.)

The Qemu virtual machine makes for a good generic example since individual machine configurations basically consist of text files that are easy to access. So, the following example contains some instructions for doing this with Qemu. Other virtual machine software may provide some other sort of process/method/interface to be able to make a copy of a virtual machine. (Details about how to use the software may be in the section about virtual machines.)

Qemu

Find the file that was used to create an instance of a pre-configured virtaul machine (a.k.a. the pre-existing script file that is used for starting a virtual machine).

(e.g. ${VMDir}/bin/exc_base_system may have been used in the guide to making a virtual machine. However, if the VMDir variable is unset, then referring to some documentation (or hunting) may be needed to locate the file.)

Copy that file to the documented location for the file that is going to be used to start the new virtual machine. (This, of course, requires that the location for this file was or is documented. Document it if that hasn't been done. Before trying to copy the file, list filesystem contents as needed to make sure that the destination directory is created. Then, if needed, see the section about how to copy a file.)

Running the new system at this time would likely be an unideal idea. First, follow the upcoming guide for customizing the new virtual machine.

Creating a new configuration

If, for some reason, no existing configuration has been made yet, then make a new configuration.

Customizing the new virtual machine

Some references, such as filenames, should be unique for this machine. A great idea is to determine a standardized format for the filename which includes the name of the machine. Then, set a variable (perhaps with a name of VMNMABBR) that contains the virtual machine's abbreviated/short name. Include that variable in references to a filename. (On the command line, the reference could look like AFileFor${VMNMABBR}Machine, which could create a file named something like AFileForFirewallMachine.) By doing this on the first script, the first script can be copied and there will be less work to do for subsequent scripts that get created.

Handing data storage virtual devices

Perhaps the most important thing is to make sure that it is using its own hard drive image.

If the old virtual machine is set to boot off of operating installation media, that may be okay. However, if that isn't desired for the new virtual machine, change the boot order as needed.

Having an operating system boot disc being ejected from the virtual machine may be nicer, as that may mean that the virtual machine no longer requires that the disc image is at the expected location. (Then, if the disc image doesn't get regularly used, and ends up being moved or deleted, the virtual machine will no longer unnecessarily rely on that file.) Remove fragility by removing any reference to that disc.

Unique networking settings
NICs for the new machine
Determine how many NICs the machine needs. Have the virtual machine create the direct number of NICs. If applicable, make sure the NICs are the desired type (which may require determining how those NICs will communicate: e.g. whether they will communicate directly with a NIC on the host machine/etc.)
Machine number

Having a number that will be unique for the local subnet can help to quickly generate a unique “link local” hardware (EUI-64/MAC) address, logical (IP) address(es), and any other addresses that may need to be unique.

Document this machine number. (A key reason for going through the effort to document this number is just to have the documentation ready to be able to add more details that are about to be decided/set.)

MAC address
Often the next most important thing, so that communications don't get temporarily disrupted, will be to make sure that this machine will use its own unique MAC addresses. (Some virtual machine software may take care of this when creating a new virtual machine configuration. Other software, like Qemu, may require some manual configuration for this.)
NIC configuration
While focused on networking, if there is any other network configuration to customize, now may be a good time to do so. (With Qemu, the customization may involve scripts to help configure the tunnel interface on the computer that is running the virtual machine software. If so, copy those files as needed. Perhaps to help with some automation efforts down the road, it may be nice to have those files, which are run by Qemu, separate from other scripts that may be more likely to be run directly.) If there are any IP addresses mentioned by the per-NIC configurations, customize those IP addresses for the new machine.
Logical network address

If automatic network (e.g. IPv4) address assignment is going to be used, consider customizing it now for the new virtual machine. (This may not be an easy option if automatic addresing has never been set up yet, but once it is, then this step may be sensible for additional machines.) Then, the new virtual machine might get that address automatically.

This may not work quite as easily for automatic addressing based on a DHCP UID rather than a MAC address, if the DHCP UID is not easily predictable. The DHCP UID is likely rather difficult to predict if the DHCP UID is at least partially based on a timestamp. Since that is common for DHCPv6, this may be a tip that is more applicable for IPv4 than IPv6. However, if this machine is going to be multi-stack (that is, dual-stack supporting both IPv6 and IPv4), then setting this up for IPv4 makes sense.

Other
Qemu systems that listen to a TCP port for the monitor need to make sure that a unique port is specified. (The port number is generally the most sensible to customize, though having a unique IP address would also technically work.)
Other

Qemu users: Customize the unique PID file assigned for this virtual machine.

If Qemu was set to pause the machine on startup, remove the -S option. (Note that lines may be commented out, by starting them with $(# and then placing a matching ) after the -S. If such an option exists in the script, but is already commented out, then no change is needed.)

Other things, like the name of the virtual machine, would ideally be customized. Although, truthfully, this might be cosmetic, so it may not be a huge issue.

Check that the virtual machine is ready/prepared
Follow the steps of creating a virtual machine: section listing final preparations.
Starting the virtual machine
Run the virtual machine. Then check that the virtual machine is running.
Getting a remote session going
The easy method

If automatic addressing is set up on the network, and there is a known way to connect to this machine (e.g. if a reserved fixed IPv4 address will be assigned to a known MAC address), and if there is a known method of remote access, then this may be as simple as creating a remote session after waiting long enough for the machine to start its boot process, get the expected network address assigned, and starting a remote access server.

However, since this method doesn't work for all virtual machines (such as a brand new virtual machine that isn't based on a pre-existing base image), this method may not be available. A more reliable process may be needed.

Note: The process of getting a display working may not be initially needed in this case. Still, it is good to know how to get a display without requiring that remote access software has been started. This is mainly because having such a display may, at some point in the future, be useful if there is trouble loading critical software during the system's startup process.

The reliable process
Getting a display
If this step can be effectively skipped, because the hard drive will automatically support remote access on a known IP address, then great! Otherwise, go ahead and set up the local display: using the local display (e.g. enabling Qemu's VNC server).
Using remote access

The general goal is going to be to get remote access working. Note that there may be some known working credentials already set up from the base image. (It will be desirable to set up some customized credentials, although until that is done, the existing credentials may be useful.)

Testing networking
Make sure that the virtual machine can communicate with what it needs to. (Routing, Internet access, and all that fancy stuff is beyond the goal: this is more about hardware testing, to just mak esure the machine can communicate with its neighbor.)
Assigning the logical (IP) network address(es) long-term
(Don't be too concerned with this on the first machine or two, until automatic IP addressing gets set up.)
Setting up the operating system

Follow the guide to setting up an operating system. e.g., give this machine unique credentials, and let the machine know a name that it may call itself. (Some of this guide may go faster on a pre-configured machine, compared to the first time when the base image was created.)

Shutting down the operating system

One thing that may be different about setting up a long term virtual machine, compared to a base image, is that setting up remote access commands may be more sensible for a long term virtual machine. This is simply because setting up a remote access command may involve placing information about credentials onto the machine, and that may be less desirable for a base image. (A base image might be more secure by not having any older, previously-used files that store the credential keys.) When working on a machine that will be used long term, fix that problem.

The guide to setting up the operating system should have covered the ability to shut down the virtual machine, perhaps by running an ssh command (as described by the section about using a special, dedicated user account to shutdown/reboot). On the machine that is running the virtual machine software, make sure there is a script that will automatically run that ssh command (using whatever key files are needed). On that same machine, make sure the shutdown process will automatically run the file to shut down that virtual machine.

Getting the virtual machine to automatically start up

See automatically started files. Unix users: Be sure to check out the automatically started files: section about problems starting programs in Unix.

Make a second virtual machine
Follow the previous steps (for making a new virtual machine). One key difference, though, is that this virtual machine only needs a NIC to be able to communicate to the other virtual machine. This means that the computer running the virtual machine software won't be requiring a (virtual?) NIC to communicate with this virtual machine. So, the setup process may be simpler. (e.g.: For Qemu, this will mean that the script files for a NIC may not be needed.)
Determine the layout

This can certainly vary. This guide shows the creation of a network that uses an example network that is envisioned by the creator of this guide. Some people may dislike this sort of example layout for one reason or another: This guide is not making a strong statement about whether performance could be increased using other methods.

The layout is as follows: Physical hardware runs virtual machines, and routes traffic. Namely, the physical hardware routes traffic to the virtual machines. The virtual machines may perform more complex network traffic routing.

This way, the physical machines will not do much processing of the network traffic, which will hopefully make them less likely to experience some forms of attack.

If there is a problem with physical hardware, and if the virtual machines are sufficiently backed up, then getting a fully operational network involves setting up a replacement machine. If the functionality of the physical machine is limited, there may be less work needed to get the physical machine to perform the tasks that it needs to.

The “device” performing the main network traffic handling is a virtual machine. Most common customizations to how network is handled, such as supporting a new network service, is handled by adjusting the virtual machine.

All virtual machines communicate with each other using virtual hardware. By using virtual hardware, these virtual machines may communicate to each other without any need to use a physical NIC, and they won't need to use a tunnel interface on the physical machine. This may reduce overhead. One virtual machine, the firewall, can communicate with the virtual network hardware used by other virtual machines, and can also (use a tunnel interface to indirectly) communicate with a physical NIC on the physical machine.

The end result is that routing is a bit complex to initially set up, but then lots of things can be running on seperate virtual machines.

IPv6 tunneling over IPv4 Internet

Hopefully this section will be come less useful over time, as more Internet provides just simply provide IPv6 access. However, IPv6 is uncommon at the time of this writing, so a tunnel broker may be needed.

The goal here is going to be to have the virtual firewall machine be able to communicate with the remote IPv6 Internet.

Get a tunnel

First, get a tunnel. There are public places that will be able to have a tunnel ready on demand. Some places may require some turn-around time.

Identify the remote end points
e.g. using a tunnel broker
Using a tunnel broker
HE.net's TunnelBroker.Net

HE.net's TunnelBroker.Net has been known to provide an address out of 2001:470:a:0000::/64 (customizing the portion which is, in the example address shown, filled with visible zero digits). The server would then use the first usable address (ending with 1). (So the web page may refer to it as looking something like 2001:470:a:0000::1/64 with the zero digits customized, and with leading zero digits being left unshown.)

The server IPv4 address may be 216.218.26.238.

These addresses, which the tunnel broker may refer to as a “server” address, are the addresses used by servers at the tunnel broker. From the vantage point of somebody setting up a network at a location other than the tunnel broker (like, for example, a person's home Internet access), these addresses for the tunnel broker's servers would be the “remote” site of the tunnel. (So these addresses are what should be used as this guide refers to a “remote” address as it continues to discuss these tunnels.)

This documentation will use the IPv6 address block of 2001:db8:2468:aced::/64 (which is part of the 2001:db8::/16 address block reserved for documentation/examples). Actual implementations should use a different address.

This documentation will be using the IPv4 address block of 203.0.113.32 (which is part of the 203.0.113/24 “TEST-NET-3” address block reserved for documentation/examples).

Determine the tunnel's local endpoint

The endpoint needs to receive the tunneled traffic. One option may be to use the firewall. (This may be easier if the firewall is a standard computer running a regular operating system.) Another option (which may be better if the firewall is just a low-end cheapo dedicated device) may be to run the tunnel software on a different device, such as a standard computer running a regular full-fledged operating system. However, the device that is going to run the tunneling needs to be able to receive the tunnelled traffic. If this device does not receive all Internet traffic, then some special routing may need to be set up so that this device can receive traffic.

If an inner computer is going to be the tunnel endpoint initially, and it uses the same platform (OS, network stack, and software to implement firewalling/routing) as the firewall, it may be simpler to just get the tunnel working on the firewall initially. That way the tunnel can be tested without concern about traffic routing breaking. Then, the tunnel may be moved. Details about that are provided, after the sections discussion how to create the initial tunnel. However, if the eventual destination uses a different platform, it may be more worthwhile to just deal with routing traffic early, rather than investing in learning the steps for the temporary setup. (This would especially be true if the firewall is using a platform that isn't easy to make the local tunnel endpoint.)

Identify the local endpoints

The IPv4 address for the local end of the tunnel is simply the publicly accessible IPv4 address. (See finding a public IP address.)

This documentation may use the IPv4 address block of 198.51.100.64 (which is part of the 198.51.100/24 “TEST-NET-2” address block reserved for documentation/examples).

Using a tunnel broker

As far as the public Internet is converned, the traffic to the netblock gets routed to the tunnel broker. However, the tunnel broker will then use a tunnel to send most traffic to the client. (The tunnel broker may retain a single address out of the netblock, as the tunnel broker may need to use this address.)

HE.net's TunnelBroker.Net

The “Tunnel Details” page (which provides account-specific details, and so requires logging into an account) may show a “Client IPv6 Address” which is one address higher, ending with :0000:0000:0000:0002. (So it may look something like 2001:470:a:0000::2/64 with the zero digits customized, and with leading zero digits being left unshown.)

Create the tunnel

HE.Net's TunnelBroker.net shows some example configurations when logged into an account. Those provided examples, provided on that page, may use account-specific IP addresses. (At least some of these examples might be based on some of that example documentation.)

NetBSD / Mac OSX

e.g.:

ifconfig gif0 create
OpenBSD
Don't bother. The tunnel will be created when it gets used.
Assign the local and remote IPv6 end points to the tunnel

Example addresses are used in this documentation.

203.0.113.32 represents the public IPv4 address for the remote tunnel endpoint.

192.168.200.128 represents the IPv4 address that will be receiving the tunneled traffic. (This could be a public IPv4 address, e.g. 198.51.100.64, or a private IPv4 address that receives redirected traffic.)

2001:db8:2468:aced::1 represents the remote public IPv6 address. 2001:db8:2468:aced::2 represents the local public IPv6 address.

When plugging in values from these examples, be sure to keep the addresses straight (regarding which one is local, and which one is remote). (The concern there is mainly that if the addresses are not put in the right spot, the tunneling is likely to fail.)

OpenBSD
sudo ifconfig gif0 tunnel 192.168.200.128 203.0.113.32
sudo ifconfig gif0 inet6 2001:db8:2468:aced::2 2001:db8:2468:aced::1 prefixlen 128
echo The following optional command simply shows current settings.
ifconfig gif0
NetBSD / Mac OSX

It is the same as OpenBSD except that the word alias is not used (at least not by HE.Net's TunnelBroker.Net's example documentation).

[#tunlicm6]: Test the tunnel

Ping the remote end.

In Unix
ping6 2001:db8:2468:aced::1
Moving the local end of the tunnel

Hopefully that tunnel was operational, as verified by the test being successful.

Verify some Internet connectivity

On the machine that will be operating the local end point, test IPv4 connectivity.

ping 203.0.113.32
Bring down the older tunnel

This was done by rebooting. Other attempts were not successful. This portion of the process should be reviewed to determine a quicker, less disruptive approach.

Set up the local tunnel endpoint

The commands may be identical to the instructions from the “Assign the local and remote IPv6 end points to the tunnel” section.

Note that this tunnel may not work, but go ahead and set it up anyway.

Test the tunnel

Set up an ongoing ICMP6 test to the remote tunnel end point. If this fails, there may be some more work to do. However, a point to running this test isn't just verifying that all work is already done, but to allow progress to be easily checked quickly as later steps are taken.

Test the tunnel.

Adjusting routing

If the internal address of the tunnel is a private IPv4 address, the tunnel may actually work due to the wonders of NAT. However, incoming tunneled traffic which doesn't match outgoing NATted tunneled traffic may not be working yet, because the incoming tunneled traffic isn't being redirected/re-routed to the needed location.

(No clear steps are provided here for fixing this issue. Some helpful information might be at routing traffic.)

Automating some of these steps upon each reboot
OpenBSD

In this example, customize the value given to CURNIC, if desired, and customize each of the (five) occurrences of an IP address.

CURNIC=gif0
[ -f /etc/hostname.${CURNIC} ] && cpytobak /etc/hostname.${CURNIC}
echo !ifconfig \$if tunnel 192.168.200.128 203.0.113.32 | sudo tee -a /etc/hostname.${CURNIC}
echo !ifconfig \$if inet6 2001:db8:2468:aced::2 2001:db8:2468:aced::1 prefixlen 128 | sudo tee -a /etc/hostname.${CURNIC}
echo Test... Disruption will exist but be minimal if all is well...
sudo ifconfig ${CURNIC} destroy
sudo $SHELL -c ". /etc/netstart ${CURNIC}"
sudo ifconfig ${CURNIC}
CURNIC=
ping6 2001:db8:2468:aced::1
Route IPv6 traffic (for the system with the endpoint?)
Let the local endpoint know where to send IPv6 traffic

Presumably traffic destined for the remote tunnel's endpoint is successful. The machine can look at its own routing tables and realize that the local tunnel endpoint is on the same subnet, and so it knows how to reach the remote tunnel's endpoint. Now, the trick is determining how to make other traffic go out the tunnel. Rephrased: the trick is determining how to make other traffic go to the tunnel's remote endpoint.

The way to accomplish this is to create routes. More specifically, creating an IPv6 default gateway setting, pointing at the remote tunnel endpoint, should work. This will cause the traffic to get sent to the tunnel's remote endpoint. e.g. the following may work for Microsoft Windows and for Unix.

route add default 2001:db8:2468:aced::1

For further details, see the section about adding network routes.

Sending traffic to the remote tunnel endpoint
The common method
Allow traffic forwarding

If this hasn't been done yet, it will need to be done. See: implementations of enabling traffic forwarding.

Have other machines route to the local endpoint

Other machines may set their IPv6 default gateway to any reachable IPv6 address on the machine that forwards and routes traffic.

The idea is that the machine which forwards and routes traffic should then receive and process the traffic. When processing the traffic, this computer determines where the traffic is supposed to go, and use its routing tables to (notice the IPv6 default gateway, and) realize that the traffic needs to go to the tunnel's remote endpoint.

The more direct method

If other machines know how to reach the remote tunnel endpoint, then forwarding may not be needed. Any machine which can communicate with the remote tunnel endpoint may simply set the default gateway to be the remote machine.

This may be fairly uncommon, unless the firewall is like some sort of bridge. The simple problem is that although the tunnel endpoint can communicate with the remote end, most other machines probably cannot. One (probably less desirable) solution will be to adjust local traffic routing so that more local machines know how to communicate directly to the remote tunnel endpoint. However, the more common method may be to allow machines to indirectly communicate to the remote tunnel endpoint.

Have the virtual machines communicate with each other

Double-check the network settings. For instance, if using Qemu's socket without multicast, make sure that one virtual machine is using socket,...,listen and that all other machines are using socket,...,connect.

Choose an IP addressing scheme to use. Network addressing has details about creating plans/methods of addressing, most notably some of the newer (so, non-traditional) general guidelines on selecting private address ranges and advice about IPv6 addresses. Know what subnets are going to be used: this means knowing what IP addresses and what “prefix lengths” (which is a newer term: the older term that was often used for IPv4 was “subnet mask”).

Assign each NIC a manual networking address (e.g. have each machine be given an IP address in the same subnet as the other machine). Details for performing this are in the section about manually setting a network address.

Make sure the machines can communicate with each other. (Further details here could be useful.) e.g.:

IPv6 usage in Unix

sudo ifconfig if0 inet6 2001:db8::127:100 prefixlen 64
ping6 -I if0 2001:db8::127:123

If using an address which is not a link local address (which would be preferred), the “-I if0 ” may not be necessary.

[#rtvmtun]: Get firewall/routing working

For a general overview/guideline, see the section about routing traffic. (This guide provides a bit more specifics.)

In this text, the example setup is based on the idea of multiple virtual machines which can communicate with each other, with one of these virtual machines also having a network interface that can communicate to physical equipment (the computer running the virtual machine software, and the Internet).

On the second machine, using ping on a public Internet address may be failing. Using ping to communicate with the maching running the virtual machine software may also be a ping command that is failing. Unsurprisingly (if it has already been discovered that ping is failing), more complex communications, such as an encrypted SSH connection, may also be failing. There may actually be a few things that need to be changed to let this work.

At this time, this guide uses manual routing to resolve routing issues. Automatic routing protocols might actually end up being easier to set up: This guide may be updated in the future. For now, the presented text works.

Quick networking basics note: order of ping commands

Note: Some of these examples may describe running a ping command on a nearby machine and running a ping command for a remote host on the Internet. There is no super strong rule about which ping command to run first. The best answer may be to just trust gut feeling on what is likely to work. Running a ping command to check further Internet connectivity will save time if it works, because that will be a strong indication that communicating with a closer device will also be successful. In that case, testing communication with the closer device might actually be quite unnecessary. However, if the problem is with communicating with the closer device, then running the ping command on the nearby device may show a problem, and then the more advanced test (communicating with the further device) may be unnecessary. Running only the necessary test might be sufficient in some cases, and may be informative, but guessing which test will be necessary would involve guessing what the results are likely to be.

So, don't worry if this guide suggested running one ping command first if another ping command seemed like the test that was naturally desirable to test first. There's no hard and fast rule about which test must come first.

Preliminary things to check

The first thing is to make sure that the virtual machines can communicate with each other. (It doesn't matter if they could communicate before, when temporary settings were in place until the last reboot. Verify that they can communicate now.) Perhaps chances are statistically high that they can communicate now, but if they can't, a lot of time may potentially be saved by figuring out this problem first. Another thing to be careful about is to make sure that the specified IP address is the correct one.

Routing issue part one: The second machine needs direction

The ping command may be providing a clue on how to start fixing traffic, as the ping command may be reporting “No route to host”. The first step is for the second virtual machine to know where it needs to send its traffic. Follow the step in the section about adding a network route to (manually) add a default gateway. That will take care of the first problem. The second virtual machine may then stop complaining about not having a route to the host, but it isn't getting any responses yet.

Forwarding
Identifying the problem

To narrow down the next problem, one may wish to watch/monitor network traffic on the first virtual machine, which is the virtual machine that needs to be forwarding traffic between the physical and virtual interfaces. To do so, consider running a command like:

sudo tcpdump -IpSni if1

(If there is a syntax error, try running just “ sudo tcpdump -i if1 ”. If even that is failing, then do realize that if1 is a customizable part of the command line. That needs to refer to the name of the NIC being used. The section about available NICs may help.)

Hmm, this may show incoming packets, but no reply packets. Go ahead and press Ctrl-C to abort the tcpdump command, and run tcpdump on the network interface that connects this virtual machine to the computer that is running the virtual machine software.

sudo tcpdump -IpSni if0

Hmm, no relevant packets are showing up. Take a moment to verify that this virtual machine can communicate with the desired destination. ping an address on the Internet. Hmm, that works. Unsurprisingly, then, pinging the computer that runs the virtual machine software also works.

So this virtual machine can communicate to the needed destination, but it is not forwarding traffic to the needed destination when that traffic is originating to another machine.

Resolving this
See the section about implementing basic traffic forwarding.
Expected results

At this point, monitoring the network traffic on the firewall machine may show that the forwarded traffic is now being passed onto the computer running the virtual machine software. Still, no responses are coming from the computer running the virtual machine software. That remains a problem, but at least it seems that this routing machine is no longer causing further problems by refusing to send out traffic.

Making routing work round-trip
Identifying the problem

Listening to the network traffic on the host server may confirm that traffic is indeed being received by the host server, but the host server isn't sending any response traffic. Some further tests on the host server may find that a ping command will work to communicate with the firewall machine if using the NIC that connects directly to the host server, but not if trying to ping another NIC on the firewall machine. An inability to really reach that further NIC does explain why the host server is not reaching the other virtual machine. What is making that NIC act differently?

The simple answer is that the host server is not routing traffic to the further NIC. The closer NIC is on the same subnet, so the routing rules have been sufficient. (This may have become clear if test ping commands were showing the results “No route to host”, similar to when other routing had been an earlier issue.)

Resolving the issue (short-term)

On the computer running the virtual machine software, adding a network route to (manually) add a route which is NOT going to be the default gateway.

Test (and celebrate, temporarily)

From the computer running the virtual machine software, not only does ping work, but so does SSH. Woohoo!

However, have these issues been resolved long term? What happens when the machines reboot? There may still be more work to do.

Fixing things long term

Note that if things have been fixed in the short term, using remote access may now be a way to fix what still needs to be fixed.

Overview: steps to take

If there was any setting of the IP address, make sure that will be taken care of long term. (See network addressing after the “Plans/Methods for addressing” section. Specifically, if automatic addressing isn't set up yet, see manually handling network addresses: setting the current address.)

For the first problem that was fixed, the issue was simply adding a default gateway. For most machines, that might be resolved through automatic sharing of the default gateway. With IPv4, that is done using DHCP which is the automatic addressing protocol. However, if this machine is going to be used as the automatic addressing server, this machine may need to have some more manual network configuration.

On the machine running the virtual machine software, the route needs to be added. Perhaps the most sensible place for this route to be added is to automatically occur when running the virtual machine that contains the NIC that all other virtual machines communicate with. After all, if that virtual machine isn't running (for whatever reason), the route probably won't be useful anyway.

Implementing the needed changes

Implementations may vary. Here is at least one example.

OpenBSD

On the machine that had the first problem, identify the name of the name of a network interface. Specifically, identify the name of the network interface that will be used. In this example, which in all likelihood does need to be customized, the example network interface being used will be called if0. The network settings (namely, especially likely the IP addresses) should also need to be customized.

FirstNIC=if0
cpytobak /etc/hostname.${FirstNIC}
echo inet 192.0.2.8 255.255.255.0 NONE | sudo tee -a /etc/hostname.${FirstNIC}
echo !route add default 192.0.2.1 | sudo tee -a /etc/hostname.${FirstNIC}

See: basic network traffic forwarding: implementations, and in the OpenBSD section, see the section about “Making long term changes”.

As for adding the route to the machine running the virtual machine software, if Qemu is used, place the route command in the script that initializes the tunnel interface. Placing the command elsewhere can be problematic, because the route command may fail if the tunnel interface does not yet have the needed IP subnet assigned yet. (Hopefully the location of that script is information that is documented, and is readily available.) Then edit a text file. If some other solution is used, either find an equivilent way or maybe just make the route when the system is starting up. That can be done by editing the /etc/rc.local file.

There is one further step to do, which is to test things. However, the best way to test things will be to reboot the computer, and it does make sense to take care of a couple of other things before performing that reboot.

UNFINISHED NOTES

		If the second machine tries to ping the host server, it will get
“No route to host”.  Add routing.  e.g. in Unix:
route add default 10.0.5.1


Then the second machine will stop complaining, but it isn't getting any
responses yet.

On the firewall machine, verify the traffic by using:
sudo tcpdump -i em1 host 10.0.4.1

that works

sudo tcpdump -i em0 host 10.0.5.8

that shows no traffic

On firewall:
sudo sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1
sudo sysctl net.inet6.ip6.forwarding=1
net.inet6.ip6.forwarding: 0 -> 1

You now see that the second machine's traffic is being sent to the host
system.

On the host system, one can use:

sudo tcpdump -i tun0 host 10.0.5.8

That shows the host system is getting the traffic, but isn't sending the
traffic.
Now, really, the host system is sending the traffic, but it isn't getting
where it needs to go.
On the host system, use:
sudo route add 10.0.5/24 10.0.4.2

That should cause the pings to fully work.
At this point, the second machine might even be able to reach the Internet.

If all that worked, then be sure to add the routes and firewall rules to some
	sort of file that will have effect each time the relevant system
reboots.  (That is a total of three systems to modify.)

The non-firewall machine can try to ping the virtual machine's host server, and a known IP address that publicly responds to ICMP and which the host server can reach. Chances may be high that this communication won't initially work: Make sure the ping commands will continue until they are interrupted. Then customize the firewall machine until that communication is working.

This may require setting up traffic forwarding/routing, and NAT. If the host server responds, but the Internet doesn't, then perhaps the host server needs to send traffic to the firewall virtual machine.

More routing: accessing the Internet

Set up firewall rules to block outgoing SMTP traffic except for authorized traffic. This is simply a precaution that is widely recommended to deter becoming blacklisted due to accidentally sent outgoing SMTP traffic. Ideally this is done before enabling Internet access, so that there will not be a single unauthorized piece of mail going to a remote SMTP server. (There may still be unauthorized traffic going to remote submission TCP ports (TCP port 587), although that is far less likely to lead to blacklisting. The reason is simply because a/the common configuration is that the remote mailservers will typically not accept mail sent to the submission TCP port, unless the computer trying to send mail is first able to successfully authenticate.

Does the main firewall know how to contact the Internet? If not, set the default gateway. Details on how to do that are at/near the section about adding a networking route.

Handling shutdowns and startups
Handling shutdowns

Make sure that the computer running the virtual machine software has a way to shut down the second machine. (This may been a task that was sensible to delay until network communication was working better. Now that this network communication works, make sure that this ability exists.)

Machine shutdown script

The following is a script file that may work well to shut down a virtual machine. (This is designed for OpenBSD, and may not work so well for a platform without the pgrep command.) Note: If the script itself contains the name of the machine shutting down, that may affect the pgrep command. (A solution for that hasn't yet been determined.)

Some customizations (namely to the ssh command line) may be needed for this script to fit well to a specific environment.

At the time of this writing, this script has not been thoroughly tested...
#!/bin/sh

if [ "$2" = "0" ]; then
echo Syntax: $0 remotesys systemName
echo remotesys is an IP address or DNS name. The specified systemName is a short name used on the virtual machine's command line.
return
fi

which pgrep
if [ "$?" = "0" ]; then
PGREPCMD=pgrep -f
else
echo Required pgrep command not found.
return
fi

echo Stopping $2 @ $1

$PGREPCMD $2.pid >> /dev/null
if [ "$?" = "0" ]; then
echo File listing for $2.pid
ls -l /var/run/vmpids/$2.pid
echo About to ping twice
ping -c 2 $1
echo In case is IPv6, about to ping6 twice
ping6 -c 2 $1
echo Starting SSH command
ssh -p 22 -l _stopsys -i ~adminusr/keys/shutdnvm/shutdnvmc $1
echo SSH results were $?
echo File listing for $2.pid
ls -l /var/run/vmpids/$2.pid
echo pgrep results:
$PGREPCMD $2.pid >> /dev/null
echo Done reporting on shutdown
else
echo $2 was not detected as running. Skipping communications with $1
fi

Also, on the machine that runs the virtual machine software, modify the shutdown sequence to do something with this virtual machine, such as stopping the virtual machine or pausing/suspending it or moving it onto another computer that runs virtual machine software.

If network traffic is being used in the implementation of how a virtual machines are being shut down, make sure that the last virtual machine to be shut down is that virtual machine that routes traffic to other virtual machines. Otherwise, the traffic to shut down the other virtual machines may not get suitably routed.

After initiating a process like having the virtual machine shut down, allow some time to check if the process completes. For example, in OpenBSD one can check if a Qemu machine is running by using:

pgrep -f qemu >> /dev/null
if [ "$?" = "0" ]; then
sleep 20
fi

Users of an environment without pgrep can use something like the following:

ps -auxww | grep -i qemu
if [ "$?" = "0" ]; then
sleep 10
fi

That may need to be customized as different operating systems may have different syntax for the ps command. That example may also show quite a bit of output if there is still a virtual machine running. (Of course, such output may not be visible. It may, depending on the operating system, generally go to the first terminal, which may be the local console.) Even if such output is seen, the information, about what is running, is rather relevant. So, that output may not be bad to have.

Here is a more streamlined option that won't clutter in quite as uncontrolled of a fashion.

echo Now sleeping 10 seconds, and showing details, if a virtual machine is still running...
ps -auxww | (grep -i qemu && sleep 10) >> /dev/null

Note that 10 seconds might not be enough time. One option, which is not preferred, may be to just adjust that number of seconds, or to repeat the commands to wait as needed. A slicker option may be to put those commands in a conditional loop, but make sure that there is a maximum timeout. This way, if a virtual machine gets hung up, the virtual machine may be shut down improperly and the computer running the virtual machines may be automatically shut down instead of being left in a sort of half-shutdown, hung/stuck state. Having the virtual machine be shut down improperly is generally undesirable, but that fate may be inevitable, and there may be no better option available if the computer running virtual machines is starting to become less responsive (e.g. having remote access become unavailable) as a normal part of its shutdown process.

Handling startups

While making changes on the computer that runs the virtual machine software, go ahead and modify the startup sequence of this computer. Make it so that every time this computer starts up, it starts up the newly created virtual machine.

(Qemu-specific note: It may be a good idea to wait three seconds after the first machine is created, before starting the second machine. That is simply designed to let the first virtual machine create the socket...listen before the other machine starts and tries to use socket...connect. The reason to keep the time short (three seconds) is so that the first machine can then, later in the process, be able to use an automatic addressing client that will hopefully not time out before the automatic addressing server gets started on the second machine. Other machines may want to wait even longer, to give the automatic addressing server a head start. Note that solving race conditions by just waiting a pre-defined number of seconds is generally an inexact method, often considered a poor design from a programmer's point of view. However, the clearly visible alternative would involve interacting with the source code for the virtual machine software. That is probably overkill for this problem.)

Verify rigorousness of startup

Got it all done? Good. Now, break things! Shut down the virtual machines, and the machine running the virtual machine software. Then start the machine that runs the virtual machine software. If, and then) start the virtual machines.

[#rtvmnoip]:

Note: There is one thing to mention as being likely to be broken at this point. On the virtual machine that is used to route traffic, it may have a NIC that connects to other virtual machines. That NIC may not have an IP address being assigned yet. These instructions did not recommend resolving this problem by using a temporary static address. (The reason that recommendation was not made is simply that the very next step on this network is likely to get automatic addressing working, so if the problem was fixed one way before then it would need to be changed momentarily.) For the short term, such an address may need to be assigned temporarily, again. So, once the virtual machines are restarted, fix that one problem. (Details are in the section on manually handling network addresses: setting the current address.)

Then, see if the network traffic works as well as it did before. If not, well, the problem will probably be easier to troubleshoot now rather than later (when this problem may have caused some other problem which, if detected first, may have caused more troubleshooting to be needed just to help narrow down the problem a bit), so the test was likely worthwhile.

If there's any other problems, then, well, good luck!

Time for a break?

Networking has just been worked on. Depending on the pace taken, maybe the first virtual machine has been worked on. If there isn't heavy time pressure, then evaluate muscles and determine if a stretch break is needed.

Recap

Starting this section after taking a break? If so, welcome back! And, if so, here's a brief re-cap: There may be multiple virtual machines working at this point. There is one known communication issue that may need to be fixed if the computers had been shut off: the routing machine may be more useful by following instructions for manually handling network addresses: setting the current address. If ping is working all around, then it is time to move on.

Get automatic addressing working
Warnings
Simple implementations use local subnets

Note: automatic addressing protocols may often involve implementations that are specific to a local subnet. For example, some/most/all popular DHCP servers may be designed to listen for clients on the local subnet. Other clients, meaning clients which are not part of the local subnet, can be supported by using different software called DHCP Relay. It is recommended not to start by trying some sort of elaborate multi-subnet setup: If at all possible, particularly for a first deployment, simplify things by making sure the server works with traffic released on the local subnet.

Avoid re-inventing DHCP Relay

(This is simply overview text for those who might want to consider trying to ignore the above warning. By providing a description of some of these pitfalls, perhaps this will save someone from some useless effort of egotistically trying to conjure up an elaborate setup in the hopes of self-discovering/inventing a “better” way of doing things...)

DHCP Relay is generally not recommended to be used until after a simpler DHCP is already working on a local subnet. (DHCP Relay is being mentioned early for a reason. That reason is NOT to encourage people to start out by trying to use it. The reason is so that people don't immediately move on, from creating a working implementation on a local subnet, to trying to support other machines using some sort of custom-designed elaborate setup to support additional subnets using routed traffic. (One key reason to avoid such inventive approaches is simply that this other way may widely be considered to be both more conventional and also easier.)

Someone new to networking may start putting together various ideas and think that perhaps a working solution can be reached by experimenting with some sort of custom solution like NATting broadcast traffic. The key reason not to spend a lot of effort trying to create some sort of custom solution is simply because there is a standardized method available, and that standardized method may widely be considered to be both more conventional, and also easier to implement. However, to prevent some pain for the curious, here is what happens if trying to go down that route.

First, who knows whether or not broadcast traffic may be NATted. Perhaps network implementations don't even work well with that concept. However, assuming that problem doesn't exist (or gets resolved), there may still be troubles.

Even if somehow the DHCP client's traffic somehow reached the DHCP server, things may still not work. In that case, changes could have been made to the network's design only to find out that the designer of these these changes hadn't yet considered how the return traffic will get to the system that needs a response. Until that is also rectified, any apparent achievements amount to accomplishing nothing significant, because the ultimately desired results still aren't happening. (Perhaps the problem isn't instantly obvious, and so it is only after a detailed investigation that one concludes that the DHCP client's traffic does end up reaching the server, and that the remaining issue is that the server isn't successfully getting a response sent back to the DHCP client.)

An initial reaction may be to feel emboldened by the level of success managed so far. After all, if traffic is routed one way, then traffic may be routed the other way, right? Ermmm... no. Perhaps particularly with IPv4, the DHCP/IPv4 client software might even be running on a system that hasn't really started listening to a real, unique, assigned IPv4 address yet. IPv6 may not have the exact same issue, and using some multicast address might seem to be an enticing idea, but such mutlicast addresses will still be link-local. This means that there is still some need for a device on that link/segment/subnet to be sending traffic to the DHCP client, so the situation isn't fully resolved in IPv6 either. Using either generation of Internet protocol, the identical results are that simply using standard Internet-style routing may not really be an available possibility/solution. So far, the problem remains an unresolved conundrum, with just half of the required traffic getting routing to the needed destination.

Since routing may not, by itself, be a solution, something on the same subnet as the DHCP system will need to help to get the remote DHCP server's traffic back to the desired DHCP client. Following are some details about a couple of approaches to try to deal with this discovered problem.

One approach to try to handle this problem would be to get the devices to somehow share Layer 2 traffic. In effect, this is what bridging will do. If bridging happens, then the devices are no longer considered to be on different network segments, so they are essentially on the same link/subnet. (The complications don't end up getting resolved through tricky mastery of network routing, but rather the complications end up simply getting avoided using a rather standard routing technique. As a result, things work, but not because of any new crafty invention that conquers the challenges that only really exist when multiple subnets are being used.)

Another approach may require that the DHCP server gets traffic routed to the desired network, perhaps using a tunnel with an end-point that is on the same network. This may end up being fairly similar to bridging (just like a VPN is), since the result is that the network traffic ends up being treated like it is on the same subnet. Perhaps this sort of approach may also be able to somehow work. However, there may be little (e.g. no) advantage to using this sort of a custom solution, and may even be some disadvantage(s). This sort of approach would require that a device, specifically a device on the same network as the DHCP client, to be acting like a tunnel endpoint. Instead of taking the approach of having special software on that device run so the device could be a tunnel endpoint, such software could simply be running a copy of DHCP Relay. The advantage is that the DHCP server has built-in support for DHCP Relay (if the DHCP server is fully implementing the standard DHCP specifications), so the machine running the DHCP server won't also be needing to have a tunnel endpoint.

Multiple NICs

Another warning: Using DHCP client software for multiple NICs on a single system may often a bad idea. This warning is not trying to suggest that people should not run DHCP client software for multiple NICs on a single computer. On the contrary, this guide will show how to do that. However, note that there can be problems. To avoid breaking something that currently works, until this guide covers the details about how to safely do so, avoid running DHCP client software for multiple NICs on a single computer.

For simplicity, this guide is going to start by focusing on getting automatic addressing to work on a local subnet. If this guide's basic design has been followed, there may be two (virtual) machines that can communicate with each other when an address is manually set for both NICs. One of those machines may now start acting as an automatic addressing server, and the other will start working as a client.

With those warnings aside, the general guidelines are in the section about automatic network addressing. For IPv4, those wanting to simply use the established method of getting this to work, see DHCPd for details about running a DHCP server.

Setting up the automatic addressing server

Here are some more specific details about running the DHCP server.

IPv4

For simplicity, this guide currently does not have exenstive information about implementing DHCP security. (There is just a brief section about security at the top of the section on DHCP.)

OpenBSD

OpenBSD comes with ISC DHCP, although there may also be an ISC DHCP package. See the section about DHCPd in order to create a useful /etc/dhcpd.conf file. To start out simple, only bother having one subnet block.

Running the command will require knowing the name of NIC that will be receiving the DHCP traffic. (If needed, get any necessary details from the section about obtaining the names of available NICs). The syntax may be as simple as mentioning the NIC. As an example, consider running:

date
dhcpd if0

Some older versions of OpenBSD may have supported a -q switch (before the interface nam) which, though not mentioned by OpenBSD's manual page for dhcpd, did seem to be supported. (At very least, the option does not seem to cause the program to quit with an error about an unknown option.)

If the program had any errors, it likely was not nice enough to complain. Instead, it likely just logged a problem and quit. First, see if the program is still running. (See the section about finding out what is running.) If it is not still running, check the end of the /var/log/messages file. After any problems have been fixed, check the current time (as reported by the clock of the computer that will be running the dhcpd server). That way old messages in /var/log/messages can be easily ignored if that file needs to be consulted after trying again to run. (The danger there is simply accidentally wasting time troubleshooting something that was fixed, and not getting to the remaining problem.)

Once the program is running in the short term, continue on to read the instructions about using a DHCP client. (Later instructions will cover running this long term.)

Automatically getting configuration for a multi-NIC system

(Assuming the rest of this guide has been followed, these steps could be done on the virtual machine which will serve as the firewall.)

There may be some important impacts on routing when using DHCP on a system with multiple NICs. At the time of this writing, this is rather specific to DHCP for IPv4 since routing information may not be getting passed using DHCPv6 as much as it had been with IPv4. (Perhaps that is due to change... a section on Routing informaiton in DHCPv6 may cover that in more detail. However, until/unless that becomes more widespread, this information may largely be more applicable to DHCP/IPv4 deployments.)

Inducing chaos

Now, some further detail is warranted about a warning given earlier. Running multiple DHCP clients can cause frequent updates of the system-wide name resolution settings, and when the different DHCP clients use different name resolution settings, then chaos can occur: The DHCP clients automatically renew things. The system running two DHCP clients may be working just fine, and then the second DHCP client will update and break the system-wide name resolution settings. After the problem is noticed, a person may try to log into the system to identify the problem. However, the first DHCP client on the computer might then renew settings, fixing things. The poor human may be perplexed to learn that name resolution is fully working, and then later become aggravated when the second DHCP client breaks things again.

If this example has been followed, there are two (virtaul) computers. One of them is running a DHCP client on one NIC, and that same machine has another NIC which is used to connect to the other computer which is running the DHCP server.

In a test network, this sort of chaotic situation may be temporarily acceptable. And there may be some real curiosity to see whether the recently set up DHCP server is working as expected. In such a case, go ahead and satisfy curiosity by causing this chaotic situation, and then the chaotic situation can be fixed later. By performing that process, earlier verification will be achived to show that the DHCP server is working as desired, and then attention can focus on the client.

To test the DHCP server, go to the machine that will be running the DHCP client, and then following the steps for manually setting an IP address: specifically see the section about manually (re-)requesting an IPv4 address.

If the desired IP address is set as desired, that is a great sign that the server is working as desired. Perhaps name resolution is broken. Whether or not problems may currently exist, it is time to stop the potential chaos. Stop the second DHCP client. (In Unix, see the section about adjusting what is running and stop any relevant dhclient process that mentions the NIC. (There may be two such processes. Stopping the process that is running as the username _dhcp may stop the other process that is running with the username of root.) In Microsoft Windows, there may be a single instance of the DHCP client software. The way to handle that in Microsoft Windows would be to adjust the NIC's settings to stop using automatic addressing.)

In Unix, if name resolution is still broken, then restart the automatic addressing client for the other NIC.

Avoiding chaos

When the dhclient command starts, it contacts the server and asks for certain information.

Unix: Using a custom config

In Unix, this is commonly implemented as follows: the dhclient command starts, and it checks a configuration file. The configuration file has a list of options. The dhclient will then ask the server for details related to those options. In the default central configuration file, one of those options is to request the name server. If that option is in the configuration file being used, then the client will try to get the nameserver details, and may even insist upon getting such details. If those details are provided when requested, then the client will adjust the system-wide nameserver settings.

The default system-wide configuration file may work fine for one NIC, so there's no reason that file has to change. However, other NICs should use a different file. As a first step, create that file.

cpytobak /etc/dhclient.conf
sudo cp /etc/dhclient.conf /etc/dhclnons.cfg
sudo $VISUAL /etc/dhclnons.cfg

Change:

request subnet-mask, broadcast-address, routers, domain-name,
domain-name-servers, host-name;

into this...

request subnet-mask, broadcast-address, routers, domain-name, host-name;
# domain-name-servers, host-name;

After saving that change, try running a DHCP client with this alternate configuration file:

CURNIC=if2
sudo dhclient -c /etc/dhclnons.cfg ${CURNIC}
ifconfig ${CURNIC}

Like the results? Great: then make them long-term.

OpenBSD: Running a DHCP client with a custom configuration file

This could potentially be done using various files that automatically start up, but the standard location for network configuration in OpenBSD is the network configuration script. If the CURNIC variable is still set, the following may work well without modification. Otherwise, if that environment variable isn't set, just modify the filename as needed.

echo !dhclient -c /etc/dhclnons.cfg \$if | sudo tee -a /etc/hostname.${CURNIC}

To test that in OpenBSD, the following may be done to remove the currently assigned IPv4 address, and hopefully to get it successfully back. (This test may be very similar to what happens during a reboot, but may go much faster than a full system reboot.)

sudo ifconfig ${CURNIC} delete 192.0.2.1
sudo $SHELL -c ". /etc/netstart ${CURNIC}"

Done working with this NIC? Then unset the temporary custom variable.

CURNIC=
Making sure the automatic addressing server automatically runs
This information may currently be rather specific to IPv4.
Unix

(If changes have just been made to the system that will be using the DHCP client, make sure the following changes are being done to the system that is running the DHCP server.)

Knowing what command line was used, to (temporarily) run the DHCP server, will be helpful: The quickest way to recall the exact command line that was used to run dhcpd may be to check command line history or finding out what is running. Determine what that syntax was: it may simply have been a reference to the names of the NICs.

Back up the /etc/rc.conf.local file (if it exists).

cpytobak /etc/rc.conf.local
OpenBSD

Specify all of the command line arguments in a file that is used when the operating system is automatically started. (Make sure not to use the following example(s) verbatim, but customize the name of the NIC.)

In OpenBSD, a command to do that would be:

echo dhcpd_flags=\"if0\" | sudo tee -a /etc/rc.conf.local

If all goes well, and if this is a test environment where downtime is perfectly acceptable, perhaps a reboot is in order. This can help make sure that automatic addressing works, that virtual machines get started, and that routing works as expected. (Previously, if this guide was followed, routing may not have automatically worked because a NIC needed to have an address manually assigned. That should now be fixed.) If any of this is still somehow broken, then it may be easier to make changes now rather than after the network gets more complex by adding more services.

Supporting the multi-hard drive layout

Current status: This is an idea that will hopefully work well. These details may not have been fully tested at the time of this writing. Until this notice is removed, expect that proceeding with this section may not go smoothly (and a willingness to solve any problems may be needed).

Consider what changes have been made to each machine. Copy any changed files to the second hard drive. Here is a way to respond to various changes that may have been made.

Capturing data that was created by specific actions
Creating a new user

Get a copy of the user database. (Backing up users may have details about backing up the entire user database. For Unix, this may involve getting /etc/passwd and /etc/group files, although those may be slimmed-down files that are not, by themselves, complete.)

Copy the user's home directory. (One useful file to be copying over is the .ssh/authorized_keys file, but another part of this is simply copying the fact that a home directory is created with the right permissions. If the startup sequence was modified, e.g. to show the prompt, that will also be taken care of.)

Installed programs
Did any programs get downloaded and installed? Are the installation files still available (e.g. in $PKGCACHE)? Perhaps instead of copying every single file that the program added, it would make just as much sense to just copy the installation files, and then have the program be automatically installed.
Customized configuration files

Here is a case where it is useful to know precisely what files got modified. (Backing up too many files, including unchanged files, will likely not break this process terribly, although that may cost more time when processes are automatically performed, and a manual process like reviewing all customized/changed files could take more time.)

In Unix, many times changes are made to files in the /etc/ directory. If such files were not made in the subdirectories, one can run
ls -laFt /etc/

If this is a child image, and if one or more file integrity checking programs were used when the base image was created/solidified, then using such a program (by comparing with previous data) could provide a list of files that have been changed.

If this guide is being followed, many (ideally all) changes may have backed up copies of the old versions of the files. Reviewing those backups could show what files were later changed. One can ignore any files that were backed up when creating the base image, which would be backups from before when the child image was created. One way to determine when that occurred may be to look at when the child image was created (perhaps by looking at the creation date of the subdirectory that is storing the hard drive image). Another method may be reviewing the last modification date of a script file, or some other sort of configuration file, that was modified when the system was created.

Recent file integrity data

The following backs up some files from multiple file integrity checking programs. It could be that not all of these exist, so feel free to verify that files exist before trying to copy them.

mkdir -p /srv/sysset/sysdata/$( hostname -s)/.
cp -pR /var/db/mtreedat/. /srv/sysset/sysdata/$( hostname -s)/.
cp -pR /var/db/aide/. /srv/sysset/sysdata/$( hostname -s)/.
cp -pR /var/db/aide/aide.*
cp -pR /var/db/aide/aide_*
/srv/sysset/sysdata/$( hostname -s)/.
Backing up backup files

e.g., if backups: section on copying files is done to a centralized location, copy the contents of the location where files got backed up to. e.g.:

mkdir -p /srv/sysset/sysdata/$( hostname -s)/.
cp -pR /origbak/. /srv/sysset/sysdata/$( hostname -s)/.

After all of the static files (meaning the files that are not expected to be changing) are at /srv/sysset/sysdata/$( hostname -s)/ then perhaps compress the data. (As discussed by compressed hard drive myths, extracting compressed data can go faster than reading a larger amount of data directly from a slower drive. This can also reduce the long term wear and tear on the data storage device. Details about how to compress this data are in the section about compressing a hard drive image.)

Varying files
e.g. logs. For Unix systems, perhaps all of the /var/ directory. e.g. If the computer will be serving as a DHCP server, then the DHCP leases
Enabling useful communications

Once automatic addressing is allowing one virtual machine to automatically get addresses, make sure that the automatic addressing server also has an address on its subnet. e.g.:

echo inet6 alias 2001:db8::8 64 | sudo tee -a /etc/hostname.if0

The desired goal is that the physical machine has both an IPv6 address and an IPv4 network address, and both of these addresses are reachable by all virtual machines, and this machine may also reach all virtual machines. Once that is working, the next goal is to make sure that those settings are all saved in files, so things continue to work even after the system is rebooted.

However, for the time being, do not connect to any untrusted networks that may be attacking the system. (It will be fine to do so by the time this guide gets to the point of recommending that routing to the Internet is working.)

For simplicity, for now, do not try to use automatic addressing for any machine which is on a different link: if the machine running the virtual machine software needs to have traffic routed through one virtual machine to get to the machine using a DHCP(/IPv4) or DHCPv6 server, then either explore DHCP Relay or, perhaps better (to just get some more critical progress working sooner), just use static addressing for the time being.

As this is largely a re-hash of actions similar to previously performed techniques, this guide is currently refraining from providing a whole series of step by step processes. (Such a guide would involve lots of IP addresses that need to be customized anyway.) So, now that this guide has taught how to fish, take the given wings and fly! Read the next paragraph and then troubleshoot as needed.

Some key pieces of documentation that may help may be routing virtual machines to the physical machine's tunnel, manual network address assignment, adding a networking route, system start process (especially Unix users may want to check out system start process: problems running programs during startup, and sudo reboot.

Security check

For any of the machines being used (the machine running the virtual machine software, and any virtual machines), are the passwords being used some sufficiently protective passwords?

One could even take the step of relying on key files, and removing support for the passwords. However, keep in mind that, compared to a key file, a password may be much easier to use at a local console session. It may be good to have an administrator account that keeps using a password. One option may be to have an administrator account that uses a password, but which not in the group that allows SSH communications. Determine how much security is desired, and be sure the implement it early. As in, now, before moving on. Making things more secure later, after connecting to a network which is potentially hostile or, worse, known to be hostile, may be too late. (It would be like installing new locks on a door after a thief has already entered the building and installed a new, hidden, unknown door. The analogy may seem silly, but if it happened then such a thief would have sufficient reason to laugh at all the work that is being done futilely. And, unlike a real physical thief in a real physical building, skilled computer attacks might be able to install a new hidden “door” relatively easily.)

Communicating with the Internet

What addresses have been getting handed out? If they are private IP addresses, then they need to be NATted. (For details about how to implement NAT with a firewall, see the sections about firewall implementations and search the implementation-specific details for information about using NAT.)

Time for a new machine?

There are various design approaches that can be taken. One is to cram as many services as possible onto as few of machines as possible. That may be particularly attractive when working with a substantially limited budget, and when there is a significant cost for each machine.

Another approach is to use plenty of machines, and to have as few services as possible for each machine.

This guide recommends the latter approach, if for no other reason than this one simple reason: educational. Having software on separate machines and controlling network traffic may help to demonstrate details about the network communications are happening between two pieces of software.

For those who have different goals, feel free to perform things differently.

So, go ahead and create a new machine. Details are similar to what was done before: making a new vm.

However, one difference now is that when a MAC address is selected, one may modify the automatic IPv4 addressing server to recognize the MAC address.

Overview/planning: Choosing the next step

When designing a new multi-segmented network, once automatic addressing is set up so that at least one client is able to automatically get the desired addresses, one possible option is to spread that functionality.

Another option that makes quite a bit of sense, though, is to put that on hold briefly. Instead, set up name resolution. The reason is because recursive name resolution can be fairly quick to set up, and then the configuration of the automatic addressing servers may be adjusted to support the new nameservers. Then, when automatic addressing is spread throghout the network, name resolution is also working.

Another reason to switch gears is that, especially for a first-timer, quite a bit of time may have been spent with setting up virtual machines and getting automatic addressing working on even a small scale. It may be encouraging to be able to cut another notch in the belt: to have another useful service successfully set up.

So, this guide focuses on getting a recursive nameserver up and going next. Then it focuses more on spreading the functionality.

Those who are wanting to serve multiple computers as much as possible, and as quickly as possible, may wish to focus more on spreading the automatic addressing functionality first. That way, more clients can benefit from the functionality that is available. They may work around the non-working internal DNS (perhaps using external DNS, or by just not needing DNS.) Later, when name resolution is working, machines that are set up for relying on automatic addressing for DNS information (even though it wasn't working yet) will have their settings fixed after they renew their lease (which is commonly done when the device is started up, after being rebooted or powered off).

Name services

The primary method of name services, which most software will be compatible with, is DNS. DNS

The first step in this guide is to set up name server and to use a fairly simple configuration file. Specifically, this example will start by setting up a recursing nameserver. Beyond that, becoming a primary nameserver will basically involve simply updating configuration files and starting to use the updated configuration files.

Creating a recursive nameserver
Predicting success

Can a computer other than the nameserver successfully communicate with (e.g. with ping6 in KAME, or ping -6 in Microsoft Windows) the computer that will be running the nameserver?

Go ahead and identify a usable DNS server.

Can DNS traffic reach the usable DNS server? For instance, try running:

nslookup wikipedia.org 2001:db8::3

(This information may become out of date.) That is looked forward to! Note: Looking up Google may not be a great test. Google's FAQ on IPv6: Why Google does not just enable IPv6 on Google websites or, more generally, overview of Google's support for IPv6.

nslookup google.com 8.8.8.8
Set up a recursive name server

If needed, install a nameserver. (It may be time-saving to check if a name server is already installed. For instance, OpenBSD FAQ 1: section about what is included with OpenBSD notes BIND is included, and running “ which named ” may show a result. If the desired nameserver is not pre-installed, install software as appropriate.

See: DNS

Spreading the functionality internally
Overview/goals/desires

At this point, most of the critical infrastructure is in place. A computer is running virtual machine software and there may be multiple virtual machines running. Although there might not be more than a couple of critical services (namely automatic addressing and name resolution may seem most prominent, although firewalling and some traffic routing may also have been done), these are the most critical components. These may also be among the more time consuming services, as they may need to be correctly customized to support specific details about a setup (like which software is being used, and what network (IP) addresses are being used).

This means most of the work on infrastructure has been made functional.

More services (like web serving) can be added, but first let's try to make sure that the already existing services are available for as many machines as they should be. By doing so, it will be likely that future services will be able to easily/automatically support those additional machines.

For example, everything done so far can be done by installing an operating system, getting a remote access server set up, and then haivng everything else be done remotely. A person may be sitting down at one machine using a remote access client, and then using that computer to interact with the server which all the other work is being done with. If that is happening, then what is the status of the machine running the remote access client software? Specifically, is that machine using the automatic addressing services and, perhaps more significantly, the name resolution services of the newly configured network?

The key idea here is to allow more machines/subnets/links/“broadcast domains” to be able to start utilizing the new services.

Automatic addressing relay

The machine that runs the virtual machine software may be a useful method for connectiving to various media, since presumably this computer may be a physical machine.

IPv4: DHCP Relay

The following were some notes made about DHCPv6 Relay:

ping6 fd00:0:0:4::2
ping6 fd00:0:0:4::1
sudo route add fd00
sudo route add -inet6 default fd00:0:0:5::1




root     20811  0.0  0.3   600   728 ??  Is    Sun10PM    0:00.07 /usr/local/sbin/dhcp6c -c /etc/dhc6cem0.cfg -p /var/run/dhc6em0.pid em0
_dhcp     5524  0.0  0.4   524   980 ??  Is    Sun10PM    0:00.22 /usr/sbin/dhcpd em0
root     14027  0.0  0.3   544   876 ??  Ss     5:54AM    0:00.01 /usr/local/sbin/dhcp6s -D -c /etc/dhcp6s.conf -P /var/run/dhcp6s.pid em0
root      5015  0.0  0.0   692     4 p0  R+     5:54AM    0:00.00 grep -i dhc (ksh)
[22]root@ttyp0:autoaddr:~/# sudo kill 20811


win7: dhcp6s on autoaddr
win4: tcpdump
win5: dhcprelay
win3: rtadvd


Feb/23/2012 06:54:53: server6_send: transmit advertise to fd00:0:0:4::1 failed




DHCPv6:
	Allow adv
		sudo sysctl net.inet6.ip6.accept_rtadv=1

sudo dhcp6relay -D -f -s fd00:0:0:5::8 -p /var/run/dhcp6r_msk1 msk1		


rtadvd -c /etc/rtadvd-em1.cnf em1
rtadvd -c /etc/rtadvd-msk1.cnf msk1
sudo /usr/local/sbin/dhcp6s -c /etc/dhc6cem0.cfg -p /var/run/dhc6em0.pid em0
sudo /usr/local/sbin/dhcp6s -D -f -c /tmp/dhcp6s.conf -p /var/run/dhc6msk1.pid msk1


default:\
        :chlim#64:raflags#0:rltime#1800:rtime#0:retrans#0:\
        :pinfoflags="la":vltime#2592000:pltime#604800:mtu#0:
em1:\
        :addr="fd00:0:0:5::":prefixlen#64:raflags#64:
#        :addr="fd00::5:0":prefixlen#64:raflags#192:
#        :addr="fd01:2345:6789:abcd::":prefixlen#64:raflags#192:
#        :addr="fd01:2345:6789:abcd::":prefixlen#64:raflags#64:
#        :addr="fd01:2345:6789:abcd::":prefixlen#64:raflags#128:
#        :addr="fd01:2345:6789:abcd::":prefixlen#64:tc=default:raflags#80:

DHCPv6 protocol in the Linux operating system shows a relay6_loop function that includes the lines:

#ifdef notyet
icmp6_react();
#endif

This suggests that some code may not yet be fleshed out. Now, granted, this site seems to suggest Linux code, and this may be old code. However, code-sharing is a known practice, so perhaps WIDE-DHCPv6 might just not be fully fleshed out yet.

It seemed sensible, at the time of making this version of this guide, to abandon DHCPv6 Relay for the time being. Within months, OpenBSD may support ISC-DHCP and perhaps that will just work better.

Expanding name resolution
Supporting internal names

Plan to head over to the section on DNS servers for details about implementing this.

BIND
The example documetnation shows how to use Split-DNS. The example configuration causes the data in /var/named/etc/prvzones.cnf to be loaded.
Public names
Of course, this won't work until the firewalls are set to allow traffic to reach the DNS server. After the DNS server's configuration seems correct, review firewall details.
Supporting Reverse DNS
...
Reviewing DNS configuration
File sharing
Transfering a file
file transferring
File sharing protocols

Determine what is meant to be shared. For example, if the content to be shared is just one directory/folder, and other parts of the filesystem are not supposed to be shared, then SMB/CIFS may be a more suitable choice than many (common) implementations of NFS which only share entire filesystems. (Such implementations might accept a command line parameter to have a folder specified, but in practice the server ends up making the entire filesystem volume's data available to a client. Clients (or, perhaps more specifically, at least some clients including crafy ones meant to try to access data which wasn't intended to be shared) could access data from other directories/folders.)

Determine what software is available on each platform. For instance, some platforms might only commonly support SMB when using certain methods of authentication encryption. There may be some compatibility issues, which may be relevant when selecting data.

filesystems from network file sharing servers untested notes, not currently cleaned up very well...

UNTESTED:

for the interfaces line, specify the local IP address and subnet mask of the interface that will be receiving the traffic.

for the hosts allow line, specify what clients are allowed.

These specifications may be partial IPv4 addresses (e.g. 192.168. is equivilent to 192.168.0.0/16) or IPv6 addresses (e.g. 2001:db8::5/64 works).

The workgroup line is unimportant in ipv4 and has really no impact in ipv6: e.g. line of output: 2001:db8:1234:5678::1 is an IPv6 address -- no workgroup available


if [ -x /usr/local/libexec/smbd ]; then
        echo -n ' smbd'
        /usr/local/libexec/smbd # Start the Samba server component
fi
if [ -x /usr/local/libexec/nmbd ]; then
        echo -n ' nmbd'
        /usr/local/libexec/nmbd # Start the Samba naming service
fi

add local user

user may have nologin as shell

was put in daemon class and wheel; probably unneeded if user can access the files.

sudo smbpasswd -a -U _smbuser

Optional: Install Samba on the client. Verify the share with smbclient -N -L serverNameOrIP Here's the trick, though: If the server is referenced by IP, that's fine. However, if the server is referenced by name, then the server's name is looked up in /etc/hosts it is not looked up by DNS.

Web server

Note: The system that accepts web requests does not need to be the same machines where the data physically resides. Also, the system that accepts web requests does not need to be the same machine as the one with a lot of web-specific customizations.

One option may be to have the data be physically residing on a remote server, and to use file sharing. This could result in a slower delivery of the web site content. Hopefully disk and/or website caching may be able to help alleviate slowness caused by that. An advantage is that this allows the difficulty of making content redundant (simply as a step of implementing resiliancy through redundancy) a problem that can be focused more narrowly on the machine(s) providing the files. The machine(s) having web hosting configurations may be seperately redundant. (The idea here is that multiple simple tasks may be easier than combining lots of potential problems into one huge gigantic monolithic task.

Web file transfers

E-Mail Server
Incoming E-Mail

First, check if an E-Mail server is installed. For many Unix operating systems, it may be. It also may be running by default, perhaps in a relative secure fashion of only listening to local ports. In that case, one may be able to find out about the running software by using:

netstat -na | grep -i 25

For details about setting up such a server, see the section about servers getting E-Mail. e.g., for Unix, see Receiving E-Mail with the sendmail program.

Sendmail
Outgoing E-Mail

Before trying to send outgoing E-Mail from the server, check firewall rules. Some firewall rules may block outgoing traffic to TCP port 25 except from authorized machines. If a telnet client is installed, one can use that to see if the TCP connections are creatable.

Sendmail
Sendmail does support TCP Wrappers. So: echo sendmail : ALL >> /etc/hosts.allow
Allowing forwarding
...
Checking for changes

Several changes may have been made to the network. When was the last time that a file integrity check was performed? Is there certainty that any changes made to a machine's configuration, if these changes were made after the latest file integrity check, have been fully documented? If not, consider re-running file integrity check(s) on all the machines (namely/particularly all of the servers) and prepare to spend some time documenting the changes. (This documentation may be a bit difficult to do now, after other tasks have been performed. However, it may be even more challenging to do months later. So, doing it earlier would have been potentially nicer, and doing it now may be better than putting it off until later.)

Eventually, an improvement to the network may be to automate the file integrity checks. If that would be an additional change to the network, perhaps that simply re-inforces the reason for doing the checks manually now (so that the reports are smaller/simpler now). (Then, when automating the file integrity checks is added, the initial reports may reflect just those changes, and so be smaller.)

Things may be different this time around. For instance, unlike the last time the file integrity checks were run, there may be less likelihood of new home directories being created for new users.

Updating credentials

If this network was created from scratch, perhaps it took a while to make. During that time, were passwords stored in documentation? Was that documentation ever copied/stored into a location that other people may have had access to? If those people are part of an authorized team, that may be okay. If not, and if this was a test environment, that may be okay.

Consider whether the newly created network would be more secure if all passwords got changed (perhaps to stronger passwords), and if those passwords were then kept secure (more carefully).

Implementing processes like one-time credentials may also be an avenue to explore (even if that technology was put off earlier).

Checking E-Mail

OpenBSD runs automated security reports. Go ahead and check the E-Mail of every such machine (or at least every such server) that was set up.

Clearly, a more useful long-term solution will be to try to centralize such reports. In the meantime, the information was generated so that it may be processed (whether by humans or by machines), so go ahead and check out the information rather than having it just entirely go to waste.

Disaster recovery
How are the backups? See backup.
Error reporting

If the system has a problem, is it reported?

File integrity checking is a great start. Now how about reporting the results? Send the results via an E-Mail.

Follow up
Break time!

Setting that up is, at the time of this writing, likely a large series of steps. Take a break. Depending on the scenario (e.g. for work? For personal hobby setup? In a military engagement where the enemy may start attacking at any time?), an appropriate amount of break could last anywhere from a couple of seconds to a week.

There are further things that can be done to enhance the network. Some of them, such as adding redundancy, could be better to do sooner rather than later (in scenarios where downtime results in higher costs, or lower desired performance, etc.). Others, such as reviewing logs, make more sense to do after a length of time (such as, perhaps, a week). For instance, if logs are checked when a system is deployed, reviewing the logs again after a week will be more likely to make sense/be useful, rather than performing that task just eight minutes later.

Data area lock down

Depending on how important security is (which may be less crucial for a machine that isn't yet deployed in a production environment, where functionality testing can be monitored in an environment that is still controlled/trusted/safe), some security settings may have been relaxed. For example, there may have been a reason to keep some areas of the disk writable when programs are being installed, and perhaps users might be more prone to be added. If some time has passed, can that be locked down further? For instance, if programs/users are not likely to be added, can /usr/ (and any other mount point underneath /usr/) be locked down more by being mounted as read-only?

Are routine tasks (e.g. backups, file integrity checking updates, anti-virus updates) completing automatically and successfully? Are there error messages in the logs? Did the creators of the operating system cause the computer to send any automated E-Mail to the superuser account? (If so, what does that E-Mail say?) Are there updates available for the operating system? (Some of these topics are discussed on this site, and may be hyperlinked to in the future.)

Are there any ports that are being listened to, but without explanation? Do all of the changes mentioned in the file integrity check make sense? If uncertain what some changes are, perhaps they are legitimate, but some self-education is needed?

Additional/future enhancements

(This information may end up getting moved to another guide.)

[#redundan]: Redundancy

(This section will likely be getting split off to a separate guide.)

[#redndata]: Duplicating data

If data can be cloned (particularly if the amount of data is small, and the budget is high enough to accomodate such a thing), perhaps the data can be physically duplicated on hardware that provides fast (real-time) access to the data. This can be a lot easier to implement for some types of data, such as small DNS configurations and DHCP leases and firewall states, while large databases may be (significantly) more of a challenge.

In cases where data is not truely duplicated in real-time, it may still be useful to perform some sort of methods where the data could easily be shared. For example, accessing some files via network-sharable paths (like a UNC), and using system names that are named after functionality provided by the system (rather than being named after some specific hardware). This can help changes to the infrastructure to be transparent, which means that other computers may not need to notice any significant change when changes are done to the infrastructure. This can mean that other software on the network will not require reconfiguration/updating when changes are made to the core infrastructure, and that is nice for multiple reasons. Network design can be simpler, and implementation may be easier (due to being less work), and work may be done more quickly (reducing downtime and other costs that may be less necessary.)

Note that important/cruicial data ought to be duplicated in some sort of fashion. If that isn't true, that would mean that backups are not occurring.

File redundancy
(Further information is expected to be added at a later time.)
Database redundancy
(Further information is expected to be added at a later time.)
Duplicating functionality

Computers need data. One of the first steps is going to be to determine what data is accessed. Any data which is on a local hard drive will need to be effectively duplicated, and synchronized with other copies, in order for the computer's functionality to be truely duplicated. That could be as easy as just copying the configuration files to multiple machines, once. (However, that will likely only work if the files don't get changed. Counting on them never changing is probably not going to be the nicest design in the long term. If the data does need to be updated, then the data will need to be updated on multiple machines.)

Beyond that, the requirements will vary depending on what functionality needs to be duplicated. Restated: Different software may have different steps to follow to perform full functionality.

Name resolution

For static and forwarded DNS, this may be as simple as just running another DNS server with the same configuration data. This is one key reason why DNS has often been one of the easiest services to deploy redundantly: deployment may be relatively easy. (Dynamic DNS might be a bit more of a challenge, so make sure that data is synchronized to any other DNS servers. However, if the Dynamic DNS isn't critical, so that losing the dynamically updated DNS data wouldn't be quite as critical, then such synchronization may be less ideal.

Automatic addressing

Automatic addressing has sometimes been more of a challenge, as multiple automatic addressing servers need to make sure that they do not hand out the same addresses (and, by doing so, create a network “address conflict”).

This could be done by simply having different pools. For instance, one server could hand out a pool of addresses ranging from IPv4 192.0.2.100 to 192.0.2.149, and another server could hand out addresses from IPv4 192.0.2.150 to 192.0.2.199. That may work fine for a network of less than fifty machines, where an IPv4 /24 subnet may be common. (The pools can, and probably should, be smaller than the subnet size.)

There is another approach that is generally better/nicer. That approach is having the multiple automatic addressing servers communicate with each other so that they are sufficiently synchronized, and then being able to serve data from the same sort of pool.

Firewalling

For stateful firewalling, synchronizing the states can be beneficial. e.g. Wikipedia article for MAC addresses: “See also” section has mentioned some alternatives: “Hot Standby Router Protocol or standard alternative VRRP Virtual Router Redundancy Protocol, which allows multiple routers to share one IP address and MAC address to provide router redundancy. The OpenBSD project has an open source alternative, the Common Address Redundancy Protocol (CARP). On Linux, iptables has a CLUSTERIP target.”

(Further information is expected to be added at a later time.)

File sharing
(Further information is expected to be added at a later time.) (This might be covered more in the section about data duplication.)
Improving defenses
Intrusion detection

Perhaps make an additional file integrity check, which contains not just a report of all files, but just reports on key/critical files. Then if there is an issue, create a different type of alert that generates more attention.

Centralized reporting/logging/etc.
...
Offline alerts

Have systems “report” to a centralized system. The reporting method can vary. For systems that don't need an immediate report, checking for a daily event such as a successful backup or an update to file integrity check data can alert to an unresponsive system. For systems that need more frequent reports, have those systems automatically send some detectable traffic, such as an HTTPS connection that uniquely identifies the machine. Perhaps even better is if the connection is useful, such as if the machine reports problems from its local logs while doing a report.

Then, if the check-in fails for a certain length of time, the centralized system may send an alert. As examples, this sort of system may cause an alert to be generated due to events such as a power failure, a different type of hardware failure, a local network outage, or other problems with the Internet. All of these may require intervention, and so may be of interest to a network administrator. By having many systems report to a small number, problems with any of the many systems will be reported (as long as the small number of centralized reporting systems is working okay). Dozens, or even hundreds, of systems may then have their issues being auto-detected.

Auto-response

It is great to know about problems: even better if the slaves of today (a.k.a. machinery, computers, software, robots, etc.) can take care of the problems quickly and without even requiring intervention.

One example may be “Intrusion Prevention”. This may be implemented in the form of features of an “Intrusion Prevention System” (“IPS”) which may be similar in nature to the software that implements “Intrusion Detection System” (“IDS”) features.

The idea there is so that attackers get stopped quickly, possibly without even being provided much/any access to the server. (A tricky aspect to this may be to make sure the IPS doesn't deny service to authorized users: especially administrators who may need to fix things when problems occur.)

Other automated responses may come in the form of responding to centralized logging. One way a computer may respond is to generate a ticket in a ticket-tracking system. However, it is generally desired that the ticket-tracking system checks whether an existing ticket is currently “open”. If so, until that ticket gets moved to a “completed”/“closed” status, it may be desirable to suppress re-alerts.

[#oldmltvm]:

The older guide

This guide is meant to guide a user through the process of creating multiple virtual machines.

[#mltvmreq]: Requirements

(Much of this text is similar/identical to what is found in the tutorial of making a virtual machine. This list may be reduced a bit to remove some redundancy, if possible. However, the items in that other tutorial are part of the process of following this tutorial, so perhaps many of the requirements will remain.)

This list may seem like a fairly long list of requirements; it is fairly long at least in part due to the amount of material this guide covers. However, this guide recommends starting by making sure that requirements are fulfilled so that showstoppers don't cause a screeching halt shortly down the road. Hopefully many, or even all, of these requirements are fulfilled, so reading through the entire list will be a waste of time. However, if any of these are not fulfilled, substantial time might be saved by having these addressed early on.

  • A host machine capable of running several virtual machines. It is recommended that this machine have multiple NICs so that it may effectively firewall traffic.
  • [#docedos]: Using an operating system that this documentation sufficiently covers:
    • For now, this host machine should be using some sort of Unix-like platform.
    • The most recommended course of action is for the host machine to use an operating system that this document is meant to be used with. Otherwise, there may be some differences between the documentation and what is expected. For now, the list of operating systems that are likely covered fairly well by this operating system consists of the following operating systems:
      • OpenBSD (currently the best hyperlink anchor for this operating system is: BSD page: OpenBSD section. That is likely to be elaborated on fairly soon.)
  • Knowledge about how to install an operating system, including handling disk partitions. Eventually there should be a hyperlink here that points to such a guide, or such details should be incorperated into this guide.
  • (This may not yet be true, but is expected to be true in the future, after this guide is updated.) Some of the steps in this guide will benefit from having IPv6 support. If using IPv4-only Internet access, this guide does provide some detail for setting up the IPv6 support. However, this is listed as a requirement because some steps in the guide may not be available if IPv6 Internet is not available (directly or through the tunnel brokers). (The choices of worldwide tunnel brokers has not regularly been changing dramatically, so being banned from such a tunnel broker may prevent some of this guide from being completed.)
  • Virtual machine software:

    An early step which is briefly mentioned is to have virtual machine software installed. Much of the rest of this guide may operate under the assumption that virtual machine software is already installed. (It may not be configured to start with, but having such software being initially installed may be helpful.

    Instructions for installing this in some operating systems may be elaborated upon by this tutorial, but for now here's some starting information:

    [#vmhstdoc]: List of virtualization software options that are supported by this guide

    There are multiple virtual machine software options available. However, this guide may not (fully/sufficiently) cover each one of them.

    For now, if the intent is to closely follow this guide, the virtual machine software being used should be one of the following:

    • Qemu (as noted by the overview of the “Qemu” software)
      Note on Qemu within Win9x
      Using Qemu under an operating system using the Win9x code base may not be sufficient for much of this guide, simply because the Qemu release for Win9x may not sufficiently support networking.

    (Additional virtual machine software packages may be covered in more detail in the future. Currently, at the time of this writing, the plan is to definitely add information about more virtual machine software solutions, including KVM. (Some other solutions might be added at a later time, after IPv6 information is further elaborated upon, and when/after additional operating systems are supported by this guide.)

    Other options
    Many of these techniques might be able to be implemented using some other available options of virtual machine software (mentioned on the virtualization page). However, do not expect that this guide will fully walk a user through all of the steps if that software isn't in the list of virtualization software options which are supported by this guide.
    Installation instructions
    (Instructions for installing this in some operating systems may be elaborated upon by this tutorial, but for now, some starting information (particularly for modern operating systems other than Microsoft Windows) might be available at general software installation instructions.)
  • NIC requirements:
    • Clearly, at least one network interface will be needed for this machine to interact with other networks (such as the Internet).
    • The machine can effectively serve as a firewall. For that to happen, this physical machine should have at least two NIC connectors. One of these will go out to the Internet, and the other will be protected machines. Working around this may be possible with some additional work (which may or may not be well documented on this site) and may result in less ideal network protection.
    • Optionally: A third network connector, such as an antenna for WiFi, and a forth connector to go to some other trusted service (such as shared files). It is the plan to have details for this, though they may not be up yet.
  • Virtual networking: the relevant information has been moved to: virtual machine tutorial: requirements related to supporting a NIC that is virtual.
  • An understanding of networking, including when traffic goes to a default gateway and understanding the concepts of CIDR notation (a.k.a. IPv4 prefix length) and subnet masks. (Wildcard masks are also good to know about, but probably won't be getting used here.) Understanding the notation of IPv6 addresses and address ranges is expected for one to easiliy fully understand what is said. A tutorial on this may be made available later, but isn't yet (such a tutorial, with heavy RFC references, is probably already made but hasn't been integrated with these instructions yet), so for now just find one on a search engine. (Search for learning about subnet masks and how AND logic is used.)

    Understanding of the purposes of common protocols (e.g. for automatic addressing and name resolution) may help: detailed knowledge of how they are implemented in IPv6 is not a requirement and this tutorial will help people familiar with how that works in IPv4 to gain familiarity and experience with how these things work in IPv6. Having an understanding of the different layers of networking. However, lacking these details might still be sufficient enough to make one's way through the tutorial.

  • These instructions may assume an active IPv4-based Internet connection. (It is hoped that this will change so the assumed connection may be support IPv4-based and/or IPv6 communications.)
  • Some way to view the contents of a virtual machine. Some quick notes about how that may be accomplished:
    • A working X Windows session will work
      • If using Qemu and not using VNC, simply leave off the portion of the command line that says “ -vnc none ”. Note that using VNC (via an SSH tunnel) is generally preferred over trying to use X forwarding.
    • Instructions on how to perform the viewing of the virtual machine are provided (and referenced in later instructions). However, there could be some difficulties if trying to use a text-mode interface to control a virtual machine running a graphical user interface. Even such an unlikely thing is being mentioned because, if such a scenario happened, it would be an issue that could be a showstopper (or at least a showslower).
  • We'll also need some IP addresses. Because these may come from ranges of addresses reserved for private use, there should be plenty of addresses available at no financial cost. (However, if working with another network that also uses such IP addresses, make sure the IP addresses chosen don't conflict.)
  • [#werqptnm]:

    (Should this go to: mkavmreq/mkavmreq.htm ?))

    We'll need some port numbers. Lots of port numbers...

    Several port numbers will need to be used. Fortunately, there is generally a surplus of port numbers that may be used. However, it will be even nicer is to have a nice range of port numbers which are fully unused (instead of trying to use a bunch of non-sequential port numbers).

    Usability of dynamic/private/ephemeral ports

    IANA's list of TCP and UDP port numbers says “The Dynamic and/or Private Ports are those from 49152 through 65535”.

    However, although private port numbers may be used from this range, a lot of software will use these as the range of dynamic ports. Dynamic ports were also referred to as ephemeral ports by old BSD versions, and that name is also used. In addition to being used by outgoing connections, these port numbers may be used by some ot her software. For instance, OpenBSD's man page for FTP: section on “Port Allocation” shows that OpenBSD's ftp program “will listen to a random high TCP port.” This is an example of how TCP ports on that operating system may be used from within the range of the sysctl called net.inet.ip.porthifirst (defaulting to 49152) through the sysctl net.inet.ip.prothilast (defaulting to 65535). (There are additional sysctl values which are just as existant: sysctl's net.inet.ip.portfirst (which defaults to 1024) through the sysctl net.inet.ip.portlast (which defaults to 49151). However, rather than using those lower numbers, the software does default to using the traditional ephemeral port range.)

    (This section is simply informative, and the information may not be essential for the following steps to work. That doesn't mean that an academic program wouldn't be allowed to use it in a test.)

    Usability of port numbers from the “Registered Ports”

    There may be some desire to use the ports which are less likely to be used, within the range of what IANA's list of port numbers calls “Registered Ports”, since they are unlikely to be used, even though such behavior violates IANA's note on that page which says “UNASSIGNED PORT NUMBER SHOULD NOT BE USED.” After all, as just shown in the “Usability of dynamic/private/ephemeral ports” section, those higher port numbers may not always be entirely available for private use.

    The reality is that using a port number for private use is likely to work, and is unlikely to cause problems, unless another application attempts to use that port (either by trying to actually use the port or, more sanely, checking that the port is available before trying to use it). An application can feasibly require a specific port number, and can do so without violating some of the most important networking standards, if the program is trying to support a protocol that specifies a port number that is assigned by IANA. However, since requiring a specific port can cause some problems (or at least some inconvenience caused by inflexibilty), even though such a requirement may be considered to be valid behavior, most programs are configurable to be able to avoid such problems by using a customizable port number.

    The restriction about not using unassigned port numbers may not even be meant to be universally followed: Software developers may be expected to test software in controlled environments (where problems would not affect any systems except for those of the developer). IANA may not assign a port until after successful implementations exist. (An example of implementations being required for a port assignment is shown by the “final note” in OpenBSD 3.5 commmentary about petitioning IANA for a port number for CARP. Not having multiple actual implementations may not have been the only issue: Wikipedia's documentation on CARP not having an official port number describes a lack of some documentation being an issue. Still, this may be an example of how there was a general expectation that an unassigned port was being used.) Clearly ports are likely to be getting used during software development (including testing) before the possible existance of successful implementations.

    Some port numbers that may be available

    Looking for some large ranges of numbers that do not have ports assigned by IANA?

    The following paragraph was true at the time of being initially written (but may not still be true). Some ports that are still unassigned by IANA (as determined by IANA's list of TCP and UDP port assignments) may be around port numbers 4610 or 5370 or 5510 or 6720. For VNC, there are several ports from 5900 to 5998 available, although not all the numbers in that range are unassigned by IANA, so the best practice would be to check before using them. Ports 47809 and higher provide 191 unassigned ports. 8475 represents a range of 25 unassigned followed closely by a range of over 50 unassigned ports. (Above that range, the author of this text stopped scanning the list quite so closely for trying to find blocks of about 30 or much larger.) Port 32500 is in a range over a hundred unassigned ports. So is port 34400. 43442-44320 is a range of 879 addresses. 44323-44552 is 230 addresses, including 44400 which may be nice to count up from.) There is a block of over 500 unassigned ports at 47002-47556 and another block of over 500 unassigned ports from 48620-59150. In between those ranges (from 47557 to 48632), there are just over a dozen assigned ports, and multiple blocks of hundreds of unassigned ports.

    All of those just-mentioned numeric ranges, with many sequential unused port numbers, are subject to change: IANA may simply add a number to the list of used port numbers. So, before trusting any numbers of that list, do check IANA's list of TCP and UDP port assignments for the latest information.

That completes this “requirements” section.

About using this tutorial
Knowledge requirements

(Much of this text is similar/identical to what is found in the tutorial of making a virtual machine.)

This tutorial may assume a fairly high amount of basic computer knowledge, including basic network terminology and setups.

Hopefully most/all additional knowledge required (other than what was just referred to) is either listed in the “requirements section” or, better yet, is documented on this website and is hyperlinked to by this tutorial. For some of the topics in the “Requirements” section, hopefully those details will be added to the site (and hyperlinked to from this guide) at a later time.

Some basic concepts, such as running a program from a command line (with one or more command line parameters) and editing text files, may be assumed. (This guide may provide details about how to accomplish those tasks in a specific operating system, but having the general concepts understood may be helpful.) If some of that knowledge is lacking, this guide may be a bit on the advanced side. It may still be useful, and perhaps more useful than some other resources, but do anticipate there may be some troubles and/or other challenging aspects.

Getting through the material efficiently

(Much of this text is similar/identical to what is found in the tutorial of making a virtual machine.

Please accept some hints about how to read this tutorial. It was designed to be followed in a specific pattern in order to minimize the amount of time needed to get through all the needed material. Perhaps a fault of this guide is spending too much time early on focusing on the expected overall time savings. There is a clear reason that was done: to simply drill home a specific point: There really is expected to be time savings compared to learning this material searching on your own. Some significant effort really was put into place to try to accomplish that goal, even though it is acknowledged that such time savings may not be quite as apparent during the first quarter or so of the guide. So, please, have faith, and start by investing a bit of time reading these recommendations.

This tutorial is part of the Cyber Pillar web site, and makes frequent references to material found elsewhere on the site. If the goal is to complete this tutorial in a streamlined fashion, then the recommended way to proceed through this tutorial is to complete just the section described by the hyperlink (including any sub-sections), and then to read no further on that page. This may sound simple but be more difficult than it sounds, and taking the time to understand this guideline may save quite a bit of time overall. After the hyperlinked section, the website may have lots of additional information that is related to what is just read. However, reading about lots of similar and related material is not going to result in completing this guide most quickly. So, after reading the referenced text assigned by this tutorial, head back to the tutorial (in order to, most quickly, complete more of the guide).

It is sincerely believed that sticking to the guide will likely result in a more streamlined process, providing learning of many fundamental aspects much more quickly than the alternative method of branching off to explore lots of related material.

To understand what “section” is meant to be read, check out the description in the hyperlink that refers to that section. There are a few things to help identify where the “sections” start and end. One clue is the indentation levels. In the many cases where an HTML “definion list” is being used, another clue to the end of a section is the color coding (which uncoincidentally happens to correspond to the indentation levels). The third clue is that hyperlink anchors (which start with a “left square bracket” and then an anchor/number/pound/hash sign, and end with a “right square bracket”) is often found at the start of a new section. If you see one of those visible hyperlink anchors, consider whether the title of the upcoming text is related to the current goal. If not, back out and make more progress on the current goal.

Some people may prefer to read in a less controlled fashion. There is nothing particularly wrong with gaining extra knowledge, but simply realize that doing so may take longer and may end up getting into more advanced concepts without the required background knowledge. This tutorial is meant to be a controlled guide to help cover information in a quick and logical order. How much this advice is followed may be dependent on the individual reader/follower.

Making multiple virtual machines

This section (which, at the time of this writing, may be the largest part of this guide) may benefit from some substantial overhaul. A lot of this was originally part of the guide to creating a virtual machine, but was later moved to this guide.

If you have gone through the tutorial of creating a single virtual machine, and were extremely comfortable performing those steps, and have experience working on a network, consider reading the following sections/references...

...and then skipping past all of the rest of this “Making mulitple virtual machines” section. Get to the interesting section(s) about starting up services. (To do this, users of web browsers with JavaScript enabled may just click on the “collapse” icon next to the title of this section.) (It is intended that the recommendation, of skipping this large section, will be removed after some additional work has been done to this section of text.)

Creating/using the first virtual machine : Initial actions

This section may be a bit similar to: Creating/using a virtual machine : Initial actions

[#mvmi6on4: Get IPv6 support when (only) IPv4 is already available

See: Get IPv6 support when (only) IPv4 is already available

Documentation

Creating documentation for a single machine (whether the machine is physical or virtual) can be useful, especially so that the single machine may be controlled remotely.

Documentation may be increasingly important for a network of computers. For example, noting what services are on each machine can help to determine some information about the machine, like what kind of work the machine is responsible for and what impacts would happen if the machine were taken down. Also, knowing what services are on each machine can help to determine which machines provide each service, so there is some indication of which machine(s) to log into when there is a need to change how a service works.

While a single document may be suitable for a single machine, the documentation of a network will likely include the documentation needed for each machine. Therefore, network documentation may consist of a significantly larger document, or a directory/folder that contains multiple document files.

This section might (still) be largely/mostly duplicated with the tutorial on making a virtual machine.

It will likely be worthwhile to at least determine where the documentation will be stored. That way, incompleteness in documentation can be quickly created while going through the remaining steps. So, do two things. First, determine where documentation will be stored. (A structured method, which may be more suitably designed for an educational group, may have specific details on where such documentation should be. For those following this guide as part of pursuing a more individualized study, this may be as easy as just making a decision about where a file will be stored. ) Get whatever details are needed, make any decisions that may be needed, and know where the documentation is going to be stored.

If the location (e.g. a directory/folder) that will be storing documentation is not yet accessible, then, create a directory (or more than one directory) as needed. Naturally, also make changes to the file's “ownership” and/or file permissions if such changes are needed. (This site might not yet contain a really good guide for handling such ownerships/permissions. What information is available may be at file attributes/ownerships/permissions.) Then, plan to put documentation/text files in that location.

Security note: At least little thought about how information will be accessed is prudent. Such information would assist, if not completely permit and enable, security breaches. This sort of information may be some of the most sensitive information related to the network. Therefore, consider how that data will be safeguarded. Keeping the only copy of critical data on a removable storage unit may not be a good idea, since removable storage is often more physically vulnerable to impacting damage, and may also be more easily lost (or stolen). If such considerations lead to the conclusion that the location of the documentation should be altered, then go ahead and make what improvements are needed (early in the process).

Documenting some basic things immediately, ahead of time before other steps are taken, may help other steps be implemented more quickly. If the desire is to have the smoothest possible experience, the best recommendation is to pre-plan some things. For those who are expecting to take time to try to learn things using the most preferred method, and reduce risk of hitting an unforeseen roadblock, go ahead and perform the following steps:

  1. See Virtualization: Initial items to decide/document when creating a virtual machine. This is meant to be documentation for a single virtual machine, although a few of those items (namely locations of things like keys) might routinely be the same for multiple virtual machines.
  2. Treat that guide like a checklist of items that need to be documented, and document everything that is in that list. (Again, note that knowing these things now will likely speed things up later. Specifically, it is anticipated that later there will be more things to be keeping track of.)

It is recognized that some people may be seeking the fastest immediate results (and not caring if the full process, including documenting things later, might take longer overall). People in that situation may understandably decide to skip past the step of starting to document some of the useful information right away, choosing instead to fill out the document file as they make decisions while performing other steps. However, even in such a haphazard approach, creating at least sparse documentation will likely save time later and overall. (At minimum, it is recommended to document some key things such as document file locations, machine names, and authentication credentials.)

Creating a virtual machine

(The information which was in this section has now been moved to: making a virtual machine.

Tweaking the virtual machine

(The information which was in this section has now been moved to: tweaking the new virtual machine.

Using the virtual machine

(The information which was in this section has now been moved to: using the new virtual machine.

Implementing networking

(The information which was in this section has now been moved to: Implementing networking in a VM.

Wrapping up info on creating the first virtual machine

(The information which was in this section has now been moved to: Wrapping up info.

After the first virtual machine is made

(The information which was in this section has now been moved to: After a VM is made.

Other items to decide for a network of virtual machines
Deciding on, and documenting, ranges of network addresses
[#vmsnetsz]: Determining the subnet size
Subnet size for IPv6 SLAAC

If even one machine within the range is going to use dynamic addresses using SLAAC (for IPv6) (see IPv6 automatic addressing), it is probably best to reserve at least a /64, and expect that SLAAC will be using a /64.

(If one is not very familiar with using IPv6 (and may be following a guide right now to gain some experience with IPv6), it is likely wise to plan to use SLAAC for automatic IPv6address assignment. If for no other reason, a good reason to use SLAAC may be to get some experience using SLAAC in order to increase familiarity with it.)

Since a /64 offers so many addresses, if that subnet size is going to be used for compatibility with SLAAC, then many of the other considerations may be unnecessary steps that are simply designed to make sure that a sufficiently large subnet size is being used.

Reserving individual IP addresses for services

The main first step to determining a subnet size is to determine how many IP addresses to use.

Determine how many different services and NICs are going to be offered in the initial design of the network. Add them up, although subtract one service for each NIC that has a service on that NIC.

The following sections may help to describe this approach.

An IP address for each NIC

In general, every single active NIC, including virtual hardware (and including any NIC on any computer that provides the host environment that is running one or more virtual machines), should have an IP address which is unique to that NIC. So, the first step may be to count how many machines will be used, and know that at least that many IP addresses will be desired. However, also take into consideration if machines have multiple NICs. If so, and if there is any substantial likelihood that the NIC may be on the subnet (neither now, nor in the future), then count the NIC as something that needs an address.

(By considering this rule, if there are multiple server-class machines providing the DNS service, then there will be multiple IP addresses allocated for the multiple machines that provide the service.)

Having a unique IP for each service
Recommended design

In addition to making sure that each NIC has a unique IP address, there may be some benefit to making sure that each machine has at least one IP address for each service that the machine provides for the benefit of other devices on the network. (This would include services meant to be used by automated software running on other devices on the network, and/or for the benefit of people who may be interacting with those devices). What this basically means is that any machine that provides multiple fundamentally different services to the network will have multiple IP addresses. (Examples of such fundamentally different services may include being a server for name resolution (such as a DNS server), and a server for automatic network address assignments.

Not counting certain services

As a point of clarification, this recommended design doesn't necessarily mean that every single service on a computer needs its own IP address. Machines may run some services that are typically not primarily meant to provide benefits for the other devices on the network. Such services may include, as examples, providing access to a remote command line prompt, or providing a remote desktop, and sharing certain files so that Administrators may access those files (like a \C$ share that shares a C:\ to administrators), and providing access for Administrators to perform certain tasks like restarting software (perhaps by using Microsoft Windows MMC consoles over RDP, or maybe something like PsKill?).

(Err, this text may be good to review to make sure that it is communicating clearly.) The main reason why there is less likely to be a benefit in having a unique IP address for such a service is that there isn't likely to be a compelling reason to ever want such a service (like remote access for an Administrator) to be moved to a different machine than other services that might be currently running on the same machine. For instance, if a computer is a DNS server and a DHCP (for IPv4 server, there may be some benefit to moving the DNS service to a new machine, and to not move the DHCP server to the same new machine. However, if somebody wants to use remote access to control the machine that is using the DNS service, the convenient thing will be for that administrator to use remote access to connect to the same IP address as the DNS service. Hopefully the remote access will not commonly be using up significant bandwidth, so there's no need to try to have remote access be moved to a different machine dedicated to remote access.

  • If these services are meant for administrators, then end users may not even have the permissions needed to access those services. For that reason, it seems unlikely that any software meant to be used by the end user will have ever been (intentionally or accidentally) relying on having those services be available at a specific IP address. Therefore, even if these services break, the lack of those services might not be likely to (accidentally) cause downtime in functionality used by an end user.
  • Also, such services might not, by themselves, be a justifiable cause to keep a machine running (if the time has come that people are considering ending the use of a machine).

OLD NOTES

Redundancy Note: The following text may be quite similar to nearby text (like the prior text). It may be worthwhile to confirm the redundancy, merging in any extra details, and removing the extraneous text.

It may make sense to compare to number of services, and compare that to the number of machines. Note that the number of machines counted should include any pre-anticipated physical and virtual machines. So if one physical machine will be running three virtual machines, then count the collection as a total of four machines.

When counting services, simply count the services that are intended to be used by multiple other machines on a regular basis, such as hosting files that many end users may access.

Use whichever number is larger, either the number of machines or the number of services, as a starting value of how many IP addresses will be getting used. Then, for every service that has multiple machines, ignore one machine and then add the number of additional machines. For every machine that is actively providing multiple services for use by the various people and services on the network, allocate an IP address for each service after the first service.

For example, if there are going to be eleven services (automatic addressing, name resolution, traffic routing/firewalling, authentication, file sharing, E-Mail hosting, web site hosting, and perhaps a few others), and if each service may need up to three IP addresses to fully implement all of the redundancy that will be used, perhaps 33 IP addresses are going to be needed.

Overview: Reasons why it may be desirable to have a unique IP for each service

(This section is basically about describing reasons why some recommendations are made, rather than providing instructions or new recommendations.)

Using more IP addresses than the number of NICs (or the number of machines)
Addressing potential reasons not to use this approach

Some people may prefer to limit each machine to having only one IP address for the computer, even if the machine provides multiple services. Perhaps reasons include tradition, nostalgia, or ignorance; none of which are particularly compelling reasons to keep doing things an old way.

Perhaps another reason to limit each machine to a single IP address is concern of how a computer will handle multiple IP addresses. For example, there may be some concern that a machine using multiple NICs and multiple IP addresses may reply to traffic, but that the reply traffic will use a different IP address (and NIC) than the address that received the network traffic that is being replied to. Although such concerns may be more valid (because such problems could exist), there may be some better approaches. One approach that may be worthwhile and preferable is to address the concerns, by determining and controlling how the computer will be handling the network traffic. (Note that in this early stage of the network design, researching the details of the operating system may not be a task that needs to be done immediately. This section of the network design is simply to be identifying how many IP addresses to be using. By at least performing the task of reserving enough IP addresses, the task of implementing any such elaborate design won't require dealing with an unnecessarily shortage of IP addresses (among other tasks that may be required, such as researching and configuring how network traffic gets handled.) An advantage may be gained flexibility (by being able to keep services easily separatable).

Advantages

Most common/popular modern operating systems will support multiple IP addresses on one machine. By assigning two addresses to the same computer, one address can be used for one service, and an additional network address may be used for another network service. This provides easier flexibility to change how a service is implemented. A service may be moved to another machine (which might use the same software including the same operating system), or the service may be implemented using another piece of software. If a service has a uniquely-assigned IP address, the service and the IP address may be moved without as much likelihood that the change may affect another service that uses the same IP address.

Redundancy Note: The following text may be quite similar to nearby text (like the prior text). It may be worthwhile to confirm the redundancy, merging in any extra details, and removing the extraneous text.

This approach may allow flexibility to move one service to a new machine, without the new machine being compelled to handle the load of all services that were on the old machine. Changes to a different operating system can be performed to support a new implementation of a single service, without needing to worry about whether the migration will smoothly support other, unrelated services that just happened to be on the same old machine.

Why to assign unique IP addresses, and not just unique DNS names, to each service

Note: Assigning unique DNS names is recommended. This section is about why to have unique network addresses in addition to unique DNS names, instead of just relying on unique DNS names.

In theory, it is nicer if each service should be using DNS rather than IP addresses, for reasons including flexibility and the greater ease (for recognition and memorization) that names provide over numbers. Therefore, one may consider just making a change to DNS to point to a different IP address if anything gets changed.

The approach of changing the DNS configuration to reflect the changes is an approach that may work (well), but there are some advantages to being able to just move the IP address as desired.

Many pieces of software do use IP addresses (even if DNS might be a better choice), and ARP caches tend to be renewed more frequently than DNS caches. Additionally, another benefit to using the same IP address (even if a MAC address changes) is that DNS won't even need to be changed at all. Not needing to make changes to the configuration on the DNS server may make the entire process (slightly) simpler. Also, keeping DNS the same may reduce the the likelihood of errors caused when changes to DNS do not occur as expected. That reduction may simplify the scope of likely problems if any troubleshooting becomes needed.

Increasing the requirements

Once the desired number of IP addresses is obtained, use that as a basis of an amount of required IP addresses. However, there may be some other requirements for subnets.

For IPv6 subnets that will be using automatic name resolution, address ranges may need to be a fixed size of /64. That will be sufficiently large for a very large number (18,446,744,073,709,551,616) of addresses.

For other subnets, take the number of addresses that need to be actively usable, and add to that number the amount of needed but unusable IP addresses. For IPv4, the last address of each subnet block is the typically-used “broadcast ID”. The first address of each subnet block is generally not considered to be usable (as discussed in the glossary entry for “network ID”). So, if 31 addresses need to be usable, add two addresses, for a total of 33 addresses.

Then increase the number of addresses as needed so that the number is a power of 2. So if 33 addresses are needed, increase the number up to 64 addresses (which is 2 raised to the sixth power).

Future growth

If there's 27 machines, but there's plans to potentially acquire another half dozen devices (whether those devices are computers, printers, or something else, perhaps things like managed switches), then instead of reserving a block of 32 addresses, plan to reserve a block of at least 64 addresses. Try to anticipate all likely growth that will happen to the subnet. Of course, networks and portions of networks may grow over time, perhaps to unanticipated levels. The cost of not making a subnet large enough is simply that there may be a need to re-design the network, including performing “renumbering” of network addresses. That's a bit of work. In some cases, it may be worthwhile to reserve too many addresses, and even implement NAT (particularly for IPv4) despite the costs of the extra work to deal with NAT, instead of making a subnet too small so that it doesn't accomodate network growth which will actually happen.

[#chsubnts]: Determining which subnets to use

A note especially for those who have subnetted for a long time: see the “Advantages”...“subnet” section.

The task

Once the size of the subnet is calculated (by following the previous steps on determining the size of the subnet), specific addresses may start to be reserved. There may be some various approaches that will successfully work: this guide presents some that have advantages over some older methods.

[#yrdhsbnt]: Advantages (even for the experienced) to reading about how to subnet

The plan here is to use methods that may have advantages over some traditional IPv4 approaches (even if those traditional methods have been used many times by many experienced subnetters).

First, here is a question for those who believe they have plenty of familiarity with subnetting (and may have experience doing this, and may even have obtained formal training in some classic IPv4 methods which minimized usage (to lower waste of addresses)), and who routinely assign the smallest possible subnet sizes to sequentially numbered subnets. The question is:

Are you ready to learn?

Many people who have been classically trained to minimize waste of addresses in IPv4 subnets may have been given reasons why those classic methods were useful. Even for such trained and experienced experts may benefit by reaidng about some of the alternate methods described by some in this section. Decide to learn how the methods work, and what the advantages are, before determining whether their advantages should be ignored.

People who have successfully subnetted for years may benefit from some of the ideas from RFC 3531 (or for IPv4, RFC 1219). Both RFC 3531 and 1219 have reasons provided for why that method is more beneficial than the common method of simple sequential counting. So...

A recommendation, even for old hats at Internet networking, is to proceed to see Network Addressing Plans/Methods section before determining which subnets to use.

IPv4 ranges

Choose some IPv4 addresses. Unless working with restrictions from limits providing by an organization that specifies what IP addresses to use, most likely a good choice would be to use some from the ranges specified by BCP 5. (BCP 5 is a name that should be stable, although RFC 1918 seems to be a name that more people are familiar with and it probably shows up in more documentation, even though the BCP 5 may, in theory, someday point to a newer RFC.)

However, see Network Addressing plans: IPv4 addressing for some more specific advice about which private addresses to choose.

IPv6 ranges

To identify a /64 to use, either identify a /64 that follows the rules of using the fd00::/8 range, or identify a larger block of addresses available to use (probably a /48, but perhaps just a /56), and then subnet that larger block to set aside a specific /64 subnet to use.

See Network Addressing plans: IPv6 addressing.

Overview: Using an IPv4 address scheme as the basis for an IPv6 address scheme

If an address scheme worked well for IPv4, then using an extremely similar scheme for IPv6 does make total sense to do.

If any machines are going to be identified by numeric IPv6 addresses (rather than just using DNS), which is usually the case, then it does make some sense to have the chosen addresses mirror IPv4 addresses being used. There is no compelling reason why the IPv6 and IPv4 must match or have a similar pattern. However, if the organization of the IPv4 works well, then the organization will likely work well for the IPv6 addresses. In that case, there's little compelling reason to try to make the IPv6 addresses different in structure than the IPv4 address scheme.

For example, if the /64 being used is 2001:0db8:4465:6d6f::/64 and the IPv4 address for a web server is at 192.0.2.29, then one approach may be to convert the last octet to hexadecimal (29 in decimal is 0x1D hexadecimal) and append that to the end of the /64, resulting in 2001:0db8:4465:6d6f::1d. Another sensible option may be to take the last octet (0x1D) and multiply it by seventy two quadrilion fifty seven trillion five hundred ninety four billion thirty seven million nine hundred twenty seven thousand nine hundred thirty six (72,057,594,037,927,936), and then add one. (Doesn't that just sound so extremely sensible? Actually, it is sensible because that number is easy to work with. Performing that task is the same thing as multiplying by 0x100000000000000 hexadecimal, and then adding one). The end result would be 2001:0db8:4465:6d6f:1d00::1, so the value from the IPv4 address's last octet, 0x1d, ended up moving left by fourteen digits (matching the number of trailing zeros in the hexadecimal number), creating a subnet that allows for a very large number of machines. (How many addresses are in the subnet? Approximately the same number as what was used in the multiplication: seventy two quadrillion...)

Clearly there's many other variations. For another example, the last octet could be multiplied by 65,536 and one could be added. This could result in an address such as 2001:0db8:4465:6d6f:0:0:1D:1. This would still allow for 16 bits to be used by subnetting, followed by 24 bits which is enough to fit the bits from the other octets of the IPv4 address, and there's still over 65,530 addresses that can be specified by using numbers after the last colon.

Yet another approach may be to reserve three (or, perhaps more sensibly, four) hexademical digits. If the IPv4 address ends with .255, then part of the IPv6 address could be :0255:. The mathematical conversion, to get from 255 (decimal) to 597 (since 597 decimal is 0x255) may not be as simple to perform in one's head, but it would then be easy to match (or identify a mismatch between) the IPv6 address in standard IPv6 notation (which is hexidecimal), and an IPv4 address written out in (common IPv4) decimal notation.

(Note that many of the addresses used just now are addresses from subnets used for the ranges used for documentation. So, be sure to use other address ranges for the subnets being used if implementing these techniques.)

An exception may be the “well-known” (site-local) IPv6 DNS server addresses. If such addresses are going to be used, simply assign one (or more, if desired for some reason) of the three well-known DNS server IPv6 addresses to one or more DNS servers. (However, for simplicity of being able to implement these things, do not assign a single one of the three well-known DNS server IPv6 addresses to more than one DNS server.) If doing this, then the addresses used for IPv4 DNS servers might not be migrated over.

[#iprngreq]: Minimum amount of ranges to select to complete this guide
Overview of the design for this guide

This guide is designed to show how to make a network that uses multiple virtual machines. The specific approach of the design used by much of the rest of the guide will involve creating a virtual machine that serves as a firewall.

(There is no clearly-adopted widespread standard that states that a network must be designed in this fashion. However, this design is simply the approach that is taken by this guide. Some networks may be designed using an entirely different approach, and that may work fine for those networks.)

The virtual machine that serves as a firewall will have multiple connections. Here are some connections that will likely be going onto the firewall.

Firewall's WAN/External Interface

The (firewall) virtual machine will, in this example, have an “external”/“WAN” NIC that is used for traffic that goes to and from the WAN. (In simple cases, the WAN may be the Internet. The idea of this connection is that approved network traffic going to and from the Internet will be using this link.) Presumably this connection goes from the virtual machine's WAN NIC, and to a virtual NIC on the host machine.

Note the IP range(s) (an IPv6 range and/or an IPv4 range) for traffic that involves the connection between the (firewall) virtual machine and a (virtual) NIC on the host machine.

Although this sort of connection may be point-to-point (e.g. using a /30 for IPv4), using more addresses may provide some desired flexibility.

Although the directions may initially look like a point-to-point connection where an IPv4 /30 subnet block (or an IPv6 /126) would be sufficiently large, that is only sufficiently large if there is only one firewall machine, and if that machine is connected to only one external Internet connection using a single externally-accessible IP address. However, for the long term, it will probably be best to plan to have firewalls for redundancy. Therefore, slap on at least a IPv4 /29 subnet block (or an IPv6 /125 subnet block), which will support up to three IP addresses for firewall redundancy, one IP address for an Internet connection, and two other addresses which may be used for either additional firewall redundancy or for additional IP addresses used with one or more Internet connections. Note that this is just the absolute minimum size that may work in many scenarios: There may be reasons to use subnet blocks of even larger sizes, particularly with IPv6 where a /125 subnet block would often be viewed as abnormally small. (It would certainly not be unheard of for an IPv6 /64 to be used for a subnet, even if there is currently only two hosts using the addresses. This would leave most of the 18,446,744,073,709,551,616 addreses of the /64 unused, but there isn't much need to be conservative with these address ranges.)

Traffic sent out from the (firewall) virtual machine will typically be going to that machine's default gateway. After the traffic reaches the gateway, there's a few possibilities of what willhappen with the traffic. Here's an overview of different ways the traffic might be processed by that default gateway.

  • In many cases, most traffic will be forwarded on to an address of a computer located at some remote location on the Internet.
  • Yet another possibility may be traffic that is sent to some other location that the default gateway knows how to forward traffic to.

    For example, the default gateway may be a router that knows that traffic on certain subnets should be sent to another building. Such network traffic might be sent over a network connection that goes straight to that other building. Such network traffic might not be sent to a broadband Internet access modem (which would likely result in the traffic using Internet access).

  • Yet another possibility is that traffic going to the default gateway may be meant to go to the default gateway for local processing, without the intention of having the gateway forward the traffic any further. An example of this scenario would be traffic used when configuring a DSL or cable modem using the HTTP protocol. (The intent in this case is for the default gateway (which would be the modem) to process the HTTP protocol. The intent would not be for the device to be forwarding traffic to the Internet.)

Traffic coming from these addresses shall be monitored with scrutiny, being treated as a potential attack from external resources.

As an example of what network addresses might be used for this connection: e.g. for IPv4: 192.168.117.0/29 (subnet mask 255.255.255.248).

Internal virtual machines providing network services

Note the IP range for connecting the VM's.

  • The VMs might be treated as machines that are tightly controlled by a network administrator. From a firewalling perspecting, this might be thought of as a DMZ. Protective firewalling is a sensible reason to place these devices on the same subnet. However, that isn't the only good and logical reason to segregate the network traffic of these devices. Another reason is the practical reality that these machines are all virtual. This may have some impact, such as not requiring that any NICs on this subnet have a way to communicate directly to any physical network infrastructure hardware.
  • (Note: Be sure to use a protocol that is compatible with the communications that are intended. With Qemu, the -net socket command line parameters may be the easiest method to be supporting communications with many other virtual machines. However, with at least some versions of Qemu (and perhaps still currently true?), only IPv4 addresses are fully supported by the network communication functionality that exists when using the -net socket command line parameters. Therefore, what may work well for an IPv4 might not work quite so smoothly when trying to use the same approach for another network that will be using IPv6.

e.g. If we will be using up to 30 “virtual machine” computers connected to this subnet, then:
e.g. 192.168.245.0/27 (subnet mask 255.255.255.224)

Interface for communicating with end users

Note the IP range(s) (an IPv6 range and/or an IPv4 range) for connecting one or more NICs on virtual machine(s) with a virtual NIC on the host machine. (This is mainly going to be meant for network traffic on downstream devices, such as devices that are part of an internal LAN.)

e.g. for IPv4: 192.168.13.0/30 (subnet mask 255.255.255.252)

Further ranges, such as a range for wireless devices, and a range for authorized remote traffic (which communicates over a VPN), may be decided on, and documented.

  • Documenting some of these additional types of network ranges is soemthing that may sensibly be done around the same time as when other network ranges are being documented. If in a documenting mood (or if there are a few spare moments of time, and it would be nice to save time later), feel free to document these other ranges now. Perhaps it will even be nice to elaborate on that documentation, by not just documenting basic ranges of subnets, but even going further by documenting individual addresses that will be used by specific devices. (Details for individual addresses may someday be hyperlinked to from here.) That documentation can be used later, so documenting out plans now may save time later (when planning would otherwise need to occur).
  • Alternatively, some of those further design details may also be decided upon later, being fairly quick and easy to implement after basic access is working well. (In many cases, “basic access” may mean using a wired LAN. Simply, it refers to the easiest method of getting devices to successfully communicate with each other.) It be nicer to delay making (and documenting) those decisions, deciding instead to focus on the minimum decisions needed for the current stage of the design, and then moving on after that minimum amount of decisions has been made. The advantage to delaying such extra planning is simply so that further action may be taken sooner.
Filling out details of the chosen address ranges

Fill out some details within those IP ranges.

Abbreviating notes

The network documentation page's section on using partial addresses is likely to be particularly useful for this section of documentation. (Other nearby text may have other hints about saving time with documentation.)

Range for external address

The instructions for selecting the IP address ranges, given just before these instructions, mentioned that an IP range is used for connecting the firewall VM's NIC that is used for WAN/external traffic.

Go ahead and note which address will be used for which machine. For example, the tunnel interface on the host machine may use the first IP address in the range. The virtual machine will use the last address in the range, which would be the sixth usable address if using a small IPv4 /29 range of usable addresses.

(One may wonder why to use the last address in the range, instead of using the second address in the range. The answer is that by having the unused addresses be in the middle of the range, either range of used addresses may expand. If one range of used addresses grows more than anticipated, that may be okay as long as there are still unused addresses in the middle of the range. The logic of handling network addresses like this is quite similar to how the values of bits are handled by the approach of RFC 1219 and 3531 (which is about the bits of a network address). By choosing the last address in the range instead of, for example, the second address, the number of Internet connections could increase up to two or even three without needing to worry about changing the first address used for the (virtual) machine that is serving as a firewall.)

Range for the other virtual machines

The instructions for selecting the IP address ranges noted to make mention of a range of IP addresses (and possibly IPv4 addresses) for virtual machines.

There is clearly a lot of flexibility that a network designer could have when determining how to assign IP addresses. One may wish to consider using (or even use) some of the guidelines/discussions/examples offered by one or more of the following resources:

  • An example layout is available at: System Address Usage. This example of assigning addresses for various tasks is meant to cover IPv4 addresses within a /25 (plus it documents the following network ID). For IPv6, the same sort of scheme may be followed, although naturally the syntax of written addresses may need to be changed. That example could be used verbatim, or as a basis which is modified as needed, or simply as a demonstration on how complicated things could get when leaving some addresses reserved for planned redundancy. Feel free to design a much simpler approach, or one that is even more elaborate.
  • example (skeletal: lacking names) layout of IPs for the LAN may also have some information about how the IP addresses may be allocated.
Range for communication with internal machines

The instructions for selecting the IP address ranges mentioned that an IP range is used for connecting the firewall VM's NIC for internal/LAN traffic.

Note that this range doesn't have to be on the LAN's subnet. Traffic to and from the LAN's subnet may need to go through this interface, so appropriate routing rules may be needed (both by the firewall virtual machine, and by a device which is on the LAN subnet). If that approach is taken, a /30 subnet block is sufficiently large. Go ahead and note which address will be used for which machine. For example, the tunnel interface on the firewall (virtual machine) may use the first IP address, and the TAP/tun interface on the guest machine may use the second IP address.

Documenting the network

Once network addresses are assigned, there should be a good grasp of how many machines will be needed. It is good to have documentation for each machine that provides services to one or more networks. For an example of what type of documentation to have on each system, see example (skeletal: lacking names) for system information that may be documented.

[#adyexmpl]: A note about addresses reserved, and used, for documentation

This website may use the following for examples:

IPv6 addresses from within 2001:0db8::/32 (as documented by RFC 3849). Another address range which may be used for examples is the IPv4 addresses from 192.0.2/24 address range. (That IPv4 range is called “TEST-NET” in RFC 3330, and is called “TEST-NET-1” by RFC 5737 and RFC 5735, and is called “TEST” by RFC 1166). Additional address ranges that may be used for examples are 198.51.100/24 (called“TEST-NET-2” in RFC 5737 and RFC 5735), and 203.0.113/24 (called“TEST-NET-3” in RFC 5737 and RFC 5735).

All of those address ranges are all meant to be used for documentation purposes (and or tests and/or research), and are not meant to be actually used when trying to get things to work. Such traffic should be blocked from working on the Internet. BCP32, also known as RFC 2606 (and not likely to have any need to be updated), reserves example.com, example.net, and example.org (in section 3) as well as four TLDs (in section 2) which are .example, .test, .invalid, and .localhost. Of course, the items with the word “example” may be used in documentation.

Even if these things appear in examples, do not use those in scenarios where the goal is for network traffic to operate normally. The documented examples might even be blocked by some hardware, so the documented examples may actually not work (even when proper usage of the examples, which involves customizing things (like addresses) that may need to be cutsomized, may result in things working just perfectly).

Next actions
(This section has been moved to the guide about making a single virtual machine.)
Creating a new virtual machine (after one already exists)

As mentioned above (in the section about how to make a child image), it may make sense to modularize by focusing first on disk images, reducing amount of details that one needs to focus on at any one time. (How true this is may be specific to the virtual machine implementation being used.)

The next important process is to make sure that documentation is up to date, including information about the virtual machine that will be used. This website contains an example file with skeletal documentation about machines.

Note that it may make sense to use the remaining steps of the process of creating a virtual machine as a method of documenting some things. For example, using virtual machines run from a command line may involve having some of the finer details about the machine stored in a script file. Simply referencing that script file from the main documentation may be an efficient method of documenting things (cutting down on time from redundancy, as well as reducing the overall amount of unneeded documentation). (However, some details may best be stored in the main documentation which is very easily accessible quickly, such as details on how to easily remotely access the system.)

If one has a choice of what to customize about the new machine first the recommended first thing to customize (in the virtual machine's configuration, such as a command used to run the virtual machine, and also in the machine's documentation) is what disk( image)s are being used. (Note that there may not be a lot of logical choice regarding which order things are customized in, such as the case of a virtual machine which is created by an interactive process that asks for certain details in a certain sequence.) This is so that the new machine doesn't accidentally start using the disk file that was the default, which may be the disk image meant for another machine (if the configuration/command and/or documentation was copied from another machine). (Many other details may cause little harm if they are wrong, such as the amount of memory that the system has. Other details may cause a bit more harm, such as using the wrong IP address, although in at least some cases the consequences of such a problem may not extend much beyond the point of time when the issue is identified. On the other hand, writing to an incorrect disk image by mistake could easily cause data loss. Such a problem may take a while to fix if it involves locating a backed up file that may be restored, and quite a while longer to fix if the restoration process involves re-creating a lot of data.)

It may make sense to assign a simple number to the machine: That number may be used for things such as assigning portions of several things such as MAC addresses, assigning IPv6 addresses, assigning IPv4 addresses, TCP/UDP ports being used (e.g. by the Qemu monitor), filenames used for disk images, the system's name, and other identifying numbers such as a “serial number” and/or an “asset tag number”.

The next thing to update is going to be details that identify this machine. That way, when modifying other details, one may easily see which machine's details are being modified.

Schadenfreude: Losing data

Here's a story of a really unpleasant thing to do.

An administrator copied the documentation about an older virtual machine that existed for some time, and used the copied file to contain the details about a newly created virtual machine. He then updated the file that contained information about the new machine. He also opened up the file that documented the older machine. So far, there does not seem to be a problem. He then made a mistake: He saw the file about the older machine, but he thought he was looking at the file with the newer machine.

Looking over the file, he thought he made a mistaken by not properly saving the new version of the documentation. This was frustrating, because he was certain that he spent some time updating the documentation. It therefore seemed that some work was lost, which was frustrating. However, his frustration paled in comparison with the frustration he was about to endure.

He proceeded to spent time fixing the perceived problem: he updated the file he was looking at, replacing details about the old system with details about the new system. He then saved the file that he modified.

The end result was that he then had two copies of the new documentation, and no details of the old documentation. He began to understand the actual error that was made when he opened up a copy of the file that he thought had the old documentation. Not only did he waste time by repeating work as he tried to fix the error that he thought he discovered, but he ended up losing the details about the old system.

Once those key factors are updated, the risks of using a wrong disk image or updating the wrong documentation are reduced. Other details that will need to be updated are network addresses (MAC addresses, assigning IPv6 addresses, assigning IPv4 addresses, TCP/UDP ports being used), resources used on the host machine (namely if using up a resource with a global identifier, such as a TAP/tun interface on a Qemu machine), and the expected local username(s) if those will be unique to this machine. In the best case scenario, failing to do this may result in an error such as the virtual machine being unable to reserve a TCP port to listen to, and causing the virtual machine to be stopped (which will likely be easily noticed and correctable). In a scenario would would be a worse case, a duplicate MAC address may not be detected right away and this machine could cause problems for machines that are already running. Therefore, making decisions about the customizations specific to this machine are recommended.

If there is a script file being run to customize a NIC (which is a feature implemented by Qemu), either ensure that the script file being referred to will work, or specify the name of a new script file.

By this time that each of these things are addressed, the new machine should start looking fairly customized.

One thing in particular when running a virtual machine right after an operating system was installed is that the order (or even existance) of boot devices will probably need to be changed. (It may be good to keep the operating system CD in the CD drive, at least for the first mount, in case other items need to be installed to the hard drive. Also, if the install files are going to be copied on the local hard drive, it will be desirable to have the CD in the drive.) If the script for the new machine was created by copying the script used for another machine, and then changing a lot of details in order to customize things for the new machine, then don't forget to check the boot order of devices. (The harm in failing to do so is likely a minimal but existant waste of time since most operating system installation boot media will prompt the user before performing destructive activities. Even if the boot media is destructive automatically, the harm may be minimal if the image being used is a new child image. But either way, it is a waste of time that is best avoided.)

Be sure to review all other details, such as how much memory the new virtual machine is using. Be sure that any script files used for configuring NICs are files that do exist. Be sure that any PID file specified is able (without having errors related to the file system's security permissions) to be written to by the user account which will be running the machine.

Before running the new virtual machine

Double-check that there are no critical details that aren't yet handled. For example: Make sure the new virtual machine points to the right hard drive images. Make sure that any “parent/base/“backing” hard drive image files have the file system attributes/security permissions set to prevent accidental writes.

If any files were specified by the virtual machine's configuration/command, make sure they exist. (This might include more than just disk images.) This is mentioned here simply because it is quite easy to refer to a file in documentation and/or virtual machine configuration including the command line being used, with an intent to make the file later, and then actually making the file is forgotten when/after attention is focused on taking care of something else.

Finally, a general rule before starting a new machine is to consider how the machine will appear on the network. Determine whether the new machine has a known hardware (EUI-48/MAC/EHA/BIA) address that needs to be supported by one or more automatic addressing servers. That likely doesn't apply for the first virtual machine when there likely isn't yet an automatic addressing server that the new virtual machine plans to be use in the long term. However, this step is mentioned here to increase familiarity with something that it would be good to be habit. If there is an automatic addressing server that needs to be customized (by adding support/recognition for a newly known hardware address), then it may make sense to customize that server (to be able to recognize the MAC address of the virtual machine) before the virtual machine starts (and then tries to communicate with the automatic addressing server). This way, the machine is properly recognized (and will receive the correct address from a server that helps with automatic address assignment).

Once everything is prepared, determine whether a break is warranted. (In some cases, finding an excuse to stop does make sense if there are time constraints, such as if performing this work in a lab that will be closing soon). However, if a fair chunk of further time is going to be used for this and if there's no further reason to delay running the virtual machine, then by all means, quadruple-check absolutely everything and anything and then delay no further: proceed with running the new virtual machine!

Make some decisions

While the virtual machine is running, consider making some choices that, in a short amount of time, will be needed.

DNS server addresses
  • What address will most IPv4-only machines use for a DNS server?
  • What address will most IPv6-only machines use for a DNS server?
  • What address will most dual-stack machines use for a DNS server? (This question will probably be easy to determine after the answer to the previous questions have been determined.)

(Further information, such as some available DNS servers, should be noted in the section about DNS. However, it may not yet be added.)

NTP server addresses
It may be that quite a few machines participate in NTP and that's great and all, but it is probably desirable to have at least one local machine serve the role of communicating NTP traffic to other local machines. Ideally, the address used for an NTP server is a DNS name, although it may also be an IP address.
Other details
  • [#ezpwuse]: Decide upon a well documented Administrator username, and also a related password, which can be easily accessed when making future child/snapshot images. This probably does not need to be complex, for reasons described in a similar situation: using an easy password.
  • Probably not needed: If there is any manual network configuration, or a plan to support multiple NICs in the base image machine, note the desired specifics.
[#nwsyscfg]: Configuring the remote machine

(This section is largely similar to configuring a newly-created virtual machine, although this section does have some differences.)

This machine will probably require a fair amount of steps to be taken. For example, the desired automatic addressing won't be available if that service is meant to be provided by this machine (or by a snapshot/child image that is based off of this machine).

There are some desired first steps other than enabling remote access. For example, some are related to security purposes. Making sure the virtual machine is protected (through firewalling or other methods) is a top priority. The very next step is quite likely going to be to make sure an instance of the virtual machine is running. Go ahead and run the virtual machine.

Once the machine is running, one may wish to check that the virtual machine is running and review the sections, if needed, on interacting with the output displayed by the virtual machine (including Creating a port forwarding rule using SSH).

MOVED INFO: Quite a bit of information was here but has started to be moved to setting up a new operating system.

Supporting shutdown options

There are multiple options provided by the section on shutting down a computer. Especially if there are multiple virtual machines, it will soon be desirable to have an easy, perhaps even automated, way to cleanly shut down virtual machines.

Using a special, dedicated user account has information about making and using a dedicated username for rebooting. This can be nice for multiple reasons, including the possibility of (immediately or eventually) using an account that has few, and perhaps no, elevated permissions except for rebooting a computer.

One method may be to cause the system to be rebooted when a user logs in. Then in initiating a reboot can be done as automatically as logging in as a user. Some remote access methods do allow such automatic logins, so this is viable. The section on “Using a special, dedicated user account” has or refers to information to accomplish this.

Other feasible methods may include:

  • Having a web interface that can initiate reboots (which, of course, must be properly secured), and automating access to that web page.
  • Perhaps using telnet and/or using a pure TCP connection (possibly with netcat/nc/socat?) to tell the virtual machine to shut down. This may not work with all options, but an example of this being a possibility could be a virtual machine using the Qemu software, if Qemu is listening for such traffic. Perhaps having the hardware send an ACPI can lead to software initiating a clean shutdown.
  • Perhaps using other methods: See the section about shutting down computers for other possibilities.
[#mltvmkid]: Preparing the machine's files for use by child/snapshot images

The hard drive image used by the virtual machine could be used as a “parent/base image”/“basing file”. If this is done, other virtual machines can be more quickly created by making “child/snapshot images”. There are some steps that can make a virtual machine's hard drive image to be more useful as a “parent/base image”/“basing file”. Following are some details of steps that can help make disk images that may be more useful (than other disk images) for fulfilling the role of being a “parent/base image”/“basing file”. These may take some time, but doing these tasks now may eliminate (or else, perhaps, reduce) the need to repeat the work for multiple child images. When considering the time taken to create the parent image, the time required for these steps certainly could be worthwhile.

Removing files
e.g. for a “parent/base image”/“basing file”, remove the keys used by SSH-related software if “child/snapshot images” are not expected to use those same keys. Especially do this in the parent image, for convenience, if a child image will check for those keys and create new keys automatically if keys don't exist. (That behavior is implemented in OpenBSD by default. It is a feature that may be manually added to the startup process of other operating systems.)
Setting network configuration to what will work well for child images
(e.g. use automatic addressing and automatically detected name servers), even if the new settings don't work well for now before certain servers are deployed.
File integrity check
For many systems, this may be done at any time (including early on) and there may be little benefit to putting this off until this late in the process. For a “parent/base image”/“basing file”, this should probably be done last. (It may be done earlier as well, but it should also be the last thing done, so that all other changes are reflected in the latest file integrity data.) This way a child image starts with a data file used by the file integrity checker, and can compare to what happened since the base image “parent/base image”/“basing file” was made. If this is done, but not as the last step, and if other steps change files, then every child image that runs will notice those other changes. That's fine for the first child image, but if any additional child images show those details then work is being duplicated when details of those additional changes are being checked for consideration.
Shut down the system
This should be done right after the file integrity check so that more files aren't accidentally modified (causing additonal differences from the last update to the file integrity checking data).
Wrapping up the initial setup of the hard drive image

(The information which was in this section has now been moved to: making a virtual machine: section on re-performing . (Hopefully the details are handled by that section. This reference might, or might not be useful to consider/remember/do-something-about when following this guide about multiple virtual machines.

Setting up a network
Test connectivity
Dealing with firewalls

Being able to test connectivity is nice. With some firewall software, including software that comes installed and active by default with Microsoft Windows, diagnostic network traffic may be blocked by default. Those settings should be adjusted, at least temporarily and at least for some early machines that are used for the core connectivity. That way, if a core technology like automatic addressing isn't working, diagnostic traffic (like IPv4's ICMP protocol) can help to determine if basic connectivity might be broken, possibly due to a configuration issue like routing issues preventing response traffic from reaching its needed destination.

See: firewall guide. Specifical details are available (at least for some firewall software implementations) about how to allow the relevant ICMP software.

Automatic addressing

For IPv6, the general expectation is that that IPv6 SLAAC will be used. Implementing SLAAC will require using NDP, so get that operational. Then check that the details of IPv6 SLAAC are being used.

For IPv4, Automatic IPv4 network addressing provides some options. See: DHCP servers.

For any other protocols, if information is available, it will be at the section about Automatic Addressing.

Routing/Firewalling

Once computers can start to communicate with each other on a network, the beauty of routing becomes desirable: Let's get traffic to be able to communicate with other networks. (Name resolution may be more interesting to some people, but the argument favoring getting routing works is as follows: once routing is enabled, it is possible for someone to use public name resolution servers. Internal names might even be able to be worked around locally by using a HOSTS file. However, even a knowledgable person on a local network will be severely limited if centralized routing isn't yet allowing traffic to the network.)

Note that while setting this up, tests that involve name resolution may not work if name resolution isn't set up yet. Most commonly people start by using a diagnostic program, ping, to verify some basic communication. This may involve needing to know some publicly accessible IP addresses, like IPv4 8.8.8.8 or others.

(There was a note here stating a need for more info. Perhaps the following hyperlink is suffcient?) See: Firewalling traffic

Name resolution

By this time, it may be true that most clients are almost capable of a rather full Internet experience. Ultimately, it will be nice to use internal name resolution servers: getting those set up very quickly may be desired. Otherwise, it may be nice to just skip that step, at least long enough to perform the second step: Change settings (perhaps with the automatic addressing protocol) so that name servers are automatically discovered by local clients. If local clients need to use external name servers, at least they will then be able to have a rather full Internet experience.

See: common DNS server software.

E-Mail

In OpenBSD, plan to use the mailwrapper command as it was designed to be used. That will involve using another piece of software to be the E-Mail server. Choices are located in the section about servers getting E-Mail.

Web serving
...
File sharing
Public file sharing
One method may be web serving. See transfering files.
Private file sharing
...
VPN server
...
[#tempnote]: Some old notes

These had been removed from the main part of this tutorial. It may contain some details useful for this tutorial, and/or for the tutorial on making a single virtual machine.

Configure the networking for the virtual machine software. (This is referring to adjusting options in the emulator, which is separate from any configuration done within the emulated virtual machine.) If using Qemu and following the earlier steps in the guide than at least one NIC startup script (if-up) probably still needs to be made.

Although one could probably get by leaving off both instances of “ -net ” and the parameter following each, that isn't recommended (mainly because the user mode (Slirp) networking in Qemu is going to be a lot slower).

export NICONETUN=tun4
sudo ifconfig $NICONETUN destroy
sudo sh /etc/netstart $NICONETUN
 -net tap,vlan=1,ifname=$NICONETUN,script=$VMDIR/bin/ifup/ifup_base_system \

  • Check if the right name is used for a tun device. (The section on making a TUN/TAP device has more detail of what the right name is.) If not, change the name of the tun device used for the first NIC (NIC#1) in the line that says:
    export NICONETUN=tun4
  • As for the script file (specified by the text after script=), the later section final virtual machine preparations has further details about a version of that file that needs to be used.
  • The example code above destroys old tunnel interfaces with “sudo ifconfig $NICONETUN destroy”, which will probably work okay. Then the above example code tries to re-make the interface by running “sudo sh /etc/netstart $NICONETUN ”. That latter command will probably work in OpenBSD, and in OpenBSD it will probably end up running “ /etc/hostname.tun? ” which may help to configure a newly created device. This might NOT work on other operating systems, so some configuration may be needed there. (This should be improved a bit...

any TUN/TAP devices are specified to use a configuration script, make sure that file exists.

e.g. for Qemu, the following will create a new, valid network configuration script (overwriting the file if it already exists):
cd $VMDIR/bin
mkdir ifup
cd ifup
echo \#! /bin/sh> ifup_base_system
echo The above backslash simply escapes the first character, and so the
echo backslash chracter does not actually appear in the text file.
echo echo Hi. The tun interface $1 will be getting used.>> ifup_base_system
chmod ugo+x ifup_base_system