Intro

This is a guide to getting one or more computers to provide services that can be useful on a computer network.

IPv6 and IPv4 are both covered by nearly all aspects of this guide, with the one current glaring exception that IPv6 automatic address assignment is not covered by the current version of this guide. Static address assignment is covered, though, and all of the other major IP technologies are supported as “dual stack”, supporting both IPv6 and IPv4.

  • The only other minor exception is related to AMaViS. Full support for IPv6, without using any IPv4 for internal communication, is not something that has been fully verified to be working, at the time of this writing. That may change relatively soon.

Note: This upcoming section provides an overview of what to expect. There are a bunch of hyperlinks. The purpose of those hyperlinks is to enable quick verification that a topic is covered. The author of this guide does not recommend simply jumping to hyperlinks to perform just part of what this guide covers, becuase some of the steps may assume that prior steps have already been taken. Being able to just “jump in”, right into the middle of the guide, is not something that this version of the guide was designed to strongly support at this time.

This guide covers implementation of at least the following technologies:

  • Virtualization
  • Network traffic routing
    • Includes NAT so that IPv4 works well. Also Enabling Network Communications/Traffic Handling for Virtual Machines covers some common problems
    • This does NOT currently cover using routing protocols such as Open Shortest Path First (“OSPF”, also specifically OSPFv6), Router Information Protocol (RIPv2 and RIPv6), EIGRP, etc. The networks designed here are just barely large enough for such things to possibly be useful, but doing things manually is not significantly more challenging when dealing with small networks. (The size of the network created by this guide is right at the tipping point. The network could potentially use such protocols, unlike smaller networks where they may not really serve any point. For networks that are much larger, such protocols tend to be beneficial even more frequently. Perhaps a future version of this guide may cover at least some of these more extensively, but coverage was considered a rather low priority.)
  • Network Defenses (Firewalling)
    • This guide actually shows how to create an extremely strict firewall, which blocks all traffic including outgoing traffic, except for the traffic which is documented. That provides more control over the network than many firewalls which only try to provide defenses by controlling incoming traffic. Having such a tightly controlled firewall forces configuration of all essential types of traffic, which effectively serves as some rather concise documentation of what actually happens on the network.
  • Network Address Assignment
    • This covers assignment of IP addresses, and similar critical networking settings. So, assignment of the network size (CIDR-style “prefix length”, which in IPv4 is commonly called a “subnet mask”), and “default gateway”.
    • For IPv4, Automatic Network Address Assignment is covered with DHCP/IPv4
    • Currently, automatic address assignment for IPv6 is not covered by the current version of this guide. (IPv6 is thoroughly supported by just about every other aspect that this guide covers.) Just as a quick run-down, the main technologies used for IPv6 automatic address assignment, which are NOT currently covered by this guide, are:
      • The Stateless Automatic Addressing Configuration (“SLAAC”) process, which uses:
        • “router advertisement” messages, which are a type of Network Discovery Protocol packet, which is a type of ICMPv6 packet. These “router advertisement” messages are sent in response after a computer sends:
        • “router solicitation” messages, which are another type of Network Discovery Protocol packet, which is a type of ICMPv6 packet.
      • The DHCPv6 protocol

      (A future version of this guide is likely to cover these technologies as well. However, implementation details have been known to vary a bit more, and so the author of this guide decided to focus on having other content be higher quality.)

  • Name Resolution (via DNS, using UDP port 53)
  • Web Server: using Hypertext Transport Protocol (“HTTP”)
    • Currently, this is implemented with OpenBSD's httpd
    • (It is quite likely that nginx will be getting covered very soon, since OpenBSD's httpd is currently lacking support for SNI)
    • (If you are seeking details about Apache, that is not currently covered by this guide's main text. However, as a quick side note, some information about Apache has been documented by the author of this text. It might be outdated, and the information might have more “rough edges” than much of the rest of this guide, but the details are available online on ][CyberPillar]['s Apache page.)
  • E-Mail Server
  • File Sharing (for internal networks), using:
    • Server Message Block (“SMB”), also known as Common Internet File System (“CIFS”), which is the primary “file sharing” protocol commonly supported on modern Microsoft Windows platforms. (This uses TCP port 445 and NetBIOS Session on TCP port 139).
    • Network File System (“NFS”) is partially covered. (At the moment, this guide covers setting up a client, and mentions some security concerns, but does not cover setting up an “NFS” server.) This involves using PortMapper and other Remote Procedure Call (“RPC”) traffic. (The amount of required TCP ports can be quite high, as discussed by file sharing section.)
  • Network-based Time Synchronization
  • Operating system installation
    • At the moment, the focus is on OpenBSD
    • When others get added, the list is likely to be added here: project variations
  • Software Package Installation
  • Remote Access
    • SSH
    • Interaction with Qemu's internal VNC/RFB server
  • Some security technologies/processes
    • Privilege Escalation
      • sudo (and related: /etc/sudoers file, visudo, sudoedit)
      • Coming soon: OpenBSD doas
    • “Key file”-based authentication (using SSH keys)
    • Tunneling: Using SSH tunnel to forward a TCP port
    • File Integrity Checking
      • On Unix-platforms, AIDE and Integrit support much of the functionality of Tripwire. Tripwire may support the Windows registry better? This guide currently uses AIDE, with instructions currently focused a bit on using AIDE in OpenBSD.

On Unix, strong efforts were made to identify every file that gets changed before it is changed, and to provide instructions to quickly make a backup copy before the change occurs. This is primarily done using cpytobak software, which provides an ability to revert rather easily (compared to not having a copy of the file), but does not perform the more elaborate functionality provided by many backup solutions that are designed to be more thorough (such as tracking snapshots, and having backups be stored in safer locations).

The following might not be part of this guide yet, but are likely to be quite soon. (Material has been created for most of these topics, and the guide might currently touch upon them.)

  • Designing a network
    • Subnetting
  • Network documentation
    • Sample documentation is anticipated to come soon... (at the moment, the example is only partially created)
    • (Some resources are currently available: network documentation)
  • One-Time Passwords
  • Identification by URL (using “OpenID”) will be getting partially covered
  • Backups
  • Effectively using a second hard drive on a virtual machine
    • There is a very specific approach that the author of this guide really likes for Unix systems. The author of this guide hopes to have covered more fully in the somewhat near future.

Support for additional software may be getting added, although that is currently a lower priority than reviewing the current material to ensure/improve quality.

Future Features

Additional software support:

  • OpenBSD doas
  • Other operating systems
    • Microsoft Windows might actually come first, due to being significantly different (so the information may be quite useful), and Linux-based sytems.
  • Additional virtualization solutions
    • OpenBSD Journal @ Undeadly.org's announcement of a native Hypervisor coming to OpenBSD/amd64 : The author of this guide currently speculates having support be added soon after the software is included in a released version of OpenBSD.
    • Hyper-V is probably going to be covered when Microsoft Windows starts being covered more heavily, or shortly thereafter. VirtualPC may also be covered.
    • Perhaps also: KVM/Xen/VirtualBox
    • Bhyve looks interesting. However, at the time this text was written, Bhyve's software compatibility was quite limited. That has led to full documentation of Bhyve being a lower priority. If it develops into an awesome solution, well, that'll be great.
    • VMWare is not currently identified as a likely priority. The main reason is that it is neither “open source”, nor freely built into Microsoft Windows. Wikipedia's article on VMware: section about VMware products mentions some software that has been released as free for non-commercial use.
    • Bochs is not currently identified as a likely priority. Although the LGPL is nicer than some of the other licensing options, the software has historically been slow.
    • Some of these technologies might be worth pursuing, but may need significant further research before they go into this guide: OpenStack, FreeBSD jails
Current status

The guide has recently completed its first version that is probably suitable for public use. Feedback is currently being sought. People who have not personally met the author of this text may find details about contacting the site staff at staff information.

This guide has only recently reached the state of covering the desired materials. There may still be quite a few rough edges that are marked with Attention-Needed-By-Tutorial-Author.

The guide is now probably rather usable in its current form. It is believed to be better, in some ways, than prior attempts to document the same sorts of things.

The guide may be a bit specific to OpenBSD, more than what will eventually be desired. Fixing that was considered to be a rather low priority, less important than being able to share what has been created already.

Overview: about overviews on this site

This guide often uses the term “Overview” to discuss how things work, or “Rationale” to describe why things work. Commands in these sections should never perform major changes. Anyone who has any troubles with commands in an “Overview” or “Rationale” section is encouraged to report the confusion, but is also encouraged to not feel worried about being unable to “move on” with progress in the guide. Perhaps the later instructions will actually clarify things as the process actually gets performed.

(People who are very familiar with the technologies described by this guide, and who are just using this guide like a checklist of steps to quickly perform, can safely skip over the “Overview” and “Rationale” sections if they're in a hurry. That is NOT recommended for people who are trying to learn this material, or for anybody who may be getting results evaluated after the end of a formal training course/session.)

In short: people with deadlines may get work done more quickly by skipping those sections, although possibly at the cost of learning less. Also, people should not get stuck by such sections.

Early Steps

This section describes steps to take before creating the first virtual machine.

This is a relatively short section, and is recommended reading. People may wish to perform (at least a quick) review, even if they have an existing, functional network (and want to quickly proceed to creating a virtual machine).

Direction Setting
Goal identification

A good step for people to do early in a project is becoming familiar with the goals of the project. This involves identifying some accomplishments that should be reached.

Another very important aspect is to identify some things that should not be receiving investments of resources (like time). Not all potentially nice things are necessarily going to be related to the project's main desired goals.

Note that people may have goals chosen for them. A person may be expected to accomplish specific goals, and those goals may have been selected by a person who has some authority to choose those goals. Making decisions can be an important task that a person does as part of the process. However, another role, which may also be important, is simply learning what the goals are.

An example of a document identifying project goals will be getting added here soon. (See <!-- ... -->).

Determine approaches/details

Determine major factors. One example may be how many machines will be getting setup/configured. Selecting software, including the “operating system”, is also a detail that will need to be decided, and choosing it early may be sensible.

Discussion of this project's design

Awareness is generally a good thing. Therefore, this guide provides some details about some things that are known to potentially be a problem, limitation, and/or drawback. An evalutation of this guide's network design has been made, and can be reviewed. (A reference is about to be provided.)

People who are part of a formal/organized/structured training program (a class) may be expected to review such considerations early on. That is really not a bad approach. People might choose to skip that discussion if they are still relatively unfamiliar with network designs, and if they simply want to get experience with accomplishing some tasks. This discussion will also be referenced later in the project, so there will be another chance to review it then. However, just keep in mind that if you ever start questioning why the network was designed a certain way, that explanation is available.

If an instructor has requested that you review the network design evaluation at this time, or if you have your own desire to check out those details now, go ahead and check out the evaluation. You might, or might not, want to read the evaluation of the network design early on.

Variations

This guide was initially designed around OpenBSD. Many other Unix-like operating systems may be similar.

Details are shown at: project variations. These details can help the project to be successful, so such details may be good to become familiar with early on.

[#reqmntok]: Verifying requirements are fulfillable

People should verify that requirements can be suitably completed. This involves becoming familiar with requirements. So, start off by becoming familiar with the project requirements.

Then, once those requirements are known, check the available resources to make sure that those requirements can be fulfilled. For instance, if a computer system is running, a person can check the available disk space to ensure there is sufficient disk space to fulfill the project's requirements.

Starting processes

This task is meant for people who are starting an actual project where things have not been set up in advance. (This may be less necessary for things like the environment of a prepared classroom, where many/most/all of the necessary resources are likely to be readily available.)

If a decision has been made to copy some software from the Internet, determine where the software can be obtained and where a copy will be made. If feasible, and if there is no compelling reason to delay, then proceed to go ahead and start making the copy.

If money needs to be spent to acquire some hardware, then submit the request to get the approval to spend that money. If the money is available, then spend the money so that the item can start to be shipped.

The point is to try to start any task that involves a delay, but which will make substantial progress once the process is started. Get those things started so that delays are taken care of.

[#ckphyenv]: Physical Environment checks
Follow the steps by: physical environment checking.
Disk checking requirements

The section about required shutdowns discusses reasons why disk checking may be needed at this point.)

When starting up hardware (to see if it is functional), many devices will get to a point where the device is designed to remain powered on until the device completes an official “graceful” shutdown process. If that is recommended by the operating system, do that.

There is a possible exception, which is that corruption of the filesystem structure is not a concern if all important data is already backed up and if entire volume (including the “meta data” that forms the filesystme structure) is going to be erased and re-created (typically as part of the process of re-installing the operating system). If in doubt, do things the safe way by properly completing the recommended shutdown process.

[#nmnetprj]: Name the net/proj
Naming the project

Determine the name of the project. (For a small network that is not expected to significantly outlast the time spent on the project, you may wish to just name the network after the project.)

If the project documentation hasn't been started yet, then go ahead and create it. Often, the project's name is sensible to use as part of a filename and/or document title. (In some cases, it may be the only part of the filename and/or document title.)

  • Place the project's name into the project documentation.
Naming the network

Some details might be available from choosing a machine name. (For instance, it references DNS names section, which cites some old standards about DNS name length limits like the 63 character limit. The network name should be under those length limits, as individual machine names will also need to be part of the long full names that should fit within the same limit.)

As an example, if the project is named “Project PureMake”, a domain name such as puremake.zz (or something a bit longer, more convoluted, and seemingly less appropriate, such as puremake.test) may be valid, as per reserved DNS names.

  • Usage of the domain name .local is discouraged, unless used as described by IETF RFC 6762: Multicast DNS. (This particular domain is being mentioned because of misuse which has historically been widespread. This domain is discussed a bit further by reserved DNS names.)
Key process notes
Conventions
  • There may be multiple times where these instructions include a step to output the contents of a variable, but then there is no clear reason why that effort was taken. Overview: outputting variables
  • There may be quite a few times that a command may be output to “| sudo -n tee -a filename” (with the “filename” being a customized portion that will varies between different examples).

    This is covered by: Overview: using sudo -n

    Such commands may be quite numerous, and may be assuming that the sudo command permits logging in without prompting for a password. Sometimes, a sudo command might be configured to never request a password in certain circumstances. Also, if a person does use the sudo command, the sudo command may start a timer which allows subsequent sudo commands to be run without a password. The length of time is related to the timestamp_timeout option.

    If this is not an option because the sudo command is not available, or because passwords always get forced, then adjustments may need to be made. The simpler adjustment may be to see if the situation can be changed. (For instance, see if sudo may be installed, which may fix the problem of the sudo command being unavailable.) Otherwise, a variation may need to be performed to the instructions. Because this situation is expected to happen numerous times, since these instructions do show the usage of “ sudo -n ” numerous times, manually creating such workarounds is not discussed at length every time “ sudo -n ” shows up in the instructions.

Warning on copy and paste

See “Warning on copy and paste

Be careful when using environments where “copy and paste” functionality is suspect, questionable, or even worse: known to not work well. In particular, at the time of this writing, that typically includes using the “local display” of a virtual machine. Unless such a display is known to work better, be quite careful about trying to use “copy and paste”-style paste functionality. (Being careful involves checking command lines for any missing characters before pressing the Enter key to submit the command. Leaving the last visible character uncopied may frequently be a relatively easy, safe way to make sure that an invisible “new line” “white space” character isn't accidentally pasted.)

Lack of an easy undo

This text will probably be unnecessary for many skilled professionals, but may be a relatively new concept for some people.

Be careful. When taking a “computer class”, many students may be used to having a computer keep track of commands and effects, and being able to reverse undesirable results by using an “undo” command.

That is often not a very available option with system administration. To “undo” an option will often require much more effort than simply pressing Ctrl-Z. Instead, the process may involve restoring data from a backup copy, and then repeating any work that has been done after that backup copy has been made.

This guide tries to create as many “checkpoints” as reasonably feasible, primarily by creating small and quick backups of data that can be restored. However, even though creating a backup might be as simple as copying a file, restoring may be a more time consuming process which includes locating the backed up data, copying the data, and having a program reload the older data.

Until the first backup has been successfully created, redoing things may require starting some processes from the very beginning (starting over from scratch).

Additionally, some actions can be taken which may require substantial work. For example, a skilled system administrator may frequently have access to commands that can modify large amounts of data (including making the data unable to be easily read at all, because the data can be “deleted” and overwritten). Throughout this guide, users of this guide are highly encouraged to be rather careful about actions that are taken, because mistakes can be rather costly (including costing quite a bit of time). Repeating work is typically less fun than doing the tasks the first time, so plan to try to avoid such stress, by being careful.

Configuring physical machine

See: Using a physical host

Stopping point

This is a guide-recognized stretch zone. If you have a little bit more time before a point when you know you want to take a longer break, you may want to dabble in just the next section: decision making.

[#afirstvm]: First Virtual Machine
Use a virtual machine
See: using first virtual machine for the earliest of steps, like:
  • Creating the virtual machine
  • Running the virtual machine
  • Installing the operating system.
See the virtual machine

The next step is to start customizing the newly-created virtual machine. Access the virtual machine's virtual display. If the virtual machine was stopped, that would mean starting out by running it:

echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}

Also, the following may be helpful: Using Qemu to see the local display.

See the virtual machine again

If you've been following along this guide, and just completed the previous steps, then the virtual machine may already be running and visible, so you can skip this step.

However, if you've performed the previous steps, but the virtual machine is no longer running, and if you want to run the virtual machine again, then you may appreciate the quick starting a created virtual machine. Those instructions are simply a condensed repeat of selected steps that have already been performed, so there's no need to follow those steps if the machine is already running. However, if the “virtual machine” software has stopped running (a.k.a., the software has been “closed”/“terminated”) for any reason, that guide shows which steps are more likely to be useful to get the running machine visible again.

Preliminary Customizing

This section includes some core configuration changes, like adjusting what account is used to perform sensitive operations, and getting proper Internet access.

To be very clear, these next steps are intended to be performed on the virtual machine. (As multiple machines get used, there starts to be a larger need to pay some more close attention to which machine a task is about to be performed on.)

(This looks like a relatively short section, but actually can be rather time-consuming.)

preliminary customizing

Regular Preparation Steps

Now that the prelimiary customizing” process took care of some of the most critical steps that should be handled early, more changes can be made. Other changes, like displaying the host name, may be less critical for security and functionality, but can be a really good idea. (Doing that reduces the chances of accidentally doing something on a different machine than intended.)

Here are some instructions to Perform Regular Preparation Steps common to newly installed computers.

If you're looking to take any sort of extended break (like a multi-day break), it may be worthwhile to do one of the tasks from the next section, which is setting up “file integrity checking”. Once that is set up, performing an actual “file integrity check” can be rather time consuming, so that may be nice to start before leaving. Otherwise, when returning to this project, you may start off with a process that can involve a fair amount of waiting, which may feel slower overall. There might even be a good reason for the feeling that it seems slower, because it may actually be rather inefficient.

Updating the host/physical system

(These directions are not necessarily meant to be followed... yet. They are just inserted for future reference as this guide gets developed further. (You may wish to explore this. Just know that it is not part of any completed section of the guide, so you don't have the availability of hand-holding if something goes awry.) Some of the tasks from Perform Regular Preparation Steps might be nice to have on the physical system, too. Especially:

  • setting environment defaults (the desired script can be re-created, or transferred)
    • (including adjusting the prompt on the main system)
  • adjusting bootup sequence
  • enable auto-reboot

(and maybe some more too...)

More Initiating

First, this guide covers some tasks that, as a generalizing, took more documentation. Follow each of these sections (as part of the process of following this guide).

Speed up system startup process

Are you still needing to manually unpause the “virtual machine” every time that it starts up?

If so, update the “virtual machine” startup process. There may have been some benefit to being able to carefully watch the system as it boots up. That time is probably passed into becoming past, so adjust that.

echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
cpytobak ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}
sudoedit ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}

Find the line that says:

[ "X${VMOptStp}" = "X" ]&&VMOptStp=" -S " #"stop" ("pause") upon start

Change it to say:

[ "X${VMOptStp}" = "X" ]&&VMOptStp="" # " -S " #"stop" ("pause") upon start

(This is a fancy way of not passing the -S option to qemu. Later, when “child” virtual machines get created, the script won't pass that option to qemu because of how the script behaves with a certain value of a variable called VMScrGen.)

Prepare an SSH key file

The following is just part of the process of supporting an SSH key file.

mkdir ~/.ssh/
[ ! -f ~/.ssh/authorized_keys ] && touch ~/.ssh/authorized_keys
chmod go-w ~/ ~/.ssh/ ~/.ssh/authorized_keys*

The remaining part(s) of the process involve inserting an actual key. Since the keys may be a bit more customized, that part of the process is being left off, for the moment. Future directions will have this file be changed.

File Integrity Checking
Overview: goals

At this time, this guide is designed around using AIDE. Follow the instructions related to AIDE to do the following steps on the “virtual machine”:

  • Install file integrity checking software
  • Configure the file integrity checking software (which this software guide refers to as “early steps”)
  • Perform the “Additional early steps” section
  • Rotate the data files
  • Update the database
  • Rotate the data files again
  • Review an AIDE report
Actions to perform

Basically, doing all that just requires:

starting the “virtual machine”
  • echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
    ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}
Installing and configuring AIDE and AIDE scripts

Then

That should cover each of these steps:

  • Installing AIDE
  • Making a group-writable /var/log/aide directory
  • Other steps that do things like configure what to scan, and initializing the database, described by AIDE early steps
  • making sure the AIDE scripts are installed
Turn current image into child

Right now, the disk image has some updates that have been worth saving.

Soon, substantial changes may be made to the disk image. Especially, the process of installing sofwtare might increase the amount of used disk space to more than double what the virtual machine has been storing so far. This may be a good time to child the disk.

  • shut down virtual machine
    • Be very careful that you are on the right machine when you do this. If the prompt shows the system name, check the prompt.
    • sudo halt -p
  • First, make very certain to verify virtualization variable values ...
    echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
  • export VMDskDir=${VMDirBas}/diskimgs/baseimgs/${VMGenNam}/${VMLILNAM}/.
    export VMCrDsNm=1
    export VMDskMom=${VMDskDir}/${VMLILNAM}${VMCrDsNm}-aide-installed.qc2

    ...

    sudo mv -i ${VMDskDir}/${VMLILNAM}${VMCrDsNm}.qc2 ${VMDskMom}

    ...

    ls -l ${VMDskMom}

    ...

    sudo chmod a-w ${VMDskMom}
    ls -l ${VMDskMom}
  • The ${VMDskMom} variable should successfully be pointing at a file. So, that last command was a check to make sure that the file exists. (By setting ${VMDskMom} here, that is overriding some default behavior in some upcoming instructions.)
  • Follow these instructions: VM: Mk Kid Disk: Common Steps
Adding an SSH key
Demonstrate file integrity checking
Testing the file integrity checking

Once the SSH key is deployed, you've effectively created a new way to log into the system. To demonstrate the effectiveness of the “file integrity check” software, you are encouraged to try running another file integrity check. (See: AIDE variable value setting, and then follow the rest of AIDE: Rotating data files. Then follow the remaining instructions at updating AIDE's database.) Then, see if the resulting report shows any changes to a file related to logging in. (AIDE: Showing report, older info: AIDE: Report view ) Search the report for any references to authorized_keys. Also, do the remaining changes look like files that you recall working with while performing actions you've done?

Installing software
  • Check the filesize of the hard drive image.
    • echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
      ls -l ${VMDirBas}/diskimgs/baseimgs/${VMGenNam}/${VMLILNAM}/${VMLILNAM}?.qc2
    • Make a note of the size. This doesn't need to be stored in long-term documentation. This just helps to show how much disk space actually gets allocated (on the “physical machine”) by installing software.
  • Log into the virtual machine
  • Install additional software onto the virtual machine
  • Check the filesize of the hard drive image.
Check network ports
  • check network ports
    • Know what type of traffic is intended for every open port. If this guide didn't cover one of the ports, research to find out what type of traffic is expected.
Some more file handling

At this point, it may be nice to compress the data (to save space), and to child the data (to have a spot that can be conveniently used as a “restore point”). That will create some new files, so then this guide walks through a bit of clean-up.

Compressing
  • shut down virtual machine
    • Be very careful that you are on the right machine when you do this. If the prompt shows the system name, check the prompt.
    • sudo halt -p

First, let's clear off any old values.

echo VMDskBeg=${VMDskBeg} VMDskSml=${VMDskSml} VMDskEnd=${VMDskEnd}
unset VMDskBeg VMDskSml VMDskEnd

Make sure to point at the right disk:

export VMCrDsNm=1
echo ${VMCrDsNm}

(Or, optionally, if the first disk is wanted, you can just have the value be unset, which will cause an upcoming instruction to set it to 1.)

Then, set the name of the compressed file.

[ "X${VMDskSml}" = "X" ] && export VMDskSml=${VMLILNAM}${VMCrDsNm}-after-first-packages.qc2
echo ${VMDskSml}

Then follow these steps:

After that, rename the older image:

sudo mv -i ${VMDskBeg} ${VMDskDir}/old/${VMLILNAM}${VMCrDsNm}-after-first-packages-uncompressed-child.qc2

These sample instructions leave no remaining ${VMDskDir}/${VMLILNAM}${VMCrDsNm}.qc2 file. The upcoming step of childing takes care of that. (If, for some reason, you choose not to child the image, you may need to link or copy the file.)

ls -l ${VMDskDir}/
Childing

First, let's clear off any old values.

echo VMDskMom=${VMDskMom}
unset VMDskMom

Point VMDskMom to the file that will become the parent image. (This is likely to be the file that was just created during the disk image compression process.)

export VMDskMom=${VMDskEnd}

This must exist:

ls -l ${VMDskMom}

Make sure that this is still set:

echo VMCrDsNm=${VMCrDsNm}
export VMCrDsNm=1

Then... VM: Mk Kid Disk: Common Steps

clean-up

At the end of VM: Mk Kid Disk: Common Steps, the child was verified to require a parent.

Let's verify that the required parent does not have any required parent.

echo VMDskMom=${VMDskMom}
qemu-img info ${VMDskMom}
qemu-img info ${VMDskMom} | grep ^"backing file"
echo ${?}

Notice how those results were different than doing the same thing for the child. (The value of ${?} is 1, because the searched-for text was not found.)

What this goes it show is that some of the *other* files in the directory are no longer needed.

echo VMDskDir=${VMDskDir}
ls -l ${VMDskDir}/
sudo mkdir ${VMDskDir}/old/old2/
sudo mv -i ${VMDskDir}/${VMLILNAM}${VMCrDsNm}-initial-changes.qc2 ${VMDskDir}/old/old2/.
sudo mv -i ${VMDskDir}/${VMLILNAM}${VMCrDsNm}-aide-installed.qc2 ${VMDskDir}/old/old2/.
ls -l ${VMDskDir}/old/old2/
ls -l ${VMDskDir}/
sudo qemu-img check ${VMDskKid}

For the next task, the virtual machine will need to be running again. Go ahead and start it up.

echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}

Then, there's a bunch more steps, some of which are relatively quick tasks. They've been grouped together in these sections. Complete these instructions:

Checking documentation
  • It might be nice to show a demonstration of what the documentation looks like so far.
    • (Currently-available sample might be a bit un-thorough, not containing SSH fingerprints)
Making child disks from parent data

If you're looking for a lengthier stopping point (when it makes sense to take a lengthy break), the end of this section isn't the recommended spot. There's just a little bit more, “Making child disks from parents”, which is related to handling the data created with this first virtual machine. When feasible, this guide recommends to continue on with that before taking a longer break.

Expanding to a network
Making child disks from parents

This tutorial is designed to have the following be performed next:

(Similar instructions were just performed at the end of Preparing for offspring, so there is no need to do this again now. The reference to these instructions, and possibly the referred-to page, may be removed after some verification...

Stopping point

This is a guide-recognized stretch zone. If you have a little bit more time before a point when you know you want to take a longer break, you may want to dabble in just the next section: decision making.

[#netchsnm]: Decision making
[#chsubnfw]: Deciding the subnets

If you haven't yet gone through the process of choosing subnets (for the child machines), then please do so now.

(The text for choosing subnets (for the child machines) was initially intended for this point. Subsequent reviewing of this guide has suggested that “choosing subnets” might happen earlier. The intro text on that page might be better to be here, instead of on that page.)

Know what address ranges will be used for these distinct subnets:

  • External/physical subnet
  • Outside VM NIC subnet
  • vmSvrs subnet
  • Inside VM NIC subnet

Those are the most critical ones. If you have additional subnets to support, such as an “internal wired LAN”, then make sure there are addresses that have been “picked out”/selected/reserved for the purpose of being used for that subnet.

(At the time of this writing, this guide is not focusing much on supporting additional subnets. However, if this guide is being followed as part of a more formal course, do see if there are additional instructions related to accomplishing that.)

Decide system information
  • Prepare details for at least one system: a firewall which is a “virtual machine”.

    Note: throughout this guide, both IPv6 and IPv4 will be supported in virtually everything that is done. The number of exceptions are few:

    • IPv6 automatic address assignment is not currently covered by this guide.
    • IPv6 tunnel broker” is designed to support communications with the IPv6 Internet when using an Internet connection that only provides IPv4. This process is not discussed the other way around.
    • Accessing remote “file systems” (NFS/“Server Message Block”) may not have been as thoroughly tested (by the author of this guide) with IPv6 (yet).

    This guide will regularly use IPv6 addresses that have endings that visually look similar to the endings of IPv4 addresses. For example, the DNS server's IPv6 address ends with “:4”, and the same system's IPv4 address ends with “.4”.

    For low-numbered addresses, that is really the only sensible way, in order to avoid confusion.

    For higher-numbered addresses, there are two rather common approaches. This guide takes the more wasteful way. So, for example, a sample NTP server with an IPv4 address ending with “.12” or “.14”, this guide will use the IPv6 address ending with “:12” or “:14” for the same equipment.

    This approach is rather wasteful of IPv6 addresses, and can even make things more challenging when working the IPv6 addresses in binary (and perhaps also hexadecimal). The reason that this approach is wasteful is because 37.5% of possible IPv6 addresses don't get used. The alternate way is to convert the ending of the visible IPv4 address from decimal to hexadecimal. So, instead of using an IPv6 address that ends with “:12” or “:14”, and an IPv6 address ending with “:c” or “:e” may get used. This is a visual mismatch when viewing in standard notation.

    This guide decided to sacrifice unnecessary address conservation, and use the addresses that would probably be easiest for people who may be rather new to networking.

Creating a virtual machine
Making the parent disk images read-only

First, these instructions will have you set a new variable. The variable named VMParNam should point to the name of the system that is going to be used as a “parent” image. If the VMLILNAM variable is still set to that, then the following may be a convenient way to set the “parent” variable.

export VMParNam=${VMLILNAM}
ls -l ${VMDirBas}/diskimgs/baseimgs/${VMGenNam}/${VMParNam}/${VMParNam}?.qc2
sudo chmod ug-w ${VMDirBas}/diskimgs/baseimgs/${VMGenNam}/${VMParNam}/${VMParNam}?.qc2
ls -l ${VMDirBas}/diskimgs/baseimgs/${VMGenNam}/${VMParNam}/${VMParNam}?.qc2
Common steps

Follow the steps described by: making first child

Steps for a firewall
Prepare firewall config

There are some steps with preparing the firewall, since the “firewall” virtual machine will have multiple NICs. For now, this guide will operate with a design that has the “firewall” virtual machine contain three (virtual) NICs.

Start the “firewall” “virtual machine”

By now, the network documentation should contain the variable names for this machine. If that's not the case, verify virtualization variable values (setting the correct values, if needed), and document those for easy reference. Also document the command that starts the virtual machine, for easy reference:

echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}
${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}

Remember, you may need to use the Qemu monitor to unpause the system, and to enable the VNC server so that you may use the VNC server built into Qemu.

If you are having troubles starting the virtual machine, using a virtual machine has some troubleshooting tips that may be helpful.

Networking notes

Almost this entire guide is meant to support IPv6 as much as possible. With the current version of the guide, logging into the firewall machine for the first time is one exception: IPv4 is recommended instead of IPv6.

This is mainly because IPv4 automatic addressing is supported, while automatic assignment of an IPv6 “ULA” addresses is not enabled with the current version of the guide. IPv6 link-local remains a perfectly viable option, and there's no reason why a person couldn't use IPv6 link-local, but it is slightly less convenient due to the need to specify an interface.

Desiring changes in the past: Adjusting the parent image

(This is not required, and should generally be skipped if everything else has been done flawlessly.)

If a desire gets created to update the parent image, there are some instructions that address the scenario of updating the parent image. Note that doing so involves repeating some work, and making all “child images” broken. Therefore, avoiding such changes is often the most preferred route; if such a change is made, then it is best to do so sooner rather than later. (An alternative to making such a change is to simply create a new child, and use that as a parent for later images.) If you really wish to replace the parent image with an updated version of the parent image, some recommended steps are covered by base image adjustments. (That contains a list of steps that may be completely unnecessary. Skimming over it might not be a bad thing. If a reason develops to desire an update to the image, then following that guide closely is recommended.)

Customizing the new machine
  • Log into the machine, and then, on that machine, proceed with:
  • configuring new child
  • One of those steps was to describe the NICs. Well, since there are multiple NICs, descriptions with some details may be rather nice. Mention more than just what TUN/TAP device is NIC is meant to connect to. To do this, be familiar with the intended network layout(INFO NEEDED). (Perhaps see also: demonstration subnets (used by this guide).)

    • We'll start by defining the VMCrTpNm variable (using the same variable name as the Firewall NIC Configuration section.)
      export VMCrTpNm=44

      Basically, this should be getting set to the VMNUM of the current machine, which is the “virtual machine” which is/“will be” serving as a firewall.

      • This is actually supposed to be run on the “virtual machine”.
        • It is not very common that the “virtual machine” uses the VMNUM. Actually, the change being made to the “virtual machine”, at this point, is simly documenting which TUN/TAP device is getting used with the virtual machine's NIC. Experience has shown this to be more useful on the firewall, compared to other virtual machines, because the firewall has multiple TUN/TAP devices.

      Then, using the same variable names from the the Firewall NIC Configuration section:

      The first of these variable names may vary based on the name of the TUN/TAP device used, which may vary between different operating systems.

      TUN/TAP base name in OpenBSD
      export OSTnDvNm=tun
      export VMTpDv1=${OSTnDvNm}1${VMCrTpNm}
      export VMTpDv2=${OSTnDvNm}2${VMCrTpNm}
      export VMTpDv3=${OSTnDvNm}3${VMCrTpNm}
    • echo description \"Connected to physhost ${VMTpDv1} : Outside VM NIC subnet\"| sudo -n tee -a /etc/hostname.em0
      echo description \"Connected to physhost ${VMTpDv2}, uses bridge0 : vmSvrs subnet\"| sudo -n tee -a /etc/hostname.em1
      echo description \"Connected to physhost ${VMTpDv#} : Inside VM NIC subnet\"| sudo -n tee -a /etc/hostname.em2
      echo ${VISUAL}
      sudoedit /etc/hostname.em0

      Then:

      • Remove incorrect data
        • Although the instructions (configuring new child) showed a sample for the first NIC, and that sample said “uses bridge0”, this system is rather unusual (different than other systems that this guide discusses) because the first NIC, em0, does not use bridge0 on the main host. (It is similar to other systems in that it is the “outside”(/“external”) NIC that gets used to communicate to the Internet.) So, if the text file contains a description with a reference to “uses bridge0”, then take out that reference.
        • (If the text file, somehow, does not contain that misleading/inaccurate information about the first TUN device using bridge0, then don't worry about these instructions to remove that detail.)
    • Finally, clean up unnecessary variables.
      unset OSTnDvNm VMCrTpNm VMTpDv1 VMTpDv3
Overview: next step

The firewall's basic purpose is to handle traffic that travels through the firewall. Well, that means some traffic needs to be sent its direction.

If an IPv6 tunnel broker is going to be used, then supporting the IPv6 tunnel (related to that IPv6 tunnel broker), at this time, may be a nice step to do. However, this guide is (unnecessarily) delaying that until a later point (when some other routing also gets handled), so directions for that are at a later time.

Configuring the firewall is more pleasant when effects can be seen immediately. Configuring a system is more pleasant when implementations can be verified right after changes are made. For these reasons, this guide suggests proceeding to make another system before trying to tinker much more with the firewall. That way, changes to the firewall and be seen right after the changes are made.

Fortunately, quite a bit of the process to creating the firewall was supporting mulitple NICs. Creating the next system ought to be easier/quicker. (Rough guess: about 1/3 less work.)

(As a quick overview: after the new machine is created, routing will need to be fixed, which will probably be a more tedious and slower process. Then things will pleasantly speed up again. So now you have some details about what the pace will probably be like in the near future.)

Creating the next system

The next service to be provided will be an “automatic addressing” server for IPv4. So, the system will be running software which is a “DHCP/IPv4 server”.

Considering variations

Some experienced technicians might be wondering why this guide has people creating a new “virtual machine”, rather than using a “virtual machine” that was already created. (So, the question is why the next service, DHCP/IPv4, gets placed on its own brand new “virtual machine”, instead of simply running the “DHCP/IPv4 server” software on an existing “virtual machine”.)

In short: using an already-available “virtual machine” might be a feasible alternative. This guide does not try to pretend otherwise.

This guide uses a network design which involves having a “virtual machine” which is dedicated to the task of running the network's first “DHCP/IPv4 server”. Some people may prefer to use different network designs. If this guide is being followed as part of some sort of formal instruction/training, then be sure to follow the instructions/advice provided by whomever is in charge of that training. In other cases, there may be some possibility that some variations from this guide's design might be able to produce more results faster. However, keep in mind that “automatic addressing” technologies, including “DHCP/IPv4 server” (and SLAAC for IPv6) may operate a bit different than many other protocols.

How DHCP/IPv4 is unusual

For instance, DHCP/IPv4 uses IPv4 broadcast, and multiple UDP ports. (The more famous is UDP port 67 which the server listens on, but there is also UDP port 68 which the client uses as a “source” UDP port, starting in the first (UDP) datagram that is sent.)

The first bit of communication will be an IPv4 packet which comes from a device that doesn't have an IPv4 address assigned. Software that acts like a DHCP/IPv4 server (which may be an actual full DHCP/IPv4 server, or a DHCP Relay”/IPv4) needs to be on the same “link”(/“subnet”).

For many protocols, “split horizon” (referring to devices obtaining different information based on their location in a network) is not an issue, because devices are likely to get the same information no matter where they are on the network. For example, NTP should give accurate time. However, DHCP/IPv4 may need to respond differently based on which subnet a request is coming from. In some cases, a DHCP/IPv4 might need to ignore a request, so that there are not conflicts with another server.

If you're follwoing this guide independently, and feel up for the challenge of customizing your experience, feel free to deviate from these instructions. Otherwise, following this guide is recommended. Following this guide precisely might take a little bit longer, while being far more likely to have things work out well. The little bit of time that might be saved trying to skip some steps might be insignificant compared to the much larger amount of time that troubleshooting may be quite likely to require.

Creating networking for the virtual machines

So far, the firewall's second NIC is not using TUN/TAP. However, this guide is currently designed to have machines communicate to the firewall using bridging to an interface that the firewall is running as TUN/TAP. So, let's prepare to do that.

Enabling TUN/TAP on the firewall

First, some variable checking. (verify virtualization variable values)

echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}

The ${VMLILNAM}'s value should be referencing the firewall. If that is not the case, fix that now.

export VMLILNAM=vmfire

Edit the virtual machine's “startup script” file.

echo ${VISUAL}
sudoedit ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}

Err, nevermind this. The intent was to adjust VMNICTWOTYP to use VMNITUNTAP but that seems to already be done. Also, the intent was to adjust VMNICTWOVLANNUM as needed, and make the TUN/TAP interface, but that seems to already be done. If that is the case, these instructions aren't needed here. (Otherwise, the general process may be similar to use TUN/TAP dev.)

Enabling bridging

This process creates a bridge, and causes the “virtual machine” to join that bridge.

First, some variable checking.

echo VMDirBas=${VMDirBas} VMGenNam=${VMGenNam} VMLILNAM=${VMLILNAM}

The ${VMLILNAM}'s value should be referencing the firewall. If that is not the case, fix that now.

export VMLILNAM=vmfire

Edit the appropriate NIC “startup script” file.

ls -l ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/
echo ${VISUAL}
sudoedit ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/upif1
Creating the bridge in OpenBSD

First, the TUN/TAP interface needs to be in TAP mode, so make sure that the LINK0 flag is set on the TUN/TAP interface. That needs to be done before trying to add the TUN/TAP interface to the bridge. (Otherwise, the bridge refuses to accept the interface. The bridge cannot work with a TUN interface that is not in TAP mode.) (Enabling LINK0 was seen at an earlier point, Layer 2 tunneling in OpenBSD.)

Actually, these instructions are given as a command line, so that there is just one variable that probably needs to be customized. (That is the value of CURTUNTP. The bridge0 might also be good to customize, although many systems may be able to just simply use that value literally, without requiring customization.

The value of CURTUNTP uses the VMNUM from the “virtual machine” which is a firewall.

export CURTUNTP=tun244
  • Customize that as needed.
    • (If you thought that you just had that value in an environment variable that got “cleaned up” (erased)... well, yes. But that environment variable was with a different command line shell. In fact, that environment variable was being used on a different computer system. So it wouldn't have been useful here.)
echo ifconfig ${CURTUNTP} link0| sudo -n tee -a ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/upif1
echo ifconfig ${CURTUNTP} up| sudo -n tee -a ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/upif1
echo ifconfig bridge0 add ${CURTUNTP} blocknonip ${CURTUNTP} up| sudo -n tee -a ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/upif1
echo ${VISUAL}
sudoedit ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/upif1

The process has been known to change. (In older versions of OpenBSD, a command called brconfig was used.) OpenBSD FAQ 6 (“Networking” section), “Setting up a network bridge in OpenBSD” (FAQ 6.9) discusses the creation of bridges. (For instance, it references the blocknonip option to ifconfig.)

You can run those commands from the command line, one at a time, manually. Or, if all of the commands in the script file look safe, and at least some of them look desirable, you could just “source” the script file (which is effectively a way of running all of the commands in the script file). For example:

sudo ${SHELL} -c ". ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/nicscr/upif1"
ifconfig ${CURTUNTP}
Creating a new virtual machine

Create a new virtual machine, which will use a “child” disk image.

The purpose of the new virtual machine will be to run “DHCP/IPv4 server” software. (That detail can be helpful to know about when creating a new machine.)

Customize the new “child” virtual machine
  • Log into both the newly created machine (which will be the DHCP/IPv4 server) and the “virtual machine” which will be the firewall.
    • Each machine should have some documented commands for setting the environment variables to the values needed to easily start the specific virtual machine. Check the documentation for the commands. (If you have not yet made such documentation, then update the network documentation.)
    • The command to start a machine, after the environment variables have the values needed to start that machine, is:
      ${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}
    • The newly created machine will probably not have an IPv4 address automatically set with DHCP/IPv4, so you'll likely need to interact by either using the local display (e.g., see: Qemu's VNC server), or by using IPv6 (perhaps using the link-local address). If you choose to try to use IPv6, the client software may need to be run on the firewall, because current network routing limitations may prevent connections from other locations.
      • Furthermore, the IPv6 non-“link-local” address might not work if the corresponding address hasn't been set up on the firewall yet.

        Sample commands to run on the firewall:

        sudo ifconfig inet6 up
        cpytobak /etc/pf.conf
        cpytobak /etc/pf/pfnics.cnf
        echo vmSvrs_if=em1| sudo -n tee -a /etc/pf/pfnics.cnf
        echo \enable SSH out to vmSvrs on \$vmSvrs_if| sudo -n tee -a /etc/pf.conf
        echo pass out on \$vmSvrs_if proto tcp from self to any port ssh keep state| sudo -n tee -a /etc/pf.conf
        sudo pfctl -nf /etc/pf.conf
        echo ${?}
        echo ${VISUAL}
        sudoedit /etc/pf.conf
        • The “port ssh” refers to TCP port 22. If you are using a custom TCP port, then make sure to specify that in the rule (e.g., “port 2222” instead of “port ssh”).
        sudo pfctl -nf /etc/pf.conf&&sudo pfctl -ef /etc/pf.conf
        ssh -p 22 -l userNameForNewKidSys fe80::5054:ff:fe12:##01%em1

        If the SSH connection works, that's great. It indicates that networking was working as desired. Since we've just been making changes to the firewall's configuration, let's exit the SSH session and do a bit more configuration on the firewall.

        cpytobak /etc/hostname.em1
        echo inet6 2001:db8:1::1 125| sudo -n tee -a /etc/hostname.em1
        echo up| sudo -n tee -a /etc/hostname.em1
        echo ${VISUAL}
        sudoedit /etc/hostname.em1
        • The 125 is only showing as “125” to be consistent with demonstration subnets (used by this guide). A much more common/typical value is “64”. Note that there is no slash before the prefix length. (Do not put a slash there, as doing so is a syntax error.)
        sudo ${SHELL} -c ". /etc/netstart em1 "
        ssh -p 22 -l userNameForNewKidSys 2001:db8:1::1
  • Overview of what is next

    Because internal network routing has not been fully set up yet, the new machine has some limits regarding its ability to communicate on the network. (The limitations will be more severe than any limitations that the firewall may have experienced already, because this system is behind the firewall, meaning that this system is one layer further away from the other networks.) Because of that, only some of the steps from Early actions for “child” virtual machines (which are about to be identified) are worth taking.

  • Follow these steps:

    (The following steps are commonly done with new machines, but get skipped for now.)

    • Skip: “Passwordless login” (since files are not being transferred before networking is implemented)
    • Skip: “Defending the system”
    • Skip: “Getting network access working” (and “Troubleshooting networking”)
    • Skip: “Check the time”
    • Skip: “Update anti-malware definitions”
Checking the networking
  • Manually assign an IP address to the second NIC on the firewall, as well as the DHCP/IPv4 server. The addresses should come from the subnet that has already been chosen for use by a network for the virtual machines. That subnet should already documented. (Perhaps it is simply being documented by the documentation of the firewall's second NIC.)
  • One of the TUN/TAP interfaces should have IP addresses (ideally both an IPv6 non-“link-local” address, and also an IPv4 address).
  • Every TUN/TAP interface related to a running “virtual machine” should be UP.

Issues like a lack of a network route, possibly including a lack of the IPv4 NAT that is commonly necessary for public communications, could cause some other communications to fail. So, failure may be expected if someone tries to perform the basic test of sending a ping to a memorized remote address on the Internet.

Verify that the systems can communicate with each other. Namely, setup and check these things:

  • Temporarily assign an IP address to a TUN/TAP device on the “physical machine” (which is running the “virtual machine software”). This isn't completely necessary (and can be skipped if it is somehow problematic), but is recommended because it can help some basic troubleshooting to help verify that bridging is functioning as necessary.
    • Can that TUN/TAP device ping the virtual machine that it is connected to?
      • If not, check IP addresses.
      • A firewall on the “virtual machine” could also cause this.
      • All network interfaces should be UP. (This includes each TUN/TAP devices, even if it does not have an IP address.)
    • Can that virtual machine ping the TUN/TAP device?
      • (If not, consider if firewall rules may be running on the machine with the TUN/TAP device.)
    • Can the other virtual machine communiate with the TUN/TAP device (on the “physical system”)?
      • (If not, check network settings like IP addresses, and check firewall rules.)
      • Bridging could be an issue. (The second virtual machine, which is not connected to the TUN/TAP device that has the IP address, might not be on the bridge.)
  • Can the two virtual machines communicate with each other?
    • If communiation between the virtual machines is not working, then bridging could be an issue.
      • One of the virtual machines might not be on the bridge.
      • This may be especially likely if each virtual machine can communicate with an IP address on the close TUN/TAP device.
    • All network interfaces (including the TUN/TAP devices) should be UP.
    • Double check the network settings (the IP addresses must be different, yet should be in the same subnet).

Getting this basic networking functional is a task that should be done before trying more elaborate things, like trying to set up a server for a network protocol like DHCP/IPv4. By doing a simple test, potential issues like problems with bridging, or active firewall rules that might affect traffic, can be identified and troubleshot without also involving the potential complexities of setting up the network protocol's server.

Quick Networking Note

Once some basic network tests work, one option could be to get traffic routing to work more fully. Another approach is to set up the DHCP/IPv4 server first, so that focus remains on the subnet that includes both of the virtual machines (the one running the “DHCP/IPv4 server” software, and the machine acting as a firewall). Either way makes some sense.

Handling Routing

Some remaining network problems are expected at this point. Follow these directions to experience, and overcome, these problems.

  • On the new “virtual machine” which will be the DHCP server, handle IPv4 routing by adding this default gateway:

    cpytobak /etc/hostname.em0
    echo ${VISUAL}
    echo !route add default 198.51.100.249| sudo -n tee -a /etc/hostname.em0
    sudoedit | sudo -n tee -a /etc/hostname.em0

    (The address to specify here will be the IPv4 address for the second NIC of the gateway/firewall “virtual machine”.)

    After confirming that all looks good:

    netstat -na | grep default
    sudo ${SHELL} -c " . /etc/netstart em0 "
    netstat -na | grep default
  • Handle firewall early routing.
Status
This section should be moved; having it here/now is premature: hostname.* may be updated after DHCP.

At this point, it is assumed that all routing is taken care of, except for support for using a IPv6 tunnel broker. Within the network, any machine that has sufficient authorization should be able to:

  • ping another machine
  • SSH to another machine

Here is what might not be working yet:

  • DHCP
  • DNS (even to external servers, as the firewall might not pass the traffic)
  • Other protocols (e.g., NTP) might not be permitted by the firewall at this point.
  • Traffic from the Internet
  • Wi-Fi support (by creating a sandbox) might not be working

Although some other protocols might not be working yet (perhaps simply because there isn't a server for them yet, or perhaps because the firewall hasn't been sufficiently relaxed to permit that traffic yet), simple ICMP diagnostics (“ping”/“ping6”) ought to be working okay, except as noted above (e.g., Wi-Fi). If something that ought to be working (e.g., ping6 between authorized internal machines) isn't working, then don't expect future instructions to help address that. (If something isn't working, and you want it to be working, then there's no compelling reason to wait in hopes that future directions will provide a fix.) No such magical “future directions” are expected, because setting this up has been covered at this point.

Note (to be added here, earlier, or on ping's page/section?)
ping may show a message like "No route".
(HTML clean-up needed?)

That could be caused by:
- no route to a remote subnet (in this case, the message is actually fairly appropriate)
- item is on local subnet, but connection cannot be initiated
- no IP address on remote end
- firewall on local end prohibits connection from being made
Set up DHCP/IPv4 server
Making the DNS server's “virtual machine”
Creating a new virtual machine

Create a new virtual machine, which will use a “child” disk image.

The purpose of the new virtual machine will be to run the software which makes this computer act as a “name resolution” (“DNS”) server. (That detail can be helpful to know about when creating a new machine.)

Customizing the new machine
Enabling package installation
Enabling DNS

On the firewall:

Sloppy notes: clean-up warranted...
may want this on a sub-page, actually
This is PRE and might hide tables since they are between < and >, so view HTML source

on firewall:

at somepoint, this was done:



# Permit ICMP from vmSvrs

pass in quick on $vmSvrs_if inet proto icmp from $vmSvrs_if:network \
        keep state
pass out quick inet proto icmp from $vmSvrs_if:network keep state
pass out quick inet6 proto icmp6 from $vmSvrs_if:network keep state

table  { $vmSvrs_if:network \
        $ext_if:network }

pass in quick inet from  to $vmSvrs_if:network keep state
pass out quick on $vmSvrs_if proto icmp from  to any \
	keep state
pass in quick inet6 from  to $vmSvrs_if:network keep state
pass out quick on $vmSvrs_if inet6 proto icmp6 from  to \
	any keep state


#Allow SSH from trusted servers to whereever they want to go

table  { $vmSvrs_if:network \
        $ext_if:network }

pass in quick proto tcp from  to \
        $vmSvrs_if:network port customSSHPortNumber keep state
pass out quick on $vmSvrs_if proto tcp from  to \
        $vmSvrs_if:network port customSSHPortNumber keep state

#allow DNS for the moment
pass in quick on $vmSvrs_if proto udp from $vmSvrs_if:network to any \
        port domain
pass out quick on { $extern_nics } proto udp from $vmSvrs_if:network to any \
        port domain

###Actually, could probably use: to  port domain




Then, on endpoint system, look up the package host:




Then, on firewall system:

pkgsvrip=129.128.5.191
#old name:
ftpopenbsdorg=129.128.5.191

pass in quick on $vmSvrs_if proto tcp from $vmSvrs_if:network to \
        $ftpopenbsdorg port www keep state
pass out quick on $ext_if proto tcp from $vmSvrs_if:network to \
        $ftpopenbsdorg port www keep state

pass in quick on $vmSvrs_if proto tcp from $vmSvrs_if:network to \
        $ftpopenbsdorg port ftp keep state
pass out quick on $ext_if proto tcp from $vmSvrs_if:network to \
        $ftpopenbsdorg port ftp keep state

pass out quick on $ext_if proto tcp from self to \
        $ftpopenbsdorg port ftp keep state

pass out quick on $ext_if proto tcp from self to \
        $ftpopenbsdorg port www keep state

block quick log









On endpoint system:

pkgsvrip=129.128.5.191
#old name ippkgsvr=129.128.5.191
pass out quick on $ext_if proto tcp from self to $ippkgsvr \
        port www keep state
pass out quick on $ext_if proto tcp from self to $ippkgsvr \
        port 443 keep state

pass out quick log
block log







on firewall:





#allow DNS for the moment
pass in quick on $vmSvrs_if proto udp from $vmSvrs_if:network to any port domain keep state
pass out quick on $ext_if proto udp from $vmSvrs_if:network to any port domain keep state

also, check the address for ftp.openbsd.org
e.g.: 203.0.113.80 # https://tools.ietf.org/html/rfc5737#section-3
(actually has been known to be: e.g., 129.128.5.191 )

ippkgsvr=203.0.113.80

pass in quick on $vmSvrs_if proto tcp from $vmSvrs_if:network to \
	$ippkgsvr port www keep state
pass out quick on $ext_if proto tcp from $vmSvrs_if:network to \
	$ippkgsvr port www keep state

(if that seems problematic due to a DNS entry having multiple IP addresses, you could try "any" instead of the IP address)

on host system:
in /etc/resolv.conf
add 8.8.8.8 as a resolver

add to /etc/pf.conf:
ippkgsvr=203.0.113.80 
pass out quick on $ext_if proto tcp from self to $ippkgsvr port www keep state

Setting up DNS
A few things that should be getting updated soon:
to do: clean-up instructions about BIND.
Make sure Unbound has info about firewall rules, instead of just relying on NSD.
Make a section for Unbound to be resolver only, not relying on NSD (and not supporting internal).
Have another section rely/refer to that, but add info about supporting the internal domains.
Supporting IPv6

These days, more and more ISPs have started to use IPv6. However, for a long time, IPv6 was not supported by most ISPs. A work-around was to use an “IPv6 tunnel broker”.

Using an IPv6 tunnel broker

This section describes supporting IPv6 with only an IPv4 connection.

At the time that this tutorial was written, the author's most recent experience indicated that all of the worldwide tunnel brokers required creating an account. The accounts were free, but creating an account was needed. (Any of the tested IPv6 tunnel brokers that did not require an account, sadly, weren't working.) If you'd like to see if different options might be available at this time, one possible resource to consider checking may be Wikipedia's List of IPv6 tunnel brokers.

The current version of this guide is going to focus on using “Hurricane Electric (HE.Net)”/tunnelbroker.net for a couple of reasons:

  • This has been verified to work rather well.
  • There is a nice “Hurricane Electric (HE.Net)”/tunnelbroker.net IPv6 certification that is freely offered, but which tracks progress using an account. So, if details are being remembered right... creating an account with them allows both useful IPv6 tunnel creation now, and later on the account will also provide the feature of being useful for getting that IPv6 certification.

Note that some of these instructions involve a website that is operating by someone other than the author of this text. The instructions may become outdated if the website changes, and that is beyond the control of the author of this text. If that happens, please let the author of this text know (see Contact Information/Interaction page at ][CyberPillar][), and try to adapt as necessary.

Details are not currently provided, like signing up, or creating a tunnel. (The author of this text intends to work with another person, who actually signs up, so that the documentation can be nice (accurate, and sufficiently precise).)

Even some of the details that are provided, have not been fully tested at the time of this writing... There may be errors. You are hereby warned.

Offering overview

IPv6.HE.net certification FAQ refers to being able “to create a 6in4 tunnel”, so presumably that is the type of protocol used. (It is even more clear that “Protocol 41” is used, but “6to4” also uses Protocol 41.)

Signing up
(Details are not currently provided.)
Logging in

Go to: https://tunnelbroker.net.

In the upper-left corner, Login.

This will bring the web browser to the “Main Page”. (The words “Main Page”, in the “Account Menu”, are both a subject header, and a hyperlink. Using that hyperlink will return to this page.)

Getting tunnel info

The main page will list tunnels. Choose a tunnel.

This will show a “Server IPv4 address” and a “Server IPv6 address”

  • Both of those are operated by Hurricane Electric, and should respond to ICMP Echo Requests.
    • (Sending ICMP Echo Requests is done with the ping command for IPv4. For ICMPv6, either the ping command or the ping6 command is used, depending on what network stack the programs came bundled with.)
    • The web page provides the “Server IPv6 address” right next to the prefix length; if using “copy and paste”, be careful to not paste the prefix length (or delete the prefix length) before trying to send an Echo Request, because the ping6 (or ping, depending in what network stack is being used) doesn't use that prefix length, and will not work if the undesirable prefix length is supplied.

This “Tunnel Details” page (“IPv6 Tunnel” tab) also shows a “Client IPv4 address” and a “Client IPv6 address”.

The “Client IPv4 address” is something that is specified by the holder of the Hurricane Electric account. (This address presumably belongs to this network which is being set up.) This network must be online if this tunnel is going to work. One way to check that is to see if ICMP/IPv4 Echo Requests sent to this address will generate replies, although a lack of replies does not necessarily mean that the tunnel won't work. (It could just mean that the network doesn't send ICMP/IPv4 Echo Response packets.)

The “Client IPv6 address” is designed to be reachable by using the IPv6 tunnel, so that “Client IPv6 address” may be unreachable from the Internet until the IPv6 tunnel is operational.

On the “firewall” “virtual machine”
OpenBSD
[ -f /etc/hostname.gif0 ] && cpytobak /etc/hostname.gif0
echo tunnel 198.51.100.250 203.0.113.251| sudo -n tee -a /etc/hostname.gif0
echo inet6 alias 2002:CB00:7100::2 128| sudo -n tee -a /etc/hostname.gif0
echo dest 2002:CB00:7100::1| sudo -n tee -a /etc/hostname.gif0
echo !route -n add -inet6 default 2002:CB00:7100::1| sudo -n tee -a /etc/hostname.gif0
echo description \"Uplink to tunnelbroker.net\"| sudo -n tee -a /etc/hostname.gif0
echo ${VISUAL}
sudoedit /etc/hostname.gif0
  • In the previous example, 198.51.100.250 represents the IPv4 address that this machine uses to communicate with the Internet.
    • If a tunnel is created using a machine that has a public IPv4 address, this could be a public IPv4 address, in which case it very well could be the “Client IPv4 address” reported by tunnelbroker.net.
    • However, in many cases the address might be a private IPv4 address. Basically, look at the network interface that is used to communicate to the Internet, and use the IPv4 address that is on that interface, even if it is a private address.
  • In the previous example, the 203.0.113.251 represents the “Server IPv4 address”. This is probably not entirely unique; presumably different tunnelbroker.net accounts may use this same address. (Perhaps it is specific to which “tunnel-server” is being used?)
  • The first IPv6 address, appearing right after “ inet6 alias ” (which is 2002:CB00:7100::2 in the above sample), is the “Client IPv6 address”.
  • The remaining IPv6 addresses are both the same, and are the “Server IPv6 address”. In the previous example, that address shows as “2002:CB00:7100::1”.
    • In both cases, this address is used as a “destination address”.
      • In the first ocurrence, the address is used as a “destination” address for the tunnel.
      • On the route command line, the “Server IPv6 address” is used as the destination for a route.
Differences are okay

The syntax for the line which starts with “ inet6 alias ” may look a bit different than the corresponding ifconfig command from the instructions provided by tunnelbroker.net's “Tunnel Details” page (“Example Configurations” tab). Namely:

  • Instead of placing the destination address between the alias and the prefix length, the destination address shows up on the next line of the text file.
  • The syntax in this text file does not include the phrase “prefixlen”, which is part of the sample ifconfig command lines provided by tunnelbroker.net's “Tunnel Details” page (“Example Configurations” tab).

These difference was intentionally created by the author of this documentation, and were done because the syntax for the /etc/hostname.* files (documented by OpenBSD page for “hostname.if” (covering the /etc/hostname.* files) is different than the syntax for OpenBSD's ifconfig command (documented by OpenBSD's manual page for the ifconfig command).) In many cases, the syntaxes are really similar, and may even be identical (like what is shown in the first line), but that is not always the case.

Every single one of those addresses should be customized, except for the word “default” in the route command.

Make sure that all the information in the text file appropriately matches the details that came from the tunnelbroker.net web page. If any of those addresses are wrong, the tunnel is not expected to work. (A key reason to go into the text editor is to make any updates that may be required.)

sudo ${SHELL} -c ". /etc/netstart gif0"
sudo ifconfig gif0
Enable the tunneled traffic through firewall
cpytobak /etc/pf/pfnics.cnf
echo brokertunv4rmt=203.0.113.251| sudo -n tee -a /etc/pf/pfnics.cnf

Customize that address to be the “Server IPv4 address”.

cpytobak /etc/pf.conf
echo| sudo -n tee -a /etc/pf/pfnics.cnf
echo # Support 6in4/Proto41 tunnel| sudo -n tee -a /etc/pf/pfnics.cnf
echo pass out quick on \{ $extern_nics \} proto 41 from self to \$brokertunv4rmt| sudo -n tee -a /etc/pf.conf
echo pass in  quick on \{ $extern_nics \} proto 41 from \$brokertunv4rmt to self| sudo -n tee -a /etc/pf.conf
Enable some more traffic

Also, make sure some traffic is allowed. For example:

cpytobak /etc/pf.conf
echo| sudo -n tee -a /etc/pf.conf
echo # Support ping6 to tunnel remote| sudo -n tee -a /etc/pf.conf
echo pass out quick on gif0 proto icmp6 from self to gif0:peer icmp6-type echoreq \\| sudo -n tee -a /etc/pf.conf
echo \\tkeep state| sudo -n tee -a /etc/pf.conf
echo pass in  quick on gif0 proto icmp6 from gif0:peer to self icmp6-type echoreq \\| sudo -n tee -a /etc/pf.conf
echo \\tkeep state| sudo -n tee -a /etc/pf.conf

Review config syntax:

sudo pfctl -nf /etc/pf.conf
echo ${VISUAL}
sudoedit /etc/pf.conf

Apply config:

sudo pfctl -nf /etc/pf.conf&&sudo pfctl -ef /etc/pf.conf
On the physical host
cpytobak /etc/pf/pfnics.cnf
echo fwvmtun_if=tun\1\#\#| sudo -n tee -a /etc/pf/pfnics.cnf
echo brokertunv4rmt=203.0.113.251| sudo -n tee -a /etc/pf/pfnics.cnf
echo fwvmipv4=198.51.100.250| sudo -n tee -a /etc/pf/pfnics.cnf

Customize that as needed, which is heavily:

  • fwvmtun_if : The TUN/TAP interface that is used to communicate with the firewall's external interface. Demonstration subnet (used by this guide) calls the related subnet the “Outside VM NIC subnet”.
  • brokertunv4rmt : This is the “Server IPv4 address”... the IPv4 address for the remote tend of the tunnel broker's “tunnel” device.
  • fwvmip : The IPv4 address used by the firewall's external NIC. (That NIC communicates with fwvmtun.)
cpytobak /etc/pf.conf
echo| sudo -n tee -a /etc/pf/pfnics.cnf
echo # Support outgoing traffic from virtual machines to tunnel broker| sudo -n tee -a /etc/pf.conf
echo \\t# Step one: from virtual machine to self| sudo -n tee -a /etc/pf.conf
echo pass in  quick on \$fwvmtun_if inet proto 41 from \$fwvmtun_if:network to \$brokertunv4rmt| sudo -n tee -a /etc/pf.conf
echo \\t# Step two: from self to tunnel broker| sudo -n tee -a /etc/pf.conf
echo pass out quick on \$ext_if inet proto 41 from \$fwvmtun_if:network \\| sudo -n tee -a /etc/pf.conf
echo \\tto \$brokertunv4rmt nat-to \(\$ext_if:0\)| sudo -n tee -a /etc/pf.conf
echo # Support incoming traffic from tunnel broker to virtual machines| sudo -n tee -a /etc/pf.conf
echo \\t# Step one: from tunnel broker to self| sudo -n tee -a /etc/pf.conf
echo pass in  quick on \$ext_if inet proto 41 from \$brokertunv4rmt to \$ext_if \\| sudo -n tee -a /etc/pf.conf
echo \\trdr-to \$fwvmipv4| sudo -n tee -a /etc/pf.conf
echo \\t# Step two: from self to virtual machine| sudo -n tee -a /etc/pf.conf
echo pass out quick on \$fwvmtun_if inet proto 41 from \$brokertunv4rmt to \$fwvmtun_if:network| sudo -n tee -a /etc/pf.conf

Review config syntax:

sudo pfctl -nf /etc/pf.conf
echo ${VISUAL}
sudoedit /etc/pf.conf

Apply config:

sudo pfctl -nf /etc/pf.conf&&sudo pfctl -ef /etc/pf.conf
On the “firewall” “virtual machine” (again)
Making even more traffic work
Add gif0 to be recognized as a NIC for external traffic
cpytobak /etc/pf/pfnics.cnf
echo ${VISUAL}
sudoedit /etc/pf/pfnics.cnf

e.g., if it said:

extern_nics=$ext_if # NIC(s) used to communicate to Internet

Make it say this instead:

extern_nics=$ext_if gif0 # NIC(s) used to communicate to Internet
Check for network references

If your firewall rules refer to “$extern_nics:network”, then that won't work as desired if $extern_nics expands to multiple NICs. So replace them.

cpytobak /etc/pf.conf
echo ${VISUAL}
sudoedit /etc/pf.conf

For example, if you have:

table <tbl_pingSvrsOK> { $vmSvrs_if:network \
$extern_nics:network }

Then replace that with:

table <tbl_pingSvrsOK> { $vmSvrs_if:network \
$ext_if:network gif0:network }

Simply search for all occurrences of $extern_nics:network and make sure that every occurrence is changed to properly point to both $ext_if:network and gif0:network

[#vmautogo]: “virtual machine” auto-starting

Some software might require a terminal to run well. Unfortunately, it seems that the system's startup process might not provide a full-looking terminal until the system gets to the login prompt. Fortunately, a full-looking terminal session can be generated by using one of the terminal multiplexers.

Starting up virtual machines within tmux
echo ${VISUAL}
sudoedit${VMDirBas}/execbin/vmgo
#!/bin/sh
[ ! "X${3}" = "X" ]&&sleep ${3}
VMDirBas="/srv/bigspace/."
VMGenNam="${1}"
VMLILNAM="${2}"
${VMDirBas}/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}
sudo chmod ug+x /srv/virtmach/execbin/vmgo
/srv/virtmach/execbin/vmgo 1 2 3
echo \#!/bin/sh| sudo -n tee -a ~root/startvms
echo env TERM=screen /usr/bin/sudo /usr/bin/tmux -f /etc/virtmach.tmx| sudo -n tee -a ~root/startvms
echo ${VISUAL}
sudoedit/etc/rc.local
sudo chmod ug+x ~root/startvms
sudoedit /etc/virtmach.tmx
new -d -s vmgennameVMs
neww -n Firewall '/srv/virtmach/execbin/vmgo vmgennam sysnmfw 0 ; /bin/sh'
neww -n AutoAddr '/srv/virtmach/execbin/vmgo vmgennam sysnmdhc 5 ; /bin/sh'
neww -n AutoAddr '/srv/virtmach/execbin/vmgo vmgennam sysnmdns 45 ; /bin/sh'
neww -n LastWindow /bin/sh
detach-client
list-sessions

Things to customize:

  • sessionName
  • vmgennam where the script is found
  • the names of each individual virtual machines

The following is meant for OpenBSD, and differs among other operating systems:

cpytobak /etc/rc.local
echo ~root/startvms| sudo -n tee -a /etc/rc.local
echo ${VISUAL}
sudoedit/etc/rc.local
(Clean-up/testing still needed...)

$ cat ${VMDirbas}/execbin/vmgo

#!/bin/sh
[ ! "X${3}" = "X" ]&&sleep ${3}
export VMDirBas=/srv/virtmach
export VMGenNam=nulif
export VMLILNAM=${2}
/srv/virtmach/execbin/${VMGenNam}/${VMLILNAM}/sysstart/exc_${VMLILNAM}
chmod ug+x vmgo and ~root/startvms
Using screen

If screen is readily available, and tmux is not, then using screen may be an acceptable alternative. (In fact, that is how the author of this guide performed such tasks before tmux existed.) Unfortunately, instructions for using screen are not readily available here at the moment, but these instructions will note that it is a viable solution.

  • Before rebooting, record the routes and IP on both the physical machine (which runs the virtual machine software) and the firewall.
    • On the physical machine (which runs the virtual machine software):
      netstat -nr | tee -a ~/netstat-nr.log
      ifconfig -Aa | tee -a ~/ifconfig.log
    • On the firewall, do the same thing.

    Hopefully those logs won't be needed, because hopefully everything will automatically just start up, pre-configured, working. However, having that information can be very satisfying if routing needs to be troubleshot after the reboot.

Making the next “virtual machine”
Decide what's next

The next topic that this guide covers is sharing data files. So, that will be used as an example. However, there's no reason why that service must be, or even should be, the next service that gets installed. Having basic protection (via a firewall), routing (including NAT for IPv4), automatic address assignment, and name resolution are key critical services that are commonly found on nearly every network because they are extremely useful. The presense of these services enable significant abilities (like communication) or simplicity (allowing people to auto-configure).

The next topics are fairly interesting concepts that many people may be interested in. However, there may also be quite a few networks that don't have need for some of these specific services. For example, running an internal web server may be a feature that is far more useful, or at least interesting, on some networks compared to other networks.

If this guide is being used as part of an organized course, or if you don't have any specific plans on what else to do next, feel free to continue using services in the order that they are described by this guide. However, if you're simply following this guide on your own, feel very free to change things up. Having that freedom is something that's been intended throughout this entire guide, but some things are really recommended to address early on (especially security, since adding a specific security measure to an already violated network can be a useless endeaver). With the upcoming services, there is less compelling reason why many of these must be done in a specific order.

(There might still be some cases where some services are sensible to deploy before some other services, so think about what makes sense before just doing things in a backwards order just because that freedom probably exists in many cases.)

Internal Network Time Protocol Server
Next

Create a new system

Handling NTP

An NTP server has been set up now. If enough time has passed (10 min? 20 min?), hopefully NTP is reporting peers, and is synchronized.

If all that is working, then modify all other computers on the internal network, including internal servers and other systems, to rely on the internal NTP server(s), instead of using public pools.

(In fact, to avoid accidentally overloading the pools, it may be a good idea to add a firewall rule that blocks NTP traffic except from the authorized internal server(s).)

Implementing file sharing

This guide provides details about setting up a “virtual machine” dedicated to file sharing. (There is no requirement that “file sharing” needs to be on its own dedicated virtual machine. That is simply how the guide is written; people with other needs may adapt.)

If you are using an existing network, you may want to heavily customize the instructions from this section. People who are taking part of some sort of formal training are recommended to check additional instructions.

Here is a basic process that has been known to work well:

  • Follow creating and configuring a “child image” to create a virtual machine for file sharing.
  • On that machine, make sure it had some content to share.
    • People who are part of a formal class may wish to see if there is pre-available content available.
    • If there is content already available elsewhere on the network, use the “Accessing files” section of: “virtual machine” for file sharing
    • Otherwise, just make a sample file. Unless additional instructions (as part of a formal traning enviornment) provide more requirements, this sample “content” can be a very plain HTML file or even a single text file. (If it is a text file, it should have some non-whitespace, so that the content can be easily recognized. Later, that will simplify the step to verify that data transfer is working as intended.) (If you need to make an HTML file, or would simply like to, details for doing that are not part of this guide, but are available at hypertext markup language.)
      • Example task: make a web page that has italicized text, and text of different colors such as red text, light blue text, and light green text. (If there are challenges with colors, such as a monochrome display or a person who is “color blind”, then skip the section about colors.) Also, try placing white text on a black background. Details on how to do all that are provided by the section on hypertext markup language
    • Then, the next part of the process was going to involve sharing files to another machine. To really test that thoroughly, another system would be needed:
      • The next machine in this guide is a “web server”. That is probably an ideal candidate.
      • To do this, follow creating and configuring a “child image” to create a virtual machine which would later become a “web server”. Although the initial purpose of this machine is simply to be a way to test the ability to access a remote filesystem, the eventual purpose of the machine is to be a “web server”. Therefore, configure the system (naming the system, and choosing IP addresses) after the intended away that the system would be used in the long term.
    • Followed directions from “virtual machine” for file sharing to share data from the “file sharing” server, and to access that data from the “web server”.
      • Setting up the sharing was a task done on the “file share” machine, not the web server.
      • Because the web server's content was allowed to be static, the web server did not need to have permission to write to the content of the shared files. Server Messaging Block permissions were used to implement that. For a Unix-ish system, sharing with Samba was a usable approach.
      • Once the shared directory is confirmed to be accessible (e.g., using Samba's client to see a list of contents in the directory), mounting the directory can be done right away, or later. If the data is going to be accessed by a program, then mounting that data might be a task that is sensible to delay until after setting up that program (at least a little bit).
        • For example, if the web content is going to be accessed by a web server, figure out where the “web server” software will be looking for the content, and then mount the content under that location.

    In this guide, the final step, which is to handle mounting, is provided as part of the steps of setting up a web server. (Although the guide is not recommending these steps at this time, some references are provided here, for possible convenience: The instructions are available: accessing shares, e.g. using Sharity-Light.)

Web server

Virtual Machine: HTTP server

E-Mail server
  • See: Virtual Machine: SMTP server (installation and configuration notes).
    • Despite its title, this section covers more than just installing a functional SMTP server. It also discusses configuration, including support for features such as content scanning.
Backup
Wrap-up topics
Review documentation
  • Make sure documentation is accurate.
    • Hint to instructors: Submitting documentation may be an excellent assignment for training environments where the “trainee”/“student” is being evaluated.)

Especially, make sure to document the following, which can break a substantial amount of a network (probably the whole thing) if these things are not properly set. (Do this before performing the upcoming steps that relate to rebooting the machine, because otherwise troubleshooting can be more challenging.)

  • document all IP addresses and routes on physical machine
    • Exception: don't bother documenting the IPv6 link-local addresses on the virtual machines, as they are based on the MAC addresses, and (if this guide was being followed, then) those MAC address follow a calculatable standard.
    • To get the primary IP addresses: ifconfig in Unix ipconfig in Microsoft Windows. getting current addresses
    • To get routes: “ netstat -nr ”. viewing routes
  • document all IP addresses and routes on firewall virtual machine
Shut down all the virtual machines

In theory, that could be done as simply as:

ls -l ${VMDirBas}/execbin/${VMGenNam}/*/sysstop/stop_
for x in ${VMDirBas}/execbin/${VMGenNam}/${x}/sysstop/stop_${x} ; do echo Now handling ${x}; ${x} ; done

However, that will try to shut down all of the systems at the same time. There may be some benefit to allowing some systems, such as the DNS server or the firewall, to remain active while other systems shut down, so that the other systems may be able to use the critical services during their shutdown process.

Back up the data of the virtual machines
  • Back up the disk images
    • If there is suitable disk space available, back up the disk images.
      • Doing this now, while the machines are not running, is probably ideal.
    • If there is not suitable disk space available, that might be okay for a training environment where the student wasn't going to be accessing the material on an ongoing basis anyway. However, for a network which is going to continue to be actively used, having current backups is highly recommended (and should be considered “required” for many projects), so fixing that issue may be a very important priority.
      • Note that usage of the “second hard drive” may permit each system to require only a small amount of space (at least in some cases). That may help to enable the amount of required disk space to be fairly low. (Where are the instructions for that?)
  • Also, back up the machine configuration details
    • (in Qemu, those are stored in the startup scripts, and the NIC configuration scripts.)
Verify post-reboot survivability

If the “physical system” reboots, how easy is it to get things started again? Automatically starting each desired “virtual machine” is actually the next task. The current task is simply to make sure that this can work fairly easily when done manually, before adding a bunch of automation. Is the process as simple as rebooting, and then manually starting each virtual machine, or does it end up being more complicated than that?

Controlling system start

(Information has been moved to: “virtual machine” auto-starting.)

Checking some things

Is all well?

  • Do the early servers use internal NTP servers?
  • Can each server be logged into using SSH keys? Note: it is currently recognized that the answer might be: No. The guide might not have covered that much, yet...
  • Network connectivity tests:
    • Make sure that each of these things works for both IPv6 and IPv4:
      • Can the “physical” system ping a virtual machine other than the firewall?
      • Can the “physical” system ping every other system?
      • Can every system ping a public machine on the Internet?
      • Can every system ping a public machine on the Internet?
      • Can at least one non-firewall “virtual machine” ping another?
Questionnaire

Please see: Multi-Server Feedback Questionnaire.

Follow-up tasks
review questions
See: review questions
Terminal multiplexing
Overview

Becoming familiar with a terminal multiplexer is recommended. tmux is recommended, although screen may be installed on many more older machines. Before tmux was made, screen had become quite popular because the “detach” feature was particularly useful in having tasks survive network disruptions. As a default behavior, a network disruption could lead to the closer of a terminal session. Now that remote reliable network connectivity has become much more widespread than it was during the days of “dial-up” connections using POTS, the value provided by these terminal multiplexing is less tremendous. (In theory, surviving network connections could also be done using nohup, but some experimentation has indicated that nohup doesn't seem to actually do that job quite as well...

However, the terminal multiplexing can still be handy due to features such as:

  • Network survivability
  • Controlling window size
    • Splitting screen between multiple prompts
  • Providing “copy and paste” ability
  • Providing scrollback
  • Ability to generate a new command prompt with a small number of keystrokes.
  • Session sharing, allowing multiple users to interact with the same screen

Some of those features might be offered by some other technologies, such as a GUI providing windowing features and a graphical terminal program (xterm) providing scrollback. In such cases, there may simply be multiple possible solutions to be able to do the same tasks. One nice thing about the terminal emulators is that a person can use SSH to log into a remote device, and the overhead may be much lower, including the speed of setting up the required functionality.

A guided tutorial, showing off specific features, has not been made available here yet. However, documentation of some supported keystrokes is covered by terminal multiplexers.

Hurricane Electric (HE.net)/TunnelBroker.net certification

If this is provided as part of a class, then pursuing the Hurricane Electric (HE.net)/TunnelBroker.net certification can be a great way to follow up with setting up many of these things that were previously set up. It is a free certification, and the majority of the required time may be to set up a bunch of technologies. Since those technologies have just been set up, performing the early tasks (not including all of the daily submissions) may be good to do now, while the servers are still operational and easy to interact with (because you can probably readily recall some details, like how to log into the servers, what the names of the servers are, and where the authentication credentials are recorded).

Create an account with TunnelBroker.net.

Then, see: tutorial on using IPv6.

That's it!

The remaining content on this page are basically some scratch notes; as this tutorial gets even more polished, the content is likely to be removed from the “scratch notes” section, possibly after being merged with some of the other content.

Misc/scratch notes
  • otp (okonce/...)
  • SSH keys SSH keys
  • OpenID: idbyurl
    • Maybe a great way to log into Stack Exchange network... However, (at least, at this time), this portion of the guide may be requiring some more interaction with “third party” services to be able to really verify that the functionality is working well.

  • HTTPS
  • Centralized authentication (e.g., Microsoft Windows security “domain” implemented with Active Directory)
  • Graphical interface
    • Seeing graphics
    • Window managers
MISC NOTES:
  • support IPv6 tunnel
  • shift configs to secondary drives
  • support alternate WWW (not just OpenBSD httpd, but also nginx/Apache), alternate SMTP (not just sendmail, but also OpenSMTPd, etc.)
  • support web proxy-type setup
for reference; this should be deleted later, as there are many other copies:
Some notes about using ps

Note that although the above syntax will work okay with OpenBSD, ps syntax differences are significant enough that no single syntax is preferred by all the Unix implementations. Users of other operating systems may need to adjust the example shown. (Common modifications may include taking out the hyphen right after the word ps and/or removing some parameters, such as one or both of the w characters.)

Multiple redirections are shown. The final redirection is discussed in the section about “Having grep exclude itself”.

Misc notes:

One thing that wasn't elaborated on too much was that dhcpd uses a bpf You can see that with: fstat | grep bpf tcpdump also uses bpf devices. You can run out (Stuart Henderson's post on interface limit) Finding other packages e.g., freshports/freshmeat/whatever...

Check for more software

If this network is being created as part of a project, or as part of a formal training session, then it may not be terribly wise to go too crazy on installing software. Experimenting with software might be something better to do with a “child” image.

OpenBSD
FreeBSD
FreshPorts
Debian
Debian Packages (Perhaps see also: software repositories?)
To do: Add startup/shutdown scripts to process of adding each machine
Add startup script to system's startup process Perhaps Sharity-Light must load within terminal multiplexer when part of system start?