Using OpenBSD Source Code To Update the OS

Current status:
Overview/Introducting some options

If compilers are going to be involved, it may be best for many people if they follow the “-stable” flavor of source code releases. The Errata/Patches may be easier to obtain, although applying them may be more time consuming than using “-stable”, and using “-stable” may contain more updates.

(That comment may require some explanation, since OpenBSD's “manual page” about building an OpenBSD release says “This branch only contains errata, no new features.” However, that comment was likely referring to the base operating system. It seems likely that newer versions of ports(/packages) that get added to the “-stable” branch may include feature changes in addition to just security updates. At least some OpenBSD releases have had a “-stable” ports tree. OpenBSD FAQ 15 (Ports): section about Security Updates For Ports (FAQ 15.3.10) discusses this. (Historical note: Announcements related to these sort of updates have been made, e.g. Matthias's comment on OpenBSD Journal article, “A Tale of Mozilla Ports”, OpenBSD Journal announcement of 4.6-stable ports.)

Spreading details about the build environment

When a kernel is made, and then that kernel is used, information about how the kernel was made gets shown to the end user. Simply be aware of this potential spread of information, so that undesired loss of confidentiality doesn't occur.

Some information is shown during the system startup sequence (which may later be viewed by running dmesg, where this information may be the second line of output). This information may include:

  • The name assigned to the computer system
  • The directory/location of the kernel file that was compiled
  • The username that was used to create a build

Also, additional information may be included in the kernel, and be visible in other locations like the first line of dmesg's output as well, as when running “ uname -v ” or when using the config. Such information includes:

  • The name of the build. This name may come from a file or directory being used.
[#tmpobsrc]: Usage of /tmp/

The examples use /tmp/, despite some good reasons why /tmp/ might not be a very good spot. A key reason that /tmp/ is used is simply due to permissions issues, because the location is likely writable to the end user. This made /tmp/ seem like a convenient directory to use as an example, in part because using /tmp/ was a simple method for creating documentation that didn't look overly cluttered. Also, it allowed a simple method that would likely work if people copied the example rather verbatim. Using /tmp/ in this way does not have to be a bad thing for people performing this task on a tightly-controlled private machine. However, there are a number of potential drawbacks, and so using /tmp/ in this fashion should probably not be a regular habit (and perhaps would be better to not be done at all). This text discussions at least some of those potential drawbacks that may be worth knowing or considering before implementing this practice.

Many of the commands in this guide may have resulted in log files being through into /tmp/. That may not be the best spot for such log files anyway, but if there is any interest in having the files long term, then /tmp/ will be a particularly bad spot when the system gets rebooted, since some operating systems (including OpenBSD) may tend to perform an action like deleting the contents of /tmp/ when the machine is rebooted.

Especially before rebooting, but also when done working on the project for a while, copy off any contents of /tmp/ if there is any desire to keep a copy of those contents.

Note that one reason why /tmp/ may not be the best space is simply because it's not considered to be a very private location. If multiple people tried to perform the same task on the same computer, then they may end up using the same file. This could be avoided simply by writing such temporary files to a different directory, such as a newly-created ~/tmp/ directory. Such a directory may be unique to the account that is logged in, although even that might not be completely unique to a person if the logged in account may be accessed by multiple people.

Expanding on the issue of privacy a bit further: many of the files placed in /tmp/ may default to being owned by the user who created the files, and the default permissions may grant some privacy. However, the presense of the filename may be visible to others; it may be that all users are able to see the names of filenames in /tmp/ (even if the user doesn't have access to read the contents of the file).

Also, ~ is account specific, so if something like “sudo -i” were to get used, then ~ may refer to different directories depending on when it is referenced. (This isn't meant to indicate that this guide makes heavy use of “sudo -i”, but rather is pointed out as a way that this practice/habit could lead to some difficulties/confusion.) That could be worked around by using something like ~root/tmpdir instead, and that may work if permissions are appropriately set.

[#obsprpar]: Preparation

The following are some initial steps, which might be worthwhile to review before trying to start any long processes for working with the source code. If this is delayed until the time of starting with source code, these initial concerns should be addressed before proceeding with any later steps.

Some considerations for -current

There are various “flavors” of OpenBSD to upgrade to. An overview of the different flavors is available at OpenBSD FAQ 5: section about operating system “flavors” (section 5.1). If a decision is made to start using -current, there are some things worth knowing about the installion process (before performing such an installation).

One note, about using source code from the “-current” code base, is found at in the “Final notes” section, underneath the OpenBSD's Anonymous CVS Guide: section about setting up a mirror. That section notes, “upgrading from a release to the current tree by rebuilding the sources can be rather difficult due to dependencies that are often not obvious. Therefore, it is suggested that you first install the latest snapshot before attempting a tree build from source.” The quoted phrase “install the latest snapshot” refers to upgrading to a “snapshot” version of OpenBSD (using provided binary/executable files). Yes, this means upgrading/(re-)installing the operating system is something that should be done before trying to build/compile/make the latest version of the code. Upgrading OpenBSD before compiling updated source code may be generally good advice with any flavor of the operating system. However, to avoid quite difficult problems, this generalized advice may be especially important when working with code from “-current”.

It can also be nice to have binary snapshots of the latest ports tree. Note that binary snapshots for the operating system, and especially binary snapshots of the latest ports tree, might not always be available from the OpenBSD project's distribution sites. In that case, this method cannot be started, and so starting at a later date (perhaps by checking back in a few days) may be preferable.

If re-installing the operating system at the start of the process seems too challenging to be a desirable solution (possibly because it seems like the time or effort commitments may be too great), then “-current” may not be good to be pursuing until obtaining more available resources such as familiarity, experience/expertise, abililty to risk and recover, time, and/or willingness to expend such effort in using “-current”. An attempt to successfully use “-current” involves the potential of certain problems that may be easily avoided simply by using a different “flavor” of the operating system. Even experienced administrators may find that certain situations aren't as condusive/welcoming to implementations that involve such unnecessary (and avoidable) potential of certain types of problems. So, don't be afraid to change plans by deciding not to use “-current” until a later time.

Although the OpenBSD team does welcome people to use -current, there are some considerations. Those who think they might be (or are) adventurous enough to try using anything based on the “flavor” of source code known as “-current”, including (but not limited to) the “snapshot” releases, may benefit greatly by being familiar with those relevant warnings. So, being familiar with these warnings is recommended: Many of those details are described in the section called Considerations, warnings, and potential problems related to using OpenBSD snapshots (and/or “-current”). (That section recommended reading: read that section before following the processes of installing anything using source code from the “-current” flavor).

Having needed software

The file sets for installing the OpenBSD operating system lists the various file sets. The following will probably be needed (or worthwhile):

bsd* (a kernel is needed: is probably preferred, if it exists and if using new enough of hardware that SMP could be used), base*.tgz (needs to be installed for the operating system), comp*.tgz (which generally should be installed if working with source code), etc*.tgz (which needs to be installed for the operating system, unless an upgrade is happening), xbase*.tgz (because this is said to include some base libraries), xetc*.tgz (which is not typically part of an upgrade).

A common/standard implementation of OpenBSD may include many of those pieces: the most common exception may be having the needed compiler tools. Make sure that the system is equipped to be building binary code from source code. OpenBSD is distributed with some optional software development packages. Although those packages may be considered “optional” for many systems, a computer lacking this “optional” software may not be equipped to handle trying to compile OpenBSD's “-current” code. (Such a system would need to start by installing the software development software.) (The entry in the OpenBSD FAQ about filesets, OpenBSD FAQ 4.7: Filesets for the OpenBSD operating system, has a subsection called “I don't want to install the compilers”. One of the statements in this section is, “If you do decide to not install the compiler, you will probably need another system to maintain and build updated software on.”)

To see if a compiler exists, check if /usr/bin/cc exists. (Run “ ls -l /usr/bin/cc ”.)

Disk space requirements

This guide may seem to focus quite a bit on checking that there is available disk space. This may not need to be a strong focus if there are many available gigabytes. A reason the guide mentions this is simply because a lack of disk space can cause issues (and was known to, multiple times, when this guide was initially created).

An 8GB drive can be manipulated to be able to compile the OpenBSD base operating system and also the Xenocara (X-Windows) system. However, the default automatic drive layouts(hyperlink note: It would be nice to narrow this down to the section called “Determining some sizes”, although that section is currently in a larger section to be reviewed) may not assign enough space to some of the mount points to accomplish this. Meanwhile, other mount points may have more space. This isn't necessarily an error in OpenBSD's design; it just means that the default disk layout wasn't necessarily made to accomodate this specific purpose. On such a system (e.g. a virtual machine that was given an 8GB hard drive), it may be worthwhile to adjust the disk layout before starting to download source code. As one example, the /home directory may not need to be very large, and the /usr/local directory may be, by default, made larger than what will be needed. (It would be nice to provide some numbers so that people have some idea of what changes to make...) Another (and perhaps more reasonable) option may be to just use a larger (virtual) hard drive.

Other requirements to know about
  • Rebooting the system may be needed.
  • In addition to rebooting, some other steps will also need “superuser” (a.k.a. “system administrator”) access (to install software, and preferably to also be able to write to certain directories).
  • Some of this process (particularly installing X?) may involve not only building the software, but installing the software. In general, make sure this is acceptable on the machine that will be doing the compiling. (It may be possible to compile the operating system kernel without installing it. Other pieces may really benefit from superuser access.)
  • There will be some disk space that will be used up. (As a rough summary, which may benefit from closer inspection to verify, perhaps about 3.5GB may be needed in /usr/ and/or subdirectories such as /usr/src/ and /usr/obj/.) (Feel free to verify that needed directories have enough free disk space before proceeding.) (Some of that 3.5GB may be for Size requirements for /usr/obj/? At one time that was measured to be about 1.2 GB.)
  • One step, OpenBSD FAQ: Building the userland notes, “This is a fairly time consuming step -- a very fast machine may be able to complete it in well under an hour, a very slow machine may take many days.” (This statement may have been made historically, when a “slow machine” typically referred to machines that were much slower than many machines that would be considered “slow” today.)
  • Network bandwidth may also be required to obtain source code (and/or patches used to provide updates to source code).

[#obupwnew]: The best experience, in getting things to compile well, may come by starting with an upgrade of the version of the operating system on the machine that will be building the updated version. (This means that one of the early recommended steps may be to update the compiling computer, so that it uses a more recent version of the operating system as other code gets compiled.) OpenBSD FAQ 5: section on “Upgrading to the closest available binary” (section 5.3.2) (the quote is the description from the hyperlink in section 5.3.1) has more details, but the simple summary are these rules: If the desired end result doesn't involve using “-current”, then start by upgrading to the latest version of “-release”. (Actually, a “-stable” that is at least as new as the latest “-release” should also be sufficient.) Otherwise, in the case that “-current” is desired, consider using the latest snapshot. (That advice is covered by OpenBSD Source Preparations's “Some considerations for -current” and/or Considerations, warnings, and potential problems related to using OpenBSD snapshots (and/or “-current”), both of which are recommended reading for people who are interested in using code from “-current”.)

The entire process of updating everything as much as possible can be broken down into four major steps: creating and installing the kernel, building userland utilities that are part of the operating system, working with X (Xenocara), and working with third party packages.

Note that even computers that aren't going to be needing to display graphical output may still benefit from building the X (Xenocara) binary files. This is documented by the OpenBSD team with OpenBSD FAQ 4.7: Required files, which has a section called “Why do I have to install X for my non-graphical application?” Specifically, this section notes, “Even if you have no intention of running X, some third party packages require the libraries or other utilities in X to be installed on your system.”

Handling permissions
It may be nice to join a group (in OpenBSD) called wsrc. This isn't at all required. Much of the process can be done as a standard user. The little bits that cannot be done as a standard user may be done by temporarily elevating privileges by running software as a superuser. However, to follow a priciple of least privilege, some tasks may be able to be done without requiring full superuser access if the user at least has the needed permissions to write to certain directories. These directories may have a group ownership of wsrc, and may be group writable (or may be easily set to be group writable). So, although this step is absolutely not required, it may be a good, recommended practice to be performing.
[#obsdscgt]: Getting the source code

All the steps described in the OpenBSD source code updates: preparation section should be covered/addressed before proceeding with any following steps. (The “preparation” section is not some sort of general overview for novices, but mentions requirements, and other specific items worth knowing about before proceeding. For example, it references some warnings about common problems that may not be obvious ahead of time. Attemping to save time by skipping that section may substantially, greatly increase the likelihood of problems that will cost much more time.) So, make sure all those steps are covered before proceeding with the following instructions.

OpenBSD's “manual page” about building an OpenBSD release describes multiple steps of building an OpenBSD release. This section of this documentation covers some of the same sort of material as section 1 (“Update sources”) of that guide.

Overview: Documentation showing that the source is needed for patching

As a person downloads an entire operating system, just to apply a small number of updates, it would make some sense to wonder, “Is all this really necessary?” The answer is: if a smooth experience is desired, then, yes. Here is some official documentation that backs up that answer:

OpenBSD FAQ 10.15: the section on “Applying patches in OpenBSD” says, “In general, you should have the entire source tree available.”

(Although OpenBSD FAQ 5: section about Why (not) to compile the system from source (section 5.2) may suggest that building from the source is not desired, that FAQ content is a discussion about the idea of building the entire operating system. In that case, source code may not be required if binary/executable files are available: OpenBSD Binary Updates has more details about such methods. Many methods of updating the operating system do actually involve using source code, as seen by the “Fetch & build” instructions seen in the OpenBSD FAQ 5: section on installing or upgrading to closest available binary (FAQ section 5.3.2) which describes how to reach a specific version.)

Getting the updated code

This is covered in the section about getting OpenBSD source code. The following outline shows some of the more major topics covered.

Loctation(s) for the extracted local copy of the source code

Before obtaining the data, it simply makes sense to consider where the data will go. This is covered in the section about getting OpenBSD source code. (The first section of the text lists where, and the second looks at how to prepare that section, to cause a speed increase when working with files.)

Pre-loading the local source tree

Though pre-loading may take a bit of time, it is either required or, to save time even when just getting the code even once, recommended. Details about performing this task are available, in the section about pre-loading the local copy of OpenBSD's source tree.

Getting newer source code

The precise details will depend on what source code is used.

Getting the latest source code (e.g. of the patch branch)

This is covered in the section about getting OpenBSD source code.

Overview of process

This is covered in the section about getting OpenBSD source code.

Using Anonymous CVS

This is covered in the section about getting OpenBSD source code.

Running the Anonymous CVS software to initially get the new source code

This is covered in the section about getting OpenBSD source code.

Updating source code

This is covered in the section about getting OpenBSD source code.

Using -current

This is covered in the section about getting OpenBSD source code.

This text may initially look much shorter than the text for handling the “-stable”/patch branch. However, that is because it largely refers to the text for handling the “-stable”/patch branch. This text, then, just contains the little bit of information about how the process differs when using “-current” instead.

Getting Errata patches

This is covered in the section about getting OpenBSD source code.

Building newer executable code with the obtained source code

The following guide follows a basic process outlined by OpenBSD's guide to following the “-stable” flavor.

This may be done in multiple pieces, installing and using a new kernel before (making and) installing and using other new binary executables.

Handling Eratta, and a comparison to other options

If the approach being taken is to use the Eratta patches, read this short guide. (For other approaches, some text after this short section will be more applicable/useful.)

While Eratta for a particular “-release” may be rather quick to obtain, applying it (especially automatically) may not be quite so easy. That is why using Eratta isn't presented as being the overall easiest approach to patching.

Handling Eratta

At this point is where the Errata method, so far the easiest method (because getting the errata code didn't involve messing around with learning how to use an Anonymous CVS server), starts to get a bit more complicated. To generate the new source code, patching needs to be applied to the old source code. For specific details about how to apply an individual patch, review the text at the top of the patch file. A bit of a more detailed guide is shown at OpenBSD FAQ 10.15: the section on “Applying patches in OpenBSD”. Basically the goal is going to make sure that all necessary “hunks” succeessfully apply on all files that are affected by a patch. It will be clear when a specific “hunk” is successful, because there will be a message such as “Hunk #1 succeeded at 1815.” (This quoted example is quoting the OpenBSD FAQ section referred to above.)

It looks like this process may become ridiculously time-consuming. Perhaps it would be more tolerable if it was more clear about what is guaranteed to happen if there is an error. For example, perhaps a(n) exit/return code can be checked for. Perhaps running this through a filter, like piping to tee (so details are placed in a log in case an error occurs) and piped through grep, would be an efficient way to check the results. However, the official guide, OpenBSD FAQ 10.15: the section on “Applying patches in OpenBSD”, doesn't seem to provide such details (at the time of this writing).

[#obsdnwkn]: Creating and installing a new kernel file
Customizing the kernel (before it is built)

If there are Errata patches that were downloaded, these instructions assume that they have already been applied (to get the source code updated to the desired version).

This step, customizing the kernel before it is built, can generally be skipped. Details are simply being provided anyway, for those who may be interested in performing such changes.

For these details, see: OpenBSD Kernel Source Modification

Rebuilding a new kernel

There's a few steps to this process, so this process is documented in a seperate section dedicated to the details about (Re-)Building OpenBSD's kernel.

Note: Once the building process starts, consider starting to download the source code for Xenocara if that download hasn't yet started. (After all, this is a multi-tasking operating system.) This might not be wise if there are concerns about running out of disk space when completing the build, but otherwise there may be no reason to put off such downloading. (If such downloading has already been started, then great!)

Configuring the new kernel

Like configuring the source code, this isn't necessarily recommended. However, if configurations are desired, it may make sense to configure the kernel before installing it. If the old kernel had any special configuration, it may make sense to make the new kernel similarly configured. Just be sure, if following any guides to making such a change, to specify the right file to modify. (A guide will likely reference the location of a kernel file being used, but the kernel file to modify will be the newly created file.)

For example, if the hardware clock uses a local time which differs from GMT, the kernel can compensate for that hardware clock if the kernel has been configured to do so. (Details are in the section about setting the clock's time.) If the old kernel does this, and the new kernel does not, then the software that relies on the kernel may experience a notable change in the system's time when the new kernel is started. (Furthermore, such software may be unlikely to be designed to compensate for such a thing.)

To do this, make sure changes are made to the desired file. (Be sure to be in the right directory!) Go to the directory that the OpenBSD kernel was compiled from. Then backup the original file, and make the desired customizations. For instance, if using the cpytobak command, perhaps run:

cpytobak ./bsd
config ./bsd

Further details would depend on what customizations are desired.

Installing and running a new kernel

This will require being a superuser. It will also involve rebooting. See installing the OpenBSD kernel for details.

[#obldulnd]: Rebuilding the userland
(binaries included in base, other than the kernel)
The plan...

OpenBSD's “manual page” about building an OpenBSD release describes multiple steps of building an OpenBSD release. This section of this documentation covers some of the same sort of material as most of section 3 (“Build a new system”) of that guide.

This process essentially follows the guideines provided by OpenBSD's guide to following the “-stable” flavor (particularly the “Rebuilding the binaries” section), which has the same commands as OpenBSD FAQ 5: section on “Building the userland” (section 5.3.5).

[#obsrmobj]: Clearing the object directory

Clearing the object directory: The time has come to prepare to erase all of the object directory that is being used. (That means removing all of the contents of that folder.)

Clearing the correct object directory
Object directory for the OS userland
When building the main operating system “userland” files (not the Xenocara (X) files), the content to clear is /usr/obj/*
Object directory for Xenocara (X Windows)

When building Xenocara (X Windows), time may be saved by leaving the /usr/obj/* content alone.

The content to clear, when building Xenocara (X), a variable called XOBJDIR may be defined in /etc/mk.conf (which may be a file that doesn't pre-exist). Otherwise, the default location that will be used is /usr/xobj/* (not /usr/obj/* ).

Whether to proceed with clearing the object directory

Naturally, the first thing to do, before just blindly following an example that involves deleting all of the contents of a directory, is to have some idea of what is currently in that directory. Chances are it will be empty. If not, then like handling any data which may have been unexpected, it may be good to find out whether there is any need of that data before going around deleting that data. (For instance, in an environment with a team of administrators, is another system administrator using that directory?)

OpenBSD FAQ 5: section on “Building the userland” (section 5.3.5) states, “Note that the use of the /usr/obj directory is mandatory. Failing to do this step before building the rest of the tree will likely leave your src tree in bad shape.” OpenBSD FAQ 5#snake: section about when “make build” fails with “cannot open output file snake: is a directory says, “Building the tree without a /usr/obj directory is not supported.”

Reserve the directory

OpenBSD FAQ: Building the userland notes, “This is a fairly time consuming step -- a very fast machine may be able to complete it in well under an hour, a very slow machine may take many days.”

Also note that after the build is complete, it may be desirable to keep the contents of this directory. OpenBSD FAQ 5: section on Building a Release (section 5.4) notes that after this upcoming build step is complete, it may be undesirable to delete the /usr/obj/ directory before a release is made. (The reason it would be undesirable is mainly since this build step may take some time. If /usr/obj/ is a memory disk (a.k.a. “RAM drive”), rebooting would effectively delete the contents of the directory. So, if /usr/obj/ is indeed a memory disk, then making a release is something to try to take care of before rebooting. Or, if rebooting needs to happen before making a release, perhaps time could be saved overall by just copying the contents from the memory disk, onto something that isn't a memory-based disk, before rebooting.)

If it is possible to reserve this directory for this purpose, do so. (In a multi-user organization and on a machine where the build could take days, this may mean trying to reserve the system in a way that other people in the organization won't be using this directory. Of course, details on how to make such a reservation, which will affect things organization-wide, may vary between different organizations. For a fast system, used only by a single user, this may simply mean planning to personally not use the directory, for any other purpose, until its contents are no longer needed.)

[#usrobmtp]: Having /usr/obj/ use a special mount point

OpenBSD FAQ 4.8: Details on how to partition a disk discusses the possibility of having a specific mount point for /usr/obj/. The provided reasoning is as follows: “This directory is populated during the build process with the object and binary files. Having this directory a separate mount point allows it to be formatted rather than erased file by file, which can be significantly faster.”

It might be true that making the directory be a mount point will take no effort at all, because perhaps it already is a mount point. (At least some versions of OpenBSD, starting with version 4.8? will create that mount point, if the mount points are decided upon automatically during the installation process, and if the automated process determined that there was sufficient room.)

Before making a separate mount point, it may be sensible to first check what the contents are of any existing /usr/obj/ directory.

It may be good to check how much free space there is. (This is true even if the directory is pre-empty: it will need to be big enough. (Details on what the required size is, may come here at a later time.)

[#usrobjsz]: Size requirements for /usr/obj/
Some documentation (which might be third party) related to OpenBSD have shown using a 256MB partition, but an actual /usr/obj/ after building OpenBSD/amd64-stable 4.8 was a much larger 1.2GB. As growth is predicted (OpenBSD's Anonymous CVS Guide: section about setting up a mirror indicates expected growth in code), a long-standing precise minimum size is not sensible to predict. If there is any doubt about how big to make such a partition, and if there is plenty of real disk space available, then it may probably be best to just use disk space rather than risk running out of space. (That is even more likely to be true when resources are abundant, and especially if creating a temporary disk area, such as a “memory disk”, that can easily be destroyed and which probably will be, automatically (when the system reboots).)
[#obsobmfs]: Using a “memory disk” (a.k.a. a “RAM drive”)

OpenBSD FAQ 5: section on Building a Release (section 5.4) does refer to the possibility of /usr/obj/ being a “memory disk” (a.k.a. a “RAM drive”) “for a little extra performance in the build process”. If there is sufficient memory, that may be something to consider doing.

The following is some info about making a memory disk. It may not have been personally verified, by the author of this text, yet. (Some of this information may be getting moved to a section about RAM drives (and memory filesystems).)

The /etc/fstab entry may look like:

/dev/wd0b /usr/obj mfs rw,async,-s=1 0 0

One part of the following example, which will need to be updated, is the -s=1 which determines the size. The number after the equal sign is the number of 512 byte sectors, and one is certainly insufficient. (See Size requirements for /usr/obj/ for details.)

The above assumes the swap partition is at /dev/wd0b (which is the common location for non-SCSI drives). /usr/obj is where the drive is mounted. The second parameter after the file system type, which is the first 0, indicates a dump level of zero. Like the reasoning behind using asynchronous mounting, the assumption is that all data on the drive will be completely unusable if the system shuts down uncleanly, so there's no harm in using these options that limit data recovery options.

Allocating disk space for /usr/obj
If having /usr/obj fit into memory doesn't seem feasible, disk space may be allocated to this. This will require sufficiently reserved free space with whatever sort of disk layout (a.k.a. partitioning scheme/layout) is being used. If such sufficiently-sized space isn't readily available in the partition layout, then taking this approach may be require more effort, and time, then what the speed payoff will be worth. If there is interest in taking this approach anyway, note that any existing /usr/obj/ directory (on disk) that may pre-exist may be best to move/rename (before trying to use a mount point at that same location), and then see the section about making a filesystem volume has details that may help with the process of taking this approach.
[#clrusrob]: How to clear the object directory (e.g. /usr/obj/*)

If not a superuser, check to make sure that permissions are suitable. If not, then either join a group (in OpenBSD) called wsrc, or plan to use sudo as needed.

See see what is in the directory. Run:

ls -la /usr/obj/.

This will likely show at least two filders, “.” and “..”. Example output may be something similar to:

drwxrwxr-x ### root  wsrc   #### Mon DD HH:MM .
drwxrwxr-x ### root  wheel  #### Mon DD HH:MM ..

If that is all the information that is shown, the job is done. If there are additional lines, the content may be in the way, and so should be backed up as needed (if needed), and then deleted.

Determine whether to do things using the method which may be more complicated, but is likely to be faster. That method is to clear the entire mount point. The other method which may be much slower, but which is more certain to work simply, is to delete the files. So, choose one of the following methods.

Clearing the entire mount point

See if the /usr/obj/ is a mount point. To do this, run:

mount | grep /usr/obj

If this directory is not mounted, then perhaps it is simply an unmounted mount point. To see if that is the case, find the relevant line that describes the mount point. Do this by running:

grep -i /usr/obj /etc/fstab

If the partition looks available, but unused, try to mount the partition and then check out whether it offers enough unused free space to be worth using.

If there is still no sign of a mount point called /usr/obj, then consider whether to make /usr/obj/ into a mount point. (The section about using /usr/obj/ as a mount point may have some details about options, including using a memory filesystem to pull this off without requiring changes to any partitions on the disk.) The rest of this method requires that /usr/obj/ is already set up as a mount point, so if it isn't, discontinue using this method of clearing that directory. Instead, see the other option about removing the file system objects located at /usr/obj/*.

It is at this point that the guide is make a significant and possibly untrue and problematic assumption. That assumption is that the mount point wasn't mounted with some sort of fancy options that aren't listed in the /etc/fstab file. (If that assumption is not true, either abort this method or plan to manually work around the issue.)

Make a note of which filesystem object is mounted. There are a few ways to determine which filesystem object is mounted. This is an item located in /dev/, and is in the first column from the results of the mount command (with no parameters) if the device is already mounted, and is the first item in the related line in the /etc/fstab file.

If the mounted object is /dev/wd0m, the following commands will unmount, make a filesystem volume, and then mount the newly created filesystem volume.

umount /usr/obj
newfs /dev/wd0m
mount /dev/wd0m /usr/obj

OpenBSD FAQ 5: section about having /usr/obj/ on its own partition shows this method. The example on that site does not reference the device to be mounted as part of the mount command line. That is probably okay, since specifying that device on the command line is optional if the necessary information is properly in the /etc/fstab file.

[#rmobjfil]: Removing the file system objects located at /usr/obj/*

(If /usr/obj is not a mount point, it may be desirable to make /usr/obj/ into a mount point. The reason is for a potential speed increase.)

ls -lda /usr/obj/*
echo WARNING: Consider the effects of the next command before running it.

Now, one option is to do things the way it is documented in OpenBSD FAQ 5: section on “Building the userland” (section 5.3.5) and OpenBSD's guide to following the “-stable” flavor. To do that, simply run the following code:

sudo rm -rf /usr/obj/*

Another method is more complicated: OpenBSD's “manual page” about building an OpenBSD release, in section 3 (“Build a new system”, an alternative similar to the following is provided:

cd /usr/obj && sudo mkdir -p .old && sudo mv * .old && sudo rm -rf .old &

The main assumption of this approach is that moving directories is faster than deleting the directories (and all of their contents), and that may be done in the background (as indicated by the final & character). An assumption seems to be that one may continue, and so the directory may be used, and that there won't be any problems caused by a race condition if a new release gets made. The assumption is probably safe smiply because Technically it seems this may introduce a race condition (if a release gets made that could include the useless .old directory).

Making some more things before the main OS userland binaries
Making/rebuilding the obj directory: Rebuilding the symbolic links

The next goal is to make the /usr/obj/ directory. (OpenBSD FAQ 5: section on “Building the userland” indicates the goal of this step is to “rebuild symbolic links”. Failing to do this is “a bad thing”, as noted by OpenBSD FAQ 5: section called “Oops! I forgot to make the /usr/obj directory first!” (section 5.11.4).) Comments at the top of the /usr/src/MAKEFILE for OpenBSD system source code may also reference this step.

cd ${BSDSRCDIR:-/usr/src}/.. 2>> /dev/null || cd /usr/
sudo time make obj || echo Err=$? | tee /tmp/mkobj.txt
{ sudo time make obj ; echo Err=$? ; } 2>&1 | tee -a /tmp/mkobj.log

This might take a little while. Times are hard to predict over the long term, because source code changes and systems get faster over time, but this has been seen to take under 30 seconds.

If there is a problem with running the time command or the tee command or writing to /tmp/ then feel free to leave off the pipe and everything after it and also the reference to using the time command : a simple “ sudo make obj ” will likely be just as sufficient, but will just be less verbose.

Making more directories

The next command line is heavily based on a command line shown by OpenBSD FAQ 5: section on “Building the userland” (section 5.3.5) and also OpenBSD's guide to following the “-stable” flavor. It is a bit advanced, by using a compound command and setting a variable with env. (If the cd command encounters a non-existant folder, it will return a non-zero value and the env command will not run.) OpenBSD FAQ 5: section on “Building the userland” (section 5.3.5) indicates the purpose of this is to “Make sure all the appropriate directories are created.”

cd ${BSDSRCDIR:-/usr/src}/etc && env DESTDIR=/ sudo make distrib-dirs
echo $?

The env DESTDIR=/ bit is probably why, after the build, OpenBSD's manual page for “release” says, “At this point your system is” ... “running the code”.

Building the main OS code

Once again, free disk space may be worth checking. Checking free disk space may not be needed if there was a gigantic over-abundance before, but if the amount of free space was in the realm of gigabytes, rather than hundreds of gigabytes, it may be worth checking again. Some data may have been added since the last time this was checked. It also may be worth documenting, so that we may then see how much space is used by just the next step.

Now, the part which will take a while...

cd ${BSDSRCDIR:-/usr/src}/. 2>> /dev/null || cd /usr/src/
sudo time make build || echo Err=$? | tee -a /tmp/mksysbin.txt
df -hi 2>&1 | tee -a /tmp/mksysbin.log
{ sudo time make build ; echo Err=$? ; } 2>&1 | tee -a /tmp/mksysbin.log

Like other examples shown before, the reference to the time command, as well as the pipe character (“|”) and everything after it, are simply to provide extra information. There's no harm in trying to remove them if that helps simplify things.

The OpenBSD's “manual page” about building an OpenBSD release does have a slightly different syntax: instead of running “ make build ”, it suggests running “ make SUDO=sudo build ”.

Follow-up steps
Merging directories

Some updates, like binary executables located under /usr/, should probably be getting updated. Other updates, such as files in /etc/, may often involve customized files, and so such files may need a bit additional care (beyond just blindly updating to the default version of the file bundled with a very recent copy of operating system code).

Check out the first paragraph of the “Determining whether or not to merge” section. After that paragraph, if the rest of the section should be followed, then be aware of this cautionary note: This series of steps may not have been fully tested/verified by the author of this text. If following these steps, please review carefully and determine on your own how to proceed with the process.

Determining whether or not to merge

If upgrading to a “-stable” (from the “-release”; or from an earlier “-stable” where both the old and new “-stable” versions are updates from the same “-release” version), do not bother with this step. OpenBSD FAQ: Building the userland (FAQ 5.3.5) says, “If following -stable after a proper upgrade process or a install of the proper starting binary, this step is not needed or desired.”

There are some instructions, which may or may not be worthwhile to follow, at OpenBSD's “manual page” about building an OpenBSD release. After discussing previous steps (like running “ make SUDO=sudo build(Is that fully marked up with CSS properly?), these instructions then say, “Update /etc, /var, and /dev/MAKEDEV, either by hand or using sysmerge”. (The quote references the OpenBSD “manual page” for sysmerge (from manual section 8, “-current” flavor).) However, before attempting to perform any such action, consider whether it is desired.

If the desire is to make a self-sustained collection of installer files that can be used to install the operating system, then the following directions for using sysmerge may be nice. However, if the plan is to make such installer files solely so they can be used on machines to update their existing copies of OpenBSD to “-stable”, then perhaps the contents of /etc and /var, and the /dev/MAKEDEV script, may not even be desired. The process of upgrading may specify that those directories should not be updated. Putting in extra effort to include those files may be a waste of effort, especially if that action later requires additional extra effort to ensure that those files are skipped.

It seems that the purpose of sysmerge may be to apply updates. If that is the case, running sysmerge might not be needed if the goals are to simply make a release, and then to install from that release.

Update information
OpenBSD FAQ 5: section about “the best way to /etc, /var, and /dev pinpoints the responsibility for making changes to these folders.
Info/Warnings related to using sysmerge

It does appear that DESTDIR is one of the OS environment variables recognized by the sysmerge command. That may be good to know if that environment variable has a value set (perhaps by reading directions for a later part of the process, such as the directions at OpenBSD FAQ 5: section on “Building a release” of OpenBSD (section 5.4).) There are other enviroinmental variables that could be used, although they may be less likely to have an undesirable impact (e.g. $VISUAL), or be less likely to have an unintentionally incorrect setting (e.g. $MERGE_CMD).

OpenBSD 4.4 Upgrade: sysmerge info recommended against using sysmerge with user databases. (Later on the same page, underneath OpenBSD 4.4 Upgrade guide: Section 3b: Using sysmerge has some more details.)

OpenBSD Journal Entry introducing sysmerge explains that this OpenBSD-specific tool was derived from a port called mergemaster.

Handling /etc
After reviewing this closer, it appears these instructions may affect more than just /etc/. (It may be good to have this section be reviewed by site staff, to clarify that.)

OpenBSD “manual page” for sysmerge says, “It is HIGHLY recommended to back up the /etc directory before running this script.” Performing such backups is a good idea in general; it may especially be a good idea when the (electronic) manual that comes with the operating system makes a special recommendation to do so. To copy such a folder:

sudo mkdir /etc_bak
sudo cp -Rp /etc/. /etc_bak/.

The following about /etc/localtime and /var/db/sysmerge/ may not have been fully verified/tested: Consider: ls -l /etc/localtime and ls -l /var/db/sysmerge/ Also, consider backing up those items.

Seeing the files that may require manual merging

These commands are not strictly required, but are mainly done for confidence-building, to help ensure that there isn't some sort of unpleasant surprise such as finding out that the changes seem to unexpectedly affect hundreds of files.

Generating an automated report

To run sysmerge safely, without making changes to most files (although it appears to be true that contents of /var/db/sysmerge/ may change, and a temporary directory may be created and may remain for review), run the following:

sudo env DESTDIR=/ sysmerge -bds /usr/src

(The “s /usr/src” is not required, because that is the default. However, using a parameter to specify the location of the new source code will be needed if a diffrent directory is being used.)

This will generate a report of what files sysmerge believes will need manual processing. A directory is made that requires superuser permissions to view. The sysmerge program will report the location of that log file. Here is an example of the output of that command:

$ sudo env env DESTDIR=/ sysmerge -bds /usr/src
===> Creating and populating temporary root under
===> Starting comparison

===> ./etc/ssh/sshd_config will remain for your consideration

===> ./etc/group will remain for your consideration

===> ./etc/hosts will remain for your consideration

===> ./etc/dhclient.conf will remain for your consideration

===> ./etc/sysctl.conf will remain for your consideration

===> ./etc/master.passwd will remain for your consideration

===> ./etc/sudoers will remain for your consideration
===> Comparison complete
===> Making sure your directory hierarchy has correct perms, running mtree
===> Manual intervention may be needed, see /var/tmp/sysmerge.eW2G8/sysmerge.log
===> When done, /var/tmp/sysmerge.eW2G8/ and its subdirectories should be removed

The above shows a randomly-generated directory name. (Running that sysmerge command twice in a row will generate different temporary directories.) References to ./ (and subfolders) appear to reference that directory which is created. (The directory that the sysmerge is run from is unimportant as it has no effect whatsoever.) Permissions on the created directory, which is owned by root, allow only the owner to read, write, and view the contents of that directory. The directory has the sysmerge.log file that sysmerge programs. Also in that directory is a subdirectory called temproot which has other subdirectories (and a single symbolic link) that have names like those found in a typical top-level (“/” folder.

The sysmerge.log file will show files which did not get successfully merged. Here is a sample of the file that is created if the command's standard output looks like the above example:

===> File(s) remaining for you to merge by hand
Interactively checking out changed files

To see these changes more interactively:

sudo env DESTDIR=/ sysmerge -ds /usr/src

The sysmerge program will show the differences between a file. The file will then be shown in a pager. (If the file is long enough, the pager may not quit automatically. Most commonly the pager may be left by pressing the lowercase letter q.) Then, here is an example of some output that shows up after the pager quits:

  Use 'd' to delete the temporary ./etc/ssh/sshd_config
  Use 'i' to install the temporary ./etc/ssh/sshd_config
  Use 'm' to merge the temporary and installed versions
  Use 'v' to view the diff results again

  Default is to leave the temporary file to deal with by hand

How should I deal with this? [Leave it for later] 

The above is a perfect example of why not to just blindly apply updates. If the security of OpenSSHd was enhanced by manually adding an AllowGroups line, such a change could revert the software back to its initial state. Another possible problem, with another example file, is accidentally reverting /etc/sudoers. Basically, the concern/threat is the reversal of all configuration changes made since the operating system was installed. To avoid this problem, just press Enter to accept the default.

To continue to manually review proposed changes without making any of the proposed changes, repeat that process as needed for each file. If this is done on a recently-installed operating system, viewing each file may feel like a history of actions that were personally taken to customize the system. To keep avoiding the loss of such customizations, keep choosing to leave things for later.

As a point of clarification, the files listed are not the ones that would be changed if sysmerge were allowed to make changes. The files being mentioned are the files that have been altered manually, and so these are the files that sysmerge feels a need to check with the end user. Other files, the files which are not listed in the section of the sysmerge.log, are files that sysmerge detected no need for manual handling. Therefore, the conclusion of sysmerge is that, if it is run with command line parameters which will allow it to make changes, is that sysmerge will be able to handle all other files automatically.

If sysmerge seems to have listed every file that had a customization made, then sysmerge is probably right. (There may, of course, be exceptions: since hash values are used there is the possibility of a collision.)

Automatically merging what can be merged

This section may still need to be verified

If the list of files to be handled manually is complete (meaning that all files listed are the only files that may need manual handling), proceed to make changes to the other files (which are the files that do not need to be handled manually). This may be done rather automatically with -b, although a point of clarification is deserved before recommending to run sysmerge with the -b parameter: Running with -b but without -d causes sysmerge to remove the directory it uses to perform work. The OpenBSD 4.8 “manual page” for sysmerge says that the -b causes sysmerge to be “saving differing files for later manual processing.” However, “saving” the file from immediate processing just means that it isn't part of the group of files that will be processed. This does not mean that a copy of the file gets saved, nor that there is any other useful report that shows what files were skipped. In fact, what sysmerge may decide to do is to just delete its work directory. There is no log file left in that directory which gets removed. The lack of detail in the output is shown in the example output.

Before any sample output, here is a clear, short example of the command line that will merge what sysmerge detects as automatically mergable.

sudo env DESTDIR=/ sysmerge -bs ${BSDSRCDIR:-/usr/src}/.

(The “s ${BSDSRCDIR:-/usr/src}/.” may not be required, because /usr/src is the default. However, if a different directory is being used, then using a parameter to specify the location of the new source code will be needed.)

Here is some sample output of the above command.

===> Creating and populating temporary root under
===> Starting comparison
===> Comparison complete
===> Making sure your directory hierarchy has correct perms, running mtree
===> Removing /var/tmp/sysmerge.dYxUa

In that example, do understand what is being said by the last line of that output. The sysmerge command will completely delete the directory that it reported earlier.

OpenBSD FAQ 5: section about “the best way to /etc, /var, and /dev has some text describing some common types of changes that are made to /etc/.

Handing /var
(This section may be similar to the section about handling /etc/.)
Handling /dev/MAKEDEV

Unknown: After running “ sudo env DESTDIR=/ sysmerge -ds /usr/src ”, it was found that /var/db/sysmerge/srcsum started with a reference to ./dev/MAKEDEV so perhaps a process that handles /etc also handles /dev/MAKEDEV ???

OpenBSD FAQ 5: section about “the best way to /etc, /var, and /dev suggests a manual copy.

sudo cp -p /dev/MAKEDEV /dev/MAKEDEVBAKYYYYMMDDhhmm
sudo cp -p /usr/src/etc/etc.$( machine)/MAKEDEV /dev/.
cd /dev
sudo ./MAKEDEV all

Note: if there is some problem with using these command lines:
cd /dev
sudo ./MAKEDEV all

then check the permissions of the /dev/MAKEDEV filename, and/or run the following commands:

cd /dev
sudo ksh
. ./MAKEDEV all

That approach will use the read permissions to read each line and then execute each line, instead of executing the file. Essentially the same results happen (the file gets run), but the different approaches use different file permissions for accomplishing the same task.

[#obmkrel]: Building a release

OpenBSD's “manual page” about building an OpenBSD release does state, “You must be root to create a release”.

(Note that the following steps may be designed to use files that were created in earlier steps. Simply starting the process at this point is not expected to work. Please see the earlier steps in this guide to make sure that any required files pre-exist.)

[#obmkrlvn]: Checking that vnode disk #0 is being unused
Note: Is this text old, outdated, meant for vers prior to OpenBSD 5.0?

It is required that “the first” snvd “device, svnd0” “must not be configured”. (So, snvd0 is used by the process of making a release. Another example of what may use an snvd device in OpenBSD is mounting a CD ISO image.) To check if the snvd0 device is currently configured, use:

sudo vnconfig -l

The parameter at the end of that command line is a lowercase L, like the word “list”. (The above command line provides some unnecessary information: the status of other svnd* devices. That is probably okay: with a default setup there may be just four such devices. Alternatively, adding “ svnd0 ” to the end, so running “sudo vnconfig -l svnd0 ”, would limit the output to only the first device, /dev/svnd0. That would be okay, since information on the first device is really the only information needed for this particular process.)

Example output:

vnd0: not in use
vnd1: not in use
vnd2: not in use
vnd3: not in use

This device is probably available unless there were manually-performed efforts to use it. (For the curious, an example of a task that uses that would be mounting a CD image (in OpenBSD).)

If the svnd0 file is in use, consider whether it is safe to release it. For example, run:

mount | grep /dev/svnd0

If there is a device mounted, and if it is determined that the device is no longer needed, then run:

umount /dev/svnd0
vnconfig -u svnd0

(The above unmounts the /dev/svnd0 device. Since the umount command accepts either syntax, specifying a device in that manner is equivilent to specifying to the mount point which uses the device.)

[#obmkrldr]: Preparing directories, including setting required environment variables

OpenBSD FAQ 5: Building a Release (section 5.4) mentions that /usr/obj should have remained undisturbed since an earlier building process was created.

Setting some required variables

There are other directories that need to be used, including two directories that should be pointed to by some environment variables. Those environment variables are quite likely to not be set unless this process has been done before, so be sure to check what they might be set to. (The names stored in these variables do not need to be names of actual directories that currently exist yet; they may be actually created soon.)

The names of these variables are:

This is where the installation files will be created. This is where the most useful output will be getting stored. The directory can be empty.

Note that this variable name in particular may also be used by some other programs. It is used by OS environment variables recognized by the sysmerge command, but also is mentioned in some other documentation related to ports: “ grep -iR DESTDIR | grep -v PKG_DESTDIR ” may return over 50 results (from a smaller number of files). Huh? Was the first grep command supposed to reference /usr/ports or something? Nothing is provided on the command line about what contents to search. Note: a similar paragraph may exist elsewhere in this text file: search for PKG_DESTDIR as needed, and update that one too.)

At the start of the process, this directory may be empty, or, rather, it probably should be empty. If it isn't empty, check out some steps to erase the contents of the directory pointed to by ${DESTDIR}.

This location is used by the process as a place to store the data that will be going into the archive files that get placed into the location specified by the RELEASEDIR variable. At some points of the process, the contents of this directory may look similar/identical to what is wanted on the “destination” when the release gets installed. (Later on, installing the release may be done on other machines, using the files that will go into the location specified by the RELEASEDIR variable.)

Like a temporary directory, the contents of this directory should not be something to expect to remain available. There are multiple points of time when the directory may, or (more certainly) will, be removed (at least when following some of the official guides). Therefore, don't count on its contents sticking around.

These variables are not just for convenience of documentation: the Makefile (which creates the release data) refers to these variables by name.

The examples at OpenBSD FAQ 5: Building a Release (section 5.4) and OpenBSD's “manual page” about building an OpenBSD release both start by setting these environment variables. (Details on how to set these variables may vary based on what shell is being used: see operating system environment variables for details about setting variables.) The variables are set to the full paths of directories (not ending with a directory-separating slash). For the convenience of documentation (and possibly to reduce typing), interactions with the directories, like making sure that they exist, is something that is done after setting the variables.

The following example line shows current values (followed by a character so that the end may be recognized).


If changes need to be made, then make whatever changes need to be made. For instance, the following may work with the Bourne shell (and compatible shells).

export DESTDIR=/tmp/destdir
export RELEASEDIR=/usr/rel

Neither of the paths pointed to by these variables may refer to /mnt/ (or any subdirectories underneath there). Anyone unfortunate enough to have selected a place under /mnt/ will simply need to choose another, different location that does not include /mnt/.

Initial state of the directories
DESTDIR mustn't be populated

Since the location specified by DESTDIR needs to be an empty directory, the next step to take (shown by OpenBSD FAQ 5: Building a Release (section 5.4) and OpenBSD's “manual page” about building an OpenBSD release) is to delete that directory.

(Deleting all of the contents of this directory is done by OpenBSD's “manual page” about building an OpenBSD release section 4: “Make and validate the system release”. However, the same guide also says in section 6, “Make and validate the xenocara release”, that “the existing contents of DESTDIR will be removed”. So simply expect that information stored in that directory may be considered temporary, and may be removed.)

Specifically, the approach taken by both of these sources of documentation are to move the directory, which will presumably occur quickly, and then to delete the moved directory. Removing that directory is done in the background. (For the temporary directory which will be getting deleted, OpenBSD's “manual page” about building an OpenBSD release appends a simple hyphen in its example.) OpenBSD FAQ 5: Building a Release (section 5.4) shows this being done with the following example:

test -d ${DESTDIR} && mv ${DESTDIR} ${DESTDIR}.old && rm -rf ${DESTDIR}.old &
Notes about the directory pointed to by RELEASEDIR

Don't feel a need to erase valid files from the directory that RELEASEDIR points to.

OpenBSD FAQ 5: Building a Release (section 5.4) notes, “RELEASEDIR does not normally need to be empty before starting the release process, however, if there are changes in the release files or their names, old files may be left laying around. You may wish to also erase this directory before starting.”

If this is the first release being made, using an empty directory is probably a good idea.

(If these same directions are being followed to build X (Xenocara) after a previous release has been made, then a new directory could be used. Another option, that may make just as much sense, is to point to the directory where the base operating system files were made. Either approach can work fine for the directory that RELEASEDIR points to. It is DESTDIR that needs to have the contents removed.)

Making required new, empty directories

Now, we'll go ahead and make sure the directories exist. (OpenBSD FAQ 5: Building a Release (section 5.4) says to “create the directories if needed”, but the statement “if needed” doesn't apply to ${DESTDIR}. Of course this is needed for ${DESTDIR}: The very previous command made this needed by deleting the directory.)


(Since /usr/ might not specify that write permissions are available to anybody other than root, be sure to use sudo as needed for the mkdir command to work.)



Nicely, the above example output does not show any contents in the directory pointed to by the DESTDIR variable.

Making the release

Once the previous preparation steps are performed, the process of actually making the release varies a bit between making a release for the main operating system, or building a release for X (Xenocara). Here is the process for making the operating system.

Notes about crunchgen

As a side note, OpenBSD FAQ 5: Building a Release (OpenBSD FAQ section 5.4) says, “The release process involves a utility”, crunchgen. That's it: crunchgen is used. The FAQ does not go into significant details about how crunchgen is used.

Fortunatley, this doesn't seem to actually complicate the process of making the release. However, for clarity, some details are hereby provided (mainly to help satisfy any resulting curiosity).

Having completed the process, it seems likely that an executable made with crunchgen will end up being an executable designed to start by running the crunchgen program. executable.) (More details about crunchgen are available on OpenBSD's manual page for crunchgen.) (The quote from the OpenBSD FAQ goes on to describe crunchgen a bit: The crunchgen command “is used to create a single executable file made up of many individual binaries. The name this single executable file is invoked by determines which component binary is run. This is how a number of individual program files are squeezed into the ramdisk kernel that exists on boot floppies and other boot media.”)

Making the release (using make)

Here is the command that performs the core actions of creating a release file. (Go ahead and run it.)

Old method follows:
cd /usr/src/etc && sudo time make release || echo Err=$? | tee -a /tmp/mkrellog.txt
Newer method follows:
cd ${BSDSRCDIR:-/usr/src}/etc && { sudo time make release ; echo Err=$? ; } | tee -a /tmp/mkrellog.txt

The reason for having multiple command lines separated by && is because the second command won't run if the first command, for some reason, indicates a failure. (So, if the directory doesn't exist, and so $? is set to a value of 1, then the next command won't run.) If the above syntax causes an issue with the shell being used, these commands could just be run on separate command lines: in fact, that is how OpenBSD FAQ 5: Building a Release (section 5.4) shows the commands. However, if taking that approach, be sure to check the results of the first command before blindly running the second command.

Checking the release

Naturally, first check that the error level/return value/exit code was zero. If disk space was tight, it may be worthwhile to verify that disk space is free again.

Files should now exist in the release directory: Use:


At this point of time, the contents of ${DESTDIR} should be a bunch of directories with names like what is typically found in an OpenBSD system's root directory. (There may also be a symlink called ${DESTDIR}/sys/ which points to the ${DESTDIR}/usr/src/sys/ subdirectory.)

After the release is made, OpenBSD FAQ 5: section on Building a Release (section 5.4) notes, “After the release is made, it is a good idea to check the release to make sure the tar files are matching what is in the DESTDIR.” A command, called checkflist, may have been obtained with the source code, and may help to verify the contents of the created files. For example, when checking the main operating system's files, the following should work if using the default command line shell of ksh:

New method follows:
df -h | tee -a /tmp/chkflst.txt
cd ${BSDSRCDIR:-/usr/src}/distrib/sets && { sudo time ${SHELL:-sh} checkflist ; } | tee -a /tmp/chkflst.txt

If ksh is not being used, a bit of a simpler syntax may work better. For example, running the following instead may be useful:

Old method follows (to keep available as a publicly visible reference):
df -h | tee -a /tmp/chkflst.txt
cd /usr/src/distrib/sets && sudo time sh checkflist | tee -a /tmp/chkflst.txt

(Like the earlier command of building the file, the approach used by OpenBSD's “manual page” about building an OpenBSD release is to use && to cause a conditionally compound command, although the commands could be separated.)

About the checkflist command, OpenBSD FAQ 5: section on Building a Release (section 5.4) notes, “The output of this step should be very minimal.” It is entirely possible that sh checkflist may output an astoundingly whopping zero bits of output.

Some follow-up tasks

After creating the release (and verifying that it was successfully created), there might not be any further need for the contents of the $DESTDIR folder. (This is quite likely to be true if the goal of doing all of this compiling was just to create a release.) It may, of course, be re-created again if necessary (based on still having, or being able to re-obtain, the source code that was used when creating the release). In that case, make sure the source is sufficiently backed up (or otherwise re-obtainable, as desired), and then feel free to remove the directory pointed to by the $DESTDIR variable.

As a final step, OpenBSD FAQ 5: section on Building a Release (section 5.4) notes, “if you wish to distribute the resultant files by HTTP for use by the upgrade or install scripts, you will need to add an "index.txt" file, which contains the list of all the files in your newly created release.” To do that, follow the the steps for making an index.txt file containing information about the installation files.

[#obidxins]: Having an index file that contains details about these installation files

Find out if write access is available.

ls -ld ${RELEASEDIR}

To have an index file that contains details about the installation files, follow one of the following processes:

Steps to take if there is write access to the ${RELEASEDIR}/. directory
/bin/ls -1 ${RELEASEDIR}/. | tee -a ${RELEASEDIR}/index.txt >> /dev/null

The first command line paramater in that example is a hyphen followed by a digit, the number one.

Steps to take if ${RELEASEDIR}/. is not writable

First of all, why is the ${RELEASEDIR} variable pointing to a location with limited write access? That's rather beside the point of how we may proceed in this situation, but this situation probably does not need to happen in the first place. This may make sense if the ${RELEASEDIR} points to /usr/rel directory (which is what was shown in the examples), but it is still not completely necessary. If the directory has an owning group of “wheel”, consider making it group writable (which may or may not take care of a lack of permissions, depending on what the current username is). Otherwise, if there is sufficient free space than ${RELEASEDIR} can just be pointed to a directory that is easier to write to, such as ~/reldir/

Other options may be to just elevate privileges (using sudo), or to follow the following steps:

We need the same output that is used in the example of what to do as root.

/bin/ls -1 ${RELEASEDIR}/. | tee -a /tmp/index.txt >> /dev/null

The first command line paramater in that example is a hyphen followed by a digit, the number one.

Actually, the output directory was meant to be ${RELEASEDIR}/. However, in the above example, the specified output directory was /tmp/. That was done just in case the command was being run as a user who doesn't have access to write to the ${RELEASEDIR}/ directory.

The intent, however, is to have the output go into the ${RELEASEDIR}/ directory. The directions so far have managed to create an index.txt file with the needed information. However, we still really don't have the information where it needs to go. To take care of that, write access is needed. Then a process like the following may be used:

sudo ksh
cat /tmp/index.txt | tee -a ${RELEASEDIR}/index.txt
echo To leave the subshell that was entered earlier by
echo earlier ksh command, run the following command.

Following are simply some notes for those who might question the complexity of this approach. First of all, simply using the mv command (or, more precisely, “sudo mv”) would only have worked well if the output file didn't pre-exist. This process may be repeated if additional installation files are added to a pre-existing collection (such as if installation files for X are added to installation files for the base operating system). In such a case, it may be desirable to append to an existing output file.

A second question people may have about this approach is why the results of the first command on the command line (either the ls command line, or the cat command line) aren't just piped straight into a sudo command that then runs tee. (If the first command had tee, then the second command wouldn't even be needed.) The answer is, that would work fine as long as sudo doesn't prompt the user for a password. In many cases, sudo may be configured not to prompt for the password, and then there is no reason why that approach wouldn't work well. If sudo does prompt for a password, then the piped output would be eaten up by sudo's password prompt. The resulting problem isn't just that the needed output doesn't get to the tee command. The problem that occurs first is that the tee command would never run because the user would not be able to enter the password.

To eliminate these problems, it was decided to just use the sudo command in a separate command from the one that redirects output.)

Many of the example commands, shown in this guide to the process of building OpenBSD from source code, involved demonstrating using tee to create an optional output file. Unlike many of those examples, in this case, the results of the tee output are essential for the intent of why this command is being run. OpenBSD FAQ 5: section on Building a Release (section 5.4)) recommended making this file, and did so using a simple redirection instead of using the “ tee ” command. That approach works fine if the user running the command has sufficient privileges to write to the output directory.

Review the release
[#obsckrel]: Check release files's “blog post “upgrading to OpenBSD-current, the stupid way” (a posting which deserves extra points for the humorous title) provoked this idea:

pwd | tee /tmp/mypwd
time cksum -c ${RELEASEDIR}/SHA256
cd $( cat /tmp/mypwd )

This may create several lines of output similar to the following:

(SHA256) base49.tgz: OK

Really, there shouldn't be any problems. (To rephrase, it is not anticipated that problems are likely to occur from the check just performed.) Therefore, this guide does not have substantial details about addressing problems if they do occur.

Final step(s)

As noted earlier, the contents of the DESTDIR folder are no longer needed. (However, the contents of RELEASEDIR probably ARE desired, since that is what so many of these steps were designed to create. So don't delete that!) If this hasn't yet been done, removing the contents of the DESTDIR directory that was used may be a good idea, so that useless files aren't sitting on the filesystem volume. (Removing the contents may not be a required step to get things to function, but properly implementing the task is a recommended idea.)

Note that the variable name of DESTDIR may also be used by some other programs. (It is used by OS environment variables recognized by the sysmerge command, but also is mentioned in some other documentation related to ports: “ grep -iR DESTDIR | grep -v PKG_DESTDIR ” may return over 50 results (from a smaller number of files).) So, unless there are plans to use that directory again in the future, it makes good sense to eliminate some potential troubles (by human(s) who may become confused, and/or by software) by unsetting the variable. It may be a good idea to:


There is likely no further need for its current value. Some other programs related to ports may actually use that hard-coded variable name.

OpenBSD's “manual page” about building an OpenBSD release section 4, “Make and validate the system release”, and OpenBSD FAQ 5: section on Building a Release (section 5.4) then recommends the following, which really is probably not necessary:


However, there is likely no real harm in leaving RELEASEDIR set if it may be convenient to keep. Thorough directions of later steps may have the envirionment values just set again when making a release of Xenocara.

If Xenocara (X) is going to be built shortly, do not remove the contents of /usr/obj/ directory. (Because the directions for Xenocara indicate that /usr/src/ may be used, it seems likley that /usr/obj/ may also be utilized.) If there is any chance that the same version of the source code may need to be used later, saving the contents of the /usr/obj/ directory may save some time later (by causing one or more make commands to go significantly faster). However, the directory should be cleared if a newer version of source code is obtained. If it is not expected that the files will be re-used before newer source code gets used, and if a release of the operating system was just created, then feel free to erase unneeded content of (/usr/obj/. (The most dire consequences are just that the content will be re-created later, taking up a bit of time, and the consequence of adding, hopefully insignificantly, to the system's wear and tear.) (Naturally, if Xenocara has just been compiled, then most of this commentary applies to /usr/xobj/ instead.)

[#bildxcra]: X

The process of building X (Xenocara) is similar to some of the steps of building executable binary files for the OpenBSD operating system.

These steps not only build X (Xenocara), but also install it.

Getting/updating the source

(This is similar to the section about getting OpenBSD source code which was followed when creating and installing the kernel and operating system.)

Determine/set the location where to put the extracted local copy of the source code. Then optionally (and recommended), pre-load the local source tree if that hasn't yet been done. Finally, if this hasn't yet been done, determine the needed details (like what CVS root to use, and what CVS tag to use) and then get the newer Xenocara source code. Details might not be exactly the same as creating building the operating system userland: the default local directory is /usr/xenocara/ instead of /usr/src/ and pre-loading comes from the xenocara.t*gz file and downloads come from the remote system's xenocara directory instead of a directory named src, but all in all, these changes are pretty minor. This documentation covered those differences while discussing how to build a userland, so refer to the earlier directions as needed.

OpenBSD's “manual page” about building an OpenBSD release section 5: “Build and install xenocara” says, “The /usr/src tree is also needed while building xenocara.” Presumably that will still be around from when the kernel was made: If not, follow the steps involving source code for the Operating System tree so that those required files are in place.

Clearing the specific object directory that will be used

(To compare this to the process of making the operating system, this skips anything related to the process of building and installing the kernel. This is more analogous to the section of “Rebuilding binaries”, and specifically the section of clearing the object directory.)

OpenBSD FAQ 5: Building X (Xenocara) (section 5.5) shows a specific object directory having contents removed. Note that this is NOT the same directory as /usr/obj/ which was used earlier. Instead of that directory, /usr/xobj/ is the one that is shown as needing to be cleared.

If using ksh, the following may work:

rm -rf ${XOBJDIR:-/usr/xobj}/*

The OpenBSD FAQ shows the following command line:

rm -rf /usr/xobj/*

Note that this directory to be erasing is different than /usr/obj/. Since /usr/src/ is still being used, and none of the directions did state to remove /usr/obj/, it seems probable that /usr/obj/* might still be useful.

If user permissions do not allow writing to that directory, consider joining the “wsrc” group, or using sudo.


This is discussed more in the section about setting a location where to put the extracted local copy of the source code. Basically, there are two options. One is to use /usr/xenocara/ as the directory where Xenocara is stored. If that option is done, then the XSRCDIR variable does not need to be manually set.

The other option involves using a /etc/mk.conf file (which may very likely not pre-exist, and so should be created) and setting XSRCDIR variable in that file.

The configuration in that file should cause the variable to point to the desired folder. Note that just simply setting the operating system environment variable the variable to the desired value would not fulfill OpenBSD's “manual page” about building an OpenBSD release section 5's notes that say the XSRCDIR variable “should be set in mk.conf”. However, there is a file, located at /usr/xenocara/README if a standard directory was being used, that says “The variable XSRCDIR can be set either in the environment or in /etc/mk.conf to point to the xenocara source tree” when a non-standard location is used.

(OpenBSD's “manual page” about building an OpenBSD release never uses a dollar sign in front of XSRCDIR, even when it would seem more convenient.) For those who may also be using other instructions (that don't clarify whether this may be an operating system environment variable), these instructions try to avoid creating confusion, by not assuming that an operating system environment variable is being used.

Customizing Xenocara

OpenBSD FAQ 5: section on Building X (Xenocara) (section 5.5) says, “If you wish to make actual modifications to the source code, you will probably need to add several packages. Details are in the /usr/xenocara/README file.” (That is likely assuming the default directory of /usr/xenocara/: if a custom XSRCDIR is used then use that instead.

These instructions do not elaborate on customizing source code for X (Xenocara) before building the code. Such customizing is not likely to be needed for simply building code made by the main developers. However, if such customizing is going to happen, this step of the process is when such customization would make sense to perform.

Building files

OpenBSD FAQ 5: section on Building X (Xenocara) (section 5.5) says “For building the standard xenocara tree as supported by OpenBSD, no external tools are needed.” That's nice.

Change to the directory where the files are going to be built. If the XSRCDIR directory was not configured, then this is assumed to be /usr/xenocara. However, if “grep XSRCDIR /etc/mk.conf ” would show a different location was set, be sure to change to that directory instead.

For example, if using ksh then the following may work:

cd ${XSRCDIR:-/usr/xenocara}

Otherwise, for example, if using the default location:

cd /usr/xenocara

Standard recommendations of pre-building apply. Namely, make sure there is enough free disk space before building.

Then, run the following command, which will likely go pretty fast and output very little (compared to some of the other make commands).

sudo make bootstrap
sudo echo Err=$?

The next two commands seem less specific to X (Xenocara): These same sort of commands were made for creating the operating system userland binaries. However, a key difference, compared to when a process was done before, is that these commands are now being run from a location with X (Xenocara) source code.

Older method:
sudo time make obj || echo Err=$? | tee /tmp/mkxobj.log
Newer method:
df -hi 2>&1 | tee -a /tmp/mkxobj.log
{ sudo time make obj ; echo Err=$? ; } 2>&1 | tee -a /tmp/mkxobj.log

The make obj command may take under a minute or even a third of a minute, so if there is a desire to start a longer process before leaving a system alone, hang around to start the make build command.

sudo time make build || echo Err=$? | tee /tmp/mkxbld.log
Newer method:
df -hi 2>&1 | tee -a /tmp/mkx.log
{ sudo time make build ; echo Err=$? ; } 2>&1 | tee -a /tmp/mkx.log
Making a release
Preparing directories (including setting operating system environment variables)

Follow the steps of preparing directories for building the operating system userland, including setting required envirionment variables. Specifically, this involves creating two variables that need to be set: RELEASEDIR and DESTDIR. This also involves deleting the contents of the folder pointed to by the ${DESTDIR} variable.

[#obsrmdst]: Removing the contents of ${DESTDIR}/

(This section is considered to be rather basic, simply showing how to remove the contents of the directory.)

To remove the directory, the following is a fairly safe approach (without as much risk of accidentally removing the entire filesystem due to an unset variable, or an equivilent thing like a mistyped variable name that causes an unset variable to be accessed).

First, save the current folder. For supporting shells, a simple “ pushd . ” may be used. Another option is to start a subshell. A third approach is to use a file, as shown in this example:

mkdir -p ${DESTDIR}
pwd | tee /tmp/mypwd

Then, if there isn't a permission denied error (which might be able to be overcome by using the word “ sudo ” just before the word “ tee ”), then the current directory is saved.

ls -la .
echo IF the directory seems safe to delete, then proceed.

rm -rf ./*
rm ./.[a-z]*
ls -la .

If the above did not produce desirable results, address the problem. For instance, if the files were created using a command line that involved by running software as a superuser, perhaps the files are owned by root, and so limitations related to permissions may need to be dealt with.

Finally, after removing the contents of that directory, return to the previous directory. If pushd was used, then use popd. Only if a subshell was run, run exit. If the example command lines involving /tmp/mypwd were used, then run the following.

cd $( cat /tmp/mypwd )

Once the directory is no longer the current directory, it may be removable:

rmdir ${DESTDIR}

(This completes the mini-guide for removing the directory. If this text was referenced from a hyperlink, consider going back to the original text.)

It may make sense to remove the directory to test removal of the directory pointed to by the DESTDIR variable, to make sure the removal isn't prevented from limitations such as permissions, ownership, or sharing (file in use) issues. However, since the directory is needed, then either don't remove the directory, or re-create it after removing the directory. If the directory is needed, it may be recreated using:

mkdir -p ${DESTDIR}

OpenBSD FAQ 5: section on Building X (Xenocara) (section 5.5) notes, “The RELEASEDIR” used for making X (Xenocara) “can be the same directory as the main system RELEASEDIR,” (which was used if a release was made of the main operating system), “but DESTDIR will be erased and rebuilt in this process. If done carefully, this is not a problem, but using a separate DESTDIR may be "safer".” There isn't really any compelling reason to use a different value for RELEASEDIR. In fact, OpenBSD's “manual page” about building an OpenBSD release section 6: “Make and validate the xenocara release” implies that the locations were the same (since it makes a reference to the X (Xenocara) files being placed where the operating system files were placed earlier).

Creating the release

Change to the directory where the files are going to be built. If the XSRCDIR directory was not configured, then this is assumed to be /usr/xenocara. However, if “grep XSRCDIR /etc/mk.conf ” would show a different location was set, be sure to change to that directory instead.

The following may work if using ksh:

cd ${XSRCDIR:-/usr/xenocara}

Otherwise, as an example, example, if using the default location:

cd /usr/xenocara

(That should not have been necessary if no directory changes were made since running the make command with the build parameter.)

Then, run:

Older code:
sudo time make release || echo Err=$? | tee /tmp/xrellog.txt
Newer code:
{ df -hi ; sudo time make release ; echo Err=$? ; } 2>&1 | tee -a /tmp/mkxrel.log
OpenBSD's “manual page” about building an OpenBSD release section 6: “Make and validate the xenocara release” says, “At this point you have both OpenBSD system and X Window System `tarballs' in your release directory.” That quoted statement is assuming that the same value was used for the ${RELEASEDIR} variable, both during the time when making the OpenBSD base system's tarballs, and when making the X Window System's tarballs.
Follow up

Follow the steps for making an index.txt file containing information about the installation files.

When performing the steps of check newly created OpenBSD release files, the steps likely will not show anything for some files, including the X (Xenocara) files.

Take a look at the results:


At this point, the contents of the directory that DESTDIR points to are now unneeded contents. Feel free to safely proceed with removing the contents of the DESTDIR directory.

OpenBSD's “manual page” about building an OpenBSD release section 6: “Make and validate the xenocara release” has one further step, which is:

[#bldobspt]: Third party packages/ports

OpenBSD FAQ 15: section about Security Updates for Ports (and resulting Packages) (FAQ 15.3.10).

This section may still be rather abbreviated.

OpenBSD's “manual page” about building an OpenBSD release section 7, “Make the third party packages” covers this. Naturally, section 7 comes after section 1. It may, therefore, be assuming that section 1 was already followed to obtain the latest code for ports.

The process is presumably familar by now, but a quick rundown involves pre-loading the local source tree with the ports file, deciding on a(n anonymous) CVS root, setting the CVSROOT variable, determining the desired CVS tag, and getting ports updated.

(This section may still be rather abbreviated: it may be helpful if there was more documentation/information about ports, particularly regarding information about how to get the latest code for ports.)

There may be little need for this if the only system(s) of concern have compiler tools installed: a system can just install from ports as needed instead of pre-compiling a bunch of packages. However, there may be some cases where pre-compiling, to create packages, may have some nice benefit. A key example would be that the packages may then, at least in some cases, be usable by other systems which might not have compilers installed.

The first step is to make sure the “ports” files have been extracted to /usr/ports/.

Especially if performing an operation involving many ports, consider checking the amount of free disk space before performing a task that may take up a lot of disk space.

Before getting into anything more advanced (including troubleshooting), it may be worthwhile to point out the OpenBSD FAQ about ports. (There's a fair amount to read there, and it may be quite worthwhile.)

For instance, the section about using dpb may show an optimal way to build many ports.

Compiling a single port

Then, for a single port, one may run:

cd /usr/ports/CATEGORY/PORT
sudo make package
Compiling multiple ports

The following is another approach, which may still be largely untested... Use at your own risk!

Consider the disk space requirements; building all ports may take substantially more disk space than just the operating system. Check the amount of free disk space. (Admittedly, it would be nicer if an amount of required space was provided by this guide. No such luck, at this time.)

Start by creating a text file that contains a list of packages to build. (This step is not required if ALL available ports (the ports with relevant files in the right location on the hard drive) are to be compiled. However, if all such packages are going to be made, this step is fairly quick and is relatively painless.) A simple way to do this is:

cd ${BSDSRCDIR:-/usr/src}/../ports
echo */* >> /tmp/pkgtobld.txt

Another approach may be to walk the directory tree. The following shows how this may be done with a single long command line when using ksh. Other shells can likely pull of something similar, possibly easiest by using mulitple command lines. This way is a more complicated way to make the text file, but is also a bit more customizable (to be able to do similar things, if desired).

(The following may be old code, which tries to run make. To get the rest of this documentation consistent with what is noted above, have it just run pwd to output the directory names, teeing output to a file. Then provide instructions to run dpb (referencing/crediting the OpenBSD FAQ about dpb). Also, in case dpb isn't done, perhaps something like (also add reference to PortsClean for making a single port) The following was designed for running in ksh. It is not yet tested/verified whether the return code will work. Here are some ideas for improvement: It would make good sense to tee output to a log (including output of df -hi that gets run before doing the whole batch, and perhaps before each program). It may be better to have make install depend on make package, and make clean depend on make install, using &&, but perhaps that will break the for commands? (Update: May now be using BULK=Yes instead of make clean.)

cd ${BSDSRCDIR:-/usr/src}/../ports
for portscat in * ; do cd ${BSDSRCDIR:-/usr/src}/../ports/${portscat} ; for portname in * ; do cd ${BSDSRCDIR:-/usr/src}/../ports/${portscat}/${portname} ; echo make package BULK=yes ; echo make install ; echo Err=$? for ${portscat}/${portname} ; pwd ; done ; done
cd ${BSDSRCDIR:-/usr/src}/../ports

If the above looks like it may work sensibly, then cosnider simply removing the relevant copies of the word echo.


These instructions made some log files in /tmp/ which may be removed at any time. (There wasn't strictly a need for them to be created in the first place.) If there is a desire to keep them, a sensible place to put them may be a directory underneath /var/tmp/.

The contents of the ${DESTDIR} directory used to create the operating system release, and the contents of the (presumably different) ${DESTDIR} directory used when creating X (Xenocara), are directories that may not be needed any longer. (After the contents of the directory are deleting, seeing the contents that used to be in the directory should be able to be done later, simply by extracting the contents of the *.tgz file(s) that were placed into the ${RELEASEDIR} directory.) If one or more of the directories pointed to by the ${DESTDIR} variable are still around, consider removing them. (If the ${DESTDIR} variable still points to the location of the directory, see removing the contents of the DESTDIR directory.)