Filesystem Formats

Optical Disc formats

Information about formats typically used with optical discs is on the page about See: Disk/Disc images.

[#fatfs]: File Allocation Table (“FAT”) Filesystem Format Family

There are quite a few variations: the most commonly supported are FAT12 and FAT16, followed by the VFAT enhancements and the successor, FAT32. Further details about these (and formats that are less common and/or newer) are in a dedicated section about the “FAT” family of operating systems.

At least some of these formats have probably been the most widely supported, compatible filesystem formats known to grace the planet Earth (especially during the late twentieth century).

[#extfs]: “Extended” Filesystem: Ext2 and successors/similar

See: Extended 2 Filesystem Format, and Similar/Successor Filesystem Formats.

[#ffsufs]: FFS/UFS and similarly-named successors
Naming

OpenBSD and NetBSD call the filesystem type “FFS”. FreeBSD and Solaris call the filesystem type “UFS”. OpenBSD FAQ 14: section on large drives (OpenBSD FAQ 14.8) refers to the filesystem types as “FFS and FFS2 (also known as UFS and UFS2)”, which suggests they may be the same. Which name is the more proper name to use?

In short, it seems both names are still around, and unfortunately there is not a consensus as to which name to use. Even the successor to the original FFS/UFS has multiple names: UFS2 and FFS2. The most accepted name is probably going to depend on what operating system is being discussed: FFS will be the preferred name when discussing file systems used with some operating systems while UFS will be the preferred name when discussing some operating systems. That seems to be an unpleasant answer that isn't easy to digest. However, there is some evidence to indicate that neither name may clearly be more technically correct than the other. The next few paragraphs discuss this for those who would like to see this evidence.

It seems there may be multiple answers as to which name is better, and some of those may seem more correct to some people. However, the answer that may be most correct is that both names are technically accurate. According to Julipedia Blog and FAQ on difference between UFS and FFS and, there are multiple layers in the file system. Both of these resources indicate that there is an upper layer and a lower layer. Both of these resources say that the lower layer focuses more on the organization of which location of a physical disk is being used to store each of the various logical “blocks” of data. Both of these resources indicate that the upper layer has to do with managing traiditonal Unix-type information, such as directory structures. The upper layer stores its information in the logical “blocks” being kept track of by the lower layer, and so pays little or no attention at all about how those blocks get stored on the disk. Since both layers (keeping track of file system details, and keeping track of how information is on the disk) are required functionality, both UFS and FFS is being used.

Unfortunately, to complicate things, these websites differ in which layer gets which name. The FAQ indicates that the upper layer is called called FFS and it handles directory info while sitting on top of the UFS that handles disk layout. Julipedia indicates that the upper layer handles the directory information, and refers to it handling details of “Unix file-system” (which is, by the way, what UFS stands for), while this upper layer sits on top of various lower layers, one of which is called FFS.

Between the two, Julipedia's seems more likely to be accurate, as it gets into more details about multiple options of lower layers (and cites reelvant source code files, and refers to a chapter 6 of a printed book). A forum post by trasz, a FreeBSD Developer seems to favor this interpretation. Perhaps the FAQ mentioned just got the names mixed up.

Chapter 6 Section 1 of Practical Unix & Internet Security, 3rd Edition, a different book than the one Julipedia cited, indicates that UC at Berkeley's “FFS was such an improvement over the original UFS that AT&T eventually abandoned its filesystem in favor of FFS.” The Linux Documentation Project's Filesystems guide: section on UFS says “All modern Unixes use FFS filesystem, not UFS! UFS was used in early BSD versions.” Then a forum post by “Bill Hacker” says, “BTW - 'ufs' is simply the latest incarnation of what was originally called 'ffs'”.

Mounting a file system

FFS/UFS is the primary file system for some operating systems.

[#ffsftupd]: Using Soft Updates
Operating system support

OpenBSD's FAQ about Disk Setup: section about Soft Updates credits code “developed for FreeBSD by Kirk McKusick. It does seem that the OpenBSD project does recommend the usage of Soft Updates in at least some cases. Unsurprisingly, Kirk McKusick's page on Soft Updates notes this code is also available in FreeBSD. Kirk's page also notes support in NetBSD.

Some of this text may be based on OpenBSD.

Overview: The impact(s) of using Soft Updates
[#ffsftrli]: Reliability impact of using FFS Soft Updates

There is an older thread on OpenBSD Journal that suggests enabling soft updates may reduce reliability. However, Kirk McKusick's page on Soft Updates states, “In addition to performance enhancement, soft updates can also maintain better disk consistency.” That would imply an increase in reliability.

At the time of this writing, the author of this text is currently not taking a stance on whether the usage of Soft Updates is more likely to increase reliability or decrease reliability. As there seems to be conflicting information available, further research may be needed to be able to render a really informed opinion.

Background filechecking

Kirk McKusick's page on Soft Updates notes, “Snapshots and back-ground fsck are available in FreeBSD 5.0 and later.”

Older versions of OpenBSD's FAQ about Disk Setup: section about Soft Updates have stated that the FFS_SOFTUPDATES kernel option must be compiled in (which is true for GENERIC, the most common set of kernel options), and stated, “The potential of background” filesystem volume checking (sometimes referred to as “disk checking”) by using fsck and by “using Soft Updates is not yet realised in OpenBSD, so” fsck “is still required after an unclean shutdown. This may be changed in future versions.” That statement seems to have disappeared from the FAQ (after OpenBSD 3.4).

Post by Kirk McKusick does clearly show that background fsck is not as thorough in error detecting as a full regular foreground fsck.

Speed increase
OpenBSD's FAQ 14: Disk Setup: section about Soft Updates (FAQ 14.6) says, “a large performance increase is seen in disk writing performance.” (The FAQ provides a small amount of further detail.)
Memory usage
Since some machines, specifically those that have too low of an amount of kernel memory, cannot use Soft Updates, it seems sensible to presume that some kernel memory gets used in the process.
Recommended actions
Determining whether or not to use Soft Updates

If potential data loss is actually acceptable (which may be the case for a temporary copy of data, such as publicly-available source code which has been downloaded and may be re-downloaded if needed), then soft updates are desirable for FFS drives on machines with enough memory.

Otherwise, to determine whether using Soft Updates is desirable or undesirable, determine whether using Soft Updates is likely to decrease reliability. (There is some thought that reliability may even be improved.) This is covered in the section about FFS Soft Updates Reliability Impact. If data reliability is not expected to be decreased, then using Soft Updates is desirable for FFS drives on machines with enough memory.

OpenBSD's FAQ 14: Disk Setup: section about Soft Updates (FAQ 14.6) provides a “Note to sparc users: Do not enable soft updates on sun4 or sun4c machines. These architectures support only a very limited amount of kernel memory and cannot use this feature. However, sun4m machines are fine.”

How to use Soft Updates

To proceed with enabling soft updates, use the “softdep” mount option when the drive is mounted. The best way to do this is probably to add “,softdep” to the end of the 4th column of the relevant line of the /etc/fstab file, and then remount the drive with “ mount -uv /mountPoint ”.

Mounting in non-native operating systems
Using FFS/UFS in Linux
In Linux, FAQ on Linux and FreeBSD indicates that Linux 2.1.127 and higher could support writing to FreeBSD partitions. The mount command may use a -o ufstype= parameter.
Reading FFS in Microsoft Windows
There is a FFS File System Driver for Windows (for Win2K/XP/2003). According to Bo Brantén's web page called “Windows driver examples”, this is read-only.
Creating a filesystem volume

For details, see making a filesystem volume. Note that this type of filesystem is often used in BSD operating systems. In such operating systems, in addition to splitting a hard drive using MBR partitioning, it is common to also be using bsdlabel/disklabel disk segmentation.

OpenBSD's man page for newfs says, “Typically the defaults are reasonable, although newfs has numerous options to allow the defaults to be selectively overridden.” Most users looking for simplicity when formatting a hard drive will probably not have any particular need of any of the options.

However, the file system type may be worth noting. An old 4.3 BSD file format may be specified with “ -O 0 ” (that is a capital letter O, and then a zero). In theory “ -O 1 ”, although (at the time of this writing) the man page indicates that is the default for systems smaller than 1TB. (OpenBSD FAQ 14: section on large drives (OpenBSD FAQ 14.8) calls FFS “the historic OpenBSD file system”, so who knows if that will change?) -O 2 specifies FFS2, and is the default when using partitions too big for FFS.

[#ffsrcovr]: FFS/UFS data recovery

This section contains some of the details specific to the FFS(2)/UFS(2) file formats. For more information about recovering data, including warnings and other details that may help recovery be more likely to succeed, also be familiar with data recovery basics. There are some common warnings/approaches to take which should be followed (but which, to minimize redundancy, are not duplicated here.)

If a file is lost, there is some code available to try recoverying such software. Note that all of the software mentioned here has been untested by the author of this text, at the time of this writing.

There is some Open Source software at FFSUndelete CVS/SCM Repository at SF. Also, there is UFS Explorer (Raise Data Recovery) for UFS/UFS2 is Shareware for running under Windows 2000/XP/newer, allowing files under 64KB in the trial version. For Amiga, there is UnDelete.

Just as ext2 has debugfs, there is a fsdb command for those ready to manually interact with filesystem structures. (This is not expected to be an easy solution.)

If an entire partition is lost, TestDisk website: section listing supported Filesystems reports supporting “BSD disklabel ( FreeBSD/OpenBSD/NetBSD )” and “Sun Solaris i386 disklabel”.

LXer forum: post about Btrfs mentions that Free-BSD added SU+J: soft-updates with journaling.

[#btrfs]: Btrfs

LXer forum: post about Btrfs describes that Btfrs essentially stores all data using logs like journals.

At the time of this writing (June 5, 2011), Btrfs Wiki at Kernel.org says, “Note that Btrfs does not yet have a fsck tool that can fix errors. While Btrfs is stable on a stable machine, it is currently possible to corrupt a filesystem irrecoverably if your machine crashes or loses power on disks that don't handle flush requests correctly. This will be fixed when the fsck tool is ready.”

Wikipedia's entry for comparing file systems: section on operating system support for various operating systems shows that Btrfs is only supported on Linux. (Other options, like Mac OS X and FreeBSD, are not showing any support for this file system yet.)

Historical filesystems used in Linux
Design and Implementation fo the Second Extended Filesystem for some details about Minux FS and Xia FS and Ext FS (and, of course, Ext2FS).
[#hpfs]: High Performance File System (“HPFS”)
Introduction
Created by Microsoft and introduced by OS/2, HPFS was also supported by Windows NT 3. HPFS supported long filenames and extended attributes. (OS/2 could also store such “extended attributes” on a FAT drive by using file(s) called “EA DATA. SF”.
MBR Partition Type IDs

Uses 0x07 which is “OS/2 IFS” (standing for “Operating System/2 Installable Filesystem”). Other filesystems may also use this same identifier: namely NTFS and exFAT. So, partition type 0x07 might best be described as a filesystem supported by an “installable filesystem” driver.

Note: These types of filesystems are also very often found within an extended partition. (If searching for all FAT filesystems, be sure to check for the presense of an extended partition.)

Operating system support

Microsoft KB Q100108: Overview of FAT, HPFS, and NTFS File Systems states, “HPFS is only supported under Windows NT versions 3.1, 3.5, and 3.51. Windows NT 4.0 does not support and cannot access HPFS partitions.” Wikipedia's article on HPFS: section titled “Windows Native Support” says “Microsoft even removed the ability of NT 3.51 to format an HPFS file system.”

Accessing HPFS in DOS

DOS could not access HPFS drives unless third party code was used.

Hobbes Archive page about HPFS-Access provided at least some ability to write to an HPFS drive. Shareware limitations turned off write support after 16 MB or 20 minutes of the driver being loaded. This software would then turn into Read-only mode. This software was probably not the best for read-only access, due to using up more memory (which could be very limited in DOS) compared to some other options to read HPFS drives.

HPFSDOS 1.00 was a Shareware option. In addition to coming with a TSR to access drives, there were some utilities that could access ata on HPFS drives (without remaining in memory), and it came with the freeware program SwapDrv. This program is available by Hobbes OS/2 Archive: Details about HPFSDOS 1.00.

One read-only option was: options may include the software found at Hobbes Archive for iHPFS (software which uses GPL, an old home page for iHPFS mentioned version 1.28 as being the last version by that author, and noted that version 1.28 apparently didn't work so well with Caldera OpenDOS 7.02) The program comes with *.asm (assembly source code) files.

Another read-only option was Hobbes Archive of AMOS 3.20 which was shareware. Documentation indicated there would be a read/write solution by that author.

[#hpfsrcov]: HPFS data recovery software

This section contains some of the details specific to the HPFS file format. For more information about recovering data, including warnings and other details that may help recovery be more likely to succeed, also be familiar with data recovery basics. There are some common warnings/approaches to take which should be followed (but which, to minimize redundancy, are not duplicated here.)

eComStation's HPFS FAQ: section on undeleting files referred to various software including File Phoenix (fenx2.zip), which was written by IBM employee(s). OS2 World's Utilities section states, “Unfortunately it only works on partitions up to 2GB. Freeware.” For HPFS drives larger than 2GB, there is HPFSTool (Hobbes download site, Hobbes Archive of HPFSTool 1.7, shareware IIRC, possibly capable of working with both HPFS and FAT), shareware DFSee (Shareware). Other possible options might include Gammatech Utilities or Graham Utilities, or software with “Undelete”/“undelete” in the description at Hobbes Disk Utilities.

[#chdfiles]: Disk/drive compression (e.g. Compressed volumes/“Compressed Hard Drive” partitions/“CHD” files)
[#chdmyths]: Widely believed myths

First, let's clear out some myths about compressed hard drives.

Myth: Hard drive compression slows things down

Fact: The truth of this varies. In some cases, disk compression can actually speed things up.

The general theory about why disk compression slows down software is that the computer needs to read the data from the disk, and then perform decompression, and so there are multiple steps. This, according to the theory, takes longer than just performing one step, which is to read data. Also, the compression and decompression may take system resources including memory, and contribute to disk swapping which will really slow things down.

The reality is: Some implementations slow things down greatly, and that is how this “myth” started to be widely believed. However, sometimes disk compression results in data becoming available faster.

After understanding the theory of why using a compressed disk may go slower, some people may want to understand some counter logic on why two steps to read data may be faster than one step to read data. So, here is that logic: After reading a smaller amount of data from the disk, the compressed data then needs to be decompressed. However, decompression uses CPU and RAM, which is faster than many forms of disk storage such as mechanical hard drives. These two steps involve first reading a small amount of data from a slow hardware component, and then processing data using faster components. Those two steps may be performed faster than just loading a large amount of data from the slower component.

The end result on overall speed is that things can go faster when the used data is compressed data. This is particularly when data handling operations involve working with the slower devices in the systems, such as primary long term data storage (like a hard drive) or, often even slower, cheaper optical media (like a CD-ROM disc).

Software implementation can have a huge impact. A really bad software implementation could certainly cause substantial slowdown. Differences in hardware can also impact speed. The rumored results are that with some old and non-identical computers, using the exact same compression software could sometimes make one computer slower, and another computer faster. The difference had to do with factors such as how fast the CPU was, and how slow the hard drive was. For a system with a fast hard drive but a slow CPU for performing the compression routines, disk compression slowed things down. For a different computer, the same software could result in disk compression speeding things up.

Some bootable CD images, with freebie operating systems using the Linux kernel, will often use a compressed image. Whatever statements are made about hard drives being slow are likely even more true when those statements are applied to optical disc drives (which, at least historically, have been typically slower than a mechanical hard drive).

One observation that has been reported by multiple technicians is that using Microsoft Windows's NTFS compression on the folder (or entire hard drive partition/volume) with the operating system is expected to result in the whole system going much, much slower. Much slower. Compressing the page/swap file (or the folder/directory or drive that contains the page/swap file) is also said to severely cripple responsiveness. These obversations, and other evidence using implementations that have been bundled with Microsoft's operating systems, may be cause why many people believe that data compression slows things down. However, one or more bad implementations doesn't disprove a general theory (even if a bad implementation is popular). Nor should such bad implementations minimize the appreciation and usefulness of success that is achieved by better implementations.

For whatever it is worth, some popular data compression software clearly decompresses data, even tightly compressed data, much more quickly than the software compresses data (at least when compression settings specify tight compression). Therefore, any impactful performance slowdown might be more noticeable when writing data, compared to the impact noticed when reading data. (This is another reason why operating system installations may benefit more when reading from a pre-compressed image, while hard drives that need to write compressed data may not benefit as much.)

Myth: Data compression is unreliable

Researching this may yield the fact that this can be true: with at least some disk compression software, a software bug can result in data loss. However, it is also worth noting that several people distribute compressed disk images, including people behind some distributions of operating systems. This may include hard drive images or, probably more commonly, bootable images meant to be used with removable media. People generally do not complain about the unreliability of the data being distributed in this fashion, which is powerful evidence that compressed volumes can, in at least some cases, be used reliably.

As a comparison, some software, or at least some versions of popular software (at least including some 7-Zip versions, and also some versions of PKZip, e.g. PKZip 2 versions earlier than 2.04g) have been known to create Zip files that contained invalid data. However, such buggy software versions haven't prevented many people from using Zip files.

It is, therefore, unwise to throw the baby out with the bath water, by stating the compressed volumes must always be avoided due to data reliability issues.

Formats
DriveSpace
Maximum size

For DriveSpace (included with MS-DOS and Windows 95), see the section about DoubleSpace. For DriveSpace3, the maximum size was increased: Microsoft KB Q93533: Largest Possible DoubleSpace Drive says, “Because DriveSpace 3 (the version of DriveSpace included with Microsoft Plus!) can use cluster sizes up to 32K, the maximum size for a DriveSpace 3 CVF is 2 GB.” (CVF stands for “Compressed Volume File”.)

Compatability

MS-DOS 6.2 and 6.22 supported DriveSpace. Compatability is likely similar to the compatability options for DoubleSpace, with the historical exception that MS-DOS 6.0 (the first implementation of DoubleSpace) did not support DriveSpace. (See the section on DoubleSpace for compatability notes.)

[#dblspace]: DoubleSpace

Microsoft tried to license Stacker. When agreements with Stacker were not being successful, Microsoft chose to use a different implementation. Stac sued. Although some computer experts may have felt Microsoft was in the right on technical grounds, a jury (likely consisting of people who are not computer experts) sided with Stac. Microsoft then released DriveSpace. The DriveSpace format may be newer, and possibly more reliable.

MS-DOS 6.2 may have some benefits to reliability. These reliability benefits may even have been marketed using the name “DoubleGuard”. The term “DoubleGuard” might refer to the new ScanDisk command, and possibly other changes (like the disk cache implemented by the SmartDrv command). Wikipedia's article on DoubleSpace (which redirects to a page named DriveSpace): “Bugs and data loss” section indicates that DoubleGuard may be related to DoubleSpace checking whether some of its memory has been corrupted by other software running on the system. Another change in MS-DOS 6.2 was to flush SmartDrv's cache before showing a command prompt (which may have slowed things down somewhat, but may have reduced corruption from people shutting down a computer when they saw the command prompt that typically indicating that the computer was then waiting on the person). Brian Livingston's article at the bottom of page 20 of InfoWorld's September 20, 1993 issue described Blossom Software's “BUST.EXE” program demonstrating problems on drives with heavy fragmentation. (Blossom Software had supplied a copy of this program to the author of this magazine article.)

Maximum size
Microsoft KB Q93533: Largest Possible DoubleSpace Drive is 512 MB says, “The versions of DoubleSpace and DriveSpace included with MS-DOS and Windows 95 support cluster sizes up to 8K. This limits the size of a DoubleSpace or DriveSpace compressed volume file (CVF) to 512 MB.”
Cluster sizes
Microsoft KB Q67321: (FAT) Cluster Sizes (and more) says, “NOTE: DoubleSpace-compressed drives appear to have 8K clusters, but internally vary the sectors-per-cluster as necessary (this information is recorded in the MDFAT). For example, a 10K file which compresses by a factor of 2:1 actually uses 5K, or 10 sectors, of drive space.”
Command

The executable file to reference at a command line was dblspace

Compatability
Compatability in Linux
TLDP documentation on DMSDOS (for DblSpace and DrvSpace(3) and Stacker 3 and Stacker 4) and TLDP documentation on Read-only Dblspace and Drvspace in Linux refer to some freely available options. These options do not seem to be available at the referenced locations, but since they did exist freely, they can probably still be located.
Diet
Shareware. A TSR that may save some disk space, although possibly taking up RAM in an environment (MS-DOS and compatible operating systems) where memory limitations may already be severe. Rumor has it that this was not a very good implementation. TOOGAM's Software Archive: Archivers Page mentions some options for use in Linux and MS-DOS. (Search for the section called “Compressed Hard Drive Volumes”.)
[#ntfscmpr]: NTFS Compression

This is a feature provided by NTFS. SEe: NTFS Compression.

More options
TOOGAM's Software Archive: Archivers Page mentions some options for use in Linux and MS-DOS. (Search for the section called “Compressed Hard Drive Volumes”.)
[#reiserfs]: ResierFS

Meant for Linux, it was famous for having some speed advantages over ext2 (at least when working with certain types of data, such as many small files). Bo Brantén's web page called “Windows driver examples” mentions an update for ReiserDriver, an Installer File System Driver (IFSD) for Microsoft Windows. This version at http://www.acc.umu.se/~bosse/ has rfsd-0.25.5-signed.zip which is a signed version of rfsd-0.25.5.zip which is an update to support newer versions of Windows.

[#ntfs]: NT (“New Technology”) File System (“NTFS”)

See: NTFS.

[#ntfsrcov]: (Information about NTFS data recovery has been moved. See: NTFS: data recovery.)

[#fsfrmnet]: Filesystems provided over networking

Information about network-based filesystems are now in its own seperate sub-section.

[#nfs]: NFS (Network Filesystem)

The term “NFS” does refer to a specific protocol which was one of the most popular ways of sharing data on old Unix-based systems. To this day, NFS is often described as an alternative that is most popular on Unix.

Details about NFS have been moved from this location. See: Filesystems provided over networking: NFS: Network Filesystem.

Server Message Block” (“SMB”) (and “Common Internet File System” (“CIFS”))

Details have been moved: see: Server Message Block” (“SMB”) (and “Common Internet File System” (“CIFS”)).

[#afs]: AFS

AFS, more fully known as the “Andrew File System”, has a number of interesting features.

Details have been moved: see: Filesystems provided over networking: AFS.

For additional ways to transfer files, see transfering files. Some remote access software with built-in file transfering abilities may also be an option.

[#ramdrive]: “RAM Drive” / “memory filesystem”

In general, if a RAM drive is being used, avoid swapping to it :)

DOS

Although DOS may come with a \DOS\RAMDRIVE.SYS file, downloading and using TDsk is likely going to be nicer. (See FreeDOS's web page about TDSK. While at it, anybody working with DOS may want to consider reviewing DOS memory for other details about handling memory.)

Much more rarely implementated may be an interesting alternative: using RAM other than the system's main base RAM. An example of this may be to use the memory on a “Gravis UltraSound” (“GUS”) card. (Such a thing exists. Also, the contents of such memory may be able to survive a motherboard initiating a “cold boot”.) (A driver to implement this is publicly available, mentioned on TOOGAM's Software Archive: GUS Sound.)

[#mfsunix]: Memory filesystem (“mfs”) in Unix
Overview

In Unix, this is typically called a “memory filesystem”. This has often been used by people who are building/compiling programming code. For those who are heavily involved in software development, there may be no long term desire to keep the built code, as the general plan may be that an updated version of the code will be getting worked on soon, so build files have a very short shelf life of even being useful anyway. Since the (compiled/built) data being created could be rather easily re-created in the unlikely event of a power issue, and so speed ends up being the preference (over any guaranteed longevity).

Creating a memory filesystem

At the moment, the section about using a memory filesystem for OpenBSD's /usr/obj/ directory may serve as a documented example (in OpenBSD). (Relevant text may be getting moved from there, to here, relatively soon.)

Note that memory drives may be unique compared to drives where long term storage is expected. For example, OpenBSD FAQ about Disk Setup: section about using asynchronous mounting (FAQ 14.19) notes that, unlike other drives, memory filesystems “are mounted asynchronously, as they will get wiped and recreated on a reboot anyway.”

Other filesystems and/or related software

Node 17 has a list of file system software. (Search for u2fs.)