NTFS

Overview

The “NT” of “NTFS” was likely named after “Windows NT”. In that operating system, the “NT” stands for “new technology”. (However, Microsoft stopped using the acronym “NT” as part of the names of its operating systems. This was done quite a while ago, and so references to NT do not refer to technology that people currently consider to be very “new”.) The “FS” stands for “filesystem” (or, probably more officially, “file system”).

There are multiple versions of NTFS. This has been known to affect authors of NTFS drivers, which did lead to some compatibility challenges for people using operating systems that Microsoft did not release. (So, most famously this affected people using Linux-based operating systems.) However, modern Linux drivers do support modern NTFS drives. The differences in NTFS versions do not typically affect end users very much at all.

NTFS was designed with the intent of using NTFS on a system's primary storage device (which were magnetic “hard drives” back in the day when NTFS was first being used). Before NTFS was used, Microsoft's DOS and Windows platforms tended to primarily use some variation(s) of FAT.

One of the most cherished benefits of using NTFS, rather than FAT, was the ability for NTFS to store more details that were helpful for implementing security. So, NTFS permissions were considered to be a pretty major deal.

Filesystem features
[#ntfsattr]: NTFS Attributes/Ownerships/Permissions

A standard disclaimer when discussing ownerships about things like filesystem objects: Note the disclaimer about ownership: ownership is not being meant as a legal term.

Windows permissions include supporting NTFS permissions/abilities, special permissions. (Similar permissions can also be applied to registry entries.)

Overview of NTFS permissions
Obtaining permissions

Note that even if an administrator does not have permissions to access an object, a user with administrator access may have the capabilities of using built-in tools to claim ownership of any resource such as a file and/or folder. The owner may then have the ability to grant permissions to the resource.

Introduction to permissions and “special permissions”

The NTFS file system provides support for a number of “permissions”. These permissions are really simply groups of “special permissions”. The “special permissions” are more detailed, and can be assigned seperately from any of the categories/groups which are commonly referred to by the simple name of “permissions”.

Many times, and probably most of the times, when people interact with setting permissions, they use the standard “permissions”. However, these standard “permissions” are really simply a group of “special permissions”.

If there is any actively used rule that specifies that a user is denied access to a permission, then that user is denied access. (It does not matter if another rule would specify that such access could be permitted. If any actively used rule denies access, then the access is denied.) If there is not an actively used rule that denies access to a permission, then the user will only have access to the permission if an rule specifies that the user should have that permission. In general, if no rule provides access to the user, then the user basically does not have the permission.

ZDNetAsia page about Windows access control objects indicates that an ACE that allows access will override an ACE that denies access if the ACE that allows access is on the object but the ACE that denies access is inherited, or if the ACE that allows access is inherited from the parent object but the ACE that denies access is inherited from some other object which is not the parent object (like, for example, a grandparent object).

Available Special Permissions

These may vary a bit between operating systems. For Windows XP Pro, see: Microsoft KB Q308419: Handling permissions for files and folders in Win XP (Pro). This article provides definitions for the various “special permissions”.

Available Permissions

These may vary a bit between operating systems. For Windows XP Pro, the Microsoft KB Q308419: Handling permissions for files and folders in Win XP (Pro) provides a table that shows the standard permissions (listen in the top row, starting with the second column) and shows which special permissions are part of each standard permission.

Some standard permissions definitely are supersets of others. As shown by the table from Microsoft KB Q308419: Handling permissions for files and folders in Win XP (Pro), “Read & Execute” is basically the same as “Read” but also supports the “Traverse Folder/Execute File” special permissions. “List Folder Contents” contains the exact same special permissions as “Read & Execute” (but affects inheritance differently). “Modify” contains all of the permissions of “Read & Execute” and also all of the permissions of “Write”, and provies “Delte”. “Full Control” includes everything, which ends up being everything from “Modify” plus “Delete Subfolders and Files” and “Change Permissions” and “Take Ownership”.

Terminology related to access controls

An “Access Control Entry” (“ACE”) is basically a rule that provides and/or denies permission to a specified user or group. When viewing the “Security” tab in the Windows GUI, the upper portion of the tab shows a selectable list of “Group or user names”. Each such group or username shown corresponds to a separete ACE.

An “Access Control List” ACL is basically a list of ACEs. Basically, in the Windows GUI, all of the information on the “Security” tab of an object represent an ACL.

  • a list of users/groups;
  • and a list of what special permissions are either denied to such users/groups;
  • and also a list of what special permissions are specifically allowed to such users/groups (unless another ACE is denying those permissions).

An ACL is basically a collection of ACEs. To narrow things down a bit further, there are multiple types of ACLs. Furthermore, there are types of ACLs. There are at least two types of ACLs: Each ACL is one of the

The term DACL stands for “Discretionary access control list”. ZDNetAsia page about Windows access control objects states, “A DACL is attached to the Active Directory rather than being attached to the NTFS file system.” ZDNetAsia page about Windows access control objects notes, “if an object simply doesn’t have a DACL, Windows 2000 interprets it to mean that there are no security restrictions on the object and that everyone should have full control over the object. The vast majority of Active Directory objects have DACLs, so an object without a DACL should be a rare occurrence.” (This does not apply to existant but empty DACLs.)

(Some text about similar implementations on other operating systems may be seen on Wikipedia's page about “Access control list”: section about “Filesystem ACLs”.)

MSDN page about “Access Control Lists (Windows)”

Commands to change ACLs from a command line
Software by Microsoft, for Windows

[#caclsexe]: CACLS

Perhaps CACLs stands for “change access control lists”.

Microsoft KB 919240 states, “you can use” ... “the Cacls.exe utility” ... “to modify NTFS permissions in Windows Server 2003.” However, that KB article also notes, “The Icacls.exe utility is an alternative option for modifying NTFS permissions. The Icacls.exe utility resolves various issues that occur when you use the existing utilities.” (Hyperlink(s) have been added to the quoted text.) In Windows Vista, the first and third lines of output are blank, with the second line saying: “ NOTE: Cacls is now deprecated, please use Icacls.

Usage: Information is likely available from Microsoft KB 135268: “How to Use CACLS.EXE in a Batch File”.

[#icacls]: Icacls
Overview
Microsoft KB 919240 Windows Vista Integrity Mechanism Technical Reference: Appendix B: Icacls and File Integrity Levels says that Icacls supports “mandatory labels”, and states that Cacls.exe does not.
Usage
See: TechNet information about Icacls.
Obtaining the software
Microsoft KB 919240 states, “The Icacls.exe utility is included in Windows Vista and in Windows Server 2003 SP2.” (This is not to suggest Win Svr 2003 SP2 was perfect: there is Microsoft KB 943043: Inheritance bit fix to Win Svr 2003 SP2's Icacls.)
[#xcacls]: XCACLS
[#xcaclvbs]: Xcacls.vbs
Usage
http://support.microsoft.com/kb/825751 (perhaps Q825751?)
Obtaining the software

Microsoft distributes an Xcacls.vbs file. (Microsoft KB Q318754: How to use Xcacls.exe to modify NTFS permissions references XCalcs_Installer.exe page for Win2K/XP/2003, which itself references a redirection page which redirects to redirection URL that contains an embedded, encoded URL to XCacls_Installer.exe which is used to install Xcacls.vbs for Win2K/XP/2003.)

Q82751 may also reference XCalcs_Installer.exe page. A slightly different URL, Extended Change Access Control List Tool (Xcacls) notes this being available for Win2K/2003 Support Tools.

Microsoft KB 919240 states that the abiilty to “use” ... “the the Xcacls.vbs utility to modify NTFS permissions in Windows Server 2003. The Icacls.exe utility is an alternative option for modifying NTFS permissions. The Icacls.exe utility resolves various issues that occur when you use the existing utilities.”

[#xcaclexe]: Xcacls.exe
Usage
See: Microsoft KB Q318754: How to use Xcacls.exe to modify NTFS permissions
Obtaining the software

Microsoft KB Q318754: How to use Xcacls.exe to modify NTFS permissions says, “The Xcacls.exe utility is included in the Windows 2000 Resource Kit. The Xcacls.exe utility is also included in the Windows Server 2003 Support Tools.”

The same KB article (Q318754) also makes a reference to a XCacls_Installer.exe which may be downloaded, but obtaining and using that file actually results in getting a Xcacls.vbs file.

SubInACL
SubInACL information, Installer for SubInACL
[#smbacls]:smbcacls
Part of the Samba suite.
[#setacl]: SetACL
Wikipedia's page on Cacls: section about the SetACL project, SetACL files (at SF.Net, includes offering source code).
Additional code
See: MSDN information on interacting with ACLs and ACEs, MSDN Visual Basic code for removing an ACE from an ACL (C++ programmers might want to try using “new AccessControlEntry” instead of the VB call to CreateObject).
Handling ownership
Modifying ownership
Command line software
TakeOwn
Graphical approach

From the Security tab, choose Advanced, then choose the “Owner” tab. If needed, press an “Edit...” button. Either select the new owner from the list on who to “Change owner to, or specify by pressing the “Other users or groups...” button.

Viewing ownership

An option may be to use “ DIR/Q ”. Windows Server 2003 did support this, as did Windows Vista. Note that this same command line option might not have the same effect in other (older) operating systems and/or certain third party command line interpretors. This is supported by recent JP Software TCC(LE) versions.

The most common way of using a graphical interface may require Windows XP Professional or newer (including Windows Server 2003, but not quite so easily including Windows XP Home Edition: this is discussed in the area of handling file security). In an Explorer window, view the file's properties and view the Security tab. Choose the Advanced button, and then choose the “Owner” tab. In order to use the “Edit...” button in Windows Vista, User Account Control may demand elevated privileges.

[#ntfscmpr]: NTFS Compression

This is a feature provided by NTFS. Unlike many other options for compressed data at the filesystem level, this doesn't require an entire compressed volumes. Individual directories/folders, and even individual files, may be compressed while other data remains uncompressed.

Some details are in TechNet Win2KRK: Volume, Folder, and File Compression.

Multi-substantiated rumor indicates severe pain, in the form of extreme slowness (if not instability) by trying to have an active Microsoft Windows operating system run from a compressed folder. Do not compress the Microsoft Windows operating system with NTFS compression.

NTFS Encryption

This feature has only been implemented by some Microsoft operating systems. (e.g., maybe WinXP Pro but not Win XP Home? Or maybe that's just true for Explorer.exe, while another way might implement this feature?)

[#ntfaltds]: Alternate data streams

A file is a collection of metadata, such as the filename and the time that the file was last modified, and data. In most cases, a file contains just one “stream” of data. The file's reported size corresponds to how many bytes are in that single stream of data. In some filesystems, there may be the possibility of having multiple streams. Such extra streams may generically be called a “fork”. For NTFS specifically, each stream other than the main one may be called an “alternate stream” or “alternate data stream” or perhaps “aternate file stream”. Blog entry about “Attachment Execution Service” notes NTFS Alternate Data Streams support “was originally added to NTFS because of Macintosh compatibility reasons”.

Using multiple streams is quite uncommon. A commonly encountered usage of such extra streams may be malware that hides data in additional streams. Unfortunately, historical Microsoft Windows operating systems came bundled with few if any decent ways of dealing with such extra streams. Flexhex page “NTFS Alternate Streams: What, When, and How To” says, “In fact, the only stream-enabled commands are” the two commands shown in the following examples shown on that web page:

C:\>echo This is just some text. >stream.dat:text

C:\>more <stream.dat:text
This is just some text.

C:\>

Blog about Vista notes that the DIR command may now have a /R switch. (This switch may not have this effect in older operating systems, or by third party software such as JP Software's TCC/LE.)

Sysinternals program called Streams is distributed by Microsoft. PDFForge.org's “Other projects” (other than PDFCreator) may contain executable code, and source code, for StreamsViewer. Alternate Stream View is freeware with command line support and also has a GUI.

Compatability

NTFS has been supported by Windows NT, Windows 2000, Windows XP, and newer Microsoft Windows operating systems. There are multiple versions of NTFS.

[#ntfsplen]: Path length limit

Some programs, bundled with at least some versions of Microsoft Windows, may have some limits regarding how long a file's path is. Typically the limit may be something like 255 characters or 260 characters. Using File Explorer, or the traditional command prompt, may reveal that a file with such a long filename may be challenging to handle. One option may be to use subst (and then use a command line to rename a file). For example:

  • Example filename: C:\ImportantDocuments\FieldWork\AnnualReports\YearEndData\Summaries\CustomerSeven \WinterProjects\Photographs\InitialState\LocationSixteen\BuildingInterior\UpperFloor \MeetingRoomThree\LightSwitches\DualSwitchPlates\CustomFacePlates\CustomerPrferencesGreenVariety.png
  • To keep the example easier to read, the example below doesn't use the full text of C:\ ImportantDocuments\FieldWork\AnnualReports\YearEndData\Summaries\CustomerSeven\ WinterProjects\Photographs\InitialState\LocationSixteen\BuildingInterior\UpperF loor\MeetingRoomThree\LightSwitches but, instead, will simply use just: C:\WayTooLongPath (even though that shorter example text isn't actually long enough to cause the problem being discussed).
subst
O:
subst O: "C:\WayTooLongPath"
subst
start O:
O:
move MorePathLen\long-name.txt C:\Temp\short.txt
C:
subst
subst /D O:
subst
  • The first command may produce no output. That is fine; the purpose of running that is just as a point of comparison later on. When the subst command is run without any parameters, the command shows what existing drives were created from when the subst command was run earlier.
  • The second command is presumed to cause an error. The command is meant to test whether the drive exists. (If it doesn't exist, then the second command generates an error message.)
  • The quotation marks are there in case the directory(/folder) contains a space. In many cases, they are optional. If using PowerShell, back-quotes will be needed before each quotation mark.
  • That command with /D deletes the drive letter created earlier.
subst vs. mapping

Using the subst command creates a “DosDevice”. (This was verified by running dosdev which can be obtained from LTR Data's Open Code. An explanation of what a DosDevice is can be found by Preet Sangha's answer to a StackOverflow.com question by Benjamin.)

This is a different impact than trying to “map” a drive by using SMB. A drive created by using subst does have some differences from drives trying to use “net use”, or using the GUI to map a drive. Trying to use a drive letter created through mapping a UNC won't work to override the 255 character path limit. Windows will still notice the long path in the true UNC for the file. However, Windows does not notice the longer true path when subst created the drive.

(Other differences between drives created by subst or “net use” are which of those commands will show the drive. When tested in Windows 10, accessing a mapped drive's “context-sensitive menu”/“context menu”/“shortcut menu”/“alternate menu”/“secondary menu”/“right-click menu” shows a “disconnect” option, but that doesn't show for a drive created with subst.)

NTFS is supported by open source software. For years, NTFS support had been only read-only. A driver called ntfs3g changed that.

VFAT-OS2 is an GPL'ed IFS (installable filesystem) for OS/2 that provides support for NTFS (and also support for VFAT). According to TLDP documentation for FAT: section on VFAT-OS2, this “can now also access NTFS partitions in read-only mode.”

Maximum size

The maximum number of clusters is 4,294,967,296 which is equal to 2 raised to the 32nd power. TechNet: Maximum Volume Sizes states, “In theory, the maximum NTFS volume size is” 4,294,967,296 “clusters. However, even if there were hardware available to supply a logical volume of that capacity, there are other limitations to the maximum size of a volume.” For example, partition sizes may be limited by the MBR format.

Of course, cluster sizes are variable in size. The web page also cites the maximum number of allocation units to be equal to 2 raised to the 64th power: that is 18,446,744,073,709,551,616. (If the allocation units were half-kilobytes, then it would be that number of half-kilobytes. Of course, if the allocation units were larger, the limit would be even larger.) Microsoft KB Q114841: Windows NT Boot Process and Hard Disk Constraints also cites this number: “NTFS uses 64-bit fields for all sizes, permitting its data structures to handle volumes up to 2^64 bytes (16 exabytes or 18,446,744,073,709,551,616 bytes).” However, TechNet: Maximum Volume Sizes states, also cites a smaller number, 4,294,967,296 (2^32), as an implementation limit.

Microsoft Support: “Frequently asked questions about the GUID Partitioning Table disk architecture” provdes some different limits, based on cluster sizes. A half-kilobyte cluser size would top out at 2,199,023,255,040 bytes (one cluster, which would be a half kilobyte, short of 2 TB) while a cluster size of 64 kilobytes (65,536 bytes) would top out at 281,474,976,710,656 bytes (one cluster, of 65,536 bytes, less than 256 TB).

[#ntfsrcov]: NTFS Undeleting software

This section contains some of the details specific to the NTFS file format. For more information about recovering data, including warnings and other details that may help recovery be more likely to succeed, also be familiar with data recovery basics. There are some common warnings/approaches to take which should be followed (but which, to minimize redundancy, are not duplicated here.)

See: Official Guide to using TestDisk 6.11 or newer to undelete from NTFS. Perhaps NTFS-3G may have Undelete support: NTFS-3G is the successor to Ntfsprogs and Wikipedia's page on Ntfsprogs says that Ntfsprogs included an ntfsundelete program. (NTFS-3G may offer some read/write support. Wikipedia's article on NTFS-3G mentions compatibility wtih several operating systems, not including OpenBSD (which, other than just Undeleting, also has NTFS support that is more limited). Some other (perhaps still untested) options may include Brian Kato's Restoration (Version 2.5.14) for Win95/NT/newer, and other programs listed on About.com list of some recovery programs.

[#ntfsdfrg]: Defragmenting NTFS
Defragmenting NTFS in Microsoft Windows

Many tools, designed for Microsoft Windows, work well on both NTFS and FAT. Do not feel like they are sub-par just because they also support FAT. Some software there has been widely used on NTFS, quite successfully. Therefore, it is entirely worthwhile to start by reviewing the discussion on defragmenting FAT drives.

(To avoid redundancy, the information from the section of defragmenting FAT drives is not being copied here. However, much of the information (and probably nearly all of the information) in the sub-section called “More defragmenting programs available for Microsoft Windows” will also apply to NTFS. So, do check that out first.

Defragmenting the Master File Table

Using Defrag C: /A ” will perform an analysis which may say that the MFT is fragmented. However, that software cannot fix the situation, despite being able to detect and report it.

contig

Contig, released by Sysinternals, can do this. This has the nicer feature of being smaller than UltraDefrag. It is also available by WebDAV (see: WebDAV clients), making this particularly easy to run.

net start WebClient
dir \\live.sysinternals.com\Tools\Contig*.exe
copy \\live.sysinternals.com\Tools\Contig.exe .

From a prompt that, if UAC is enabled, has elevated permissions:

Defrag C: /A /V
.\Contig.exe C:\$Mft -a -v -nobanner
.\Contig.exe C:\$Mft -v -nobanner
.\Contig.exe C:\$Mft -a -v -nobanner
Defrag C: /A /V

Re-running can have an effect. If the number of fragments has decreased, try running again to see if the number of fragments can successfully be decreased yet again.

UltraDefrag

UltraDefrag Handbook: Console Interface shows to use --optimize-mft. e.g.:

udefrag --optimize-mft C: D:
Defragmenting NTFS in Linux
UltraDefrag (for Windows) Handbook says “UltraDefrag for Linux exists which is an independent port of the program based on NTFS-3G capabilities.”