> This is part 1 of 7 of the full documentation (pages 1–100 of 668). > The content is paginated: fetch every part to see all of it. > Next part: https://docs.pmem.io/ndctl-user-guide/llms-full.txt/1 > Page index: https://docs.pmem.io/ndctl-user-guide/llms.txt # NDCTL User Guide ## Introduction `ndctl` is a utility for managing the Linux LIBNVDIMM Kernel subsystem. It is designed to work with various non-volatile memory devices (NVDIMMs) from different vendors. The LIBNVDIMM subsystem defines a kernel device model and control message interface for platform NVDIMM resources like those defined by the [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) NFIT (**N**VDIMM **F**irmware **I**nterface **T**able). The latest [ACPI ](http://www.uefi.org/specifications)and [UEFI ](http://www.uefi.org/specifications)specifications can be found at [uefi.org](http://www.uefi.org). Operations supported by `ndctl` include: * Provisioning capacity (namespaces) * Enumerating Devices * Enabling and Disabling NVDIMMs, Regions, and Namespaces * Managing NVDIMM Labels # Installing NDCTL The `ndctl` utility is used to manage the libnvdimm (non-volatile memory device) sub-system in the Linux Kernel. It is required for certain Persistent Memory Developer Kit (PMDK) features. {% hint style="info" %} Microsoft Windows users should read the [Understand and Deploy Persistent Memory](https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/deploy-pmem) documentation. {% endhint %} The `ndctl` utility can be installed on Linux using either of the two options: * [Install ndctl using packages](/ndctl-user-guide/master/installing-ndctl/installing-ndctl-packages-on-linux) * [Install ndctl using the source code](/ndctl-user-guide/master/installing-ndctl/installing-ndctl-from-source-on-linux) # Installing NDCTL Packages on Linux The `ndctl` utility is available in many Linux distribution package repositories. This approach is the easiest to implement and maintain, compared with [installing ndctl from source code](/ndctl-user-guide/master/installing-ndctl/installing-ndctl-from-source-on-linux). {% tabs %} {% tab title="Fedora" %} 1\) Query the repository to identify if ndctl is delivered: Fedora 21 or earlier ``` yum search ndctl ``` Fedora 22 or later ``` dnf repoquery ndctl ``` 2\) Install the ndctl utility Fedora 21 or earlier ``` yum install ndctl ``` Fedora 22 or later ``` dnf install ndctl ``` {% endtab %} {% tab title="RHEL & CentOS" %} The ndctl package is available on CentOS and RHEL 7.0 or later. 1\) Query the repository to identify if ndctl is delivered: ``` yum search ndctl ``` 2\) Install the ndctl package ``` $ yum install ndctl ``` {% endtab %} {% tab title="SLES & OpenSUSE" %} 1\) Query the repository to identify if ndctl is delivered: ``` $ zypper search ndctl ``` 2\) Install the ndctl package ``` $ zypper install ndctl ``` {% endtab %} {% tab title="Ubuntu" %} The ndctl package is available on Ubuntu 18.10 (Cosmic Cuttlefish) or later. 1\) Query the repository to identify if ndctl is delivered using either the aptitude, apt-cache, or apt utilities ``` $ aptitude search ndctl $ apt-cache search ndctl $ apt search ndctl ``` 2\) Verify if the ndctl package is currently installed and check the version ``` $ apt list --installed ndctl ``` 3\) Install the ndctl package or update an installed package ``` $ sudo apt-get install ndctl ``` {% endtab %} {% tab title="Debian" %} {% hint style="info" %} The ndctl package is available on Debian 10 (Buster) or later. See for up to date information. {% endhint %} 1\) Query available versions in configured repos: ``` $ apt policy ndctl ``` 2\) Query the repository to identify if ndctl is delivered using either the aptitude, apt-cache, or apt utilities ``` $ apt search ndctl ``` 3\) Verify if the ndctl package is currently installed and check the version ``` $ apt list --installed ndctl ``` 4\) Install the ndctl package or update an installed package ``` $ sudo apt install ndctl ``` {% endtab %} {% endtabs %} # Installing NDCTL from Source on Linux These instructions provide a step-by-step guide for installing the `ndctl` and `daxctl` utilities from the latest GitHub [master branch](https://github.com/pmem/ndctl). ### 1. Install the Prerequisites There are a number of packages required for the build steps that may not be installed by default. For information about the required packages, see the "BuildRequires:" lines in ndctl.spec.in. To successfully compile ndctl from source with documentation, the following packages are required * autoconf * automake * asciidoc (asciidoctor) * bash-completion * doxygen * gcc * gcc-c++ (g++) * git * glib2 * glib2-devel * graphviz * json-c-devel * keyutils-libs-devel (libkeyutils-dev) * kmod * kmod-devel * libfabric * libfabric-devel * libtool * libudev-devel * libuuid-devel * ncurses * pandoc * pkg-config * rubygem-asciidoctor (asciidoctor) * xmlto To install these prerequisites, use: {% tabs %} {% tab title="Fedora" %} Fedora 21 or earlier ``` sudo yum install git gcc gcc-c++ autoconf automake asciidoc asciidoctor xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel ``` Fedora 22 or later ``` sudo dnf install git gcc gcc-c++ autoconf automake asciidoc asciidoctor xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel ``` {% endtab %} {% tab title="RHEL & CentOS" %} Some of the required packages can be found in the EPEL repository. Verify the EPEL repository is active: ``` $ yum repolist ``` If the EPEL repository is not listed, install and activate it using: ``` $ sudo yum install epel-release ``` Install the required packages ``` $ sudo yum install git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkgconfig glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel rubygem-asciidoctor keyutils-libs-devel ``` {% endtab %} {% tab title="SLES & OpenSUSE" %} ``` $ sudo zypper install -y git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel rubygem-asciidoctor keyutils-libs-devel ``` {% endtab %} {% tab title="Ubuntu & Debian" %} **For Ununtu 18.04 (Bionic) and Debian 9 (Stretch) or later:** ``` $ sudo apt install -y git gcc g++ autoconf automake asciidoc asciidoctor bash-completion xmlto libtool pkg-config libglib2.0-0 libglib2.0-dev libfabric1 libfabric-dev doxygen graphviz pandoc libncurses5 libkmod2 libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev ``` **For Ubuntu 16.04 (Xenial) and Debian 8 (Jessie):** {% hint style="info" %} Earlier releases of Ubuntu and Debian do not have libfabric1 or libfabric-dev available in the repository. If these libraries are required, you should compile them yourself. See {% endhint %} ``` $ sudo apt-get install -y git gcc g++ autoconf automake asciidoc asciidoctor bash-completion xmlto libtool pkg-config libglib2.0-0 libglib2.0-dev doxygen graphviz pandoc libncurses5 libkmod2 libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev ``` {% endtab %} {% endtabs %} ### 2. Clone the GitHub Repository 2.1) If you're behind a company proxy, configure git to work with your proxy server first. The following configures a HTTP and HTTPS proxy for all users. Refer to the [git-config documentation](https://git-scm.com/docs/git-config) for more options and information. ``` $ git config --global http.proxy http://proxyUsername:proxyPassword@proxy.server.com:port $ git config --global https.proxy https://proxyUsername:proxyPassword@proxy.server.com:port ``` 2.2) Create a working directory to clone the ndctl GitHub repository to, eg: 'downloads' ``` $ sudo mkdir /downloads $ sudo chmod +w /downloads ``` 2.3) Clone the repository: ``` $ cd /downloads $ sudo git clone https://github.com/pmem/ndctl $ cd ndctl ``` ### 3. Build The following configures ndctl to be installed in to the /usr/local directory. ``` $ sudo ./autogen.sh $ sudo ./configure CFLAGS='-g -O2' --prefix=/usr/local --sysconfdir=/etc --libdir=/usr/local/lib64 $ sudo make ``` #### **Build using an alternative compiler** **Note:** If you want to compile with a different compiler other than gcc, you have to provide the CC and CXX environment variables. For example: ``` $ sudo make CC=clang CXX=clang++ ``` These variables are independent and setting CC=clang does not set CXX=clang++. #### Build a Debug Version To compile ndctl with debugging, use the `--enable-debug` option: ``` $ sudo ./autogen.sh $ sudo ./configure CFLAGS='-g -O2' --enable-debug --prefix=/usr/local --sysconfdir=/etc --libdir=/usr/local/lib64 $ sudo make ``` For a full list of configure options use: ``` $ ./configure --help ``` ### 4. Install Once ndctl has successfully been compiled, it can be installed using the following: ``` $ sudo make install ``` ### 5. Build and Run Unit Tests (Optional) The unit tests run by `make check` require the nfit\_test.ko module to be loaded. To build and install nfit\_test.ko: 1. Obtain the kernel source. For example, `git clone -b libnvdimm-for-next git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git` 2. Skip to step 3 if the kernel version is >= v4.8. Otherwise, for kernel versions < v4.8, configure the kernel to make some memory available to CMA (contiguous memory allocator). This will be used to emulate DAX. `CONFIG_DMA_CMA=y` `CONFIG_CMA_SIZE_MBYTES=200` **or** `cma=200M` on the kernel command line. 3. Compile the libnvdimm sub-system as a module, make sure "zone device" memory is enabled, and enable the btt, pfn, and dax features of the sub-system: `CONFIG_X86_PMEM_LEGACY=m` `CONFIG_ZONE_DEVICE=y` `CONFIG_LIBNVDIMM=m` `CONFIG_BLK_DEV_PMEM=m` `CONFIG_ND_BLK=m` `CONFIG_BTT=y` `CONFIG_NVDIMM_PFN=y` `CONFIG_NVDIMM_DAX=y` `CONFIG_DEV_DAX_PMEM=m` 4. Build and install the unit test enabled libnvdimm modules in the following order. The unit test modules need to be in place prior to the `depmod` that runs during the final `modules_install` `make M=tools/testing/nvdimm` `sudo make M=tools/testing/nvdimm modules_install` `sudo make modules_install` 5. Now run `make check` in the ndctl source directory, or `ndctl test`, if ndctl was built with `--enable-test`. ### Troubleshooting The unit tests will validate that the environment is set up correctly before they try to run. If the platform is misconfigured, i.e. the unit test modules are not available, or the test versions of the modules are superseded by the "in-tree/production" version of the modules `make check` will skip tests and report a message like the following in test/test-suite.log: ``` SKIP: libndctl ============== test/init: nfit_test_init: nfit.ko: appears to be production version: /lib/modules/4.8.8-200.fc24.x86_64/kernel/drivers/acpi/nfit/nfit.ko.xz __ndctl_test_skip: explicit skip test_libndctl:2684 nfit_test unavailable skipping tests ``` If the unit test modules are indeed available in the modules 'extra' directory the default depmod policy can be overridden by adding a file to /etc/depmod.d with the following contents: ``` override nfit * extra override device_dax * extra override dax_pmem * extra override libnvdimm * extra override nd_blk * extra override nd_btt * extra override nd_e820 * extra override nd_pmem * extra ``` The nfit\_test module emulates pmem with memory allocated via vmalloc(). One of the side effects is that this breaks 'physically contiguous' assumptions in the driver. Use the `--align=4K` option to `ndctl create-namespace` to avoid these corner case scenarios. # Quick Start The `ndctl` command is designed to be user friendly. Once [installed](/ndctl-user-guide/master/installing-ndctl), a list of commands can be shown using any of the following: 1\) With no arguments or options, `ndctl` shows a simple usage message: ``` # ndctl usage: ndctl [--version] [--help] COMMAND [ARGS] See 'ndctl help COMMAND' for more information on a specific command. ndctl --list-cmds to see all available commands ``` 2\) Using `ndctl help` displays basic help and syntax: ``` # ndctl help usage: ndctl [--version] [--help] COMMAND [ARGS] See 'ndctl help COMMAND' for more information on a specific command. ndctl --list-cmds to see all available commands ``` Below is an example of using the `ndctl help` command to launch the `create-namespace` man page: ``` # ndctl help create-namespace ``` 3\) Using `ndctl --list-cmds` lists all commands as a single list. ``` # ndctl --list-cmds version enable-namespace disable-namespace create-namespace destroy-namespace check-namespace clear-errors enable-region disable-region enable-dimm disable-dimm zero-labels read-labels write-labels init-labels check-labels inject-error update-firmware inject-smart wait-scrub start-scrub setup-passphrase update-passphrase remove-passphrase freeze-security sanitize-dimm load-keys wait-overwrite list monitor help ``` An alternative method for listing commands uses the TAB key completion feature of ndctl. By typing`ndctl ` we can list the available commands, eg: ``` # ndctl check-labels disable-namespace help monitor update-firmware zero-labels check-namespace disable-region init-labels read-labels update-passphrase clear-errors enable-dimm inject-error remove-passphrase version create-namespace enable-namespace inject-smart sanitize-dimm wait-overwrite destroy-namespace enable-region list setup-passphrase wait-scrub disable-dimm freeze-security load-keys start-scrub write-labels ``` ### TAB Command and Argument Completion `ndctl` supports command completion using the TAB key. For example, typing `ndctl enable-` lists all commands beginning with 'enable', eg: ``` # ndctl enable- enable-dimm enable-namespace enable-region ``` TAB completion also works with command arguments. For example, typing `ndctl enable-dimm ` will show all available command arguments. For example, the 'enable-dimm' command can enable one, more than one, or all NVDIMMs. It will list all available NVDIMMs (nmem) devices when using the TAB completion, eg: ``` # ndctl enable-dimm all nmem0 ``` ### Getting Help NDCTL ships with a man page for each command. Each man page describes the required arguments and features in detail. Man pages can be found and accessed using the `man` or `ndctl` utilities. The following `man -k ndctl` searches for any man page containing the "ndctl" keyword: ``` # man -k ndctl ndctl (1) - Manage "libnvdimm" subsystem devices (Non-volatile Memory) ndctl-check-labels (1) - determine if the given dimms have a valid namespace index block ndctl-check-namespace (1) - check namespace metadata consistency ndctl-clear-errors (1) - clear all errors (badblocks) on the given namespace ndctl-create-namespace (1) - provision or reconfigure a namespace ndctl-destroy-namespace (1) - destroy the given namespace(s) ndctl-disable-dimm (1) - disable one or more idle dimms ndctl-disable-namespace (1) - disable the given namespace(s) ndctl-disable-region (1) - disable the given region(s) and all descendant namespaces ndctl-enable-dimm (1) - enable one more dimms ndctl-enable-namespace (1) - enable the given namespace(s) ndctl-enable-region (1) - enable the given region(s) and all descendant namespaces ndctl-freeze-security (1) - Set the given DIMM(s) to reject future security operations ndctl-init-labels (1) - initialize the label data area on a dimm or set of dimms ndctl-inject-error (1) - inject media errors at a namespace offset ndctl-inject-smart (1) - perform smart threshold/injection operations on a DIMM ndctl-list (1) - dump the platform nvdimm device topology and attributes in json ndctl-load-keys (1) - load the kek and encrypted passphrases into the keyring ndctl-monitor (1) - Monitor the smart events of nvdimm objects ndctl-read-labels (1) - read out the label area on a dimm or set of dimms ndctl-remove-passphrase (1) - Stop a DIMM from locking at power-loss and requiring a passphrase to access media ndctl-sanitize-dimm (1) - Perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s) ndctl-setup-passphrase (1) - setup and enable the security passphrase for an NVDIMM ndctl-start-scrub (1) - start an Address Range Scrub (ARS) operation ndctl-update-firmware (1) - provides for updating the firmware on an NVDIMM ndctl-update-passphrase (1) - update the security passphrase for an NVDIMM ndctl-wait-overwrite (1) - wait for an overwrite operation to complete ndctl-wait-scrub (1) - wait for an Address Range Scrub (ARS) operation to complete ndctl-write-labels (1) - write data to the label area on a dimm ndctl-zero-labels (1) - zero out the label area on a dimm or set of dimms ``` {% hint style="info" %} Note: If `man -k ndctl` returns "ndctl: nothing appropriate." or similar, see the [Troubleshooting](/ndctl-user-guide/master/troubleshooting) section to manually build the indexes. {% endhint %} Additionally, executing `ndctl help ` can be used to display the man page for the command, eg: ``` # ndctl help enable-dimm ``` A list of ndctl man pages are available online. See '[NDCTL Man Pages](/ndctl-user-guide/master/man-pages)' for a complete list. ### Displaying Bus, NVDIMM, Region, and Namespace Information The `ndctl list` command is a very powerful and feature rich command. A list of options is shown below: ``` # ndctl list -? Error: unknown switch `?' usage: ndctl list [] -b, --bus filter by bus -r, --region filter by region -d, --dimm filter by dimm -n, --namespace filter by namespace id -m, --mode filter by namespace mode -t, --type filter by region-type -U, --numa-node filter by numa node -B, --buses include bus info -D, --dimms include dimm info -F, --firmware include firmware info -H, --health include dimm health -R, --regions include region info -N, --namespaces include namespace info (default) -X, --device-dax include device-dax info -i, --idle include idle devices -M, --media-errors include media errors -u, --human use human friendly number formats ``` Using the filters is a powerful way to limit the output. ### Examples To list all active/enabled namespaces: ``` # ndctl list -N ``` To list all active/enabled regions: ``` # ndctl list -R ``` To list all active/enabled NVDIMMs: ``` # ndctl list -D ``` To list all active/enabled NVDIMMs, Regions, and Namespaces: ``` # ndctl list -DRN ``` To list all active/enabled and disabled/inactive (idle) NVDIMMs, Regions, and Namespaces: ``` # ndctl list -DRNi ``` To list all active/enabled and disabled/inactive (idle) NVDIMMs, Regions, and Namespaces with human readable values: ``` # ndctl list -iNuRD ``` # Concepts ## Persistent Memory Concepts This section describes the basic concepts when configuring and managing NVDIMMs. The following describes terms used throughout this chapter. **Interleave Sets:** two or more NVDIMMs create an N-Way interleave set to provide stripes read/write operations for increased throughput. Commonly 2-Way or 4-Way interleaving. **Region:** a grouping of one or more NVDIMMs, or an interleaved set, that can be divided up into one or more Namespaces. Regions are created within interleaved sets. **Label Storage Area (LSA):** Namespaces are defined by Labels which are stored in the Label Storage Area(s). **Namespace:** defines a contiguously-addressed range of Non-Volatile Memory conceptually similar to a hard disk partition, SCSI Logical Unit (LUN), or an NVM Express namespace. It is the unit of persistent memory storage that appears in /dev as a device usable for I/O. **Type:** defines the way in which the persistent memory associated with a Namespace or Region can be accessed. Valid Types are PMEM and BLK: * **PMEM:** Direct access to the media via load/store operations. DAX Supported. * **BLK:** Direct access to the media via Apertures (sliding memory windows). DAX is not supported. **Mode:** defines which NVDIMM software features are enabled for a given Namespace. Namespace Modes include raw, sector, fsdax(memory), and devdax(dax). Sibling Namespaces of the same parent Region will always have the same Type, but might be configured to have different Modes. ## Configuration Options This section discusses some of the common configuration options but does not cover all possible supported config. Using one or more physical NVDIMMs, the available capacity can be configure in many different ways. Figure 1 below shows a typical Filesystem-DAX (FSDAX) configuration with three physical NVDIMMs that have been interleaved. All available capacity has been used to create a single region (Region0), namespace (Namespace0.0), and dax enabled filesystem created. Namespace naming convention is commonly X.Y where X is the region number and Y is the namespace. The persistent memory pool(s) are memory mapped to the application, thus allowing direct load/store access to the NVDIMMs. ![](https://2827761497-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlBtYw13PknzW_5Tx%2Fdraw.io-gitbook-interleaved-dimms-fsdax.jpg?alt=media\&token=5f2e188f-5683-4441-8e5d-2e2c75310589) Figure 2 shows a configuration where the region has been partitioned into two namespaces. Each namespace has a single DAX Filesystem. ![](https://2827761497-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlFmTfOvpD8GUY9eA%2Fdraw.io-gitbook-interleaved-dimms-with-2-namespaces-fsdax%20\(1\).jpg?alt=media\&token=05d9d599-9773-470b-ba61-4bac6491b4d5) Figure 3 shows a configuration where NVDIMM interleaving has been disabled. In this mode, each NVDIMM becomes addressable with single regions per device. Regions and Namespaces cannot be striped or interleaved across NVDIMMs. It is possible to use software RAID such as Linux's Device Mapper to implement mirroring, striping, or concatenation of the /dev/pmem{N} devices. ![](https://2827761497-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlSA1qHvOy6St0XnM%2Fdraw.io-gitbook-non-interleaved-dimm-fsdax.jpg?alt=media\&token=2ca8c490-07d3-4fd0-84c6-49cf2f5d3457) The following sections describe each layer in more detail. # NVDIMM Devices A generic DIMM device object, named /dev/nmemX, is registered for each physical memory device indicated in the ACPI NFIT table, or other platform NVDIMM resource discovery mechanism. Physical NVDIMMs may have configuration options available via the BIOS. The features and available configuration options will be dependant on the NVDIMM device and BIOS. The Linux LIBNVDIMM core provides a built-in driver for these DIMM devices. The driver is responsible for determining if the DIMM implements a namespace label area, and initializing the kernel's in-memory copy of that label data. The kernel performs runtime modifications of that data when namespace provisioning actions are taken, and actively blocks user-space from initiating label data changes while the DIMM is active in any region. Disabling a DIMM, after all the regions it is a member of have been disabled, allows userspace to manually update the label data to be consumed when the DIMM is next enabled. Refer to [Managing NVDIMMs](/ndctl-user-guide/master/managing-nvdimms) for more information on administering NVDIMMs. # PMEM and BLK Modes The NFIT (**N**VDIMM **F**irmware **I**nterface **T**able) specification defined by the [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) standardizes not only the description of Persistent Memory (PMEM) and Block (BLK) modes, but also the platform message-passing entry points for control and configuration. ## LIBNVDIMM Sub-System The Linux LIBNVDIMM subsystem provides support for three types of NVDIMMs; PMEM, BLK, and NVDIMM. PMEM (Persistent Memory) devices allow byte addressable access. BLK (block) devices allow sector atomicity like traditional storage devices. NVDIMM devices can simultaneously support both PMEM and BLK mode access. These three modes of operation are described by the "NVDIMM Firmware Interface Table" (NFIT) in [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) or later. While the LIBNVDIMM implementation is generic and supports pre-NFIT platforms, it was guided by the superset of capabilities need to support this ACPI v6.0 definition for NVDIMM resources. The bulk of the kernel implementation is in place to handle the case where the DIMM Physical Address (DPA) accessible via PMEM is aliased with DPA accessible via BLK. When that occurs a LABEL is needed to reserve DPA for exclusive access via one mode a time. For each NVDIMM access method (PMEM, BLK), LIBNVDIMM provides a block device driver: 1. **PMEM (nd\_pmem.ko):** Drives a system-physical-address range. This range is contiguous in system memory and may be interleaved (hardware memory controller striped) across multiple DIMMs. When interleaved the platform may optionally provide details of which DIMMs are participating in the interleave. Note that while LIBNVDIMM describes system-physical-address ranges that may alias with BLK access as ND\_NAMESPACE\_PMEM ranges and those without alias as ND\_NAMESPACE\_IO ranges, to the nd\_pmem driver there is no distinction. The different device-types are an implementation detail that userspace can exploit to implement policies like "only interface with address ranges from certain DIMMs". It is worth noting that when aliasing is present and a DIMM lacks a label, then no block device can be created by default as userspace needs to do at least one allocation of DPA to the PMEM range. In contrast ND\_NAMESPACE\_IO ranges, once registered, can be immediately attached to nd\_pmem. 2. **BLK (nd\_blk.ko):** This driver performs I/O using a set of platform defined apertures. A set of apertures will access just one DIMM. Multiple windows (apertures) allow multiple concurrent accesses, much like tagged-command-queuing, and would likely be used by different threads or different CPUs. The NFIT specification defines a standard format for a BLK-aperture, but the spec also allows for vendor specific layouts, and non-NFIT BLK implementations may have other designs for BLK I/O. For this reason "nd\_blk" calls back into platform-specific code to perform the I/O. One such implementation is defined in the "Driver Writer's Guide" and "DSM Interface Example". ## Differences between PMEM and BLK Modes While PMEM provides direct byte-addressable CPU-load/store access to NVDIMM storage, it does not provide the best system RAS (recovery, availability, and serviceability) model. An access to a corrupted system-physical-address address causes a CPU exception while an access to a corrupted address through an BLK-aperture causes that block window to raise an error status in a register. The latter is more aligned with the standard error model that host-bus-adapter attached disks present. Also, if an administrator ever wants to replace a memory it is easier to service a system at DIMM module boundaries. Compare this to PMEM where data could be interleaved in an opaque hardware specific manner across several DIMMs. BLK-apertures solve these RAS problems, but their presence is also the major contributing factor to the complexity of the ND subsystem. They complicate the implementation because PMEM and BLK alias in DPA space. Any given DIMM's DPA-range may contribute to one or more system-physical-address sets of interleaved DIMMs, and may also be accessed in its entirety through its BLK-aperture. Accessing a DPA through a system-physical-address while simultaneously accessing the same DPA through a BLK-aperture has undefined results. For this reason, DIMMs with this dual interface configuration include a DSM function to store/retrieve a LABEL. The LABEL effectively partitions the DPA-space into exclusive system-physical-address and BLK-aperture accessible regions. For simplicity a DIMM is allowed a PMEM "region" per each interleave set in which it is a member. The remaining DPA space can be carved into an arbitrary number of BLK devices with discontiguous extents. # Regions A generic REGION device is registered for each PMEM range or BLK-aperture set. LIBNVDIMM provides a built-in driver for these REGION devices. This driver is responsible for reconciling the aliased DPA mappings across all regions, parsing the LABEL, if present, and then emitting NAMESPACE devices with the resolved/exclusive DPA-boundaries for the `nd_pmem` or `nd_blk` device driver to consume. Refer to [Managing Regions](/ndctl-user-guide/master/managing-regions) for more information. # Regions, Atomic Sectors, and DAX One of the few reasons to allow multiple BLK namespaces per REGION is so that each BLK-namespace can be configured with a Block Translation Table (BTT) with unique atomic sector sizes. While a PMEM device can host a BTT, the LABEL specification does not provide for a sector size to be specified for a PMEM namespace. This is due to the expectation that the primary usage model for PMEM is via DAX, and the BTT is incompatible with DAX. However, for the cases where an application or filesystem still needs atomic sector update guarantees, it can register a BTT on a PMEM device or partition. ## Example NVDIMM Platform Figure 1 shows an example platform with four NVDIMMs, two integrated memory controllers (IMC), and a single CPU . ``` (a) (b) DIMM BLK-REGION +-------------------+--------+--------+--------+ +------+ | pm0.0 | blk2.0 | pm1.0 | blk2.1 | 0 region2 | imc0 +--+- - - region0- - - +--------+ +--------+ +--+---+ | pm0.0 | blk3.0 | pm1.0 | blk3.1 | 1 region3 | +-------------------+--------v v--------+ +--+---+ | | | cpu0 | region1 +--+---+ | | | +----------------------------^ ^--------+ +--+---+ | blk4.0 | pm1.0 | blk4.0 | 2 region4 | imc1 +--+----------------------------| +--------+ +------+ | blk5.0 | pm1.0 | blk5.0 | 3 region5 +----------------------------+--------+--------+ ``` *Figure 1: Example sysfs layout* Each unique interface (BLK or PMEM) to DPA space is identified by a region device with a dynamically assigned id (REGION0 - REGION5). 1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A single PMEM namespace is created in the REGION0-SPA-range that spans most of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that interleaved system-physical-address range is reclaimed as BLK-aperture accessed space starting at DPA-offset (a) into each DIMM. In that reclaimed space we create two BLK-aperture "namespaces" from REGION2 and REGION3 where "blk2.0" and "blk3.0" are just human readable names that could be set to any user-desired name in the LABEL. 2. In the last portion of DIMM0 and DIMM1 we have an interleaved ``` system-physical-address range, REGION1, that spans those two DIMMs as ``` well as DIMM2 and DIMM3. Some of REGION1 is allocated to a PMEM namespace named "pm1.0", the rest is reclaimed in 4 BLK-aperture namespaces (for each DIMM in the interleave set), "blk2.1", "blk3.1", "blk4.0", and "blk5.0". 3. The portion of DIMM2 and DIMM3 that do not participate in the REGION1 interleaved system-physical-address range (i.e. the DPA address past offset (b) are also included in the "blk4.0" and "blk5.0" namespaces. Note, that this example shows that BLK-aperture namespaces don't need to be contiguous in DPA-space. This bus is provided by the kernel under the device `/sys/devices/platform/nfit_test.0` when CONFIG\_NFIT\_TEST is enabled and the nfit\_test.ko module is loaded. This not only test LIBNVDIMM but the acpi\_nfit.ko driver as well. # Label Storage Area (LSA) The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to store the definition of NVDIMM *namespaces.* # Namespaces The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. A REGION is the Linux term for what ACPI and UEFI call a DIMM-interleave-set, or a system-physical-address-range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label-less" namespace per region. Examples of label-less namespaces are the ones created by the kernel's memmap option, or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, fsdax, devdax, sector, and raw: * **fsdax:** Filesystem-DAX mode is the default mode of a namespace when specifying `ndctl create-namespace` with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working-sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. * **devdax:** Device-DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem-DAX. Instead of a block-device that can support a DAX enabled filesystem, this mode provides a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual-machine, register persistent memory for RDMA, or when gigantic mappings are needed. * **sector:** Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. * **raw:** Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux fsdax or devdax mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. Figure 1 below shows a typical Filesystem-DAX (FSDAX) configuration with three physical NVDIMMs that have been interleaved. All available capacity has been used to create a single region (Region0), namespace (Namespace0.0), and dax enabled filesystem created. Namespace naming convention is commonly X.Y where X is the region number and Y is the namespace ![](https://2827761497-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlBtYw13PknzW_5Tx%2Fdraw.io-gitbook-interleaved-dimms-fsdax.jpg?alt=media\&token=5f2e188f-5683-4441-8e5d-2e2c75310589) Refer to [Managing Namespaces](/ndctl-user-guide/master/managing-namespaces) for information on how to create, delete, edit, and manage the namespace configuration. # Managing NVDIMMs Managing physical or emulated NVDIMMs using `ndctl` has no functional difference. Physical NVDIMM features and options may be controlled through the system BIOS. The BIOS cannot see emulated NVDIMMs. Observe the following restrictions when managing NVDIMMs * DO NOT change the memory slot of physical NVDIMMs when they are part of an Interleave Set. Doing so changes the order of interleaving so data access will be compromised or corrupted. If interleaving is disabled, moving NVDIMMs to different slot locations is okay, but not recommended. * DO NOT disable NVDIMMs when they are part of an active Region and/or Namespace as this will prevent data access and may corrupt data. * ALWAYS backup the data and make a copy of the configuration layout prior to any changes. ## Listing active/enabled NVDIMMs The `ndct list -D` , or equivalent `ndct list --dimm` , can be used to show active/enabled NVDIMM devices on the system, eg: ``` # ndctl list -D { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29 } ``` ## Listing disabled/inactive NVDIMMs By default, `ndctl` only lists enabled/active dimms, regions, and namespaces. To include previously disabled (inactive) NVDIMMs, include the `-i` flag to show both enabled and disabled devices, eg: ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":0, "phys_id":29 } { "dev":"nmem1", "id":"9759-b5-1459-00000502", "handle":1, "phys_id":30 "state":"disabled" } ... ``` {% hint style="info" %} NVDIMM vendor specific tools can be used to display more information about the NVDIMMs from the operating system layer. For example, Intel Optane DC Persistent Memory Modules can be managed using the [ipmctl](https://github.com/intel/ipmctl) utility. Refer to the [IPMCTL User Guide](https://docs.pmem.io/ipmctl-user-guide/) for more information. For other persistent memory products, refer to the vendor specific documentation. {% endhint %} ## Disabling NVDIMMs NVDIMMs can only be disabled if they have no active Regions or Namespaces. If an active/enabled namespace and/or region exists, a message is displayed: ``` # ndctl disable-dimm nmem0 nmem0 is active, skipping... disabled 0 nmem ``` 1\) List the current active/enabled configuration ``` # ndctl list -NRD ``` 2\) Verify no fsdax or devdax namespaces are mounted or in-use by running applications 3\) Destroy or disable any active/enabled namespace(s). ``` # ndctl disable-namespace - or - # ndctl destroy-namespace ``` See [Destroying Namespaces](/ndctl-user-guide/master/managing-namespaces#destroying-namespaces) or [Disabling Namespaces](/ndctl-user-guide/master/managing-namespaces#disabling-namespaces) for more information. 4\) Disable the regions used by the NVDIMM (nmem) that needs to be disabled ``` # ndctl disable-region ``` See [Disabling Regions](/ndctl-user-guide/master/managing-regions#disabling-regions) for more information. 5\) Disable a single, subset, or all NVDIMM (nmem) devices To disable a single NVDIMM, use: ``` # ndctl disable-dimm nmem0 disabled 1 nmem ``` To disable a subset or specific list or NVDIMMs, use: ``` # ndctl disable-dimm nmem0 nmem1 nmem5 disabled 3 nmem ``` To disable all NVDIMMs, use: ``` # ndctl disable-dimm all disabled 12 nmem ``` 6\) Verify the NVDIMM(s) are disabled by listing inactive dimms and verifying the 'state': ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29, "state":"disabled" } ... ``` ## Enabling NVDIMMs 1\) Verify the nmem device, or list of nmem devices, that need to be enabled using the `ndctl list -Di` command: ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29, "state":"disabled" } ``` 2\) Enable the NVDIMM(s) To enable a single NVDIMM (nmemX) device, use: ``` # ndctl enable-dimm nmem0 enabled 1 nmem ``` To enable a subset of disabled NVDIMMs, use: ``` # ndctl enable-dimm nmem0 nmem1 nmem5 enabled 3 nmem ``` To enable all disabled NVDIMMs, use: ``` # ndctl enable-dimm all enabled 12 nmem ``` 3\) Verify the state by listing all NVDIMMs ``` # ndctl list -Di ``` A filtered list of NVDIMMs can shown using the `-d ` or `-dimm ` option, eg: ``` # ndctl list -Di -d nmem0 - or - # ndctl list -Di -dimm nmem0 ``` # Managing Regions A region is a grouping of one or more NVDIMMs, or an interleaved set, that can be divided up into one or more Namespaces. Regions are of type PMEM or BLK. See [PMEM or BLK](/ndctl-user-guide/master/concepts/libnvdimm-pmem-and-blk-modes) modes for more information. Regions can only be created, destroye, and configured using the vendor specific NVDIMM utility that manages the NVDIMMs or BIOS options, if available. ## Disabling Regions 1\) List all active/enabled regions: ``` # ndctl list -R { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of active/enabled regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -R -r region0 - or - # ndctl list -R -region region0 ``` 2\) Disable the region ``` # ndctl disable-region region0 disabled 1 region ``` 3\) Verify the region is disabled by including the `-i` option: ``` # ndctl list -Ri { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of disabled/inactive regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -Ri -r region0 - or - # ndctl list -Ri -region region0 ``` ## Enabling Regions 1\) List disabled/inactive regions ``` # ndctl list -Ri { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of active/enabled regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -R -r region0 - or - # ndctl list -R -region region0 ``` 2\) Enable the region ``` # ndctl enable-region region0 enabled 1 region ``` 3\) Verify the region is enabled using `ndctl list -R`. Note the 'state' field is not displayed for enabled regions. ``` # ndctl list -R { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "persistence_domain":"memory_controller" } ``` # Managing Label Storage Areas (LSA) The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to store the definition of any NVDIMM namespaces. The labels on the NVDIMMs cannot be edited directly. Changing the configuration can be achieved using the ndctl **create** or **destroy** commands. See [Managing Namespaces](/ndctl-user-guide/master/managing-namespaces) and [Managing Regions](/ndctl-user-guide/master/managing-regions). `ndctl` provides several label related commands shown below: * [check-labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#checking-labels) - determine if the given dimm(s) have a valid namespace index block * [init-labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#initializing-labels) - initialize the label data area on a dimm or set of dimms * [read-labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#reading-labels) - read out the label area on a dimm or set of dimms * [write-labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#writing-labels) - write data to the label area on a dimm * [zero-labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#erasing-zeroing-labels) - zero out the label area on a dimm or set of dimms ## Checking Labels The `check-labels` command validates the index block within the label storage area on the specified NVDIMMs. If issues are found, running the command in verbose mode using the `-v` option, reports the reason the index block is deemed invalid. ``` usage: ndctl check-labels [..] [] -b, --bus must be on a bus with an id/provider of -v, --verbose turn on debug ``` 1\) Identify the NVDIMMs (nmem's) in the host using the `ndctl list -D` command: ``` # ndctl list -D [ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ] ``` 2\) Check the labels of one or more devices. To check the labels on a single device: ``` # ndctl check-labels nmem3 ``` To check the labels on multiple devices: ``` # ndctl check-labels nmem3 nmem0 nmem2 ``` To check the labels on all devices: ``` # ndctl check-labels all ``` To display more information, if available, use the `-v` option: ``` # ndctl check-labels nmem3 -v ``` {% hint style="info" %} Note: The `-v` option may not produce any additional information if the labels are valid or ndctl was not built with logging or debug options. {% endhint %} ## Initializing Labels This command can be used to initialize the namespace index block if it is missing or reinitialize it if it is damaged or was previously zero'd. Note that reinitialization effectively destroys all existing namespace labels on the DIMM. {% hint style="danger" %} This is an expert mode operation. Overwriting labels will cause data loss. Backup all data prior to overwriting labels. {% endhint %} ``` usage: ndctl init-labels [..] [] -b, --bus must be on a bus with an id/provider of -v, --verbose turn on debug -f, --force force initialization even if existing index-block present -V, --label-version namespace label specification version (default: 1.1) ``` Before initializing the labels, all namespaces and regions must be destroyed or disabled. A warning message is displayed when attempting to initialize labels on active regions. ``` # ndctl init-labels nmem3 nmem2 nmem3: regions active, abort label write nmem2: regions active, abort label write ``` 1\) Identify the active/enabled NVDIMM devices, Regions, and Namespaces ``` # ndctl list -DRN { "dimms":[ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ], "regions":[ { "dev":"region1", "size":534723428352, "available_size":0, "type":"pmem", "iset_id":3026489321642266624, "mappings":[ { "dimm":"nmem3", "offset":268435456, "length":267361714176, "position":1 }, { "dimm":"nmem2", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":526366277632, "uuid":"76fc4dd1-e793-4eee-81ed-2f31275eb820", "blockdev":"pmem1" } ] }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "mappings":[ { "dimm":"nmem0", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 2\) Destroy or disable the namespace(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl destroy-namespace -f namespace1.0 - or - # ndctl disable-namespace namespace1.0 # ndctl destroy-namespace namespace1.0 ``` 3\) Disable the region(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl disable-region region1 ``` 4\) Initialize the labels ``` # ndctl init-labels nmem2 nmem3 ``` If the `init-labels` command returns "error: labels already initialized", use the `-f`, `--force` option. 5\) Enable the region(s). After initializing the labels, the region(s) may not be brought online automatically. List the active and disabled regions: ``` # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "state":"disabled", "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` Activate any disabled regions ``` # ndctl enable-region region1 # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` 6\) New namespaces can now be created on the regions. See [Managing Namespaces](/ndctl-user-guide/master/managing-namespaces) for more information. ## Reading Labels The `read-labels` command dumps the data in a dimm’s label area to stdout or a file in raw or JSON format. When multiple dimms (nmem's) are specified, the data is concatenated. Usage: ``` ndctl read-labels [..] [] OPTIONS One or more nmemX device names. The keyword all can be specified to operate on every dimm in the system, optionally filtered by bus id (see --bus= option). -b, --bus= Limit operation to memory devices (dimms) that are on the given bus. Where bus can be a provider name or a bus id number. -v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). -o, --output output file -j, --json parse the label data into json assuming the NVDIMM Namespace Specification format. ``` The following shows a label from nvdimm0 (nmem0) printed to stdout in JSON format. ``` # ndctl read-labels -j nmem0 { "dev":"nmem0", "index":[ { "signature":"NAMESPACE_INDEX", "major":1, "minor":2, "labelsize":256, "seq":2, "nslot":510 }, { "signature":"NAMESPACE_INDEX", "major":1, "minor":2, "labelsize":256, "seq":1, "nslot":510 } ], "label":[ { "uuid":"9ed56df2-c0a9-4e17-b82d-28f642f95a22", "name":"", "slot":0, "position":0, "nlabel":1, "isetcookie":710683120631319073, "lbasize":512, "dpa":268435456, "rawsize":267361714176, "type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb", "abstraction_guid":"ba006426-9ffb-7746-bcb0-968f11d0d225" } ] } read 1 nmem ``` To backup an NVDIMM label using the raw format, use: ``` # ndctl read-labels nmem0 > nmem0.lsa ``` ## Writing Labels {% hint style="danger" %} This is an expert mode operation. Overwriting labels will cause data loss. Backup all data prior to overwriting labels. {% endhint %} Writing labels overwrites the current label with the contents of the specified file, or stdin, collected from a [read-labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#reading-labels) operation from a different label, or a previous label backup. Read data from the input filename, or stdin, and write it to the given device. The device being written to must not be active in any region, otherwise the kernel will not allow write access to the device’s label data area. The following shows how a raw label from an active nvdimm can be written to an inactive nvdimm. ``` $ ndctl read-labels nmem0 -o nmem0.label.backup.raw $ ndctl write-labels nmem1 -i nmem0.label.backup.raw ``` The above read-label raw output can be piped directly to the write-label command using: ``` $ ndctl read-labels nmem0 | ndctl write-labels nmem1 ``` ## Erasing/Zeroing Labels The `zero-labels` command resets the device to its default state by deleting all labels. {% hint style="danger" %} This operation cannot be undone unless a previous backup of the labels was created using the [read-labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#reading-labels) command. All data on the device will be destroyed. Create a backup before proceeding. {% endhint %} Before erasing/zeroing the labels, all namespaces and regions must be destroyed or disabled. 1\) Identify the active/enabled NVDIMM devices, Regions, and Namespaces ``` # ndctl list -DRN { "dimms":[ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ], "regions":[ { "dev":"region1", "size":534723428352, "available_size":0, "type":"pmem", "iset_id":3026489321642266624, "mappings":[ { "dimm":"nmem3", "offset":268435456, "length":267361714176, "position":1 }, { "dimm":"nmem2", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":526366277632, "uuid":"76fc4dd1-e793-4eee-81ed-2f31275eb820", "blockdev":"pmem1" } ] }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "mappings":[ { "dimm":"nmem0", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 2\) Destroy or disable the namespace(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl destroy-namespace -f namespace1.0 - or - # ndctl disable-namespace namespace1.0 # ndctl destroy-namespace namespace1.0 ``` 3\) Disable the region(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl disable-region region1 ``` 4\) To erase the labels on a single 'nmem0' device. ``` $ ndctl zero-labels nmem0 ``` To erase labels on multiple devices, use: ``` $ ndctl zero-labels nmem0 nmem2 nmem3 ``` 5\) Enable the region(s). After initializing the labels, the region(s) may not be brought online automatically. List the active and disabled regions: ``` # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "state":"disabled", "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` Activate any 'disabled' regions ``` # ndctl enable-region region1 # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` 6\) New namespaces can now be created on the regions. See [Managing Namespaces](/ndctl-user-guide/master/managing-namespaces) for more information. # Managing Namespaces Regions can be partitioned in to one or more namespaces. Namespace operations include listing, creating, destroying, enabling, disabling, and checking (validating) using the following commands: * [list ](/ndctl-user-guide/master/managing-namespaces#listing-namespaces)- dump the platform nvdimm device topology and attributes in json format * [create-namespace](/ndctl-user-guide/master/managing-namespaces#creating-namespaces) - provision or reconfigure a namespace * [destroy-namespace](/ndctl-user-guide/master/managing-namespaces#destroying-namespaces) - destroy the given namespace(s) * [disable-namespace](/ndctl-user-guide/master/managing-namespaces#disabling-namespaces) - disable the given namespace(s) * [enable-namespace](/ndctl-user-guide/master/managing-namespaces#enabling-namespaces) - enable the given namespace(s) * [check-namespace](/ndctl-user-guide/master/managing-namespaces#checking-sector-namespaces) - check namespace metadata consistency ## Listing Namespaces #### Displaying Active/Enabled Namespaces The `ndctl list -N` command is used to display namespaces, for example: ``` # ndctl list -N { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":4294967296, "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` The output can be filtered using any combination of the following options: ``` -n, --namespace= An namespaceX.Y device name, or namespace region plus id tuple X.Y. Limit the namespace list to the single identified device if present. -m, --mode= Filter listing by the mode (raw, fsdax, sector or devdax) of the namespace(s). ``` **Examples:** List all active namespaces ``` # ndctl list -N ``` List all active namespaces and print human readable values (MB, MiB, GB, GiB, etc) ``` # ndctl list -Nu { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":"4.00 GiB (4.29 GB)", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` List a single namespace (namespace0.0) ``` # ndctl list -n namespace0.0 ``` List all active 'fsdax' mode namespaces ``` # ndctl list -m fsdax ``` #### Displaying Disabled/Inactive Namespaces By default, `ndctl list -N` lists only active/enabled namespaces. In the following example no active/enabled namespaces exist so no output is displayed: ``` # ndctl list -N # ``` Adding the `-i` flag includes both active/enabled and inactive/destroyed/disabled namespaces. The following example shows two previously destroyed namespaces. The `size` and `uuid` are clear indications that no information exists about the namespace. The `ndctl destroy` command simply NULLs the existing namespace entry and does not completely erase it from the metadata. For this reason, it may be possible to see more than one namespace using `ndctl list -Ni` ``` # ndctl list -Ni [ { "dev":"namespace0.0", "mode":"raw", "size":0, "uuid":"00000000-0000-0000-0000-000000000000", "sector_size":512, "state":"disabled", "numa_node":0 } ] ``` The `-i` option can be used in coordination with any other valid ndctl option. **Examples:** List all enabled and disabled namespaces ``` # ndctl list -Ni ``` List a single disabled namespace (namespace0.0) ``` # ndctl list -in namespace0.0 ``` List all enabled and disabled 'fsdax' mode namespaces ``` # ndctl list -im fsdax ``` ## Creating Namespaces The create-namespace command has a lot of options summarized below. The `ndctl-create-namespace` man page contains further details of each option. * **-t, --type:** Set the type of the namespace to either "pmem" or "blk" * **-m, mode:** Define the namespace mode - fsdax, devdax, sector, and raw * **-s, --size:** Specify the capacity of the namespace * **-a, --align:** Specify the alignments size, eg 4K or 2M * **-e, --reconfigure:** Reconfigure existing namespace configuration * **-u, --uuid:** Used for recovery purposes only * **-n, --name:** Specify a friendly name for namespaces that support labels * **-l, --sector-size:** Override the default sector size for block based namespaces * **-M, --map:** For 'fsdax' or 'devdax' namespaces, define whether metadata is stored in volatile memory (mem) or persistent storage (dev) * **-f, --force:** Allow the operation to continue on enabled namespaces * **-L, --autolabel, --no-autolabel:** Manage labels for legacy NVDIMMs that do not support labels * **-v, --verbose:** Emit verbose messages during the creation process * **-r, --region:** Limit the operation to a specific region * **-b, --bus:** Limit the operation to a specific bus The `mode` is the most important feature to get correct. The four modes available are defined as: * **fsdax:** Filesystem-DAX mode is the default mode of a namespace when specifying `ndctl create-namespace` with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working-sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. * **devdax:** Device-DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem-DAX. However, instead of a block-device that can support a DAX-enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual-machine, register persistent memory for RDMA, or when gigantic mappings are needed. * **sector:** Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. * **raw:** Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux fsdax or devdax mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. {% hint style="info" %} If the *-s, --size* option is used, the value provided to ndctl is used to create the namespace. Depending on the *mode*, the available capacity may be smaller than the specified size due to the calculated spared required for metadata. This is shown below in the examples. {% endhint %} ### FSDAX and DEVDAX Capacity Considerations The namespace configuration (label metadata) is stored at the beginning of the persistent memory address range. See the [Label Storage Area (LSA)](/ndctl-user-guide/master/concepts/label-area) and [Managing Label Storage Area (LSA)](/ndctl-user-guide/master/managing-label-storage-areas-lsa) sections for more information. In addition to the namespace label metadata, both fsdax and devdax modes require space to store the Page Frame Number (PFN) metadata (also known as “struct page” metadata) for each page within the namespace. A PFN is simply in index within physical memory, or on the NVDIMM, that is counted in page-sized units. For modes with PFN metadata, shown in Table 1 below, the requirement is 64 bytes per 4 KiB of persistent memory. To calculate the estimated space for metadata, use: $$ metadata \_{(bytes)} = (size \_{(bytes)}/4096)\*64 $$ **Example 1:** A 256GiB namespace requires \~3.93GiB of metadata (263182090240/4096)\*64 = 4112220160 bytes == 3.93GB **Example 2:** A 1TiB namespace requires \~17.18GiB of metadata (1099511627776/4096)\*64 = 17179869184 bytes == 17.18GiB The location of PFN metadata can either be in volatile DDR from the NVDIMM. The default is to keep metadata on the NVDIMM. The location is defined per namespace by the `-M, --map` configuration option. The following provides guidance for PFN metadata location: * **map=mem:** PFN metadata is stored in regular system memory (DDR) * Adequate for small persistent memory capacities * Frequent or high performance PFN access is required. For example, when time sensitive, or frequent, registration and deregistration of RDMA memory regions is important. Normal RDMA memory \[de]registration can be achieved with 'map=dev', but there are situations where \[de]registration needs to be very fast. This assumes the metadata fits in DDR memory. * **map=dev:** PFN metadata is stored on persistent memory (NVDIMM) * Intended for large persistent memory capacities. There might not be enough regular memory in the system if the ratio of persistent memory to DDR is high (>8:1) The persistence of the PFN metadata is not important as it's re-mapped at system boot time or when new namespaces are created/destroyed. It is convenient to store the metadata on the NVDIMM because it scales with the persistent memory capacity. The space for metadata is calculated and reserved at namespace creation time, meaning the usable capacity is lower than the physical capacity depending on the mode and map values. **Table 1: Features for each namespace mode** | Mode | Description | Device Path | Label Metadata | Atomicity | Filesystems | DAX | PFN Metadata | | ------ | -------------- | ----------- | -------------- | --------- | ----------- | --- | ------------ | | raw | raw | /dev/pmemN | no | no | yes | no | no | | sector | sector atomic | /dev/pmemNs | yes | yes | yes | no | no | | fsdax | filesystem DAX | /dev/pmemN | yes | no | yes | yes | yes | | devdax | device DAX | /dev/daxN.M | yes | no | no | yes | yes | **PFN Metadata Usage Example** The following demonstrates the difference between `--map=dev` and `--map=mem`. The example uses a \~250GiB region to create an fsdax namespace. A devdax namespace has identical behavior. When a namespace using the default `--map=dev` is created, the usable 'size' is \~4GiB smaller than the 249GiB region size due to the space reserved for PFN metdata. ``` # ndctl create-namespace -r region0 # ndctl list -RuN -r region0 { "regions":[ { "dev":"region0", "size":"249.00 GiB (267.36 GB)", "available_size":0, "type":"pmem", "iset_id":"0xb000000058000000", "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"245.11 GiB (263.18 GB)", "uuid":"19cccf5d-d227-42fe-b338-cfa882a6b6ce", "blockdev":"pmem0" } ] } ] } ``` When using `--map=mem`, the usable capacity on the namespace increases, but the amount of 'used' memory as shown by `free` has increased by \~4GiB. ``` # free total used free shared buff/cache available Mem: 362926556 5747588 356160304 2608 1018664 355200308 Swap: 32882684 0 32882684 # ndctl create-namespace -r region0 --map=mem # ndctl list -RuN -r region0 { "regions":[ { "dev":"region0", "size":"249.00 GiB (267.36 GB)", "available_size":0, "type":"pmem", "iset_id":"0xb000000058000000", "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":"249.00 GiB (267.36 GB)", "uuid":"ff626073-eef9-422c-9b09-814fc1498c1b", "blockdev":"pmem0" } ] } ] } # free total used free shared buff/cache available Mem: 362926556 9827632 352080120 2612 1018804 351120192 Swap: 32882684 0 32882684 ``` #### **Creating Namespace Examples** This example section has been split in to four, one for each mode described above: * [FSDAX Mode Examples](/ndctl-user-guide/master/managing-namespaces#fsdax-mode-examples) * [DEVDAX Mode Examples](/ndctl-user-guide/master/managing-namespaces#devdax-mode-examples) * [SECTOR Mode Examples](/ndctl-user-guide/master/managing-namespaces#sector-mode-examples) * [RAW Mode Examples](/ndctl-user-guide/master/managing-namespaces#raw-mode-examples) #### **FSDAX Mode Examples** **Example 1:** Create an 'fsdax' mode namespace using all available capacity (default behavior). The `ndctl create-namespace` command shown below is the equivalent of `ndctl create-namespace -m fsdax`. A new `/dev/pmem{X[.Y]}` device is created. This can be used to create a new DAX enabled filesystem such as XFS or EXT4. The 'X' value represents the region on which the namespace is created. This defaults to zero (0). If more than one namespace is created within a region, the naming convention of pmem{X.Y} is used, where Y represents a sequentially increasing integer value for the new namespace. If multiple namespaces within a region are created, the first namespace is always created as 'pmem0', it does not get re-enumerated to 'pmem0.0' like subsequent namespaces. ``` # ndctl create-namespace { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"a60e6a4f-274d-4cd5-8d39-c8dd263345e2", "raw_uuid":"8d28948f-5434-4c8d-8efa-581eacad265a", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ls -l /dev/pmem* brw-rw----. 1 root disk 259, 0 Jul 9 10:42 /dev/pmem0 ``` **Example 2:** Create a 50GiB 'fsdax' mode namespace. The 'size' provided to ndctl includes space required for metadata so the resulting available capacity for a filesystem will be smaller as the example below shows: ``` # ndctl create-namespace -m fsdax -s 50G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"638d67f3-4c18-4b2e-a6f2-a044bdc82253", "raw_uuid":"6c6dfdad-e12f-418d-9fb0-3a75032dd9de", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` **Note:** If the remaining capacity needs to be assigned to another namespace using the same or different mode, the remaining capacity can be assigned to the new namespace without specifying the '-s \' option. For example, executing `ndctl create-namespace` after the command above creates a new namespace in the same region using all remaining capacity (\~78GiB): ``` # ndctl create-namespace { "dev":"namespace0.1", "mode":"fsdax", "map":"dev", "size":"73.83 GiB (79.27 GB)", "uuid":"d7f9473e-97aa-48cf-aefa-128797c83e88", "raw_uuid":"6c116e57-19dd-43d8-ae03-039f1588a23a", "sector_size":512, "blockdev":"pmem0.1", "numa_node":0 } ``` **Example 3:** Create a namespace with a friendly name (tag) Tagging the namespace with a friendly name or description using the '-n, --name' option can be useful to know what a namespace is used for. This is particularly useful when provisioning multiple end-users, applications, or to identify a namespace with production or test/dev environments. Creating a namespace with a tag/name/description can be done using: ``` # ndctl create-namespace -n "PROD Web DB 1" { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"6a0abb59-5279-4731-921a-0099101e17f2", "raw_uuid":"03b40e23-56e1-407a-b5d1-f1ec929645c1", "sector_size":512, "blockdev":"pmem0", "name":"PROD Web DB 1", "numa_node":0 } ``` #### **DEVDAX Mode Examples** Create a pmem namespace with devdax mode using all available capacity ``` # ndctl create-namespace -m devdax { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"e1c1d031-71a3-4a6b-b3d7-a4037a5256c3", "raw_uuid":"2a15b77f-9f31-45e5-84a3-4ad06817302a", "daxregion":{ "id":0, "size":"123.04 GiB (132.12 GB)", "align":2097152, "devices":[ { "chardev":"dax0.0", "size":"123.04 GiB (132.12 GB)" } ] }, "numa_node":0 } # ls -l /dev/dax* crw-------. 1 root root 250, 1 Jul 9 10:39 /dev/dax0.0 ``` Create a 50GiB 'devdax' namespace ``` # ndctl create-namespace -s 50G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"b49087fc-9786-48d1-899b-6166c7c586f9", "raw_uuid":"f441a441-77ad-4286-9585-ac9511d5381a", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` #### **Sector Mode Examples** Create a sector mode namespace using all available capacity and a default 4K sector size ``` # ndctl create-namespace -m sector { "dev":"namespace0.0", "mode":"sector", "size":"124.88 GiB (134.09 GB)", "uuid":"08f4e273-bbdd-4d1d-85e8-cf1f847e1df7", "raw_uuid":"f3bd03ad-4225-4919-a370-bb6293180e4d", "sector_size":4096, "blockdev":"pmem0s", "numa_node":0 } # ls -l /dev/pmem0 brw-rw----. 1 root disk 259, 0 Jul 9 10:47 /dev/pmem0s ``` #### RAW Mode Examples Create a 'raw' mode namespace using all available capacity ``` # ndctl create-namespace -m raw { "dev":"namespace0.0", "mode":"raw", "size":"125.00 GiB (134.22 GB)", "uuid":"24c1d696-b197-4c89-96b2-f91b60f4f295", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ls -l /dev/pmem0 brw-rw----. 1 root disk 259, 0 Jul 9 10:52 /dev/pmem0 ``` ## Editing Namespace Properties Changing properties of existing namespaces can be done online using the `ndctl create-namespace -fe ` command. The `-e, --reconfigure` flag edits existing namespaces. Using the `-f` flag does not require that the namespace be manually disabled. The command may still fail if the namespace is currently being used. Supported options are listed above in the [Creating Namespaces](/ndctl-user-guide/master/managing-namespaces#creating-namespaces) section, or review the ndctl-create-namespace [man page](/ndctl-user-guide/master/man-pages). Some namespace properties are read-only and cannot be changed using the `ndctl` utility. [Resizing Namespaces](/ndctl-user-guide/master/managing-namespaces#resizing-namespaces) or [Changing Namespace Modes](/ndctl-user-guide/master/managing-namespaces#changing-namespace-modes) can be achieved by changing existing namespace properties. They are discussed below under their own headings to provide additional detail. #### Examples Add or Change a namespace name (also referred to as a tag or description): ``` # ndctl create-namespace -fe namespace0.0 -n "prod_web_db1" { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"086d4800-4dec-41f4-a3b8-bb3e53974c54", "raw_uuid":"411b37d2-284e-4626-8724-efd0855e3c39", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` ## Resizing Namespaces {% hint style="danger" %} WARNING: Data will be lost or corrupted if the new size is smaller than the existing size or extending the namespace and growing the filesystem(s). Backup all data before resizing namespaces. {% endhint %} The following example resizes an existing 'fsdax' namespace (namespace0.0) to 20GiB: ``` # ndctl create-namespace -fe namespace0.0 -s 20G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"19.69 GiB (21.14 GB)", "uuid":"75cac06f-fc80-4c51-8f0b-2d73e2d2e31b", "raw_uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` If there is an existing EXT4 or XFS filesystem on this device, the partition table may need to be manually changed to accommodate the new device capacity before the filesystem can be resized. Due to the many configuration options available when using fsdax devices, such as whether it's being managed by the Linux device-mapper, LVM (Linux Volume Manager), or just an XFS/EXT4 filesystem on top, refer to the relevant Linux distribution documentation for specific procedures to handle your configuration. ## Changing Namespace Modes The namespace mode can be changed on any enabled or disabled namespace. Valid modes are raw, sector, fsdax, and devdax. Refer to the `ndctl-create-namespace` man page or [Namespaces](/ndctl-user-guide/master/concepts/nvdimm-namespaces) chapter for more information on each mode. {% hint style="warning" %} {% endhint %} * Changing the namespace mode will destroy all current data. Backup all data before proceeding. * If a filesystem using an fsdax or sector namespace is mounted, unmount it before changing the mode. * If a devdax or raw namespace is currently being used by a running application, stop the application before changing modes. Switching modes of an enabled/active namespace can be done using `ndctl create-namespace -fe --mode=`. For example, to switch an enabled fsdax namespace to devdax mode, use: ``` # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":4294967296, "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ndctl create-namespace -fe namespace0.0 --mode=devdax { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":"3.94 GiB (4.23 GB)", "uuid":"d71504d1-0286-474d-839a-52afe7d0da5f", "daxregion":{ "id":0, "size":"3.94 GiB (4.23 GB)", "align":2097152, "devices":[ { "chardev":"dax0.0", "size":"3.94 GiB (4.23 GB)" } ] }, "numa_node":0 } # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":4225761280, "uuid":"d71504d1-0286-474d-839a-52afe7d0da5f", "chardev":"dax0.0", "numa_node":0 } ``` If the namespace is currently disabled, the `-f` flag is not required. ## Destroying Namespaces Namespaces should be inactive/disabled prior to deletion otherwise a warning message is displayed. The following shows an attempt to destroy an active namespace ``` # ndctl destroy-namespace namespace0.0 Error: namespace0.0 is active, specify --force for re-configuration error destroying namespaces: Device or resource busy ``` First disable the namespace, then destroy it ``` # ndctl disable-namespace namespace0.0 disabled 1 namespace # ndctl destroy-namespace namespace0.0 destroyed 1 namespace ``` Alternatively using the `-f` flag to the command forcibly destroys the namespace without doing any validation checks and does not require that the namespace be disabled first, eg: ``` # ndctl destroy-namespace -f namespace0.0 destroyed 1 namespace ``` ## Disabling Namespaces {% hint style="warning" %} **Warning:** Disabling namespaces while the namespace is mounted or in-use will result in undefined behavior by the application(s) using the namespace. Always stop the application and unmount filesystems (fsdax mode) before disabling namespaces. {% endhint %} 1\) Disable the namespace (namespace0.0) using: ``` # ndctl disable-namespace namespace0.0 disabled 1 namespace ``` 2\) Verify the namespace is *disabled*. The `-i` flag is required to display inactive/disabled namespaces otherwise only active/enabled namespaces are shown. The example below filters the output to a specific namespace (namespace0.0) to avoid potentially large volumes of output being shown: ``` # ndctl list -Ni -n namespace0.0 { "dev":"namespace0.0", "mode":"raw", "size":21474836480, "uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "state":"disabled", "name":"prod_web_db1", "numa_node":0 } ``` ## Enabling Namespaces 1\) Enable the namespace (namespace0.0) using: ``` # ndctl enable-namespace namespace0.0 enabled 1 namespace ``` 2\) Verify the namespace is enabled. The state is only displayed for disabled namespaces. If the state field is not listed, the namespace is assumed to be enabled. ``` # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":21137195008, "uuid":"75cac06f-fc80-4c51-8f0b-2d73e2d2e31b", "raw_uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` ## Checking Sector Namespaces The `check-namespace` command only works on namespaces in *sector mode*. Attempting to check namespaces in any other mode will yield an error similar to the following which attempts to check an fsdax mode namespace: ``` # ndctl check-namespace namespace0.0 namespace0.0: namespace_check: namespace0.0: check aborted, namespace online error checking namespaces: Device or resource busy ``` A namespace in the sector mode will have metadata on it to describe the Kernel BTT (Block Translation Table). The check-namespace command can be used to check the consistency of this metadata, and optionally, also attempt to repair it, if it has enough information to do so. The command usage is: ``` ndctl check-namespace [] ``` Where the available options include: ``` -b, --bus limit namespace to a bus with an id or provider of -r, --region limit namespace to a region with an id or name of -v, --verbose emit extra debug messages to stderr -R, --repair perform metadata repairs -L, --rewrite-log regenerate the log -f, --force check namespace even if currently active ``` **Examples** The namespace being checked has to be disabled before initiating a check on it as a precautionary measure. ``` # ndctl disable-namespace namespace0.0 # ndctl check-namespace namespace0.0 ``` The `-f, --force` option can override the need to manually disable the namespace first, although it is not recommended to do so: ``` # ndctl check-namespace -f namespace0.0 checked 1 namespace ``` Adding the `-v, --verbose` option yields more information, eg: ``` # ndctl check-namespace -f namespace0.0 -v namespace0.0: namespace_check: checking namespace0.0 namespace0.0: btt_discover_arenas: found 1 BTT arena namespace0.0: log_set_indices: arena[0]: log_index_0 = 0 namespace0.0: log_set_indices: arena[0]: log_index_1 = 1 namespace0.0: btt_check_arenas: checking arena 0 checked 1 namespace ``` Check a namespace and perform repairs if possible or required can be done using: ``` # ndctl disable-namespace namespace0.0 # ndctl check-namespace --repair namespace0.0 ``` # Managing NVDIMM Security {% hint style="info" %} Note: These security features require ndctl v64.0 or later and requires Linux Kernel 5.0 or later. Vendor specific utilities may also provide the same functionality on earlier Linux Kernel versions. For Intel(R) Optane(TM) DC Persistent Memory Modules, see the '[ipmctl](https://github.com/intel/ipmctl)' utility. {% endhint %} NVDIMMs that provide additional security mechanisms, such as the Inte(R) Optane(TM) DC Persistent Memory Modules, can be managed using the `ndctl` utility and the following sub-commands: * ndctl-load-keys * ndctl-freeze-security * ndctl-remove-passphrase * ndctl-sanitize-dimm * ndctl-setup-passphrase * ndctl-update-passphrase {% hint style="warning" %} **Note:** Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled, see '[Disabling Regions](/ndctl-user-guide/master/managing-regions#disabling-regions)'. For the *overwrite* operation, in addition to disabling the *regions*, the *nvdimms* also need to be disabled, see '[Disabling NVDIMMs](/ndctl-user-guide/master/managing-nvdimms)'. {% endhint %} The Intel(R) Device Specific Methods (DSM) specification [v1.7](http://pmem.io/documents/NVDIMM_DSM_Interface-V1.7.pdf) and [v1.8](http://pmem.io/documents/NVDIMM_DSM_Interface-V1.8.pdf) introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel-managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key-ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key-encryption-key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* ## Setup a User Passphrase The `ndctl setup-passphrase` is used to setup and enable a security passphrase for one or more NVDIMMs. To setup the passphrase for a NVDIMM or interleaved set of NVDIMMs, it is expected that the master key has previously been loaded into the user keyring. More information on how this can be done can be found in the kernel documentation at: . The *kek* encrypts the NVDIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. The command will fail if the passphrase is already in the user keyring or if a passphrase blob already exists in /etc/ndctl/keys. ## Update a User Passphrase The `ndctl update-passphrase` command updates the security passphrase for one or more NVDIMMs that have previously been setup using the `ndctl setup-passphrase` command. Prerequisites for this command to succeed: 1. The kek has been loaded into the kernel’s user keyring 2. `setup-passphrase` has successfully been executed on the NVDIMM(s) and the NVDIMM is unlocked The updated key blobs will be created in the /etc/ndctl/keys directory with a file name format of "nvdimm.blob". The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update-passphrase*: 1. Remove the encrypted passphrase from the kernel’s user keyring. 2. Rename the passphrase blob to old. 3. Load this old passphrase blob into the keyring with an “old” name. 4. Create the new passphrase and encrypt with the *kek*. 5. Send DSM with the old and new decrypted passphrases. 6. Remove old passphrase and the passphrase blob from the keyring. ## Remove a User Passphrase The `ndctl remove-passphrase` command prevents a DIMM from locking at power-loss and requiring a passphrase to access media. The *key-ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ## Crypto (Secure) Erase NVDIMMs This operation is similar to `ndctl remove-passphrase`. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use the `ndctl sanitize-dimm` command with the `--crypto-erase` option to perform this operation. The `ndctl sanitize-dimm` command performs a cryptographic destruction of the contents of the given NVDIMM. It scrambles the data, and any metadata or info-blocks, but it doesn’t modify namespace labels. Therefore, any namespaces on regions associated with the given NVDIMM will be retained, but they will end up in the raw mode. Additionally, after completion of this command, the security and passphrase for the given NVDIMM will be disabled, and the passphrase and any key material will also be removed from the keyring and the ndctl keys directory at /etc/ndctl/keys The command supports two different methods of performing the cryptographic erase. The default is `crypto-erase`, but additionally, an overwrite option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. ## Overwriting (Sanitizing) NVDIMMs One or more NVDIMMs can be sanitized using `ndctl sanitize-dimm [--crypto-erase][--overwrite]`. The command supports two different methods of performing the cryptographic erase. The default is `crypto-erase`, but additionally, an `overwrite` option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. The overwrite operation wipes the entire NVDIMM(s) and can take a significant amount of time depending on the size of the device(s) and number of devices. When the command returns successfully, it just means overwrite has been successfully started, the overwrite operation continues asynchronously in the background. Use the `ndctl wait-overwrite` command to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both `–crypto-erase` and `–overwrite` options are supplied, then crypto-erase is performed before overwrite. ## Security Freeze The `ndctl freeze-security` command does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. This is used in scenarios where the administrator has taken all expected security actions for the current boot and wants the NVDIMMs to enforce/lock the current state until system reboot. To check the current security status of a single NVDIMM (nmem{X} device): ``` $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"unlocked" } ] ``` To freeze a single NVDIMM: ``` $ ndctl freeze-security nmem0 security freezed 1 nmem. $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"frozen" } ] ``` ## Unlock NVDIMMs Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command `ndctl load-keys` is executed before the libnvdimm module is loaded by modprobe. See the 'Loading Keys' section. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1. Check the security state of the device and see if the DIMM is locked 2. Request the associated encrypted passphrase from the kernel’s user key ring 3. Use the *kek* to decrypt the passphrase 4. Create the unlock DSM, copy the decrypted payload into the DSM 5. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ## Loading Keys {% hint style="info" %} **Note#1:** All key files are expected to be in the format: nvdimm*\_hostname The '*' character is used to delimit the different components in the file name. Within the hostname, the '\_' character is allowed since it is the last component of the file name. **Note#1:** This command is typically never called directly by a user. {% endhint %} The `load-keys` command loads the master key (kek) and the encrypted passphrases for all NVDIMMs into the user keyring maintained by the kernel. The command is expected to be called during initialization and before the libnvdimm kernel module is loaded, typically from an initrd. This is typically set up using a modprobe config that calls the command before module load. See for more information on the Kernel implementation. ## Master Passphrase - Setup, Update, and Crypto Erase These operations are similar to creating and updating the user passphrase. The only difference is that a different passphrase is used. Both `setup-passphrase` and `update-passphrase` support the `-m` flag to differentiate between user and master passphrase operations. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. More information about the master key can be found in the kernel documentation at: . # NDCTL Man Pages This section links to the online man pages: * [ndctl](https://pmem.io/ndctl/ndctl.html) - Manage “libnvdimm” subsystem devices (Non-volatile Memory) * [ndctl-check-labels](https://pmem.io/ndctl/ndctl-check-labels.html) - determine if the given dimms have a valid namespace index block * [ndctl-check-namespace](https://pmem.io/ndctl/ndctl-check-namespace.html) - check namespace metadata consistency * [ndctl-create-namespace](https://pmem.io/ndctl/ndctl-create-namespace.html) - provision or reconfigure a namespace * [ndctl-destroy-namespace](https://pmem.io/ndctl/ndctl-destroy-namespace.html) - destroy the given namespace(s) * [ndctl-disable-dimm](https://pmem.io/ndctl/ndctl-disable-dimm.html) - disable one or more idle dimms * [ndctl-disable-namespace](https://pmem.io/ndctl/ndctl-disable-namespace.html) - disable the given namespace(s) * [ndctl-disable-region](https://pmem.io/ndctl/ndctl-disable-region.html) - disable the given region(s) and all descendant namespaces * [ndctl-enable-dimm](https://pmem.io/ndctl/ndctl-enable-dimm.html) - enable one more dimms * [ndctl-enable-namespace](https://pmem.io/ndctl/ndctl-enable-namespace.html) - enable the given namespace(s) * [ndctl-enable-region](https://pmem.io/ndctl/ndctl-enable-region.html) - enable the given region(s) and all descendant namespaces * [ndctl-freeze-security](https://pmem.io/ndctl/ndctl-freeze-security.html) * [ndctl-init-labels](https://pmem.io/ndctl/ndctl-init-labels.html) - initialize the label data area on a dimm or set of dimms * [ndctl-inject-error](https://pmem.io/ndctl/ndctl-inject-error.html) - inject media errors at a namespace offset * [ndctl-inject-smart](https://pmem.io/ndctl/ndctl-inject-smart.html) - perform smart threshold/injection operations on an NVDIMM * [ndctl-list](https://pmem.io/ndctl/ndctl-list.html) - print the platform nvdimm device topology and attributes * [ndctl-load-keys](https://pmem.io/ndctl/ndctl-load-keys.html) - load the master key (kek) and encrypted passphrases into the keyring * [ndctl-monitor](https://pmem.io/ndctl/ndctl-monitor.html) - Monitor the SMART events from NVDIMMs * [ndctl-read-labels](https://pmem.io/ndctl/ndctl-read-labels.html) - read out the label area on a NVDIMM or set of NVDIMMs * [ndctl-remove-passphrase](https://pmem.io/ndctl/ndctl-remove-passphrase.html) - Stop an NVDIMM from locking at power-loss and requiring a passphrase to access media * [ndctl-sanitize-dimm](https://pmem.io/ndctl/ndctl-sanitize-dimm.html) - Perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s) * [ndctl-setup-passphrase](https://pmem.io/ndctl/ndctl-setup-passphrase.html) - setup and enable the security passphrase for one or more NVDIMMs * [ndctl-start-scrub](https://pmem.io/ndctl/ndctl-start-scrub.html) - start an Address Range Scrub (ARS) operation * [ndctl-update-firmware](https://pmem.io/ndctl/ndctl-update-firmware.html) - provides for updating the firmware on an NVDIMM * [ndctl-update-passphrase](https://pmem.io/ndctl/ndctl-update-passphrase.html) - update the security passphrase for one or more NVDIMMs * [ndctl-wait-overwrite](https://pmem.io/ndctl/ndctl-wait-overwrite.html) - wait for an overwrite operation to complete * [ndctl-wait-scrub](https://pmem.io/ndctl/ndctl-wait-scrub.html) - wait for an Address Range Scrub (ARS) operation to complete * [ndctl-write-labels](https://pmem.io/ndctl/ndctl-write-labels.html) - write data to the label area on a dimm * [ndctl-zero-labels](https://pmem.io/ndctl/ndctl-zero-labels.html) - zero out the label area on a dimm or set of dimms # Troubleshooting The `ndctl` utility is designed to be user friendly and informative. This section describes some of the common messages seen when using `ndctl`. ## "DAX unsupported by block device. Turning off DAX." **Issue:** During system boot or mounting of an EXT4 or XFS filesystem, the following message may be seen in dmesg: ``` [125.755367] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk [125.763736] XFS (pmem0): DAX unsupported by block device. Turning off DAX. ``` The filesystem should still mount, but it will not have the 'dax' (Direct Access) flag set as shown by the mount command: ``` $ mount | grep pmem0 /dev/pmem0 on /mnt/pmem0 type xfs (rw,relatime,attr2,inode64,noquota) ``` **Solution:** Check the namespace mode using `ndctl list -Nu`. In the following example the mode is 'raw': ``` # ndctl list -Nu { "namespaces":[ { "dev":"namespace0.0", "mode":"raw", "size":"704.00 MiB (738.20 MB)", "blockdev":"pmem0", "numa_node":0 } ] } ``` From the `ndctl-create-namespace` man page, 'raw' mode does not support DAX (Direct Access). ``` -m, --mode= · "raw": expose the namespace capacity directly with limitations. Neither a raw pmem namepace nor raw blk namespace support sector atomicity by default (see "sector" mode below). A raw pmem namespace may have limited to no dax support depending the kernel. In other words operations like direct-I/O targeting a dax buffer may fail for a pmem namespace in raw mode or indirect through a page-cache buffer. See "fsdax" and "devdax" mode for dax operation. ``` The namespace needs to be changed to either 'fsdax' or 'devdax' using the following: {% hint style="warning" %} **WARNING:** Changing the namespace mode will destroy any existing data. Backup all data before changing the mode. {% endhint %} ``` $ sudo ndctl create-namespace -f -e namespace0.0 --mode=fsdax { "dev":"namespace0.0", "mode":"memory", "size":"690.00 MiB (723.52 MB)", "uuid":"d1cc8945-7bf7-4b57-83b5-0a79565e9c15", "blockdev":"pmem0", "numa_node":0 } ``` The filesystem needs to be re-created, mounted with the 'dax' option, then any data restored: ``` $ sudo mkfs.xfs /dev/pmem0 $ sudo mount -o dax /dev/pmem0 /mnt/pmem0 $ mount | grep pmem0 /dev/pmem0 on /mnt/pmem0 type xfs (rw,relatime,attr2,dax,inode64,noquota) ``` ## "nmem0 is active, skipping..." **Issue:** When disabling an NVDIMM (nmem) device, a notice is displayed indicating it is active, eg: ``` # ndctl disable-dimm nmem0 nmem0 is active, skipping... disabled 0 nmem ``` **Cause:** The message indicates there's at least one active/enabled Region and/or Namespace using the NVDIMM. **Solution:** All active/enabled Regions and Namespaces must be destroyed an/or disabled prior to disabling the dimm. 1\) List the current configuration: ``` # ndctl list -NRD ``` 2\) Verify no fsdax or devdax namespaces are mounted or in-use by running applications. 3\) Destroy or disable the namespace(s). 4\) Disable the regions used by the NVDIMM (nmem) that needs to be disabled. 5\) Disable the NVDIMM (nmem). ## "ndctl: error while loading shared libraries: libjson-c.so.2: cannot open shared object file: No such file or directory" **Issue:** Executing the `ndctl` utility, with or without commands or options, yields the following missing library error: ``` # ndctl --version ndctl: error while loading shared libraries: libjson-c.so.2: cannot open shared object file: No such file or directory ``` **Cause:** The issue could be caused by one of the following issues: * The json-c package is not installed * A version mis-match between the `json-c` and `ndctl`. More recent versions of json-c deliver `libjson-c.so.4` rather than `libjson-c.so.2`. * The json-c library is installed in a non-default location and the LD\_LIBRARY\_PATH environment variable needs to be updated. Verify which libraries are missing from `ndctl` using the `ldd` utility and identifying libraries that are 'not found': ``` # ldd `which ndctl` linux-vdso.so.1 (0x00007ffc677c4000) libndctl.so.6 => /lib64/libndctl.so.6 (0x00007f31fca49000) libdaxctl.so.1 => /lib64/libdaxctl.so.1 (0x00007f31fc843000) libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f31fc63c000) libjson-c.so.2 => not found libc.so.6 => /lib64/libc.so.6 (0x00007f31fc27d000) libudev.so.1 => /lib64/libudev.so.1 (0x00007f31fc057000) [...] ``` **Solution:** Verify that the json-c package is installed. For example, on Fedora use `dnf info --installed json-c`. If it is installed, information about the package will be displayed, eg: ``` # sudo dnf info --installed json-c Installed Packages Name : json-c Version : 0.13.1 Release : 2.fc28 Arch : x86_64 Size : 65 k Source : json-c-0.13.1-2.fc28.src.rpm Repo : @System From repo : updates Summary : JSON implementation in C URL : https://github.com/json-c/json-c License : MIT Description : JSON-C implements a reference counting object model that allows you : to easily construct JSON objects in C, output them as JSON formatted : strings and parse JSON formatted strings back into the C representation : of JSON objects. It aims to conform to RFC 7159. ``` If the package is not installed, a message similar to the following will be returned: ``` # sudo dnf info --installed json-c Error: No matching Packages to list ``` To install the json-c package if it is missing, use ``` # sudo dnf install json-c ``` To query the json-c package to identify the library version use the following: ``` # dnf repoquery -l json-c | grep libjson-c.so Last metadata expiration check: 2:07:37 ago on Fri 06 Jul 2018 06:18:47 AM MDT. /usr/lib64/libjson-c.so.4 /usr/lib64/libjson-c.so.4.0.0 /usr/lib64/libjson-c.so.4 /usr/lib64/libjson-c.so.4.0.0 /usr/lib/libjson-c.so.4 /usr/lib/libjson-c.so.4.0.0 /usr/lib/libjson-c.so.4 /usr/lib/libjson-c.so.4.0.0 ``` Verify the LD\_LIBRARY\_PATH includes the location of libjson-c.so.\*. Note: `/usr/lib` and `/usr/lib64` are automatically included. ``` # echo $LD_LIBRARY_PATH /usr/local/lib:/usr/local/lib64 ``` If the package and LD\_LIBRARY\_PATH are correct, the version of ndctl will need to be updated. Using `ndctl --version` won't work and will simply return "*ndctl: error while loading shared libraries: libjson-c.so.2: cannot open shared object file: No such file or directory*". If the `ndctl` utility was installed using the ndctl package from the operating system's repository, update the package to the latest version. On Fedora: ``` # sudo dnf update -y ndctl ``` If the latest version within the package repository is old with no new versions available, download, compile, and install from source code. Detailed instructions can be found in the [Installing NDCTL](https://github.com/sscargal/pmem-docs-ndctl-user-guide/tree/bad2d0a9e4528c1cc9d2aa6271f8a427dad45420/getting-started-guide/installing-ndctl.md) chapter. If the `ndctl` utility was previously compiled and installed using source code, download the latest version from the [ndctl GitHub repository](https://github.com/pmem/ndctl), compile, and install. Detailed instructions can be found in the [Installing NDCTL](https://github.com/sscargal/pmem-docs-ndctl-user-guide/tree/bad2d0a9e4528c1cc9d2aa6271f8a427dad45420/getting-started-guide/installing-ndctl.md) chapter. ## Error: namespace0.0 is active, specify --force for re-configuration **Issue:** Destroying a namespace mode may fail with the following error: ``` # ndctl destroy-namespace namespace0.0 Error: namespace0.0 is active, specify --force for re-configuration ``` Changing a namespace mode may fail with the following error: ``` # ndctl create-namespace -e namespace0.0 -m fsdax Error: namespace0.0 is active, specify --force for re-configuration failed to reconfigure namespace: Device or resource busy ``` **Cause:** The error indicates the namespace is currently **active** and potentially mounted (FSDAX) or in use, so the operation is not permitted. **Solution:** 1\) Verify the current namespace mode using `ndctl list -N` to show all namespaces or filter the result by specifying namespace name using `ndctl list -N namespace{X.Y}`, eg: ``` # ndctl list -N { "dev":"namespace0.0", "mode":"raw", "size":134217728000, "uuid":"0a66b46c-5146-45bb-a0f1-320f28db4d40", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` If the namespace mode is 'fsdax', verify any associated filesystems are unmounted. If the namespace mode is 'devdax', verify any application using the device is stopped. {% hint style="info" %} The 'fuser' command on Linux can be used to identify any running processes with the /dev/dax\* or /dev/pmem\* devices open. {% endhint %} 2\) Either disable the namespace prior to deleting it or changing the mode or use the `-f`, `--force` options to override the active status checks. To change the mode of an active namespace without disabling it first, use the `-f`, `--force` option: ``` # ndctl create-namespace -f -e namespace0.0 -m fsdax { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"1cf25e33-8dfa-4ad2-b409-e236d598bc1e", "raw_uuid":"9368c235-26d3-4220-9048-e85f3ccd8469", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` To delete a namespace without disabling it first, use the `-f`, `--force` option: ``` # ndctl destroy-namespace -f namespace0.0 destroyed 1 namespace ``` Alternatively, disable the namespace then perform the destroy/mode change operation. Note: Changing a namespace mode automatically activates it. To change the namespace mode from 'raw' to 'fsdax': ``` # ndctl list -N { "dev":"namespace0.0", "mode":"raw", "size":134217728000, "uuid":"ce15f005-d3bd-4a94-8565-55be77bed6f6", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ndctl disable-namespace namespace0.0 disabled 1 namespace # ndctl create-namespace -e namespace0.0 -m fsdax { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"f45a6725-b80e-4174-a366-b07c91e66a58", "raw_uuid":"ee359fec-0a83-436a-91b6-44e3f45b04e0", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` To destroy a namespace: ``` # ndctl list -N { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":132118478848, "uuid":"f45a6725-b80e-4174-a366-b07c91e66a58", "raw_uuid":"ee359fec-0a83-436a-91b6-44e3f45b04e0", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ndctl disable-namespace namespace0.0 disabled 1 namespace # ndctl destroy-namespace namespace0.0 destroyed 1 namespace ``` ## "failed to create namespace: Resource temporarily unavailable" **Issue:** Creating a new namespace using 'ndctl create-namespace', may return the following error: ``` # ndctl create-namespace -r region0 failed to create namespace: Resource temporarily unavailable ``` **Cause:** There are many potential causes including: * There's no available capacity within the region because one or more namespaces exist and have consumed all the space. * The region is disabled. * There's an issue with the labels for the NVDIMMs belonging to the region. **Solution:** Use the `-v` option to print more information to help identify the cause. A debug version of `ndctl` may be required to get useful information. See [Installing NDCTL](https://github.com/sscargal/pmem-docs-ndctl-user-guide/tree/bad2d0a9e4528c1cc9d2aa6271f8a427dad45420/getting-started-guide/installing-ndctl.md) for instructions to build `ndctl` with debug options from source code. For a scenario where there's no space left within the region, a message similar to the following will be shown: ``` # ndctl create-namespace -r region0 -v ... namespace_create:772: region0: insufficient capacity size: 0 avail: 0 failed to create namespace: Resource temporarily unavailable ``` When a region is disabled, the following will be shown: ``` # ndctl create-namespace -r region0 -v ... validate_namespace_options:473: region0: disabled, skipping... failed to create namespace: Resource temporarily unavailable ``` Identifying label issues requires more investigation. Start with [Validating Labels](/ndctl-user-guide/master/managing-label-storage-areas-lsa#checking-labels). The `Resource temporarily unavailable` message improvements is being handled by ## "man -k ndctl" returns "ndctl: nothing appropriate." **Issue:** When searching the short descriptions and manual page for a *keyword*, the man utility may return an empty set, for example: ``` # man -k ndctl ndctl: nothing appropriate. ``` **Cause:** If you built ndctl from source, the man page indexes may not have been automatically generated for you. Additionally, if you installed ndctl to a non-default location, your $MANPATH shell environment variable may not been updated to point to to the new man page locations. **Solution:** 1\) Verify the current system global search paths for manual pages ``` # manpath -g /usr/man:/usr/share/man:/usr/local/man:/usr/local/share/man:/usr/X11R6/man:/opt/man ``` 2\) Identify if these paths are overridden by a local MANPATH setting, for example the following shows no additional search paths ``` # echo $MANPATH # ``` 3\) If the install location of ndctl is not included in the above output, add the path to your MANPATH shell environment, otherwise skip this step: ``` # export MANPATH=$MANPATH: ``` 4\) Manually scan and build the man page indexes ``` # sudo mandb -c ``` This command may take a few minutes depending on the number of search paths and man pages it has to index. The output is verbose as it scans and indexes. Once complete, it will provide a summary: ``` ... 118 man subdirectories contained newer manual pages. 10550 manual pages were added. 0 stray cats were added. ``` 5\) If the ndctl man pages were scanned and indexed, `man -k ndctl` will now work as expected, eg: ``` # man -k ndctl ndctl (1) - Manage "libnvdimm" subsystem devices (Non-volatile Memory) ndctl-check-labels (1) - determine if the given dimms have a valid namespace index block ndctl-check-namespace (1) - check namespace metadata consistency ndctl-clear-errors (1) - clear all errors (badblocks) on the given namespace ... ``` # Glossary **BLK:** Block Mode: A set of one or more programmable memory mapped apertures provided by an NVDIMM to access its media. This indirection precludes the performance benefit of interleaving, but enables NVDIMM-bounded failure modes. **BTT:** Block Translation Table: Persistent memory is byte addressable. Existing software may have an expectation that the power-fail-atomicity of writes is at least one sector, 512 bytes. The BTT is an indirection table with atomic update semantics to front a PMEM/BLK block device driver and present arbitrary atomic sector sizes. **DAX:** Direct Access File system extensions to bypass the page cache and block layer to mmap persistent memory, from a PMEM block device, directly into a process address space. **DCR:** NVDIMM Control Region Structure: It defines a vendor-id, device-id, and interface format for a given DIMM. See [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) Section 5.2.25.5. **DPA:** DIMM Physical Address: is an NVDIMM-relative offset. With one NVDIMM in the system there would be a 1:1 system-physical-address:DPA association. Once more NVDIMMs are added a memory controller interleave must be decoded to determine the DPA associated with a given system-physical-address. BLK capacity always has a 1:1 relationship with a single-NVDIMM's DPA range. **DSM:** Device Specific Method: ACPI method to to control specific device - in this case the firmware. **LABEL:** Metadata stored on an NVDIMM device that partitions and identifies (persistently names) storage between PMEM and BLK. It also partitions BLK storage to host BTTs with different parameters per BLK-partition. Note that traditional partition tables, GPT/MBR, are layered on top of a BLK or PMEM device. **NVDIMM:** A Non-Volatile DIMM device (also called a module) installed in to a memory socket on the system board. Data is written to the device and stored persistently, meaning data is retained across power-cycles. **PMEM:** Persistent Memory Mode: A system-physical-address range where writes are persistent. A block device composed of PMEM is capable of DAX. A PMEM address range may span an interleave of several NVDIMMs. # Reference Material The following is a list of reference material: ## Documents See the latest documentation for the NVDIMM kernel sub-system here: ACPI v6.0: [http://www.uefi.org/sites/default/files/resources/ACPI\_6.0.pdf ](http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf) ACPI Specification (Latest): NVDIMM DSM Interface Specification (v1.8): NVDIMM DSM Interface Example: [http://pmem.io/documents/NVDIMM\_DSM\_Interface\_Example.pdf ](http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf) NVDIMM Driver Writer's Guide: NVDIMM Namespace: [http://pmem.io/documents/NVDIMM\_Namespace\_Spec.pdf ](http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf) UEFI Specification (Latest): ## Source Code Repositories LIBNVDIMM: [https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git ](https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git) LIBNDCTL: [https://github.com/pmem/ndctl.git ](https://github.com/pmem/ndctl.git) NDCTL: PMEM: # NDCTL User Guide ## Introduction `ndctl` is a utility for managing the Linux LIBNVDIMM Kernel subsystem. It is designed to work with various non-volatile memory devices (NVDIMMs) from different vendors. The LIBNVDIMM subsystem defines a kernel device model and control message interface for platform NVDIMM resources like those defined by the [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) NFIT (**N**VDIMM **F**irmware **I**nterface **T**able). The latest [ACPI ](http://www.uefi.org/specifications)and [UEFI ](http://www.uefi.org/specifications)specifications can be found at [uefi.org](http://www.uefi.org). Operations supported by `ndctl` include: * Provisioning capacity (namespaces) * Enumerating Devices * Enabling and Disabling NVDIMMs, Regions, and Namespaces * Managing NVDIMM Labels ## What's new in v66 This release incorporates functionality up to the 5.3 kernel, and adds a number of bug fixes, and improvements. Highlights include a new command to reconfigure dax devices to different modes (devdax - default, and system-ram - to hotplug the dax device as system memory), improvements to ndctl-{read,write,init}-labels allowing smaller sized reads/writes, usability fixes to ndctl-monitor, and ndctl-create-namespace, and a fix to ndctl-check-namespace allowing it to be used on systems with different page sizes. Commands: * [daxctl-reconfigure-device](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device): new command for device mode management * [daxctl-online-memory](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3), [daxctl-offline-memory](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory): new commands for devices in system-ram mode * [monitor](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-monitor): logging improvements, allow sending to background * [inject-error](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error): refuse to operate on active BTT namespaces * [read-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels), [write-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-write-labels), [zero-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels): improvements to minimize data transfer * [create-namespace](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace): usability improvements around region search APIs: * ndctl\_cmd\_cfg\_read\_set\_extent * ndctl\_cmd\_cfg\_write\_set\_extent * ndctl\_dimm\_read\_label\_extent * ndctl\_dimm\_read\_label\_index * ndctl\_dimm\_zero\_label\_extent * daxctl\_dev\_disable * daxctl\_dev\_enable\_devdax * daxctl\_dev\_enable\_ram * daxctl\_dev\_get\_ctx * daxctl\_dev\_get\_memory * daxctl\_dev\_get\_resource * daxctl\_dev\_get\_target\_node * daxctl\_dev\_is\_enabled * daxctl\_memory\_get\_block\_size * daxctl\_memory\_get\_dev * daxctl\_memory\_get\_node\_path * daxctl\_memory\_is\_online * daxctl\_memory\_num\_sections * daxctl\_memory\_offline * daxctl\_memory\_online # Installing NDCTL The `ndctl` utility is used to manage the libnvdimm (non-volatile memory device) sub-system in the Linux Kernel. It is required for certain Persistent Memory Developer Kit (PMDK) features. {% hint style="info" %} Microsoft Windows users should read the [Understand and Deploy Persistent Memory](https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/deploy-pmem) documentation. {% endhint %} The `ndctl` utility can be installed on Linux using either of the two options: * [Install ndctl using packages](/ndctl-user-guide/v66/installing-ndctl/installing-ndctl-packages-on-linux) * [Install ndctl using the source code](/ndctl-user-guide/v66/installing-ndctl/installing-ndctl-from-source-on-linux) # Installing NDCTL Packages on Linux The `ndctl` utility is available in many Linux distribution package repositories. This approach is the easiest to implement and maintain, compared with [installing ndctl from source code](/ndctl-user-guide/v66/installing-ndctl/installing-ndctl-from-source-on-linux). {% tabs %} {% tab title="Fedora" %} 1\) Query the repository to identify if ndctl is delivered: Fedora 21 or earlier ``` yum search ndctl ``` Fedora 22 or later ``` dnf repoquery ndctl ``` 2\) Install the ndctl utility Fedora 21 or earlier ``` yum install ndctl ``` Fedora 22 or later ``` dnf install ndctl ``` {% endtab %} {% tab title="RHEL & CentOS" %} The ndctl package is available on CentOS and RHEL 7.0 or later. 1\) Query the repository to identify if ndctl is delivered: ``` yum search ndctl ``` 2\) Install the ndctl package ``` $ yum install ndctl ``` {% endtab %} {% tab title="SLES & OpenSUSE" %} 1\) Query the repository to identify if ndctl is delivered: ``` $ zypper search ndctl ``` 2\) Install the ndctl package ``` $ zypper install ndctl ``` {% endtab %} {% tab title="Ubuntu" %} The ndctl package is available on Ubuntu 18.10 (Cosmic Cuttlefish) or later. 1\) Query the repository to identify if ndctl is delivered using either the aptitude, apt-cache, or apt utilities ``` $ aptitude search ndctl $ apt-cache search ndctl $ apt search ndctl ``` 2\) Verify if the ndctl package is currently installed and check the version ``` $ apt list --installed ndctl ``` 3\) Install the ndctl package or update an installed package ``` $ sudo apt-get install ndctl ``` {% endtab %} {% tab title="Debian" %} {% hint style="info" %} The ndctl package is available on Debian 10 (Buster) or later. See for up to date information. {% endhint %} 1\) Query available versions in configured repos: ``` $ apt policy ndctl ``` 2\) Query the repository to identify if ndctl is delivered using either the aptitude, apt-cache, or apt utilities ``` $ apt search ndctl ``` 3\) Verify if the ndctl package is currently installed and check the version ``` $ apt list --installed ndctl ``` 4\) Install the ndctl package or update an installed package ``` $ sudo apt install ndctl ``` {% endtab %} {% endtabs %} # Installing NDCTL from Source on Linux These instructions provide a step-by-step guide for installing the `ndctl` and `daxctl` utilities from the latest GitHub [master branch](https://github.com/pmem/ndctl). ### 1. Install the Prerequisites There are a number of packages required for the build steps that may not be installed by default. For information about the required packages, see the "BuildRequires:" lines in ndctl.spec.in. To successfully compile ndctl from source with documentation, the following packages are required * autoconf * automake * asciidoc (asciidoctor) * bash-completion * doxygen * gcc * gcc-c++ (g++) * git * glib2 * glib2-devel * graphviz * json-c-devel * keyutils-libs-devel (libkeyutils-dev) * kmod * kmod-devel * libfabric * libfabric-devel * libtool * libudev-devel * libuuid-devel * ncurses * pandoc * pkg-config * rubygem-asciidoctor (asciidoctor) * xmlto To install these prerequisites, use: {% tabs %} {% tab title="Fedora" %} Fedora 21 or earlier ``` sudo yum install git gcc gcc-c++ autoconf automake asciidoc asciidoctor xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel ``` Fedora 22 or later ``` sudo dnf install git gcc gcc-c++ autoconf automake asciidoc asciidoctor xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel ``` {% endtab %} {% tab title="RHEL & CentOS" %} Some of the required packages can be found in the EPEL repository. Verify the EPEL repository is active: ``` $ yum repolist ``` If the EPEL repository is not listed, install and activate it using: ``` $ sudo yum install epel-release ``` Install the required packages ``` $ sudo yum install git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkgconfig glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel rubygem-asciidoctor keyutils-libs-devel ``` {% endtab %} {% tab title="SLES & OpenSUSE" %} ``` $ sudo zypper install -y git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel rubygem-asciidoctor keyutils-libs-devel ``` {% endtab %} {% tab title="Ubuntu & Debian" %} **For Ununtu 18.04 (Bionic) and Debian 9 (Stretch) or later:** ``` $ sudo apt install -y git gcc g++ autoconf automake asciidoc asciidoctor bash-completion xmlto libtool pkg-config libglib2.0-0 libglib2.0-dev libfabric1 libfabric-dev doxygen graphviz pandoc libncurses5 libkmod2 libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev ``` **For Ubuntu 16.04 (Xenial) and Debian 8 (Jessie):** {% hint style="info" %} Earlier releases of Ubuntu and Debian do not have libfabric1 or libfabric-dev available in the repository. If these libraries are required, you should compile them yourself. See {% endhint %} ``` $ sudo apt-get install -y git gcc g++ autoconf automake asciidoc asciidoctor bash-completion xmlto libtool pkg-config libglib2.0-0 libglib2.0-dev doxygen graphviz pandoc libncurses5 libkmod2 libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev ``` {% endtab %} {% endtabs %} ### 2. Clone the GitHub Repository 2.1) If you're behind a company proxy, configure git to work with your proxy server first. The following configures a HTTP and HTTPS proxy for all users. Refer to the [git-config documentation](https://git-scm.com/docs/git-config) for more options and information. ``` $ git config --global http.proxy http://proxyUsername:proxyPassword@proxy.server.com:port $ git config --global https.proxy https://proxyUsername:proxyPassword@proxy.server.com:port ``` 2.2) Create a working directory to clone the ndctl GitHub repository to, eg: 'downloads' ``` $ sudo mkdir /downloads $ sudo chmod +w /downloads ``` 2.3) Clone the repository: ``` $ cd /downloads $ sudo git clone https://github.com/pmem/ndctl $ cd ndctl ``` ### 3. Build The following configures ndctl to be installed in to the /usr/local directory. ``` $ sudo ./autogen.sh $ sudo ./configure CFLAGS='-g -O2' --prefix=/usr/local --sysconfdir=/etc --libdir=/usr/local/lib64 $ sudo make ``` #### **Build using an alternative compiler** **Note:** If you want to compile with a different compiler other than gcc, you have to provide the CC and CXX environment variables. For example: ``` $ sudo make CC=clang CXX=clang++ ``` These variables are independent and setting CC=clang does not set CXX=clang++. #### Build a Debug Version To compile ndctl with debugging, use the `--enable-debug` option: ``` $ sudo ./autogen.sh $ sudo ./configure CFLAGS='-g -O2' --enable-debug --prefix=/usr/local --sysconfdir=/etc --libdir=/usr/local/lib64 $ sudo make ``` For a full list of configure options use: ``` $ ./configure --help ``` ### 4. Install Once ndctl has successfully been compiled, it can be installed using the following: ``` $ sudo make install ``` ### 5. Build and Run Unit Tests (Optional) The unit tests run by `make check` require the nfit\_test.ko module to be loaded. To build and install nfit\_test.ko: 1. Obtain the kernel source. For example, `git clone -b libnvdimm-for-next git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git` 2. Skip to step 3 if the kernel version is >= v4.8. Otherwise, for kernel versions < v4.8, configure the kernel to make some memory available to CMA (contiguous memory allocator). This will be used to emulate DAX. `CONFIG_DMA_CMA=y` `CONFIG_CMA_SIZE_MBYTES=200` **or** `cma=200M` on the kernel command line. 3. Compile the libnvdimm sub-system as a module, make sure "zone device" memory is enabled, and enable the btt, pfn, and dax features of the sub-system: `CONFIG_X86_PMEM_LEGACY=m` `CONFIG_ZONE_DEVICE=y` `CONFIG_LIBNVDIMM=m` `CONFIG_BLK_DEV_PMEM=m` `CONFIG_ND_BLK=m` `CONFIG_BTT=y` `CONFIG_NVDIMM_PFN=y` `CONFIG_NVDIMM_DAX=y` `CONFIG_DEV_DAX_PMEM=m` 4. Build and install the unit test enabled libnvdimm modules in the following order. The unit test modules need to be in place prior to the `depmod` that runs during the final `modules_install` `make M=tools/testing/nvdimm` `sudo make M=tools/testing/nvdimm modules_install` `sudo make modules_install` 5. Now run `make check` in the ndctl source directory, or `ndctl test`, if ndctl was built with `--enable-test`. ### Troubleshooting The unit tests will validate that the environment is set up correctly before they try to run. If the platform is misconfigured, i.e. the unit test modules are not available, or the test versions of the modules are superseded by the "in-tree/production" version of the modules `make check` will skip tests and report a message like the following in test/test-suite.log: ``` SKIP: libndctl ============== test/init: nfit_test_init: nfit.ko: appears to be production version: /lib/modules/4.8.8-200.fc24.x86_64/kernel/drivers/acpi/nfit/nfit.ko.xz __ndctl_test_skip: explicit skip test_libndctl:2684 nfit_test unavailable skipping tests ``` If the unit test modules are indeed available in the modules 'extra' directory the default depmod policy can be overridden by adding a file to /etc/depmod.d with the following contents: ``` override nfit * extra override device_dax * extra override dax_pmem * extra override libnvdimm * extra override nd_blk * extra override nd_btt * extra override nd_e820 * extra override nd_pmem * extra ``` The nfit\_test module emulates pmem with memory allocated via vmalloc(). One of the side effects is that this breaks 'physically contiguous' assumptions in the driver. Use the `--align=4K` option to `ndctl create-namespace` to avoid these corner case scenarios. # Quick Start The `ndctl` command is designed to be user friendly. Once [installed](/ndctl-user-guide/v66/installing-ndctl), a list of commands can be shown using any of the following: 1\) With no arguments or options, `ndctl` and `daxctl` show a simple usage message: ``` # ndctl usage: ndctl [--version] [--help] COMMAND [ARGS] See 'ndctl help COMMAND' for more information on a specific command. ndctl --list-cmds to see all available commands ``` ``` # daxctl usage: daxctl [--version] [--help] COMMAND [ARGS] See 'daxctl help COMMAND' for more information on a specific command. daxctl --list-cmds to see all available commands ``` 2\) Using `ndctl help` and `daxctl help` displays basic help and syntax: ``` # ndctl help usage: ndctl [--version] [--help] COMMAND [ARGS] See 'ndctl help COMMAND' for more information on a specific command. ndctl --list-cmds to see all available commands ``` ``` # daxctl help usage: daxctl [--version] [--help] COMMAND [ARGS] See 'daxctl help COMMAND' for more information on a specific command. daxctl --list-cmds to see all available commands ``` Below is an example of using the `ndctl help` command to launch the `create-namespace` man page: ``` # ndctl help create-namespace ``` 3\) Using `ndctl --list-cmds` and `daxctl --list-cmds`lists all commands as a single list. ``` # ndctl --list-cmds version enable-namespace disable-namespace create-namespace destroy-namespace check-namespace clear-errors enable-region disable-region enable-dimm disable-dimm zero-labels read-labels write-labels init-labels check-labels inject-error update-firmware inject-smart wait-scrub start-scrub setup-passphrase update-passphrase remove-passphrase freeze-security sanitize-dimm load-keys wait-overwrite list monitor help ``` ``` # daxctl --list-cmds version list help migrate-device-model reconfigure-device online-memory offline-memory ``` An alternative method for listing commands uses the TAB key completion feature of `ndctl` and `daxctl`. By typing`ndctl ` or `daxctl ` we can list the available commands, eg: ``` # ndctl check-labels freeze-security setup-passphrase check-namespace help start-scrub clear-errors init-labels update-firmware create-namespace inject-error update-passphrase destroy-namespace inject-smart version disable-dimm list wait-overwrite disable-namespace load-keys wait-scrub disable-region monitor write-labels enable-dimm read-labels zero-labels enable-namespace remove-passphrase enable-region sanitize-dimm ``` ``` # daxctl version list help migrate-device-model reconfigure-device online-memory offline-memory ``` ## TAB Command and Argument Completion `ndctl` and `daxctl`supports command completion using the TAB key. For example, typing `ndctl enable-` lists all commands beginning with 'enable', eg: ``` # ndctl enable- enable-dimm enable-namespace enable-region ``` TAB completion also works with command arguments. For example, typing `ndctl enable-dimm ` will show all available command arguments. For example, the 'enable-dimm' command can enable one, more than one, or all NVDIMMs. It will list all available NVDIMMs (nmem) devices when using the TAB completion, eg: ``` # ndctl enable-dimm all nmem0 ``` ## Getting Help NDCTL ships with a man page for each command. Each man page describes the required arguments and features in detail. Man pages can be found and accessed using the `man` or `ndctl` utilities. The following `man -k ndctl` searches for any man page containing the "ndctl" keyword: ``` # man -k ndctl ndctl (1) - Manage "libnvdimm" subsystem devices (Non-volatile Memory) ndctl-check-labels (1) - determine if the given dimms have a valid namespace index block ndctl-check-namespace (1) - check namespace metadata consistency ndctl-clear-errors (1) - clear all errors (badblocks) on the given namespace ndctl-create-namespace (1) - provision or reconfigure a namespace ndctl-destroy-namespace (1) - destroy the given namespace(s) ndctl-disable-dimm (1) - disable one or more idle dimms ndctl-disable-namespace (1) - disable the given namespace(s) ndctl-disable-region (1) - disable the given region(s) and all descendant namespaces ndctl-enable-dimm (1) - enable one more dimms ndctl-enable-namespace (1) - enable the given namespace(s) ndctl-enable-region (1) - enable the given region(s) and all descendant namespaces ndctl-freeze-security (1) - Set the given DIMM(s) to reject future security operations ndctl-init-labels (1) - initialize the label data area on a dimm or set of dimms ndctl-inject-error (1) - inject media errors at a namespace offset ndctl-inject-smart (1) - perform smart threshold/injection operations on a DIMM ndctl-list (1) - dump the platform nvdimm device topology and attributes in json ndctl-load-keys (1) - load the kek and encrypted passphrases into the keyring ndctl-monitor (1) - Monitor the smart events of nvdimm objects ndctl-read-labels (1) - read out the label area on a dimm or set of dimms ndctl-remove-passphrase (1) - Stop a DIMM from locking at power-loss and requiring a passphrase to access media ndctl-sanitize-dimm (1) - Perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s) ndctl-setup-passphrase (1) - setup and enable the security passphrase for an NVDIMM ndctl-start-scrub (1) - start an Address Range Scrub (ARS) operation ndctl-update-firmware (1) - provides for updating the firmware on an NVDIMM ndctl-update-passphrase (1) - update the security passphrase for an NVDIMM ndctl-wait-overwrite (1) - wait for an overwrite operation to complete ndctl-wait-scrub (1) - wait for an Address Range Scrub (ARS) operation to complete ndctl-write-labels (1) - write data to the label area on a dimm ndctl-zero-labels (1) - zero out the label area on a dimm or set of dimms ``` ``` # man -k daxctl daxctl (1) - Provides enumeration and provisioning commands for the Linux kernel Device-DAX facility daxctl-list (1) - dump the platform Device-DAX regions, devices, and attributes in json. daxctl-migrate-device-model (1) - Opt-in to the /sys/bus/dax device-model, allow for alternative Device-DAX instance drivers. daxctl-offline-memory (1) - Offline the memory for a device that is in system-ram mode daxctl-online-memory (1) - Online the memory for a device that is in system-ram mode daxctl-reconfigure-device (1) - Reconfigure a dax device into a different mode ``` {% hint style="info" %} Note: If `man -k ndctl` returns "ndctl: nothing appropriate." or similar, see the [Troubleshooting](/ndctl-user-guide/v66/troubleshooting) section to manually build the indexes. {% endhint %} Additionally, executing `ndctl help ` can be used to display the man page for the command, eg: ``` # ndctl help enable-dimm ``` A list of ndctl and daxctl man pages are available online. See [NDCTL Man Pages](/ndctl-user-guide/v66/ndctl-man-pages) and [DAXCTL Man Pages](/ndctl-user-guide/v66/daxctl-man-pages) for a complete list. ## Displaying Bus, NVDIMM, Region, and Namespace Information The `ndctl list` command is a very powerful and feature rich command. A list of options is shown below: ``` # ndctl list -? Error: unknown switch `?' usage: ndctl list [] -b, --bus filter by bus -r, --region filter by region -d, --dimm filter by dimm -n, --namespace filter by namespace id -m, --mode filter by namespace mode -t, --type filter by region-type -U, --numa-node filter by numa node -B, --buses include bus info -D, --dimms include dimm info -F, --firmware include firmware info -H, --health include dimm health -R, --regions include region info -N, --namespaces include namespace info (default) -X, --device-dax include device-dax info -C, --capabilities include region capability info -i, --idle include idle devices -M, --media-errors include media errors -u, --human use human friendly number formats -v, --verbose increase output detail ``` Using the filters is a powerful way to limit the output. ### Examples To list all active/enabled namespaces: ``` # ndctl list -N ``` To list all active/enabled regions: ``` # ndctl list -R ``` To list all active/enabled NVDIMMs: ``` # ndctl list -D ``` To list all active/enabled NVDIMMs, Regions, and Namespaces: ``` # ndctl list -DRN ``` To list all active/enabled and disabled/inactive (idle) NVDIMMs, Regions, and Namespaces: ``` # ndctl list -DRNi ``` To list all active/enabled and disabled/inactive (idle) NVDIMMs, Regions, and Namespaces with human readable values: ``` # ndctl list -iNuRD ``` ## Increasing output verbosity The `-v, --verbose` option increases the output verbosity of the command. Using `-vv` or -`-vvv` further increases the output and verbosity. # Concepts ## Persistent Memory Concepts This section describes the basic concepts when configuring and managing NVDIMMs. The following describes terms used throughout this chapter. **Interleave Sets:** two or more NVDIMMs create an N-Way interleave set to provide striped read/write operations for increased throughput. Commonly 2-Way or 4-Way interleaving. **Region:** a grouping of one or more NVDIMMs, or an interleaved set, that can be divided up into one or more Namespaces. Regions are created within interleaved sets. **Label Storage Area (LSA):** Namespaces are defined by Labels which are stored in the Label Storage Area(s). **Namespace:** defines a contiguously-addressed range of Non-Volatile Memory conceptually similar to a hard disk partition, SCSI Logical Unit (LUN), or an NVM Express namespace. It is the unit of persistent memory storage that appears in /dev as a device usable for I/O. **Type:** defines the way in which the persistent memory associated with a Namespace or Region can be accessed. Valid Types are PMEM and BLK: * **PMEM:** Direct access to the media via load/store operations. DAX Supported. * **BLK:** Direct access to the media via Apertures (sliding memory windows). DAX is not supported. **Mode:** defines which NVDIMM software features are enabled for a given Namespace. Namespace Modes include raw, sector, fsdax(memory), and devdax(dax). Sibling Namespaces of the same parent Region will always have the same Type, but might be configured to have different Modes. ## Configuration Options This section discusses some of the common configuration options but does not cover all possible supported config. Using one or more physical NVDIMMs, the available capacity can be configure in many different ways. Figure 1 below shows a typical Filesystem-DAX (FSDAX) configuration with three physical NVDIMMs that have been interleaved. All available capacity has been used to create a single region (Region0), namespace (Namespace0.0), and dax enabled filesystem created. Namespace naming convention is commonly X.Y where X is the region number and Y is the namespace. The persistent memory pool(s) are memory mapped to the application, thus allowing direct load/store access to the NVDIMMs. ![](https://579286785-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlBtYw13PknzW_5Tx%2Fdraw.io-gitbook-interleaved-dimms-fsdax.jpg?alt=media\&token=5f2e188f-5683-4441-8e5d-2e2c75310589) Figure 2 shows a configuration where the region has been partitioned into two namespaces. Each namespace has a single DAX Filesystem. ![](https://579286785-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlFmTfOvpD8GUY9eA%2Fdraw.io-gitbook-interleaved-dimms-with-2-namespaces-fsdax%20\(1\).jpg?alt=media\&token=05d9d599-9773-470b-ba61-4bac6491b4d5) Figure 3 shows a configuration where NVDIMM interleaving has been disabled. In this mode, each NVDIMM becomes addressable with single regions per device. Regions and Namespaces cannot be striped or interleaved across NVDIMMs. It is possible to use software RAID such as Linux's Device Mapper to implement mirroring, striping, or concatenation of the /dev/pmem{N} devices. ![](https://579286785-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlSA1qHvOy6St0XnM%2Fdraw.io-gitbook-non-interleaved-dimm-fsdax.jpg?alt=media\&token=2ca8c490-07d3-4fd0-84c6-49cf2f5d3457) The following sections describe each layer in more detail. # NVDIMM Devices A generic DIMM device object, named /dev/nmemX, is registered for each physical memory device indicated in the ACPI NFIT table, or other platform NVDIMM resource discovery mechanism. Physical NVDIMMs may have configuration options available via the BIOS. The features and available configuration options will be dependant on the NVDIMM device and BIOS. The Linux LIBNVDIMM core provides a built-in driver for these DIMM devices. The driver is responsible for determining if the DIMM implements a namespace label area, and initializing the kernel's in-memory copy of that label data. The kernel performs runtime modifications of that data when namespace provisioning actions are taken, and actively blocks user-space from initiating label data changes while the DIMM is active in any region. Disabling a DIMM, after all the regions it is a member of have been disabled, allows userspace to manually update the label data to be consumed when the DIMM is next enabled. Refer to [Managing NVDIMMs](/ndctl-user-guide/v66/managing-nvdimms) for more information on administering NVDIMMs. # PMEM and BLK Modes The NFIT (**N**VDIMM **F**irmware **I**nterface **T**able) specification defined by the [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) standardizes not only the description of Persistent Memory (PMEM) and Block (BLK) modes, but also the platform message-passing entry points for control and configuration. ## LIBNVDIMM Sub-System The Linux LIBNVDIMM subsystem provides support for three types of NVDIMMs; PMEM, BLK, and NVDIMM. PMEM (Persistent Memory) devices allow byte addressable access. BLK (block) devices allow sector atomicity like traditional storage devices. NVDIMM devices can simultaneously support both PMEM and BLK mode access. These three modes of operation are described by the "NVDIMM Firmware Interface Table" (NFIT) in [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) or later. While the LIBNVDIMM implementation is generic and supports pre-NFIT platforms, it was guided by the superset of capabilities need to support this ACPI v6.0 definition for NVDIMM resources. The bulk of the kernel implementation is in place to handle the case where the DIMM Physical Address (DPA) accessible via PMEM is aliased with DPA accessible via BLK. When that occurs a LABEL is needed to reserve DPA for exclusive access via one mode a time. For each NVDIMM access method (PMEM, BLK), LIBNVDIMM provides a block device driver: 1. **PMEM (nd\_pmem.ko):** Drives a system-physical-address range. This range is contiguous in system memory and may be interleaved (hardware memory controller striped) across multiple DIMMs. When interleaved the platform may optionally provide details of which DIMMs are participating in the interleave. Note that while LIBNVDIMM describes system-physical-address ranges that may alias with BLK access as ND\_NAMESPACE\_PMEM ranges and those without alias as ND\_NAMESPACE\_IO ranges, to the nd\_pmem driver there is no distinction. The different device-types are an implementation detail that userspace can exploit to implement policies like "only interface with address ranges from certain DIMMs". It is worth noting that when aliasing is present and a DIMM lacks a label, then no block device can be created by default as userspace needs to do at least one allocation of DPA to the PMEM range. In contrast ND\_NAMESPACE\_IO ranges, once registered, can be immediately attached to nd\_pmem. 2. **BLK (nd\_blk.ko):** This driver performs I/O using a set of platform defined apertures. A set of apertures will access just one DIMM. Multiple windows (apertures) allow multiple concurrent accesses, much like tagged-command-queuing, and would likely be used by different threads or different CPUs. The NFIT specification defines a standard format for a BLK-aperture, but the spec also allows for vendor specific layouts, and non-NFIT BLK implementations may have other designs for BLK I/O. For this reason "nd\_blk" calls back into platform-specific code to perform the I/O. One such implementation is defined in the "Driver Writer's Guide" and "DSM Interface Example". ## Differences between PMEM and BLK Modes While PMEM provides direct byte-addressable CPU-load/store access to NVDIMM storage, it does not provide the best system RAS (recovery, availability, and serviceability) model. An access to a corrupted system-physical-address address causes a CPU exception while an access to a corrupted address through an BLK-aperture causes that block window to raise an error status in a register. The latter is more aligned with the standard error model that host-bus-adapter attached disks present. Also, if an administrator ever wants to replace a memory it is easier to service a system at DIMM module boundaries. Compare this to PMEM where data could be interleaved in an opaque hardware specific manner across several DIMMs. BLK-apertures solve these RAS problems, but their presence is also the major contributing factor to the complexity of the ND subsystem. They complicate the implementation because PMEM and BLK alias in DPA space. Any given DIMM's DPA-range may contribute to one or more system-physical-address sets of interleaved DIMMs, and may also be accessed in its entirety through its BLK-aperture. Accessing a DPA through a system-physical-address while simultaneously accessing the same DPA through a BLK-aperture has undefined results. For this reason, DIMMs with this dual interface configuration include a DSM function to store/retrieve a LABEL. The LABEL effectively partitions the DPA-space into exclusive system-physical-address and BLK-aperture accessible regions. For simplicity a DIMM is allowed a PMEM "region" per each interleave set in which it is a member. The remaining DPA space can be carved into an arbitrary number of BLK devices with discontiguous extents. # Regions A generic REGION device is registered for each PMEM range or BLK-aperture set. Regions represent the physical persistent memory devices in an interleaved or non-interleaved set. LIBNVDIMM provides a built-in driver for these REGION devices. This driver is responsible for reconciling the aliased DPA mappings across all regions, parsing the LABEL, if present, and then emitting NAMESPACE devices with the resolved/exclusive DPA-boundaries for the `nd_pmem` or `nd_blk` device driver to consume. Refer to [Managing Regions](/ndctl-user-guide/v66/managing-regions) for more information. # Regions, Atomic Sectors, and DAX One of the few reasons to allow multiple BLK namespaces per REGION is so that each BLK-namespace can be configured with a Block Translation Table (BTT) with unique atomic sector sizes. While a PMEM device can host a BTT, the LABEL specification does not provide for a sector size to be specified for a PMEM namespace. This is due to the expectation that the primary usage model for PMEM is via DAX, and the BTT is incompatible with DAX. However, for the cases where an application or filesystem still needs atomic sector update guarantees, it can register a BTT on a PMEM device or partition. ## Example NVDIMM Platform Figure 1 shows an example platform with four NVDIMMs, two integrated memory controllers (IMC), and a single CPU . ``` (a) (b) DIMM BLK-REGION +-------------------+--------+--------+--------+ +------+ | pm0.0 | blk2.0 | pm1.0 | blk2.1 | 0 region2 | imc0 +--+- - - region0- - - +--------+ +--------+ +--+---+ | pm0.0 | blk3.0 | pm1.0 | blk3.1 | 1 region3 | +-------------------+--------v v--------+ +--+---+ | | | cpu0 | region1 +--+---+ | | | +----------------------------^ ^--------+ +--+---+ | blk4.0 | pm1.0 | blk4.0 | 2 region4 | imc1 +--+----------------------------| +--------+ +------+ | blk5.0 | pm1.0 | blk5.0 | 3 region5 +----------------------------+--------+--------+ ``` *Figure 1: Example sysfs layout* Each unique interface (BLK or PMEM) to DPA space is identified by a region device with a dynamically assigned id (REGION0 - REGION5). 1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A single PMEM namespace is created in the REGION0-SPA-range that spans most of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that interleaved system-physical-address range is reclaimed as BLK-aperture accessed space starting at DPA-offset (a) into each DIMM. In that reclaimed space we create two BLK-aperture "namespaces" from REGION2 and REGION3 where "blk2.0" and "blk3.0" are just human readable names that could be set to any user-desired name in the LABEL. 2. In the last portion of DIMM0 and DIMM1 we have an interleaved ``` system-physical-address range, REGION1, that spans those two DIMMs as ``` well as DIMM2 and DIMM3. Some of REGION1 is allocated to a PMEM namespace named "pm1.0", the rest is reclaimed in 4 BLK-aperture namespaces (for each DIMM in the interleave set), "blk2.1", "blk3.1", "blk4.0", and "blk5.0". 3. The portion of DIMM2 and DIMM3 that do not participate in the REGION1 interleaved system-physical-address range (i.e. the DPA address past offset (b) are also included in the "blk4.0" and "blk5.0" namespaces. Note, that this example shows that BLK-aperture namespaces don't need to be contiguous in DPA-space. This bus is provided by the kernel under the device `/sys/devices/platform/nfit_test.0` when CONFIG\_NFIT\_TEST is enabled and the nfit\_test.ko module is loaded. This not only test LIBNVDIMM but the acpi\_nfit.ko driver as well. # Label Storage Area (LSA) The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to store the definition of NVDIMM *namespaces.* # Namespaces The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. A REGION is the Linux term for what ACPI and UEFI call a DIMM-interleave-set, or a system-physical-address-range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label-less" namespace per region. Examples of label-less namespaces are the ones created by the kernel's memmap option, or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, fsdax, devdax, sector, and raw: * **fsdax:** Filesystem-DAX mode is the default mode of a namespace when specifying `ndctl create-namespace` with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working-sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. * **devdax:** Device-DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem-DAX. Instead of a block-device that can support a DAX enabled filesystem, this mode provides a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual-machine, register persistent memory for RDMA, or when gigantic mappings are needed. * **sector:** Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. * **raw:** Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux fsdax or devdax mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. Figure 1 below shows a typical Filesystem-DAX (FSDAX) configuration with three physical NVDIMMs that have been interleaved. All available capacity has been used to create a single region (Region0), namespace (Namespace0.0), and dax enabled filesystem created. Namespace naming convention is commonly X.Y where X is the region number and Y is the namespace ![](https://579286785-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlBtYw13PknzW_5Tx%2Fdraw.io-gitbook-interleaved-dimms-fsdax.jpg?alt=media\&token=5f2e188f-5683-4441-8e5d-2e2c75310589) Refer to [Managing Namespaces](/ndctl-user-guide/v66/managing-namespaces) for information on how to create, delete, edit, and manage the namespace configuration. # Managing NVDIMMs Managing physical or emulated NVDIMMs using `ndctl` has no functional difference. Physical NVDIMM features and options may be controlled through the system BIOS. The BIOS cannot see emulated NVDIMMs. Observe the following restrictions when managing NVDIMMs * DO NOT change the memory slot of physical NVDIMMs when they are part of an Interleave Set. Doing so changes the order of interleaving so data access will be compromised or corrupted. If interleaving is disabled, moving NVDIMMs to different slot locations is okay, but not recommended. * DO NOT disable NVDIMMs when they are part of an active Region and/or Namespace as this will prevent data access and may corrupt data. * ALWAYS backup the data and make a copy of the configuration layout prior to any changes. ## Listing active/enabled NVDIMMs The `ndct list -D` , or equivalent `ndct list --dimm` , can be used to show active/enabled NVDIMM devices on the system, eg: ``` # ndctl list -D { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29 } ``` ## Listing disabled/inactive NVDIMMs By default, `ndctl` only lists enabled/active dimms, regions, and namespaces. To include previously disabled (inactive) NVDIMMs, include the `-i` flag to show both enabled and disabled devices, eg: ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":0, "phys_id":29 } { "dev":"nmem1", "id":"9759-b5-1459-00000502", "handle":1, "phys_id":30 "state":"disabled" } ... ``` {% hint style="info" %} NVDIMM vendor specific tools can be used to display more information about the NVDIMMs from the operating system layer. For example, Intel Optane DC Persistent Memory Modules can be managed using the [ipmctl](https://github.com/intel/ipmctl) utility. Refer to the [IPMCTL User Guide](https://docs.pmem.io/ipmctl-user-guide/) for more information. For other persistent memory products, refer to the vendor specific documentation. {% endhint %} ## Disabling NVDIMMs NVDIMMs can only be disabled if they have no active Regions or Namespaces. If an active/enabled namespace and/or region exists, a message is displayed: ``` # ndctl disable-dimm nmem0 nmem0 is active, skipping... disabled 0 nmem ``` 1\) List the current active/enabled configuration ``` # ndctl list -NRD ``` 2\) Verify no fsdax or devdax namespaces are mounted or in-use by running applications 3\) Destroy or disable any active/enabled namespace(s). ``` # ndctl disable-namespace - or - # ndctl destroy-namespace ``` See [Destroying Namespaces](/ndctl-user-guide/v66/managing-namespaces#destroying-namespaces) or [Disabling Namespaces](/ndctl-user-guide/v66/managing-namespaces#disabling-namespaces) for more information. 4\) Disable the regions used by the NVDIMM (nmem) that needs to be disabled ``` # ndctl disable-region ``` See [Disabling Regions](/ndctl-user-guide/v66/managing-regions#disabling-regions) for more information. 5\) Disable a single, subset, or all NVDIMM (nmem) devices To disable a single NVDIMM, use: ``` # ndctl disable-dimm nmem0 disabled 1 nmem ``` To disable a subset or specific list or NVDIMMs, use: ``` # ndctl disable-dimm nmem0 nmem1 nmem5 disabled 3 nmem ``` To disable all NVDIMMs, use: ``` # ndctl disable-dimm all disabled 12 nmem ``` 6\) Verify the NVDIMM(s) are disabled by listing inactive dimms and verifying the 'state': ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29, "state":"disabled" } ... ``` ## Enabling NVDIMMs 1\) Verify the nmem device, or list of nmem devices, that need to be enabled using the `ndctl list -Di` command: ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29, "state":"disabled" } ``` 2\) Enable the NVDIMM(s) To enable a single NVDIMM (nmemX) device, use: ``` # ndctl enable-dimm nmem0 enabled 1 nmem ``` To enable a subset of disabled NVDIMMs, use: ``` # ndctl enable-dimm nmem0 nmem1 nmem5 enabled 3 nmem ``` To enable all disabled NVDIMMs, use: ``` # ndctl enable-dimm all enabled 12 nmem ``` 3\) Verify the state by listing all NVDIMMs ``` # ndctl list -Di ``` A filtered list of NVDIMMs can shown using the `-d ` or `-dimm ` option, eg: ``` # ndctl list -Di -d nmem0 - or - # ndctl list -Di -dimm nmem0 ``` ## Sanitize NVDIMMs Sanitizing one or more NVDIMMs perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s). The sanitize-dimm command performs a cryptographic destruction of the contents of the given NVDIMM. It scrambles the data, and any metadata or info-blocks, but it doesn’t modify namespace labels. Therefore, any namespaces on regions associated with the given NVDIMM will be retained, but they will end up in the raw mode. Additionally, after completion of this command, the security and passphrase for the given NVDIMM will be disabled, and the passphrase and any key material will also be removed from the keyring and the ndctl keys directory at /etc/ndctl/keys. The command supports two different methods of performing the cryptographic erase. The default is crypto-erase, but additionally, an overwrite option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. {% hint style="info" %} Sanitizing NVDIMMs requires the region to be disabled first. If there are existing namespaces, those too must be destroyed or disabled. See the [Managing Regions](/ndctl-user-guide/v66/managing-regions) section for more information. {% endhint %} ### Examples Sanitize a single NVDIMM. This requires the region is disabled. The following destroys all namespaces, disables the region, the sanitizes the DIMMs within the region. 1\) Stop all applications that use a namespace associated with the NVDIMMs that need to be sanitized. 2\) Unmount any mounted file systems ``` $ sudo umount ``` 3\) Destroy the namespaces ``` $ sudo ndctl destroy-namespace --force [all | --region X] ``` 4\) Identify the NVDIMMs associated with the region using the `ndctl list -Rvv` or `ndctl list -DR` command. `-D` displays DIMMs, `-R` displays regions. The JSON formatted output shows the hierarchy. ``` $ sudo ndctl list -DR { ... "regions":[ { "dev":"region1", "size":1623497637888, "available_size":0, "max_available_extent":0, "type":"pmem", "iset_id":-2506113243053544244, "mappings":[ { "dimm":"nmem11", "offset":268435456, "length":270582939648, "position":5 }, { "dimm":"nmem10", "offset":268435456, "length":270582939648, "position":1 }, { "dimm":"nmem9", "offset":268435456, "length":270582939648, "position":3 }, { "dimm":"nmem8", "offset":268435456, "length":270582939648, "position":2 }, { "dimm":"nmem7", "offset":268435456, "length":270582939648, "position":4 }, { "dimm":"nmem6", "offset":268435456, "length":270582939648, "position":0 } ], "persistence_domain":"memory_controller" }, { "dev":"region0", "size":1623497637888, "available_size":1597727834112, "max_available_extent":1597727834112, "type":"pmem", "iset_id":3259620181632232652, "mappings":[ { "dimm":"nmem5", "offset":268435456, "length":270582939648, "position":5 }, { "dimm":"nmem4", "offset":268435456, "length":270582939648, "position":1 }, { "dimm":"nmem3", "offset":268435456, "length":270582939648, "position":3 }, { "dimm":"nmem2", "offset":268435456, "length":270582939648, "position":2 }, { "dimm":"nmem1", "offset":268435456, "length":270582939648, "position":4 }, { "dimm":"nmem0", "offset":268435456, "length":270582939648, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 5\) Disable the region(s) ``` $ sudo ndctl disable-region 1 disabled 1 region ``` 6\) Sanitize the DIMM(s) associated with the region(s) using the overwrite (`-o`) procedure. The -`c` (cryptographic) option works on DIMMs that were previously encrypted. ``` $ sudo ndctl sanitize-dimm -o nmem6 nmem7 nmem8 nmem9 nmem10 nmem11 overwrite issued for 6 nmems. ``` {% hint style="info" %} Note: If the DIMMs were not encrypted with a passphrase, you may see harmless messages such as the following for each DIMM ``` failed to open file /etc/ndctl/keys/nvdimm_8089-a2-1837-00000e88_{hostname}.blob: No such file or directory ``` {% endhint %} 7\) You may need to re-provision the NVDIMMs using the vendors utility. For example, use [ipmctl ](https://docs.pmem.io/ipmctl-user-guide/)for Intel Optane DC Persistent Memory to recreate the goal. Then use `ndctl` to [create the namespace configuration](/ndctl-user-guide/v66/managing-namespaces). # Managing Regions A region is a grouping of one or more NVDIMMs, or an interleaved set, that can be divided up into one or more Namespaces. Regions are of type PMEM or BLK. See [PMEM or BLK](/ndctl-user-guide/v66/concepts/libnvdimm-pmem-and-blk-modes) modes for more information. Regions can only be created, destroye, and configured using the vendor specific NVDIMM utility that manages the NVDIMMs or BIOS options, if available. ## Disabling Regions 1\) List all active/enabled regions: ``` # ndctl list -R { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of active/enabled regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -R -r region0 - or - # ndctl list -R -region region0 ``` 2\) Disable the region ``` # ndctl disable-region region0 disabled 1 region ``` 3\) Verify the region is disabled by including the `-i` option: ``` # ndctl list -Ri { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of disabled/inactive regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -Ri -r region0 - or - # ndctl list -Ri -region region0 ``` ## Enabling Regions 1\) List disabled/inactive regions ``` # ndctl list -Ri { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of active/enabled regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -R -r region0 - or - # ndctl list -R -region region0 ``` 2\) Enable the region ``` # ndctl enable-region region0 enabled 1 region ``` 3\) Verify the region is enabled using `ndctl list -R`. Note the 'state' field is not displayed for enabled regions. ``` # ndctl list -R { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "persistence_domain":"memory_controller" } ``` # Managing Label Storage Areas (LSA) The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to store the definition of any NVDIMM namespaces. The labels on the NVDIMMs cannot be edited directly. Changing the configuration can be achieved using the ndctl **create** or **destroy** commands. See [Managing Namespaces](/ndctl-user-guide/v66/managing-namespaces) and [Managing Regions](/ndctl-user-guide/v66/managing-regions). `ndctl` provides several label related commands shown below: * [check-labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#checking-labels) - determine if the given dimm(s) have a valid namespace index block * [init-labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#initializing-labels) - initialize the label data area on a dimm or set of dimms * [read-labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#reading-labels) - read out the label area on a dimm or set of dimms * [write-labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#writing-labels) - write data to the label area on a dimm * [zero-labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#erasing-zeroing-labels) - zero out the label area on a dimm or set of dimms ## Checking Labels The `check-labels` command validates the index block within the label storage area on the specified NVDIMMs. If issues are found, running the command in verbose mode using the `-v` option, reports the reason the index block is deemed invalid. ``` usage: ndctl check-labels [..] [] -b, --bus must be on a bus with an id/provider of -v, --verbose turn on debug ``` 1\) Identify the NVDIMMs (nmem's) in the host using the `ndctl list -D` command: ``` # ndctl list -D [ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ] ``` 2\) Check the labels of one or more devices. To check the labels on a single device: ``` # ndctl check-labels nmem3 ``` To check the labels on multiple devices: ``` # ndctl check-labels nmem3 nmem0 nmem2 ``` To check the labels on all devices: ``` # ndctl check-labels all ``` To display more information, if available, use the `-v` option: ``` # ndctl check-labels nmem3 -v ``` {% hint style="info" %} Note: The `-v` option may not produce any additional information if the labels are valid or ndctl was not built with logging or debug options. {% endhint %} ## Initializing Labels This command can be used to initialize the namespace index block if it is missing or reinitialize it if it is damaged or was previously zero'd. Note that reinitialization effectively destroys all existing namespace labels on the DIMM. {% hint style="danger" %} This is an expert mode operation. Overwriting labels will cause data loss. Backup all data prior to overwriting labels. {% endhint %} ``` usage: ndctl init-labels [..] [] -b, --bus must be on a bus with an id/provider of -v, --verbose turn on debug -f, --force force initialization even if existing index-block present -V, --label-version namespace label specification version (default: 1.1) ``` Before initializing the labels, all namespaces and regions must be destroyed or disabled. A warning message is displayed when attempting to initialize labels on active regions. ``` # ndctl init-labels nmem3 nmem2 nmem3: regions active, abort label write nmem2: regions active, abort label write ``` 1\) Identify the active/enabled NVDIMM devices, Regions, and Namespaces ``` # ndctl list -DRN { "dimms":[ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ], "regions":[ { "dev":"region1", "size":534723428352, "available_size":0, "type":"pmem", "iset_id":3026489321642266624, "mappings":[ { "dimm":"nmem3", "offset":268435456, "length":267361714176, "position":1 }, { "dimm":"nmem2", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":526366277632, "uuid":"76fc4dd1-e793-4eee-81ed-2f31275eb820", "blockdev":"pmem1" } ] }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "mappings":[ { "dimm":"nmem0", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 2\) Destroy or disable the namespace(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl destroy-namespace -f namespace1.0 - or - # ndctl disable-namespace namespace1.0 # ndctl destroy-namespace namespace1.0 ``` 3\) Disable the region(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl disable-region region1 ``` 4\) Initialize the labels ``` # ndctl init-labels nmem2 nmem3 ``` If the `init-labels` command returns "error: labels already initialized", use the `-f`, `--force` option. 5\) Enable the region(s). After initializing the labels, the region(s) may not be brought online automatically. List the active and disabled regions: ``` # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "state":"disabled", "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` Activate any disabled regions ``` # ndctl enable-region region1 # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` 6\) New namespaces can now be created on the regions. See [Managing Namespaces](/ndctl-user-guide/v66/managing-namespaces) for more information. ## Reading Labels The `read-labels` command dumps the data in a dimm’s label area to stdout or a file in raw or JSON format. When multiple dimms (nmem's) are specified, the data is concatenated. Usage: ``` ndctl read-labels [..] [] OPTIONS One or more nmemX device names. The keyword all can be specified to operate on every dimm in the system, optionally filtered by bus id (see --bus= option). -b, --bus= Limit operation to memory devices (dimms) that are on the given bus. Where bus can be a provider name or a bus id number. -v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). -o, --output output file -j, --json parse the label data into json assuming the NVDIMM Namespace Specification format. ``` The following shows a label from nvdimm0 (nmem0) printed to stdout in JSON format. ``` # ndctl read-labels -j nmem0 { "dev":"nmem0", "index":[ { "signature":"NAMESPACE_INDEX", "major":1, "minor":2, "labelsize":256, "seq":2, "nslot":510 }, { "signature":"NAMESPACE_INDEX", "major":1, "minor":2, "labelsize":256, "seq":1, "nslot":510 } ], "label":[ { "uuid":"9ed56df2-c0a9-4e17-b82d-28f642f95a22", "name":"", "slot":0, "position":0, "nlabel":1, "isetcookie":710683120631319073, "lbasize":512, "dpa":268435456, "rawsize":267361714176, "type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb", "abstraction_guid":"ba006426-9ffb-7746-bcb0-968f11d0d225" } ] } read 1 nmem ``` To backup an NVDIMM label using the raw format, use: ``` # ndctl read-labels nmem0 > nmem0.lsa ``` ## Writing Labels {% hint style="danger" %} This is an expert mode operation. Overwriting labels will cause data loss. Backup all data prior to overwriting labels. {% endhint %} Writing labels overwrites the current label with the contents of the specified file, or stdin, collected from a [read-labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#reading-labels) operation from a different label, or a previous label backup. Read data from the input filename, or stdin, and write it to the given device. The device being written to must not be active in any region, otherwise the kernel will not allow write access to the device’s label data area. The following shows how a raw label from an active nvdimm can be written to an inactive nvdimm. ``` $ ndctl read-labels nmem0 -o nmem0.label.backup.raw $ ndctl write-labels nmem1 -i nmem0.label.backup.raw ``` The above read-label raw output can be piped directly to the write-label command using: ``` $ ndctl read-labels nmem0 | ndctl write-labels nmem1 ``` ## Erasing/Zeroing Labels The `zero-labels` command resets the device to its default state by deleting all labels. {% hint style="danger" %} This operation cannot be undone unless a previous backup of the labels was created using the [read-labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#reading-labels) command. All data on the device will be destroyed. Create a backup before proceeding. {% endhint %} Before erasing/zeroing the labels, all namespaces and regions must be destroyed or disabled. 1\) Identify the active/enabled NVDIMM devices, Regions, and Namespaces ``` # ndctl list -DRN { "dimms":[ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ], "regions":[ { "dev":"region1", "size":534723428352, "available_size":0, "type":"pmem", "iset_id":3026489321642266624, "mappings":[ { "dimm":"nmem3", "offset":268435456, "length":267361714176, "position":1 }, { "dimm":"nmem2", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":526366277632, "uuid":"76fc4dd1-e793-4eee-81ed-2f31275eb820", "blockdev":"pmem1" } ] }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "mappings":[ { "dimm":"nmem0", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 2\) Destroy or disable the namespace(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl destroy-namespace -f namespace1.0 - or - # ndctl disable-namespace namespace1.0 # ndctl destroy-namespace namespace1.0 ``` 3\) Disable the region(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl disable-region region1 ``` 4\) To erase the labels on a single 'nmem0' device. ``` $ ndctl zero-labels nmem0 ``` To erase labels on multiple devices, use: ``` $ ndctl zero-labels nmem0 nmem2 nmem3 ``` 5\) Enable the region(s). After initializing the labels, the region(s) may not be brought online automatically. List the active and disabled regions: ``` # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "state":"disabled", "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` Activate any 'disabled' regions ``` # ndctl enable-region region1 # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` 6\) New namespaces can now be created on the regions. See [Managing Namespaces](/ndctl-user-guide/v66/managing-namespaces) for more information. # Managing Namespaces Regions can be partitioned in to one or more namespaces. Namespace operations include listing, creating, destroying, enabling, disabling, and checking (validating) using the following commands: * [list ](/ndctl-user-guide/v66/managing-namespaces#listing-namespaces)- dump the platform nvdimm device topology and attributes in json format * [create-namespace](/ndctl-user-guide/v66/managing-namespaces#creating-namespaces) - provision or reconfigure a namespace * [destroy-namespace](/ndctl-user-guide/v66/managing-namespaces#destroying-namespaces) - destroy the given namespace(s) * [disable-namespace](/ndctl-user-guide/v66/managing-namespaces#disabling-namespaces) - disable the given namespace(s) * [enable-namespace](/ndctl-user-guide/v66/managing-namespaces#enabling-namespaces) - enable the given namespace(s) * [check-namespace](/ndctl-user-guide/v66/managing-namespaces#checking-sector-namespaces) - check namespace metadata consistency ## Listing Namespaces #### Displaying Active/Enabled Namespaces The `ndctl list -N` command is used to display namespaces, for example: ``` # ndctl list -N { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":4294967296, "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` The output can be filtered using any combination of the following options: ``` -n, --namespace= An namespaceX.Y device name, or namespace region plus id tuple X.Y. Limit the namespace list to the single identified device if present. -m, --mode= Filter listing by the mode (raw, fsdax, sector or devdax) of the namespace(s). ``` **Examples:** List all active namespaces ``` # ndctl list -N ``` List all active namespaces and print human readable values (MB, MiB, GB, GiB, etc) ``` # ndctl list -Nu { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":"4.00 GiB (4.29 GB)", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` List a single namespace (namespace0.0) ``` # ndctl list -n namespace0.0 ``` List all active 'fsdax' mode namespaces ``` # ndctl list -m fsdax ``` #### Displaying Disabled/Inactive Namespaces By default, `ndctl list -N` lists only active/enabled namespaces. In the following example no active/enabled namespaces exist so no output is displayed: ``` # ndctl list -N # ``` Adding the `-i` flag includes both active/enabled and inactive/destroyed/disabled namespaces. The following example shows two previously destroyed namespaces. The `size` and `uuid` are clear indications that no information exists about the namespace. The `ndctl destroy` command simply NULLs the existing namespace entry and does not completely erase it from the metadata. For this reason, it may be possible to see more than one namespace using `ndctl list -Ni` ``` # ndctl list -Ni [ { "dev":"namespace0.0", "mode":"raw", "size":0, "uuid":"00000000-0000-0000-0000-000000000000", "sector_size":512, "state":"disabled", "numa_node":0 } ] ``` The `-i` option can be used in coordination with any other valid ndctl option. **Examples:** List all enabled and disabled namespaces ``` # ndctl list -Ni ``` List a single disabled namespace (namespace0.0) ``` # ndctl list -in namespace0.0 ``` List all enabled and disabled 'fsdax' mode namespaces ``` # ndctl list -im fsdax ``` ## Creating Namespaces The create-namespace command has a lot of options summarized below. The `ndctl-create-namespace` man page contains further details of each option. * **-t, --type:** Set the type of the namespace to either "pmem" or "blk" * **-m, mode:** Define the namespace mode - fsdax, devdax, sector, and raw * **-s, --size:** Specify the capacity of the namespace * **-a, --align:** Specify the alignments size, eg 4K or 2M * **-e, --reconfigure:** Reconfigure existing namespace configuration * **-u, --uuid:** Used for recovery purposes only * **-n, --name:** Specify a friendly name for namespaces that support labels * **-l, --sector-size:** Override the default sector size for block based namespaces * **-M, --map:** For 'fsdax' or 'devdax' namespaces, define whether metadata is stored in volatile memory (mem) or persistent storage (dev) * **-f, --force:** Allow the operation to continue on enabled namespaces * **-L, --autolabel, --no-autolabel:** Manage labels for legacy NVDIMMs that do not support labels * **-v, --verbose:** Emit verbose messages during the creation process * **-r, --region:** Limit the operation to a specific region * **-b, --bus:** Limit the operation to a specific bus The `mode` is the most important feature to get correct. The four modes available are defined as: * **fsdax:** Filesystem-DAX mode is the default mode of a namespace when specifying `ndctl create-namespace` with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working-sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. * **devdax:** Device-DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem-DAX. However, instead of a block-device that can support a DAX-enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual-machine, register persistent memory for RDMA, or when gigantic mappings are needed. * **sector:** Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. * **raw:** Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux fsdax or devdax mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. {% hint style="info" %} If the *-s, --size* option is used, the value provided to ndctl is used to create the namespace. Depending on the *mode*, the available capacity may be smaller than the specified size due to the calculated spared required for metadata. This is shown below in the examples. {% endhint %} ### FSDAX and DEVDAX Capacity Considerations The namespace configuration (label metadata) is stored at the beginning of the persistent memory address range. See the [Label Storage Area (LSA)](/ndctl-user-guide/v66/concepts/label-area) and [Managing Label Storage Area (LSA)](/ndctl-user-guide/v66/managing-label-storage-areas-lsa) sections for more information. In addition to the namespace label metadata, both fsdax and devdax modes require space to store the Page Frame Number (PFN) metadata (also known as “struct page” metadata) for each page within the namespace. A PFN is simply in index within physical memory, or on the NVDIMM, that is counted in page-sized units. For modes with PFN metadata, shown in Table 1 below, the requirement is 64 bytes per 4 KiB of persistent memory. To calculate the estimated space for metadata, use: $$ metadata \_{(bytes)} = (size \_{(bytes)}/4096)\*64 $$ **Example 1:** A 256GiB namespace requires \~3.93GiB of metadata (263182090240/4096)\*64 = 4112220160 bytes == 3.93GB **Example 2:** A 1TiB namespace requires \~17.18GiB of metadata (1099511627776/4096)\*64 = 17179869184 bytes == 17.18GiB The location of PFN metadata can either be in volatile DDR from the NVDIMM. The default is to keep metadata on the NVDIMM. The location is defined per namespace by the `-M, --map` configuration option. The following provides guidance for PFN metadata location: * **map=mem:** PFN metadata is stored in regular system memory (DDR) * Adequate for small persistent memory capacities * Frequent or high performance PFN access is required. For example, when time sensitive, or frequent, registration and deregistration of RDMA memory regions is important. Normal RDMA memory \[de]registration can be achieved with 'map=dev', but there are situations where \[de]registration needs to be very fast. This assumes the metadata fits in DDR memory. * **map=dev:** PFN metadata is stored on persistent memory (NVDIMM) * Intended for large persistent memory capacities. There might not be enough regular memory in the system if the ratio of persistent memory to DDR is high (>8:1) The persistence of the PFN metadata is not important as it's re-mapped at system boot time or when new namespaces are created/destroyed. It is convenient to store the metadata on the NVDIMM because it scales with the persistent memory capacity. The space for metadata is calculated and reserved at namespace creation time, meaning the usable capacity is lower than the physical capacity depending on the mode and map values. **Table 1: Features for each namespace mode** | Mode | Description | Device Path | Label Metadata | Atomicity | Filesystems | DAX | PFN Metadata | | ------ | -------------- | ----------- | -------------- | --------- | ----------- | --- | ------------ | | raw | raw | /dev/pmemN | no | no | yes | no | no | | sector | sector atomic | /dev/pmemNs | yes | yes | yes | no | no | | fsdax | filesystem DAX | /dev/pmemN | yes | no | yes | yes | yes | | devdax | device DAX | /dev/daxN.M | yes | no | no | yes | yes | **PFN Metadata Usage Example** The following demonstrates the difference between `--map=dev` and `--map=mem`. The example uses a \~250GiB region to create an fsdax namespace. A devdax namespace has identical behavior. When a namespace using the default `--map=dev` is created, the usable 'size' is \~4GiB smaller than the 249GiB region size due to the space reserved for PFN metdata. ``` # ndctl create-namespace -r region0 # ndctl list -RuN -r region0 { "regions":[ { "dev":"region0", "size":"249.00 GiB (267.36 GB)", "available_size":0, "type":"pmem", "iset_id":"0xb000000058000000", "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"245.11 GiB (263.18 GB)", "uuid":"19cccf5d-d227-42fe-b338-cfa882a6b6ce", "blockdev":"pmem0" } ] } ] } ``` When using `--map=mem`, the usable capacity on the namespace increases, but the amount of 'used' memory as shown by `free` has increased by \~4GiB. ``` # free total used free shared buff/cache available Mem: 362926556 5747588 356160304 2608 1018664 355200308 Swap: 32882684 0 32882684 # ndctl create-namespace -r region0 --map=mem # ndctl list -RuN -r region0 { "regions":[ { "dev":"region0", "size":"249.00 GiB (267.36 GB)", "available_size":0, "type":"pmem", "iset_id":"0xb000000058000000", "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":"249.00 GiB (267.36 GB)", "uuid":"ff626073-eef9-422c-9b09-814fc1498c1b", "blockdev":"pmem0" } ] } ] } # free total used free shared buff/cache available Mem: 362926556 9827632 352080120 2612 1018804 351120192 Swap: 32882684 0 32882684 ``` #### **Creating Namespace Examples** This example section has been split in to four, one for each mode described above: * [FSDAX Mode Examples](/ndctl-user-guide/v66/managing-namespaces#fsdax-mode-examples) * [DEVDAX Mode Examples](/ndctl-user-guide/v66/managing-namespaces#devdax-mode-examples) * [SECTOR Mode Examples](/ndctl-user-guide/v66/managing-namespaces#sector-mode-examples) * [RAW Mode Examples](/ndctl-user-guide/v66/managing-namespaces#raw-mode-examples) #### **FSDAX Mode Examples** **Example 1:** Create an 'fsdax' mode namespace using all available capacity (default behavior). The `ndctl create-namespace` command shown below is the equivalent of `ndctl create-namespace -m fsdax`. A new `/dev/pmem{X[.Y]}` device is created. This can be used to create a new DAX enabled filesystem such as XFS or EXT4. The 'X' value represents the region on which the namespace is created. This defaults to zero (0). If more than one namespace is created within a region, the naming convention of pmem{X.Y} is used, where Y represents a sequentially increasing integer value for the new namespace. If multiple namespaces within a region are created, the first namespace is always created as 'pmem0', it does not get re-enumerated to 'pmem0.0' like subsequent namespaces. ``` # ndctl create-namespace { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"a60e6a4f-274d-4cd5-8d39-c8dd263345e2", "raw_uuid":"8d28948f-5434-4c8d-8efa-581eacad265a", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ls -l /dev/pmem* brw-rw----. 1 root disk 259, 0 Jul 9 10:42 /dev/pmem0 ``` **Example 2:** Create a 50GiB 'fsdax' mode namespace. The 'size' provided to ndctl includes space required for metadata so the resulting available capacity for a filesystem will be smaller as the example below shows: ``` # ndctl create-namespace -m fsdax -s 50G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"638d67f3-4c18-4b2e-a6f2-a044bdc82253", "raw_uuid":"6c6dfdad-e12f-418d-9fb0-3a75032dd9de", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` **Note:** If the remaining capacity needs to be assigned to another namespace using the same or different mode, the remaining capacity can be assigned to the new namespace without specifying the '-s \' option. For example, executing `ndctl create-namespace` after the command above creates a new namespace in the same region using all remaining capacity (\~78GiB): ``` # ndctl create-namespace { "dev":"namespace0.1", "mode":"fsdax", "map":"dev", "size":"73.83 GiB (79.27 GB)", "uuid":"d7f9473e-97aa-48cf-aefa-128797c83e88", "raw_uuid":"6c116e57-19dd-43d8-ae03-039f1588a23a", "sector_size":512, "blockdev":"pmem0.1", "numa_node":0 } ``` **Example 3:** Create a namespace with a friendly name (tag) Tagging the namespace with a friendly name or description using the '-n, --name' option can be useful to know what a namespace is used for. This is particularly useful when provisioning multiple end-users, applications, or to identify a namespace with production or test/dev environments. Creating a namespace with a tag/name/description can be done using: ``` # ndctl create-namespace -n "PROD Web DB 1" { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"6a0abb59-5279-4731-921a-0099101e17f2", "raw_uuid":"03b40e23-56e1-407a-b5d1-f1ec929645c1", "sector_size":512, "blockdev":"pmem0", "name":"PROD Web DB 1", "numa_node":0 } ``` #### **DEVDAX Mode Examples** Create a pmem namespace with devdax mode using all available capacity ``` # ndctl create-namespace -m devdax { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"e1c1d031-71a3-4a6b-b3d7-a4037a5256c3", "raw_uuid":"2a15b77f-9f31-45e5-84a3-4ad06817302a", "daxregion":{ "id":0, "size":"123.04 GiB (132.12 GB)", "align":2097152, "devices":[ { "chardev":"dax0.0", "size":"123.04 GiB (132.12 GB)" } ] }, "numa_node":0 } # ls -l /dev/dax* crw-------. 1 root root 250, 1 Jul 9 10:39 /dev/dax0.0 ``` Create a 50GiB 'devdax' namespace ``` # ndctl create-namespace -s 50G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"b49087fc-9786-48d1-899b-6166c7c586f9", "raw_uuid":"f441a441-77ad-4286-9585-ac9511d5381a", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` #### **Sector Mode Examples** Create a sector mode namespace using all available capacity and a default 4K sector size ``` # ndctl create-namespace -m sector { "dev":"namespace0.0", "mode":"sector", "size":"124.88 GiB (134.09 GB)", "uuid":"08f4e273-bbdd-4d1d-85e8-cf1f847e1df7", "raw_uuid":"f3bd03ad-4225-4919-a370-bb6293180e4d", "sector_size":4096, "blockdev":"pmem0s", "numa_node":0 } # ls -l /dev/pmem0 brw-rw----. 1 root disk 259, 0 Jul 9 10:47 /dev/pmem0s ``` #### RAW Mode Examples Create a 'raw' mode namespace using all available capacity ``` # ndctl create-namespace -m raw { "dev":"namespace0.0", "mode":"raw", "size":"125.00 GiB (134.22 GB)", "uuid":"24c1d696-b197-4c89-96b2-f91b60f4f295", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ls -l /dev/pmem0 brw-rw----. 1 root disk 259, 0 Jul 9 10:52 /dev/pmem0 ``` ## Editing Namespace Properties Changing properties of existing namespaces can be done online using the `ndctl create-namespace -fe ` command. The `-e, --reconfigure` flag edits existing namespaces. Using the `-f` flag does not require that the namespace be manually disabled. The command may still fail if the namespace is currently being used. Supported options are listed above in the [Creating Namespaces](/ndctl-user-guide/v66/managing-namespaces#creating-namespaces) section, or review the ndctl-create-namespace [man page](/ndctl-user-guide/v66/ndctl-man-pages). Some namespace properties are read-only and cannot be changed using the `ndctl` utility. [Resizing Namespaces](/ndctl-user-guide/v66/managing-namespaces#resizing-namespaces) or [Changing Namespace Modes](/ndctl-user-guide/v66/managing-namespaces#changing-namespace-modes) can be achieved by changing existing namespace properties. They are discussed below under their own headings to provide additional detail. #### Examples Add or Change a namespace name (also referred to as a tag or description): ``` # ndctl create-namespace -fe namespace0.0 -n "prod_web_db1" { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"086d4800-4dec-41f4-a3b8-bb3e53974c54", "raw_uuid":"411b37d2-284e-4626-8724-efd0855e3c39", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` ## Resizing Namespaces {% hint style="danger" %} WARNING: Data will be lost or corrupted if the new size is smaller than the existing size or extending the namespace and growing the filesystem(s). Backup all data before resizing namespaces. {% endhint %} The following example resizes an existing 'fsdax' namespace (namespace0.0) to 20GiB: ``` # ndctl create-namespace -fe namespace0.0 -s 20G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"19.69 GiB (21.14 GB)", "uuid":"75cac06f-fc80-4c51-8f0b-2d73e2d2e31b", "raw_uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` If there is an existing EXT4 or XFS filesystem on this device, the partition table may need to be manually changed to accommodate the new device capacity before the filesystem can be resized. Due to the many configuration options available when using fsdax devices, such as whether it's being managed by the Linux device-mapper, LVM (Linux Volume Manager), or just an XFS/EXT4 filesystem on top, refer to the relevant Linux distribution documentation for specific procedures to handle your configuration. ## Changing Namespace Modes The namespace mode can be changed on any enabled or disabled namespace. Valid modes are raw, sector, fsdax, and devdax. Refer to the `ndctl-create-namespace` man page or [Namespaces](/ndctl-user-guide/v66/concepts/nvdimm-namespaces) chapter for more information on each mode. {% hint style="warning" %} {% endhint %} * Changing the namespace mode will destroy all current data. Backup all data before proceeding. * If a filesystem using an fsdax or sector namespace is mounted, unmount it before changing the mode. * If a devdax or raw namespace is currently being used by a running application, stop the application before changing modes. Switching modes of an enabled/active namespace can be done using `ndctl create-namespace -fe --mode=`. For example, to switch an enabled fsdax namespace to devdax mode, use: ``` # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":4294967296, "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ndctl create-namespace -fe namespace0.0 --mode=devdax { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":"3.94 GiB (4.23 GB)", "uuid":"d71504d1-0286-474d-839a-52afe7d0da5f", "daxregion":{ "id":0, "size":"3.94 GiB (4.23 GB)", "align":2097152, "devices":[ { "chardev":"dax0.0", "size":"3.94 GiB (4.23 GB)" } ] }, "numa_node":0 } # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":4225761280, "uuid":"d71504d1-0286-474d-839a-52afe7d0da5f", "chardev":"dax0.0", "numa_node":0 } ``` If the namespace is currently disabled, the `-f` flag is not required. ## Destroying Namespaces Namespaces should be inactive/disabled prior to deletion otherwise a warning message is displayed. The following shows an attempt to destroy an active namespace ``` # ndctl destroy-namespace namespace0.0 Error: namespace0.0 is active, specify --force for re-configuration error destroying namespaces: Device or resource busy ``` First disable the namespace, then destroy it ``` # ndctl disable-namespace namespace0.0 disabled 1 namespace # ndctl destroy-namespace namespace0.0 destroyed 1 namespace ``` Alternatively using the `-f` flag to the command forcibly destroys the namespace without doing any validation checks and does not require that the namespace be disabled first, eg: ``` # ndctl destroy-namespace -f namespace0.0 destroyed 1 namespace ``` ## Disabling Namespaces {% hint style="warning" %} **Warning:** Disabling namespaces while the namespace is mounted or in-use will result in undefined behavior by the application(s) using the namespace. Always stop the application and unmount filesystems (fsdax mode) before disabling namespaces. {% endhint %} 1\) Disable the namespace (namespace0.0) using: ``` # ndctl disable-namespace namespace0.0 disabled 1 namespace ``` 2\) Verify the namespace is *disabled*. The `-i` flag is required to display inactive/disabled namespaces otherwise only active/enabled namespaces are shown. The example below filters the output to a specific namespace (namespace0.0) to avoid potentially large volumes of output being shown: ``` # ndctl list -Ni -n namespace0.0 { "dev":"namespace0.0", "mode":"raw", "size":21474836480, "uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "state":"disabled", "name":"prod_web_db1", "numa_node":0 } ``` ## Enabling Namespaces 1\) Enable the namespace (namespace0.0) using: ``` # ndctl enable-namespace namespace0.0 enabled 1 namespace ``` 2\) Verify the namespace is enabled. The state is only displayed for disabled namespaces. If the state field is not listed, the namespace is assumed to be enabled. ``` # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":21137195008, "uuid":"75cac06f-fc80-4c51-8f0b-2d73e2d2e31b", "raw_uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` ## Checking Sector Namespaces The `check-namespace` command only works on namespaces in *sector mode*. Attempting to check namespaces in any other mode will yield an error similar to the following which attempts to check an fsdax mode namespace: ``` # ndctl check-namespace namespace0.0 namespace0.0: namespace_check: namespace0.0: check aborted, namespace online error checking namespaces: Device or resource busy ``` A namespace in the sector mode will have metadata on it to describe the Kernel BTT (Block Translation Table). The check-namespace command can be used to check the consistency of this metadata, and optionally, also attempt to repair it, if it has enough information to do so. The command usage is: ``` ndctl check-namespace [] ``` Where the available options include: ``` -b, --bus limit namespace to a bus with an id or provider of -r, --region limit namespace to a region with an id or name of -v, --verbose emit extra debug messages to stderr -R, --repair perform metadata repairs -L, --rewrite-log regenerate the log -f, --force check namespace even if currently active ``` **Examples** The namespace being checked has to be disabled before initiating a check on it as a precautionary measure. ``` # ndctl disable-namespace namespace0.0 # ndctl check-namespace namespace0.0 ``` The `-f, --force` option can override the need to manually disable the namespace first, although it is not recommended to do so: ``` # ndctl check-namespace -f namespace0.0 checked 1 namespace ``` Adding the `-v, --verbose` option yields more information, eg: ``` # ndctl check-namespace -f namespace0.0 -v namespace0.0: namespace_check: checking namespace0.0 namespace0.0: btt_discover_arenas: found 1 BTT arena namespace0.0: log_set_indices: arena[0]: log_index_0 = 0 namespace0.0: log_set_indices: arena[0]: log_index_1 = 1 namespace0.0: btt_check_arenas: checking arena 0 checked 1 namespace ``` Check a namespace and perform repairs if possible or required can be done using: ``` # ndctl disable-namespace namespace0.0 # ndctl check-namespace --repair namespace0.0 ``` # Managing NVDIMM Security {% hint style="info" %} Note: These security features require ndctl v64.0 or later and requires Linux Kernel 5.0 or later. Vendor specific utilities may also provide the same functionality on earlier Linux Kernel versions. For Intel(R) Optane(TM) DC Persistent Memory Modules, see the '[ipmctl](https://github.com/intel/ipmctl)' utility. {% endhint %} NVDIMMs that provide additional security mechanisms, such as the Inte(R) Optane(TM) DC Persistent Memory Modules, can be managed using the `ndctl` utility and the following sub-commands: * ndctl-load-keys * ndctl-freeze-security * ndctl-remove-passphrase * ndctl-sanitize-dimm * ndctl-setup-passphrase * ndctl-update-passphrase {% hint style="warning" %} **Note:** Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled, see '[Disabling Regions](/ndctl-user-guide/v66/managing-regions#disabling-regions)'. For the *overwrite* operation, in addition to disabling the *regions*, the *nvdimms* also need to be disabled, see '[Disabling NVDIMMs](/ndctl-user-guide/v66/managing-nvdimms)'. {% endhint %} The Intel(R) Device Specific Methods (DSM) specification [v1.7](http://pmem.io/documents/NVDIMM_DSM_Interface-V1.7.pdf) and [v1.8](http://pmem.io/documents/NVDIMM_DSM_Interface-V1.8.pdf) introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel-managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key-ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key-encryption-key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* ## Setup a User Passphrase The `ndctl setup-passphrase` is used to setup and enable a security passphrase for one or more NVDIMMs. To setup the passphrase for a NVDIMM or interleaved set of NVDIMMs, it is expected that the master key has previously been loaded into the user keyring. More information on how this can be done can be found in the kernel documentation at: . The *kek* encrypts the NVDIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. The command will fail if the passphrase is already in the user keyring or if a passphrase blob already exists in /etc/ndctl/keys. ## Update a User Passphrase The `ndctl update-passphrase` command updates the security passphrase for one or more NVDIMMs that have previously been setup using the `ndctl setup-passphrase` command. Prerequisites for this command to succeed: 1. The kek has been loaded into the kernel’s user keyring 2. `setup-passphrase` has successfully been executed on the NVDIMM(s) and the NVDIMM is unlocked The updated key blobs will be created in the /etc/ndctl/keys directory with a file name format of "nvdimm.blob". The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update-passphrase*: 1. Remove the encrypted passphrase from the kernel’s user keyring. 2. Rename the passphrase blob to old. 3. Load this old passphrase blob into the keyring with an “old” name. 4. Create the new passphrase and encrypt with the *kek*. 5. Send DSM with the old and new decrypted passphrases. 6. Remove old passphrase and the passphrase blob from the keyring. ## Remove a User Passphrase The `ndctl remove-passphrase` command prevents a DIMM from locking at power-loss and requiring a passphrase to access media. The *key-ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ## Crypto (Secure) Erase NVDIMMs This operation is similar to `ndctl remove-passphrase`. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use the `ndctl sanitize-dimm` command with the `--crypto-erase` option to perform this operation. The `ndctl sanitize-dimm` command performs a cryptographic destruction of the contents of the given NVDIMM. It scrambles the data, and any metadata or info-blocks, but it doesn’t modify namespace labels. Therefore, any namespaces on regions associated with the given NVDIMM will be retained, but they will end up in the raw mode. Additionally, after completion of this command, the security and passphrase for the given NVDIMM will be disabled, and the passphrase and any key material will also be removed from the keyring and the ndctl keys directory at /etc/ndctl/keys The command supports two different methods of performing the cryptographic erase. The default is `crypto-erase`, but additionally, an overwrite option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. ## Overwriting (Sanitizing) NVDIMMs One or more NVDIMMs can be sanitized using `ndctl sanitize-dimm [--crypto-erase][--overwrite]`. The command supports two different methods of performing the cryptographic erase. The default is `crypto-erase`, but additionally, an `overwrite` option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. The overwrite operation wipes the entire NVDIMM(s) and can take a significant amount of time depending on the size of the device(s) and number of devices. When the command returns successfully, it just means overwrite has been successfully started, the overwrite operation continues asynchronously in the background. Use the `ndctl wait-overwrite` command to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both `–crypto-erase` and `–overwrite` options are supplied, then crypto-erase is performed before overwrite. ## Security Freeze The `ndctl freeze-security` command does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. This is used in scenarios where the administrator has taken all expected security actions for the current boot and wants the NVDIMMs to enforce/lock the current state until system reboot. To check the current security status of a single NVDIMM (nmem{X} device): ``` $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"unlocked" } ] ``` To freeze a single NVDIMM: ``` $ ndctl freeze-security nmem0 security freezed 1 nmem. $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"frozen" } ] ``` ## Unlock NVDIMMs Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command `ndctl load-keys` is executed before the libnvdimm module is loaded by modprobe. See the 'Loading Keys' section. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1. Check the security state of the device and see if the DIMM is locked 2. Request the associated encrypted passphrase from the kernel’s user key ring 3. Use the *kek* to decrypt the passphrase 4. Create the unlock DSM, copy the decrypted payload into the DSM 5. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to re-validate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ## Loading Keys {% hint style="info" %} **Note#1:** All key files are expected to be in the format: nvdimm*\_hostname The '*' character is used to delimit the different components in the file name. Within the hostname, the '\_' character is allowed since it is the last component of the file name. **Note#2:** This command is typically never called directly by a user. {% endhint %} The `load-keys` command loads the master key (kek) and the encrypted passphrases for all NVDIMMs into the user keyring maintained by the kernel. The command is expected to be called during initialization and before the libnvdimm kernel module is loaded, typically from an initrd. This is typically set up using a modprobe config that calls the command before module load. See for more information on the Kernel implementation. ## Master Passphrase - Setup, Update, and Crypto Erase These operations are similar to creating and updating the user passphrase. The only difference is that a different passphrase is used. Both `setup-passphrase` and `update-passphrase` support the `-m` flag to differentiate between user and master passphrase operations. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. More information about the master key can be found in the kernel documentation at: . # NDCTL Man Pages This section links to the online man pages: * [ndctl ](/ndctl-user-guide/v66/ndctl-man-pages/untitled)- Manage “libnvdimm” subsystem devices (Non-volatile Memory) * [ndctl-check-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels) - determine if the given dimms have a valid namespace index block * [ndctl-check-namespace](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace) - check namespace metadata consistency * [ndctl-create-namespace](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace) - provision or reconfigure a namespace * [ndctl-destroy-namespace](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-destroy-namespace) - destroy the given namespace(s) * [ndctl-disable-dimm](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm) - disable one or more idle dimms * [ndctl-disable-namespace](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace) - disable the given namespace(s) * [ndctl-disable-region](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region) - disable the given region(s) and all descendant namespaces * [ndctl-enable-dimm](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm) - enable one more dimms * [ndctl-enable-namespace](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace) - enable the given namespace(s) * [ndctl-enable-region](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region) - enable the given region(s) and all descendant namespaces * [ndctl-freeze-security](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security) - Set the given DIMM(s) to reject future security operations * [ndctl-init-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels) - initialize the label data area on a dimm or set of dimms * [ndctl-inject-error](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error) - inject media errors at a namespace offset * [ndctl-inject-smart](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart) - perform smart threshold/injection operations on an NVDIMM * [ndctl-list](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list) - print the platform nvdimm device topology and attributes * [ndctl-load-keys](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-load-keys) - load the master key (kek) and encrypted passphrases into the keyring * [ndctl-monitor](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-monitor) - Monitor the SMART events from NVDIMMs * [ndctl-read-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels) - read out the label area on a NVDIMM or set of NVDIMMs * [ndctl-remove-passphrase](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase) - Stop an NVDIMM from locking at power-loss and requiring a passphrase to access media * [ndctl-sanitize-dimm](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm) - Perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s) * [ndctl-setup-passphrase](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase) - setup and enable the security passphrase for one or more NVDIMMs * [ndctl-start-scrub](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub) - start an Address Range Scrub (ARS) operation * [ndctl-update-firmware](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-firmware) - provides for updating the firmware on an NVDIMM * [ndctl-update-passphrase](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase) - update the security passphrase for one or more NVDIMMs * [ndctl-wait-overwrite](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite) - wait for an overwrite operation to complete * [ndctl-wait-scrub](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2) - wait for an Address Range Scrub (ARS) operation to complete * [ndctl-write-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-write-labels) - write data to the label area on a dimm * [ndctl-zero-labels](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels) - zero out the label area on a dimm or set of dimms # ndctl(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/untitled#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/untitled#synopsis)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/untitled#options)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/untitled#description)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/untitled#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/untitled#see-also) ## NAME ndctl − Manage "libnvdimm" subsystem devices (Non−volatile Memory) ## SYNOPSIS *ndctl* \[−−version] \[−−help] \[OPTIONS] COMMAND \[ARGS] ## OPTIONS −v, −−version Display ndctl version. −h, −−help Run ndctl help command. ## DESCRIPTION ndctl is utility for managing the "libnvdimm" kernel subsystem. The "libnvdimm" subsystem defines a kernel device model and control message interface for platform NVDIMM resources like those defined by the ACPI 6.0 NFIT (NVDIMM Firmware Interface Table). Operations supported by the tool include, provisioning capacity (namespaces), as well as enumerating/enabling/disabling the devices (dimms, regions, namespaces) associated with an NVDIMM bus. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−create−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace), [ndctl−destroy−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-destroy-namespace), [ndctl−check−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace), [ndctl−enable−region(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region), [ndctl−disable−region(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region), [ndctl−enable−dimm(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm), [ndctl−disable−dimm(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm), [ndctl−enable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace), [ndctl−disable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace), [ndctl−zero−labels(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels), [ndctl−read−labels(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels), [ndctl−inject−error(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error), [ndctl−list(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list), *LIBNVDIMM* Overview" , *NVDIMM Driver* Writer’s Guide" # ndctl−check−labels(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels#see-also) ## NAME ndctl−check−labels − determine if the given dimms have a valid namespace index block ## SYNOPSIS *ndctl check−labels* \ \[\..\] \[\] ## DESCRIPTION The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to resolve aliasing between *pmem* and *blk* capacity by delineating namespace boundaries. In addition to checking if a label area has a valid index block, running this command in verbose mode reports the reason the index block is deemed invalid. ## OPTIONS \ One or more *nmemX* device names. The keyword *all* can be specified to operate on every dimm in the system, optionally filtered by bus id (see −−bus= option). −s, −−size= Limit the operation to the given number of bytes. A size of 0 indicates to operate over the entire label capacity. −O, −−offset= Begin the operation at the given offset into the label area. −b, −−bus= Limit operation to memory devices (dimms) that are on the given bus. Where *bus* can be a provider name or a bus id number. −v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO *UEFI NVDIMM Label Protocol* # ndctl−check−namespace(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace#description)\ [EXAMPLES](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace#examples)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace#see-also) ## NAME ndctl−check−namespace − check namespace metadata consistency ## SYNOPSIS *ndctl check−namespace* \ \[\] ## DESCRIPTION A namespace in the *sector* mode will have metadata on it to describe the kernel BTT (Block Translation Table). The check−namespace command can be used to check the consistency of this metadata, and optionally, also attempt to repair it, if it has enough information to do so. The namespace being checked has to be disabled before initiating a check on it as a precautionary measure. The `−−force` option can override this. ## EXAMPLES Check a namespace (only report errors) ``` ndctl disable−namespace namespace0.0 ndctl check−namespace namespace0.0 ``` Check a namespace, and perform repairs if possible ``` ndctl disable−namespace namespace0.0 ndctl check−namespace −−repair namespace0.0 ``` ## OPTIONS −R, −−repair Perform metadata repairs if possible. Without this option, the raw namespace contents will not be touched. −L, −−rewrite−log Regenerate the BTT log and write it to media. This can be used to convert from the old (pre 4.15) padding format that was incompatible with other BTT implementations to the updated format. This requires the `−−repair` option to be provided. WARNING: Do not interrupt this operation as it can potentially cause unrecoverable metadata corruption. It is highly recommended to create a backup of the raw namespace before attempting this. −f, −−force Unless this option is specified, a check−namespace operation will fail if the namespace is presently active. Specifying `−−force` causes the namespace to be disabled before checking. −v, −−verbose Emit debug messages for the namespace check process. −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see `−−bus=` option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−disable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace), [ndctl−enable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace), *UEFI NVDIMM Label Protocol* # ndctl−clear−errors(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-check-namespace#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-clear-errors#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-clear-errors#description)\ [EXAMPLES](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-clear-errors#examples)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-clear-errors#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-clear-errors#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-clear-errors#see-also) ## NAME ndctl−clear−errors − clear all errors (badblocks) on the given namespace ## SYNOPSIS *ndctl clear−errors* \ \[\] ## DESCRIPTION A namespace may have one or more *media errors*, either known to the kernel or in a latent state. These error locations, or *badblocks* can cause poison consumption events if read in an unsafe manner. Moreover, these badblocks also indicate that due to media corruption, any data that may have been in these locations has been unrecoverably lost. Normally, in the presence of such errors, the administrator is expected to recover the data from out of band means (such as backups), destroy the namespace, recreate it, and then restore the data. When the data is re−written, the writes will allow any errors to be cleared as they are encountered. In such a workflow, one should **never** need to use the *clear−errors* command. However, there may be special use cases, where the data currently on the namespace does not matter − for example, if a *devdax* mode namespace is being prepared for use as *system−ram*. In such cases, it may be desirable to clear any errors on the namespace prior to switching its mode to prevent disruptive machine checks due to poison consumption. **Note**\ **Only** use this command when the data on the namespace is immaterial. For any blocks that are cleared via this command, any data on the blocks in question will be lost, and replaced with content that is implementation (platform) defined, and unpredictable. **Warning**\ This is a DANGEROUS command, and should only be used after fully understanding its implications and consequences. This WILL erase your data. For namespaces in one of *fsdax* or *davdax* modes, this command will only consider the *data* area for error clearing. Namespace metadata, such as info−blocks, will not be touched. For namespaces in *raw* mode, the full available capacity of the namespace is considered for error clearing. Namespaces that are in *sector* mode are not supported, and will be skipped. **Note**\ It is expected that the command is run with the namespace *enabled*. A namespace in the *disabled* state will appear as, and will be treated as a *raw* namespace, and error clearing will be performed for the full available capacity of the namespace, including any potential metadata areas. If there happen to be errors in the metadata area, clearing them may result in unpredictable outcomes. You have been warned! Known errors are ones that the kernel has encountered before, either via a previous scrub, or by an attempted read from those locations. These can be listed by running *ndctl list −−media−errors* for a given namespace. Latent errors, as the name indicates, are unknown to the kernel. These can be found by running a scrub operation on the NVDIMMs in question. By default, the ndctl−clear−errors command only clears known errors. This can be overridden using the *−−scrub* option to clear **all** errors. **Note**\ If a scrub is in progress when the command is called, it will unconditionally wait for it to complete. ## EXAMPLES Clear errors on namespace 0.0 ``` ndctl clear−errors namespace0.0 ``` Clear errors on all namespaces belonging to region1, including scrubbing for latent errors ``` ndctl clear−errors −−scrub −−region=region1 all ``` ## OPTIONS −s, −−scrub Perform a *scrub* on the bus prior to clearing errors. This allows for the clearing of any latent media errors in addition to errors the kernel already knows about. **Note**\ This will cause the command to start and wait for a full scrub, and this can potentially be a very long−running operation. −v, −−verbose Emit debug messages. −r, −−region= A *regionX* device name, or a region id number. The keyword `all`can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see `−−bus=` option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−start−scrub(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub), [ndctl−list(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list) # ndctl−create−namespace(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#synopsis)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#theory-of-operation)\ [EXAMPLES](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#examples)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#see-also) ## NAME ndctl−create−namespace − provision or reconfigure a namespace ## SYNOPSIS *ndctl create−namespace* \[\] ## THEORY OF OPERATION The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. REGION is the Linux term for what ACPI and UEFI call a DIMM−interleave−set, or a system−physical−address−range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label−less" namespace per region. Examples of label−less namespaces are the ones created by the kernel’s *memmap=ss!nn* command line option (see the nvdimm wiki on kernel.org), or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, *fsdax*, *devdax*, *sector*, and *raw*. Here are the expected usage models for these modes: • **fsdax**: Filesystem−DAX mode is the default mode of a namespace when specifying *ndctl create−namespace* with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working−sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. • **devdax**: Device−DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem−DAX. However, instead of a block−device that can support a DAX−enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual−machine, register persistent memory for RDMA, or when gigantic mappings are needed. • **sector**: Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. • **raw**: Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux *fsdax* or *devdax* mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. ## EXAMPLES Create a maximally sized pmem namespace in *fsdax* mode (the default) ``` ndctl create−namespace ``` Convert namespace0.0 to *sector* mode ``` ndctl create−namespace −f −e namespace0.0 −−mode=sector ``` ## OPTIONS −t, −−type= Create a *pmem* or *blk* namespace (subject to available capacity). A pmem namespace supports the dax (direct access) capability to mmap(2) persistent memory directly into a process address space. A blk namespace access persistent memory through a block−window−aperture. Compared to pmem it supports a traditional storage error model (EIO on error rather than a cpu exception on a bad memory access), but it does not support dax. −m, −−mode= • "raw": expose the namespace capacity directly with limitations. Neither a raw pmem namepace nor raw blk namespace support sector atomicity by default (see "sector" mode below). A raw pmem namespace may have limited to no dax support depending the kernel. In other words operations like direct−I/O targeting a dax buffer may fail for a pmem namespace in raw mode or indirect through a page−cache buffer. See "fsdax" and "devdax" mode for dax operation. • "sector": persistent memory, given that it is byte addressable, does not support sector atomicity. The problematic aspect of sector tearing is that most applications do not know they have a atomic sector update dependency. At least a disk rarely ever tears sectors and if it does it almost certainly returns a checksum error on access. Persistent memory devices will always tear and always silently. Until an application is audited to be robust in the presence of sector−tearing "safe" mode is recommended. This imposes some performance overhead and disables the dax capability. (also known as "safe" or "btt" mode) • "fsdax": A pmem namespace in this mode supports dax operation with a block−device based filesystem (in previous ndctl releases this mode was named "memory" mode). This mode comes at the cost of allocating per−page metadata. The capacity can be allocated from "System RAM", or from a reserved portion of "Persistent Memory" (see the −−map= option). NOTE: A filesystem that supports DAX is required for dax operation. If the raw block device (/dev/pmemX) is used directly without a filesystem, it will use the page cache. See "devdax" mode for raw device access that supports dax. • "devdax": The device−dax character device interface is a statically allocated / raw access analogue of filesystem−dax (in previous ndctl releases this mode was named "dax" mode). It allows memory ranges to be mapped without need of an intervening filesystem. The device−dax is interface strict, precise and predictable. Specifically the interface: • Guarantees fault granularity with respect to a given page size (4K, 2M, or 1G on x86) set at configuration time. • Enforces deterministic behavior by being strict about what fault scenarios are supported. I.e. if a device is configured with a 2M alignment an attempt to fault a 4K aligned offset will result in SIGBUS. −s, −−size= For NVDIMM devices that support namespace labels, set the namespace size in bytes. Otherwise it defaults to the maximum size specified by platform firmware. This option supports the suffixes "k" or "K" for KiB, "m" or "M" for MiB, "g" or "G" for GiB and "t" or "T" for TiB. For pmem namepsaces the size must be a multiple of the\ interleave−width and the namespace alignment (see\ below). −a, −−align Applications that want to establish dax memory mappings with page table entries greater than system base page size (4K on x86) need a persistent memory namespace that is sufficiently aligned. For "fsdax" and "devdax" mode this defaults to 2M. Note that "devdax" mode enforces all mappings to be aligned to this value, i.e. it fails unaligned mapping attempts. The "fsdax" alignment setting determines the starting alignment of filesystem extents and may limit the possible granularities, if a large mapping is not possible it will silently fall back to a smaller page size. −e, −−reconfig= Reconfigure an existing namespace (change the mode, sector size, etc...). All namespace parameters, save uuid, default to the current attributes of the specified namespace. The namespace is then re−created with the specified modifications. The uuid is refreshed to a new value by default whenever the data layout of a namespace is changed, see −−uuid= to set a specific uuid. −u, −−uuid= This option is not recommended as a new uuid should be generated every time a namespace is (re−)created. For recovery scenarios however the uuid may be specified. −n, −−name= For NVDIMM devices that support namespace labels, specify a human friendly name for a namespace. This name is available as a device attribute for use in udev rules. −l, −−sector−size Specify the logical sector size (LBA size) of the Linux block device associated with an namespace. −M, −−map= A pmem namespace in "fsdax" or "devdax" mode requires allocation of per−page metadata. The allocation can be drawn from either: • "mem": typical system memory • "dev": persistent memory reserved from the namespace Given relative capacities of "Persistent Memory" to "System\ RAM" the allocation defaults to reserving space out of the\ namespace directly ("−−map=dev"). The overhead is 64−bytes per\ 4K (16GB per 1TB) on x86. −f, −−force Unless this option is specified the *reconfigure namespace* operation will fail if the namespace is presently active. Specifying −−force causes the namespace to be disabled before the operation is attempted. However, if the namespace is mounted then the *disable namespace* and *reconfigure namespace* operations will be aborted. The namespace must be unmounted before being reconfigured. −L, −−autolabel, −−no−autolabel Legacy NVDIMM devices do not support namespace labels. In that case the kernel creates region−sized namespaces that can not be deleted. Their mode can be changed, but they can not be resized smaller than their parent region. This is termed a "label−less namespace". In contrast, NVDIMMs and hypervisors that support the ACPI 6.2 label area definition (ACPI 6.2 Section 6.5.10 NVDIMM Label Methods) support "labelled namespace" operation. • There are two cases where the kernel will default to label−less operation: • NVDIMM does not support labels • The NVDIMM supports labels, but the Label Index Block (see UEFI 2.7) is not present and there is no capacity aliasing between *blk* and *pmem* regions. • In the latter case the configuration can be upgraded to labelled operation by writing an index block on all DIMMs in a region and re−enabling that region. The *autolabel* capability of *ndctl create−namespace −−reconfig* tries to do this by default if it can determine that all DIMM capacity is referenced by the namespace being reconfigured. It will otherwise fail to autolabel and remain in label−less mode if it finds a DIMM contributes capacity to more than one region. This check prevents inadvertent data loss of that other region is in active use. The −−autolabel option is implied by default, the −−no−autolabel option can be used to disable this behavior. When automatic labeling fails and labelled operation is still desired the safety policy can be bypassed by the following commands, note that all data on all regions is forfeited by running these commands: ndctl disable−region all\ ndctl init−labels all\ ndctl enable−region all −v, −−verbose Emit debug messages for the namespace creation process −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−zero−labels(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels), [ndctl−init−labels(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels), [ndctl−disable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace), [ndctl−enable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace), *UEFI NVDIMM Label Protocol* *Linux Persistent Memory Wiki* # ndctl−destroy−namespace(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-clear-errors#synopsis)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-destroy-namespace#theory-of-operation)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-destroy-namespace#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-destroy-namespace#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-destroy-namespace#see-also) ## NAME ndctl−destroy−namespace − destroy the given namespace(s) ## SYNOPSIS *ndctl destroy−namespace* \ \[\] ## THEORY OF OPERATION The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. REGION is the Linux term for what ACPI and UEFI call a DIMM−interleave−set, or a system−physical−address−range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label−less" namespace per region. Examples of label−less namespaces are the ones created by the kernel’s *memmap=ss!nn* command line option (see the nvdimm wiki on kernel.org), or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, *fsdax*, *devdax*, *sector*, and *raw*. Here are the expected usage models for these modes: • fsdax: Filesystem−DAX mode is the default mode of a namespace when specifying *ndctl create−namespace* with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working−sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. • devdax: Device−DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem−DAX. However, instead of a block−device that can support a DAX−enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual−machine, register persistent memory for RDMA, or when gigantic mappings are needed. • sector: Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. • raw: Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux *fsdax* or *devdax* mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. ## OPTIONS A *namespaceX.Y* device name. The keyword *all* can be specified to carry out the operation on every namespace in the system, optionally filtered by region (see −−region=option) −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −v, −−verbose Emit debug messages for the namespace operation −f, −−force Unless this option is specified the *destroy namespace* operation will fail if the namespace is presently active. Specifying −−force causes the namespace to be disabled before the operation is attempted. However, if the namespace is mounted then the *disable namespace* and *destroy namespace* operations will be aborted. The namespace must be unmounted before being destroyed. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−create−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace) # ndctl−disable−dimm(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm#see-also) ## NAME ndctl−disable−dimm − disable one or more idle dimms ## SYNOPSIS *ndctl disable−dimm* \ \[\] ## DESCRIPTION A generic DIMM device object, named `/dev/nmemX`, is registered for each memory device indicated in the ACPI NFIT table, or other platform NVDIMM resource discovery mechanism. The LIBNVDIMM core provides a built−in driver for these DIMM devices. The driver is responsible for determining if the DIMM implements a namespace label area, and initializing the kernel’s in−memory copy of that label data. The kernel performs runtime modifications of that data when namespace provisioning actions are taken, and actively blocks userspace from initiating label data changes while the DIMM is active in any region. Disabling a DIMM, after all the regions it is a member of have been disabled, allows userspace to manually update the label data to be consumed when the DIMM is next enabled. ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see `−−bus=` option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−enable−dimm(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm) # ndctl−disable−namespace(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace#synopsis)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace#theory-of-operation)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace#see-also) ## NAME ndctl−disable−namespace − disable the given namespace(s) ## SYNOPSIS *ndctl disable−namespace* \ \[\] ## THEORY OF OPERATION The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. REGION is the Linux term for what ACPI and UEFI call a DIMM−interleave−set, or a system−physical−address−range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label−less" namespace per region. Examples of label−less namespaces are the ones created by the kernel’s *memmap=ss!nn* command line option (see the nvdimm wiki on kernel.org), or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, *fsdax*, *devdax*, *sector*, and *raw*. Here are the expected usage models for these modes: • fsdax: Filesystem−DAX mode is the default mode of a namespace when specifying *ndctl create−namespace* with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working−sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. • devdax: Device−DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem−DAX. However, instead of a block−device that can support a DAX−enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual−machine, register persistent memory for RDMA, or when gigantic mappings are needed. • sector: Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. • raw: Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux *fsdax* or *devdax* mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. ## OPTIONS A *namespaceX.Y* device name. The keyword *all* can be specified to carry out the operation on every namespace in the system, optionally filtered by region (see −−region=option) −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −v, −−verbose Emit debug messages for the namespace operation ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−enable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace) # ndctl−disable−region(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region#see-also) ## NAME ndctl−disable−region − disable the given region(s) and all descendant namespaces ## SYNOPSIS *ndctl disable−region* \ \[\] ## DESCRIPTION A generic REGION device is registered for each PMEM range or BLK−aperture set. LIBNVDIMM provides a built−in driver for these REGION devices. This driver is responsible for reconciling the aliased DPA mappings across all regions, parsing the LABEL, if present, and then emitting NAMESPACE devices with the resolved/exclusive DPA−boundaries for the nd\_pmem or nd\_blk device driver to consume. ## OPTIONS A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−enable−region(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region) # ndctl−enable−dimm(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-dimm#see-also) ## NAME ndctl−enable−dimm − enable one more dimms ## SYNOPSIS *ndctl enable−dimm* \ \[\] ## DESCRIPTION A generic DIMM device object, named /dev/nmemX, is registered for each memory device indicated in the ACPI NFIT table, or other platform NVDIMM resource discovery mechanism. The LIBNVDIMM core provides a built−in driver for these DIMM devices. The driver is responsible for determining if the DIMM implements a namespace label area, and initializing the kernel’s in−memory copy of that label data. The kernel performs runtime modifications of that data when namespace provisioning actions are taken, and actively blocks userspace from initiating label data changes while the DIMM is active in any region. Disabling a DIMM, after all the regions it is a member of have been disabled, allows userspace to manually update the label data to be consumed when the DIMM is next enabled. ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−disable−dimm(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-dimm) # ndctl-enable−namespace(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace#synopsis)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace#theory-of-operation)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-namespace#see-also) ## NAME ndctl−enable−namespace − enable the given namespace(s) ## SYNOPSIS *ndctl enable−namespace* \ \[\] ## THEORY OF OPERATION The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. REGION is the Linux term for what ACPI and UEFI call a DIMM−interleave−set, or a system−physical−address−range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label−less" namespace per region. Examples of label−less namespaces are the ones created by the kernel’s *memmap=ss!nn* command line option (see the nvdimm wiki on kernel.org), or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, *fsdax*, *devdax*, *sector*, and *raw*. Here are the expected usage models for these modes: • fsdax: Filesystem−DAX mode is the default mode of a namespace when specifying *ndctl create−namespace* with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working−sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. • devdax: Device−DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem−DAX. However, instead of a block−device that can support a DAX−enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual−machine, register persistent memory for RDMA, or when gigantic mappings are needed. • sector: Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. • raw: Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux *fsdax* or *devdax* mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. ## OPTIONS A *namespaceX.Y* device name. The keyword *all* can be specified to carry out the operation on every namespace in the system, optionally filtered by region (see −−region=option) −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −v, −−verbose Emit debug messages for the namespace operation ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−disable−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-namespace) # ndctl−enable−region(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-enable-region#see-also) ## NAME ndctl−enable−region − enable the given region(s) and all descendant namespaces ## SYNOPSIS *ndctl enable−region* \ \[\] ## DESCRIPTION A generic REGION device is registered for each PMEM range or BLK−aperture set. LIBNVDIMM provides a built−in driver for these REGION devices. This driver is responsible for reconciling the aliased DPA mappings across all regions, parsing the LABEL, if present, and then emitting NAMESPACE devices with the resolved/exclusive DPA−boundaries for the nd\_pmem or nd\_blk device driver to consume. ## OPTIONS A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−disable−region(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-disable-region) # ndctl−freeze−security(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security#description)\ [EXAMPLES](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security#examples)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security#options)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security#theory-of-operation)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-freeze-security#copyright) ## NAME ndctl−freeze−security − Set the given DIMM(s) to reject future security operations ## SYNOPSIS *ndctl freeze−security* \ \[\..\] \[\] ## DESCRIPTION Prevent any further security operations on the given DIMMs until the next reboot. This is used in scenarios where the administrator has taken all expected security actions for the current boot and wants the DIMM to enforce / lock the current state. ## EXAMPLES ``` $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"unlocked" } ] $ ndctl freeze-security nmem0 security freezed 1 nmem. $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"frozen" } ] ``` ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −v, −−verbose Emit debug messages. ## THEORY OF OPERATION The Intel Device Specific Methods (DSM) specification v1.7 and v1.8 \[1] introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel−managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key−ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key−encryption−key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* For testing purposes a user key with randomized payload can also be used as a *kek*. See \[2] for details. To perform any security operations, it is expected that the *kek* has been added to the kernel’s user keyring as shown in example below: ``` # keyctl show Session Keyring 736023423 --alswrv 0 0 keyring: _ses 675104189 --alswrv 0 65534 \_ keyring: _uid.0 680187394 --alswrv 0 0 \_ trusted: nvdimm-master ``` Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled. For the *overwrite* operation, in addition to the *regions*, the *dimm* also needs to be disabled. \[1] \ \[2] The following sub−sections describe specifics of each security feature. ### **UNLOCK** Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command (ndctl *load−keys*) is executed before the libnvdimm module is loaded by modprobe. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1\. Check the security state of the device and see if the DIMM is locked 2\. Request the associated encrypted passphrase from the kernel’s user key ring 3\. Use the *kek* to decrypt the passphrase 4\. Create the unlock DSM, copy the decrypted payload into the DSM 5\. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ### **SETUP USER PASSPHRASE** To setup the passphrase for a DIMM, it is expected that the *kek* to be used is present in the kernel’s user keyring. The *kek* encrypts the DIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. ### **UPDATE USER PASSPHRASE** The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update−passphrase*: 1\. Remove the encrypted passphrase from the kernel’s user keyring. 2\. Rename the passphrase blob to old. 3\. Load this old passphrase blob into the keyring with an "old" name. 4\. Create the new passphrase and encrypt with the *kek*. 5\. Send DSM with the old and new decrypted passphrases. 6\. Remove old passphrase and the passphrase blob from the keyring. ### **REMOVE USER PASSPHRASE** The *key−ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ### **CRYPTO (SECURE) ERASE** This operation is similar to remove−passphrase. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use ndctl’s sanitize−dimm command with the −−crypto−erase option to perform this operation. ### **OVERWRITE** This is invoked using −−overwrite option for ndctl *sanitize−dimm*. The overwrite operation wipes the entire NVDIMM. The operation can take a significant amount of time. NOTE: When the command returns successfully, it just means overwrite has been successfully started, and not that the overwrite is complete. Subsequently, 'ndctl wait−overwrite’can be used to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both −−crypto−erase and −−overwrite options are supplied, then crypto−erase is performed before overwrite. ### **SECURITY FREEZE** This operation does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. ### **MASTER PASSPHRASE SETUP, UPDATE, and CRYPTO ERASE** These operations are similar to the user passphrase enable and update. The only difference is that a different passphrase is used. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. # ndctl−init−labels(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels#description)\ [EXAMPLE](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels#example)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-init-labels#see-also) ## NAME ndctl−init−labels − initialize the label data area on a dimm or set of dimms ## SYNOPSIS *ndctl init−labels* \ \[\..\] \[\] ## DESCRIPTION The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to resolve aliasing between *pmem* and *blk* capacity by delineating namespace boundaries. By default, and in kernels prior to v4.10, the kernel only honors labels when a DIMM aliases PMEM and BLK capacity. Starting with v4.10 the kernel will honor labels for sub−dividing PMEM if all the DIMMs in an interleave set / region have a valid namespace index block. This command can be used to initialize the namespace index block if it is missing or reinitialize it if it is damaged. Note that reinitialization effectively destroys all existing namespace labels on the DIMM. ## EXAMPLE Find the DIMMs that comprise a given region: ``` # ndctl list -RD --region=region1 { "dimms":[ { "dev":"nmem0", "id":"8680-56341200" } ], "regions":[ { "dev":"region1", "size":268435456, "available_size":0, "type":"pmem", "mappings":[ { "dimm":"nmem0", "offset":13958643712, "length":268435456 } ] } ] } ``` Disable that region so the DIMM label area can be written from userspace: ``` # ndctl disable−region region1 ``` Initialize labels: ``` # ndctl init−labels nmem0 ``` Re−enable the region: ``` # ndctl enable−region region1 ``` Create a namespace in that region: ``` # ndctl create−namespace −−region=region1 ``` ## OPTIONS One or more *nmemX* device names. The keyword *all* can be specified to operate on every dimm in the system, optionally filtered by bus id (see −−bus= option). −s, −−size= Limit the operation to the given number of bytes. A size of 0 indicates to operate over the entire label capacity. −O, −−offset= Begin the operation at the given offset into the label area. −b, −−bus= Limit operation to memory devices (dimms) that are on the given bus. Where *bus* can be a provider name or a bus id number. −v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). −f, −−force Force initialization of the label space even if there appears to be an existing / valid namespace index. Warning, this will destroy all defined namespaces on the dimm. −V, −−label−version Initialize with a specific version of labels from the namespace label specification. Defaults to 1.1 ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−create−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace), *UEFI NVDIMM Label Protocol* # ndctl−inject−error(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error#synopsis)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error#theory-of-operation)\ [EXAMPLES](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error#examples)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-error#see-also) ## NAME ndctl−inject−error − inject media errors at a namespace offset ## SYNOPSIS *ndctl inject−error* \ \[\] ## THEORY OF OPERATION The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. REGION is the Linux term for what ACPI and UEFI call a DIMM−interleave−set, or a system−physical−address−range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label−less" namespace per region. Examples of label−less namespaces are the ones created by the kernel’s *memmap=ss!nn* command line option (see the nvdimm wiki on kernel.org), or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, *fsdax*, *devdax*, *sector*, and *raw*. Here are the expected usage models for these modes: • fsdax: Filesystem−DAX mode is the default mode of a namespace when specifying *ndctl create−namespace* with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working−sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. • devdax: Device−DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem−DAX. However, instead of a block−device that can support a DAX−enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual−machine, register persistent memory for RDMA, or when gigantic mappings are needed. • sector: Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. • raw: Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux *fsdax* or *devdax* mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. ndctl−inject−error can be used to ask the platform to simulate media errors in the NVDIMM address space to aid debugging and development of features related to error handling. By default, injecting an error actually only injects an error to the first *n* bytes of the block, where *n* is the output of ndctl\_cmd\_ars\_cap\_get\_size(). In other words, we only inject one *ars\_unit* per sector. This is sufficient for Linux to mark the whole sector as bad, and will show up as such in the various *badblocks* lists in the kernel. If multiple blocks are being injected, only the first *n* bytes of each block specified will be injected as errors. This can be overridden by the −−saturate option, which will force the entire block to be injected as an error. **Warning**\ These commands are DANGEROUS and can cause data loss. They are only provided for testing and debugging purposes. ## EXAMPLES Inject errors in namespace0.0 at block 12 for 2 blocks (i.e. 12, 13) ``` ndctl inject−error −−block=12 −−count=2 namespace0.0 ``` Check status of injected errors on namespace0.0 ``` ndctl inject−error −−status namespace0.0 ``` Uninject errors at block 12 for 2 blocks on namespace0.0 ``` ndctl inject−error −−uninject −−block=12 −−count=2 namespace0.0 ``` ## OPTIONS −B, −−block= Namespace block offset in 512 byte sized blocks where the error is to be injected. NOTE: The offset is interpreted in different ways based on the "mode"\ of the namespace. For "raw" mode, the offset is the base namespace\ offset. For "fsdax" mode (i.e. a "pfn" namespace), the offset is\ relative to the user−visible part of the namespace, and the offset\ introduced by the kernel's metadata will be accounted for. For a\ "sector" mode namespace (i.e. a "BTT" namespace), the offset is\ relative to the base namespace, as the BTT translation details are\ internal to the kernel, and can't be accounted for while injecting\ errors. −n, −−count= Number of blocks to inject as errors. This is also in terms of fixed, 512 byte blocks. −d, −−uninject This option will ask the platform to remove any injected errors for the specified block offset, and count. WARNING: This will not clear the kernel's internal badblock tracking,\ those can only be cleared by doing a write to the affected locations.\ Hence use the −−clear option only if you know exactly what you are\ doing. For normal usage, injected errors should only be cleared by\ doing writes. Do not expect have the original data intact after\ injecting an error, and clearing it using −−clear − it will be lost,\ as the only "real" way to clear the error location is to write to it\ or zero it (truncate/hole−punch). −t, −−status This option will retrieve the status of injected errors. Note that this will not retrieve all known/latent errors (i.e. non injected ones), and is NOT equivalent to performing an Address Range Scrub. −N, −−no−notify This option is only valid when injecting errors. By default, the error inject command and will ask platform firmware to trigger a notification in the kernel, asking it to update its state of known errors. With this option, the error will still be injected, the kernel will not get a notification, and the error will appear as a latent media error when the location is accessed. If the platform firmware does not support this feature, this will have no effect. −S, −−saturate This option forces error injection or un−injection to cover the entire address range covered by the specified block(s). −v, −−verbose Emit debug messages for the error injection process −u, −−human Format numbers representing storage sizes, or offsets as human readable strings with units instead of the default machine−friendly raw−integer data. Convert other numeric fields into hexadecimal strings. −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−list(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list), # ndctl−inject−smart(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart#description)\ [EXAMPLES](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart#examples)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart#see-also) ## NAME ndctl−inject−smart − perform smart threshold/injection operations on a DIMM ## SYNOPSIS *ndctl inject−smart* \ \[\] ## DESCRIPTION A generic DIMM device object, named /dev/nmemX, is registered for each memory device indicated in the ACPI NFIT table, or other platform NVDIMM resource discovery mechanism. ndctl−inject−smart can be used to set smart thresholds, and inject smart attributes. ## EXAMPLES Set smart controller temperature and spares threshold for DIMM−0 to 32C, spares threshold to 8, and enable the spares alarm. ``` ndctl inject−smart −−ctrl−temperature−threshold=32 −−spares−threshold=8 −−spares−alarm nmem0 ``` Inject a media temperature value of 52 and fatal health status flag for DIMM−0 ``` ndctl inject−smart −−media−temperature=52 −−health=fatal nmem0 ``` ## OPTIONS −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −m, −−media−temperature= Inject for the media temperature smart attribute. −M, −−media−temperature−threshold= Set for the smart media temperature threshold. −−media−temperature−alarm= Enable or disable the smart media temperature alarm. Options are *on* or *off*. −−media−temperature−uninject Uninject any media temperature previously injected. −c, −−ctrl−temperature= Inject for the controller temperature smart attribute. −C, −−ctrl−temperature−threshold= Set for the smart controller temperature threshold. −−ctrl−temperature−alarm= Enable or disable the smart controller temperature alarm. Options are *on* or *off*. −−ctrl−temperature−uninject Uninject any controller temperature previously injected. −s, −−spares= Inject for the spares smart attribute. −S, −−spares−threshold= Set for the smart spares threshold. −−spares−alarm= Enable or disable the smart spares alarm. Options are *on* or *off*. −−spares−uninject Uninject any spare percentage previously injected. −f, −−fatal Set the flag to spoof fatal health status. −−fatal−uninject Uninject the fatal health status flag. −U, −−unsafe−shutdown Set the flag to spoof an unsafe shutdown on the next power down. −−unsafe−shutdown−uninject Uninject the unsafe shutdown flag. −N, −−uninject−all Uninject all possible smart fields/values irrespective of whether they have been previously injected or not. −v, −−verbose Emit debug messages for the error injection process −u, −−human Format numbers representing storage sizes, or offsets as human readable strings with units instead of the default machine−friendly raw−integer data. Convert other numeric fields into hexadecimal strings. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−list(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list), # ndctl−list(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list#synopsis)\ [EXAMPLE](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list#example)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list#see-also) ## NAME ndctl−list − dump the platform nvdimm device topology and attributes in json ## SYNOPSIS *ndctl list* \[\] Walk all the nvdimm buses in the system and list all attached devices along with some of their major attributes. Options can be specified to limit the output to devices of a certain class. Where the classes are buses, dimms, regions, and namespaces. By default, *ndctl list* with no options is equivalent to: ``` ndctl list −−namespaces −−bus=all −−region=all ``` ## EXAMPLE ``` # ndctl list --buses --namespaces { "provider":"nfit_test.1", "dev":"ndbus2", "namespaces":[ { "dev":"namespace9.0", "mode":"raw", "size":33554432, "blockdev":"pmem9" } ] } { "provider":"nfit_test.0", "dev":"ndbus1" } { "provider":"e820", "dev":"ndbus0", "namespaces":[ { "dev":"namespace0.0", "mode":"fsdax", "size":8589934592, "blockdev":"pmem0" } ] } ``` ## OPTIONS −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −d, −−dimm= An *nmemX* device name, or dimm id number. The dimm id number here is X in *nmemX*. Filter listing by devices that reference the given dimm. For example to see all namespaces comprised of storage capacity on nmem0: ``` # ndctl list −−dimm=nmem0 −−namespaces ``` −n, −−namespace= An *namespaceX.Y* device name, or namespace region plus id tuple *X.Y*. Limit the namespace list to the single identified device if present. −t, −−type= Filter listing by region type (*pmem* or *blk*) −m, −−mode= Filter listing by the mode (*raw*, *fsdax*, *sector* or *devdax*) of the namespace(s). −U, −−numa−node= Filter listing by numa node −B, −−buses Include bus info in the listing −D, −−dimms Include dimm info in the listing ``` { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security:":"disabled" } ``` −H, −−health Include dimm health info in the listing. For example: ``` { "dev":"nmem0", "health":{ "health_state":"non-critical", "temperature_celsius":23, "spares_percentage":75, "alarm_temperature":true, "alarm_spares":true, "temperature_threshold":40, "spares_threshold":5, "life_used_percentage":5, "shutdown_state":"clean" } } ``` −F, −−firmware Include dimm firmware info in the listing. For example: ``` { "dev":"nmem0", "firmware":{ "current_version":0, "next_version":1, "need_powercycle":true } } ``` −X, −−device−dax Include device−dax ("daxregion") details when a namespace is in "devdax" mode. ``` { "dev":"namespace0.0", "mode":"devdax", "size":4225761280, "uuid":"18ae1bbb-bb62-4efc-86df-4a5caacb5dcc", "daxregion":{ "id":0, "size":4225761280, "align":2097152, "devices":[ { "chardev":"dax0.0", "size":4225761280 } ] } } ``` −R, −−regions Include region info in the listing −N, −−namespaces Include namespace info in the listing. Namespace info is listed by default if no other options are specified to the command. −i, −−idle Include idle (not enabled) devices in the listing −C, −−capabilities Include region capabilities in the listing, i.e. supported namespace modes and variable properties like sector sizes and alignments. −M, −−media−errors Include media errors (badblocks) in the listing. Note that the *badblock\_count* property is included in the listing by default when the count is non−zero, otherwise it is hidden. Also, if the namespace is in *sector* mode the *badblocks* listing is not included and *badblock\_count* property may include blocks that are located in metadata, or unused capacity in the namespace. Convert a *sector* namespace into *raw* mode to list precise *badblocks* offsets. ``` { "dev":"namespace7.0", "mode":"raw", "size":33554432, "blockdev":"pmem7", "badblock_count":17, "badblocks":[ { "offset":4, "length":1 }, { "offset":32768, "length":8 }, { "offset":65528, "length":8 } ] } ``` −v, −−verbose Increase verbosity of the output. This can be specified multiple times to be even more verbose on the informational and miscellaneous output, and can be used to override omitted flags for showing specific information. • **−v** In addition to the enabled namespaces default output, show the numa\_node, raw\_uuid, and bad block media errors. • **−vv** Everything *−v* provides, plus automatically enable −−dimms, −−buses, and −−regions. • **−vvv** Everything *−vv* provides, plus −−health, −−capabilities, −−idle, and −−firmware. −u, −−human Format numbers representing storage sizes, or offsets as human readable strings with units instead of the default machine−friendly raw−integer data. Convert other numeric fields into hexadecimal strings. ``` # ndctl list --region=7 { "dev":"region7", "size":67108864, "available_size":67108864, "type":"pmem", "iset_id":-6382611090938810793, "badblock_count":8 } # ndctl list --human --region=7 { "dev":"region7", "size":"64.00 MiB (67.11 MB)", "available_size":"64.00 MiB (67.11 MB)", "type":"pmem", "iset_id":"0xa76c6907811fae57", "badblock_count":8 } ``` ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−create−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace) # ndctl−load−keys(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-load-keys#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-load-keys#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-load-keys#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-load-keys#options)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-load-keys#theory-of-operation)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-load-keys#copyright) ## NAME ndctl−load−keys − load the kek and encrypted passphrases into the keyring ## SYNOPSIS *ndctl load−keys* \[\] ## DESCRIPTION The *load−keys* command loads the master key (*kek*) and the encrypted passphrases for all NVDIMMs into the user keyring maintained by the kernel. The command is expected to be called during initialization and before the libnvdimm kernel module is loaded, typically from an initrd. This is typically set up using a modprobe config that calls the command before module load. **Note**\ All key files are expected to be in the format: nvdimm*\_hostname*\ *The '*' character is used to delimit the different components in the file name. Within the hostname, the '\_' character is allowed since it is the last component of the file name. **Note**\ This command is typically never called directly by a user. ## OPTIONS −p, −−key−path= Path to where key related files reside. This parameter is optional and the default location is /etc/ndctl/keys. −t, −−tpm−handle= Provide a TPM handle (should be a string such as 0x81000001). If the key path (/etc/ndctl/keys) contains a file called tpm.handle which contains the handle string, then this option may be left out, and the tpm handle will be obtained from the file. If both are present, then this option will override (but not overwrite) anything that is in the file. ## THEORY OF OPERATION The Intel Device Specific Methods (DSM) specification v1.7 and v1.8 \[1] introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel−managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key−ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key−encryption−key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* For testing purposes a user key with randomized payload can also be used as a *kek*. See \[2] for details. To perform any security operations, it is expected that the *kek* has been added to the kernel’s user keyring as shown in example below: ``` # keyctl show Session Keyring 736023423 --alswrv 0 0 keyring: _ses 675104189 --alswrv 0 65534 \_ keyring: _uid.0 680187394 --alswrv 0 0 \_ trusted: nvdimm-master ``` Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled. For the *overwrite* operation, in addition to the *regions*, the *dimm* also needs to be disabled. \[1] \ \[2] The following sub−sections describe specifics of each security feature. ### **UNLOCK** Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command (ndctl *load−keys*) is executed before the libnvdimm module is loaded by modprobe. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1\. Check the security state of the device and see if the DIMM is locked 2\. Request the associated encrypted passphrase from the kernel’s user key ring 3\. Use the *kek* to decrypt the passphrase 4\. Create the unlock DSM, copy the decrypted payload into the DSM 5\. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ### **SETUP USER PASSPHRASE** To setup the passphrase for a DIMM, it is expected that the *kek* to be used is present in the kernel’s user keyring. The *kek* encrypts the DIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. ### **UPDATE USER PASSPHRASE** The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update−passphrase*: 1\. Remove the encrypted passphrase from the kernel’s user keyring. 2\. Rename the passphrase blob to old. 3\. Load this old passphrase blob into the keyring with an "old" name. 4\. Create the new passphrase and encrypt with the *kek*. 5\. Send DSM with the old and new decrypted passphrases. 6\. Remove old passphrase and the passphrase blob from the keyring. ### **REMOVE USER PASSPHRASE** The *key−ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ### **CRYPTO (SECURE) ERASE** This operation is similar to remove−passphrase. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use ndctl’s sanitize−dimm command with the −−crypto−erase option to perform this operation. ### **OVERWRITE** This is invoked using −−overwrite option for ndctl *sanitize−dimm*. The overwrite operation wipes the entire NVDIMM. The operation can take a significant amount of time. NOTE: When the command returns successfully, it just means overwrite has been successfully started, and not that the overwrite is complete. Subsequently, 'ndctl wait−overwrite’can be used to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both −−crypto−erase and −−overwrite options are supplied, then crypto−erase is performed before overwrite. ### **SECURITY FREEZE** This operation does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. ### **MASTER PASSPHRASE SETUP, UPDATE, and CRYPTO ERASE** These operations are similar to the user passphrase enable and update. The only difference is that a different passphrase is used. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. # ndctl−monitor(1) [NAME](broken://pages/-LxhySvphPC0V4cs2877#name)\ [SYNOPSIS](broken://pages/-LxhySvphPC0V4cs2877#synopsis)\ [DESCRIPTION](broken://pages/-LxhySvphPC0V4cs2877#description)\ [EXAMPLES](broken://pages/-LxhySvphPC0V4cs2877#EXAMPLES)\ [OPTIONS](broken://pages/-LxhySvphPC0V4cs2877#options)\ [COPYRIGHT](broken://pages/-LxhySvphPC0V4cs2877#copyright)\ [SEE ALSO](broken://pages/-LxhySvphPC0V4cs2877#see-also) ## NAME ndctl−monitor − Monitor the smart events of nvdimm objects ## SYNOPSIS *ndctl monitor* \[\] ## DESCRIPTION `ndctl monitor` is used for monitoring the smart events of nvdimm objects and dumping the json format notifications to syslog, standard output or a logfile. The objects to monitor and smart events to notify can be selected by setting options and/or the configuration file at `/etc/ndctl/monitor.conf`. Both, the values in configuration file and in options will work. If there is a conflict, the values in options will override the values in the configuration file. Any updated values in the configuration file will take effect only after the monitor process is restarted. ## EXAMPLES Run a monitor as a daemon to monitor DIMMs on bus "nfit\_test.1" ``` ndctl monitor −−bus=nfit_test.1 −−daemon ``` Run a monitor as a one−shot command and output the notifications to `/var/log/ndctl.log` ``` ndctl monitor −−log=/var/log/ndctl.log ``` Run a monitor daemon as a system service ``` systemctl start ndctl−monitor.service ``` ## OPTIONS −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −d, −−dimm= A *nmemX* device name, or dimm id number. Select the devices to monitor reference the given dimm. −r, −−region= A *regionX* device name, or a region id number. The keyword `all` can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see `−−bus=` option). −n, −−namespace= A *namespaceX.Y* device name, or namespace region plus id tuple *X.Y*. −l, −−log= Send log messages to the specified destination. * "\": Send log messages to specified . When fopen() is not able to open , log messages will be forwarded to syslog. * "syslog": Send messages to syslog. * "standard": Send messages to standard output. The default log destination is syslog if "−−daemon" is specified, otherwise *standard*. Note that standard and relative path for will not work if "−−daemon" is specified. −c, −−config−file= Provide the config file to use. This overrides the default config typically found in /etc/ndctl −−daemon Run a monitor as a daemon. −D, −−dimm−event= Name of an smart health event from the following: * "dimm−spares−remaining": Spare Blocks Remaining value has gone below the pre−programmed threshold. * "dimm−media−temperature": NVDIMM Media temperature value has gone above the pre−programmed threshold. * "dimm−controller−temperature": NVDIMM Controller temperature value has gone above the pre−programmed threshold. * "dimm−health−state": NVDIMM Normal Health Status has changed * "dimm−unclean−shutdown": NVDIMM Last Shutdown Status was a unclean shutdown. The monitor will attempt to enable the alarm control bits for all specified events. −u, −−human Output monitor notification as human friendly json format instead of the default machine friendly json format. −v, −−verbose Emit extra debug messages to log. ## COPYRIGHT Copyright (c) 2018, FUJITSU LIMITED. License GPLv2: GNU GPL version 2\ . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−list(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list), [ndctl−inject−smart(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-inject-smart) # ndctl−read−labels(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-read-labels#see-also) ## NAME ndctl−read−labels − read out the label area on a dimm or set of dimms ## SYNOPSIS *ndctl read−labels* \ \[\..\] \[\] ## DESCRIPTION The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to resolve aliasing between *pmem* and *blk* capacity by delineating namespace boundaries. This command dumps the raw binary data in a dimm’s label area to stdout or a file. In the multi−dimm case the data is concatenated. ## OPTIONS One or more *nmemX* device names. The keyword *all* can be specified to operate on every dimm in the system, optionally filtered by bus id (see −−bus= option). −s, −−size= Limit the operation to the given number of bytes. A size of 0 indicates to operate over the entire label capacity. −O, −−offset= Begin the operation at the given offset into the label area. −b, −−bus= Limit operation to memory devices (dimms) that are on the given bus. Where *bus* can be a provider name or a bus id number. −v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). −I, −−index Limit the span of the label operation to just the index−block area. This is useful to determine if the dimm label area is initialized. Note that this option and −−size/−−offset are mutually exclusive. −o, −−output output file −j, −−json parse the label data into json assuming the *NVDIMM Namespace Specification* format. −u, −−human enable json output and convert number formats to human readable strings, for example show the size in terms of "KB", "MB", "GB", etc instead of a signed 64−bit numbers per the JSON interchange format (implies −−json). ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO *UEFI NVDIMM Label Protocol* # ndctl−remove−passphrase(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase#options)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase#theory-of-operation)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase#see-also) ## NAME ndctl−remove−passphrase − Stop a DIMM from locking at power−loss and requiring a passphrase to access media ## SYNOPSIS *ndctl remove−passphrase* \ \[\..\] \[\] ## DESCRIPTION Search the user keyring for an encrypted passphrase for the NVDIMM in question. If not found, attempt to load the passphrase blob. After disabling the passphrase, remove the *key−ID* from the keyring as well as the passphrase blob from the file system. ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −v, −−verbose Emit debug messages. ## THEORY OF OPERATION The Intel Device Specific Methods (DSM) specification v1.7 and v1.8 \[1] introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel−managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key−ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key−encryption−key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* For testing purposes a user key with randomized payload can also be used as a *kek*. See \[2] for details. To perform any security operations, it is expected that the *kek* has been added to the kernel’s user keyring as shown in example below: ``` # keyctl show Session Keyring 736023423 --alswrv 0 0 keyring: _ses 675104189 --alswrv 0 65534 \_ keyring: _uid.0 680187394 --alswrv 0 0 \_ trusted: nvdimm-master ``` Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled. For the *overwrite* operation, in addition to the *regions*, the *dimm* also needs to be disabled. \[1] \ \[2] The following sub−sections describe specifics of each security feature. ### **UNLOCK** Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command (ndctl *load−keys*) is executed before the libnvdimm module is loaded by modprobe. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1\. Check the security state of the device and see if the DIMM is locked 2\. Request the associated encrypted passphrase from the kernel’s user key ring 3\. Use the *kek* to decrypt the passphrase 4\. Create the unlock DSM, copy the decrypted payload into the DSM 5\. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ### **SETUP USER PASSPHRASE** To setup the passphrase for a DIMM, it is expected that the *kek* to be used is present in the kernel’s user keyring. The *kek* encrypts the DIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. ### **UPDATE USER PASSPHRASE** The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update−passphrase*: 1\. Remove the encrypted passphrase from the kernel’s user keyring. 2\. Rename the passphrase blob to old. 3\. Load this old passphrase blob into the keyring with an "old" name. 4\. Create the new passphrase and encrypt with the *kek*. 5\. Send DSM with the old and new decrypted passphrases. 6\. Remove old passphrase and the passphrase blob from the keyring. ### **REMOVE USER PASSPHRASE** The *key−ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ### **CRYPTO (SECURE) ERASE** This operation is similar to remove−passphrase. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use ndctl’s sanitize−dimm command with the −−crypto−erase option to perform this operation. ### **OVERWRITE** This is invoked using −−overwrite option for ndctl *sanitize−dimm*. The overwrite operation wipes the entire NVDIMM. The operation can take a significant amount of time. NOTE: When the command returns successfully, it just means overwrite has been successfully started, and not that the overwrite is complete. Subsequently, 'ndctl wait−overwrite’can be used to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both −−crypto−erase and −−overwrite options are supplied, then crypto−erase is performed before overwrite. ### **SECURITY FREEZE** This operation does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. ### **MASTER PASSPHRASE SETUP, UPDATE, and CRYPTO ERASE** These operations are similar to the user passphrase enable and update. The only difference is that a different passphrase is used. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−setup−passphrase(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase), [ndctl−update−passphrase(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase) # ndctl−sanitize−dimm(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm#options)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm#theory-of-operation)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm#see-also) ## NAME ndctl−sanitize−dimm − Perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s) ## SYNOPSIS *ndctl sanitize−dimm* \ \[\..\] \[\] ## DESCRIPTION The *sanitize−dimm* command performs a cryptographic destruction of the contents of the given NVDIMM. It scrambles the data, and any metadata or info−blocks, but it doesn’t modify namespace labels. Therefore, any namespaces on regions associated with the given NVDIMM will be retained, but they will end up in the *raw* mode. Additionally, after completion of this command, the security and passphrase for the given NVDIMM will be disabled, and the passphrase and any key material will also be removed from the keyring and the ndctl keys directory at /etc/ndctl/keys The command supports two different methods of performing the cryptographic erase. The default is *crypto−erase*, but additionally, an *overwrite* option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −c, −−crypto−erase Replace the media encryption key on the NVDIMM causing all existing data to read as cipher text with the new key. This does not change label data. Namespaces get reverted to raw mode. −o, −−ovewrite Wipe the entire DIMM, including label data. This can take significant time, and the command is non−blocking. With this option, the overwrite request is merely submitted to the NVDIMM, and the completion is asynchronous. Depending on the medium and capacity, overwrite may take tens of minutes to many hours. −m, −−master−passphrase Indicate that we are using the master passphrase to perform the erase. This only is applicable to the *crypto−erase* option. −z, −−zero−key Passing in a key with payload that is just 0’s. −−verbose Emit debug messages. ## THEORY OF OPERATION The Intel Device Specific Methods (DSM) specification v1.7 and v1.8 \[1] introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel−managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key−ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key−encryption−key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* For testing purposes a user key with randomized payload can also be used as a *kek*. See \[2] for details. To perform any security operations, it is expected that the *kek* has been added to the kernel’s user keyring as shown in example below: ``` # keyctl show Session Keyring 736023423 --alswrv 0 0 keyring: _ses 675104189 --alswrv 0 65534 \_ keyring: _uid.0 680187394 --alswrv 0 0 \_ trusted: nvdimm-master ``` Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled. For the *overwrite* operation, in addition to the *regions*, the *dimm* also needs to be disabled. \[1] \ \[2] The following sub−sections describe specifics of each security feature. ### **UNLOCK** Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command (ndctl *load−keys*) is executed before the libnvdimm module is loaded by modprobe. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1\. Check the security state of the device and see if the DIMM is locked 2\. Request the associated encrypted passphrase from the kernel’s user key ring 3\. Use the *kek* to decrypt the passphrase 4\. Create the unlock DSM, copy the decrypted payload into the DSM 5\. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ### **SETUP USER PASSPHRASE** To setup the passphrase for a DIMM, it is expected that the *kek* to be used is present in the kernel’s user keyring. The *kek* encrypts the DIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. ### **UPDATE USER PASSPHRASE** The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update−passphrase*: 1\. Remove the encrypted passphrase from the kernel’s user keyring. 2\. Rename the passphrase blob to old. 3\. Load this old passphrase blob into the keyring with an "old" name. 4\. Create the new passphrase and encrypt with the *kek*. 5\. Send DSM with the old and new decrypted passphrases. 6\. Remove old passphrase and the passphrase blob from the keyring. ### **REMOVE USER PASSPHRASE** The *key−ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ### **CRYPTO (SECURE) ERASE** This operation is similar to remove−passphrase. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use ndctl’s sanitize−dimm command with the −−crypto−erase option to perform this operation. ### **OVERWRITE** This is invoked using −−overwrite option for ndctl *sanitize−dimm*. The overwrite operation wipes the entire NVDIMM. The operation can take a significant amount of time. NOTE: When the command returns successfully, it just means overwrite has been successfully started, and not that the overwrite is complete. Subsequently, 'ndctl wait−overwrite’can be used to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both −−crypto−erase and −−overwrite options are supplied, then crypto−erase is performed before overwrite. ### **SECURITY FREEZE** This operation does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. ### **MASTER PASSPHRASE SETUP, UPDATE, and CRYPTO ERASE** These operations are similar to the user passphrase enable and update. The only difference is that a different passphrase is used. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−wait−overwrite(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite), # ndctl−setup−passphrase(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase#options)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase#theory-of-operation)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase#see-also) ## NAME ndctl−setup−passphrase − setup and enable the security passphrase for an NVDIMM ## SYNOPSIS ndctl setup−passphrase \ \[\..\] -k \ \[\] ## DESCRIPTION Setup and enable a security passphrase for one or more NVDIMMs. For this command to succeed, it is expected that the master key has previously been loaded into the user keyring. More information on how this can be done can be found in the kernel documentation at:\ The passphrase blobs are created in the /etc/ndctl/keys directory with a file name format of "nvdimm.blob" The command will fail if the passphrase is already in the user keyring or if a passphrase blob already exists in /etc/ndctl/keys. ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −k, −−key\_handle= Handle for the master *kek* (key−encryption−key) that will be used for sealing the passphrase(s) for the given DIMM(s). The format is: : e.g. trusted:nvdimm−master\ NOTE: The *kek* is expected to have been loaded into the user keyring. −m, −−master−passphrase Indicates that we are managing the master passphrase instead of the user passphrase. −v, −−verbose Emit debug messages. ## THEORY OF OPERATION The Intel Device Specific Methods (DSM) specification v1.7 and v1.8 \[1] introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel−managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key−ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key−encryption−key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* For testing purposes a user key with randomized payload can also be used as a *kek*. See \[2] for details. To perform any security operations, it is expected that the *kek* has been added to the kernel’s user keyring as shown in example below: ``` # keyctl show Session Keyring 736023423 --alswrv 0 0 keyring: _ses 675104189 --alswrv 0 65534 \_ keyring: _uid.0 680187394 --alswrv 0 0 \_ trusted: nvdimm-master ``` Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled. For the *overwrite* operation, in addition to the *regions*, the *dimm* also needs to be disabled. \[1] \ \[2] The following sub−sections describe specifics of each security feature. ### **UNLOCK** Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command (ndctl *load−keys*) is executed before the libnvdimm module is loaded by modprobe. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1\. Check the security state of the device and see if the DIMM is locked 2\. Request the associated encrypted passphrase from the kernel’s user key ring 3\. Use the *kek* to decrypt the passphrase 4\. Create the unlock DSM, copy the decrypted payload into the DSM 5\. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ### **SETUP USER PASSPHRASE** To setup the passphrase for a DIMM, it is expected that the *kek* to be used is present in the kernel’s user keyring. The *kek* encrypts the DIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. ### **UPDATE USER PASSPHRASE** The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update−passphrase*: 1\. Remove the encrypted passphrase from the kernel’s user keyring. 2\. Rename the passphrase blob to old. 3\. Load this old passphrase blob into the keyring with an "old" name. 4\. Create the new passphrase and encrypt with the *kek*. 5\. Send DSM with the old and new decrypted passphrases. 6\. Remove old passphrase and the passphrase blob from the keyring. ### **REMOVE USER PASSPHRASE** The *key−ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ### **CRYPTO (SECURE) ERASE** This operation is similar to remove−passphrase. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use ndctl’s sanitize−dimm command with the −−crypto−erase option to perform this operation. ### **OVERWRITE** This is invoked using −−overwrite option for ndctl *sanitize−dimm*. The overwrite operation wipes the entire NVDIMM. The operation can take a significant amount of time. NOTE: When the command returns successfully, it just means overwrite has been successfully started, and not that the overwrite is complete. Subsequently, 'ndctl wait−overwrite’can be used to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both −−crypto−erase and −−overwrite options are supplied, then crypto−erase is performed before overwrite. ### **SECURITY FREEZE** This operation does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. ### **MASTER PASSPHRASE SETUP, UPDATE, and CRYPTO ERASE** These operations are similar to the user passphrase enable and update. The only difference is that a different passphrase is used. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−update−passphrase(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase), [ndctl−remove−passphrase(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase) # ndctl−start−scrub(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub#description)\ [EXAMPLE](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub#example)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub#see-also) ## NAME ndctl−start−scrub − start an Address Range Scrub (ARS) operation ## SYNOPSIS *ndctl start−scrub* \[\ \ .. \] \[\] ## DESCRIPTION NVDIMM Address Range Scrub is a capability provided by platform firmware that allows for the discovery of memory errors by system software. It enables system software to pre−emptively avoid accesses that could lead to uncorrectable memory error handling events, and it otherwise allows memory errors to be enumerated. The kernel provides a sysfs file (*scrub*) that when written with the string "1\n" initiates an ARS operation. The *ndctl start−scrub* operation starts an ARS, across all specified buses, and the kernel in turn proceeds to scrub every persistent memory address region on the specified buses. ## EXAMPLE Start a scrub on all nvdimm buses in the system. The json listing report only includes the buses that support ARS operations. ``` # ndctl start-scrub [ { "provider":"nfit_test.1", "dev":"ndbus3", "scrub_state":"active" }, { "provider":"nfit_test.0", "dev":"ndbus2", "scrub_state":"active" } ] ``` When specifying an individual bus, or if there is only one bus in the system, the command reports whether ARS support is available. ``` # ndctl start−scrub e820 error starting scrub: Operation not supported ``` ## OPTIONS −v, −−verbose Emit debug messages for the ARS start process ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−wait−scrub(1)](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2), *ACPI* 6.2 Specification Section 9.20.7.2 Address Range Scrubbing (ARS) Overview" # ndctl−update−firmware(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-firmware#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-firmware#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-firmware#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-firmware#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-firmware#copyright) ## NAME ndctl−update−firmware − provides for updating the firmware on an NVDIMM ## SYNOPSIS ndctl update−firmware \ \[\] ## DESCRIPTION Provide a generic interface for updating NVDIMM firmware. The use of this depends on support from the underlying libndctl, kernel, as well as the platform itself. ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −f, −−firmware firmware file used to perform the update −v, −−verbose Emit debug messages for the namespace check process. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. # ndctl-update-passphrase(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase#options)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase#theory-of-operation)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-update-passphrase#see-also) ## NAME ndctl−update−passphrase − update the security passphrase for an NVDIMM ## SYNOPSIS *ndctl update−passphrase* \ \[\..\] \[\] ## DESCRIPTION Update the security passphrase for one or more NVDIMMs. Prerequisites for command to succeed: 1\. The *kek* has been loaded into the kernel’s user keyring. 2\. setup−passphrase has successfully been executed on the NVDIMM. * Alternatively, the NVDIMM is unlocked. The updated key blobs will be created in the /etc/ndctl/keys directory with a file name format of "nvdimm.blob". ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −k, −−key\_handle= Handle for the master *kek* (key−encryption−key) that will be used for sealing the passphrase(s) for the given DIMM(s). The format is: : e.g. trusted:nvdimm−master\ NOTE: The *kek* is expected to have been loaded into the user keyring. −m, −−master−passphrase Indicates that we are managing the master passphrase instead of the user passphrase. −v, −−verbose Emit debug messages. ## THEORY OF OPERATION The Intel Device Specific Methods (DSM) specification v1.7 and v1.8 \[1] introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel−managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key−ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key−encryption−key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* For testing purposes a user key with randomized payload can also be used as a *kek*. See \[2] for details. To perform any security operations, it is expected that the *kek* has been added to the kernel’s user keyring as shown in example below: ``` # keyctl show Session Keyring 736023423 --alswrv 0 0 keyring: _ses 675104189 --alswrv 0 65534 \_ keyring: _uid.0 680187394 --alswrv 0 0 \_ trusted: nvdimm-master ``` Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled. For the *overwrite* operation, in addition to the *regions*, the *dimm* also needs to be disabled. \[1] \ \[2] The following sub−sections describe specifics of each security feature. ### **UNLOCK** Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command (ndctl *load−keys*) is executed before the libnvdimm module is loaded by modprobe. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1\. Check the security state of the device and see if the DIMM is locked 2\. Request the associated encrypted passphrase from the kernel’s user key ring 3\. Use the *kek* to decrypt the passphrase 4\. Create the unlock DSM, copy the decrypted payload into the DSM 5\. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ### **SETUP USER PASSPHRASE** To setup the passphrase for a DIMM, it is expected that the *kek* to be used is present in the kernel’s user keyring. The *kek* encrypts the DIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. ### **UPDATE USER PASSPHRASE** The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update−passphrase*: 1\. Remove the encrypted passphrase from the kernel’s user keyring. 2\. Rename the passphrase blob to old. 3\. Load this old passphrase blob into the keyring with an "old" name. 4\. Create the new passphrase and encrypt with the *kek*. 5\. Send DSM with the old and new decrypted passphrases. 6\. Remove old passphrase and the passphrase blob from the keyring. ### **REMOVE USER PASSPHRASE** The *key−ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ### **CRYPTO (SECURE) ERASE** This operation is similar to remove−passphrase. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use ndctl’s sanitize−dimm command with the −−crypto−erase option to perform this operation. ### **OVERWRITE** This is invoked using −−overwrite option for ndctl *sanitize−dimm*. The overwrite operation wipes the entire NVDIMM. The operation can take a significant amount of time. NOTE: When the command returns successfully, it just means overwrite has been successfully started, and not that the overwrite is complete. Subsequently, 'ndctl wait−overwrite’can be used to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both −−crypto−erase and −−overwrite options are supplied, then crypto−erase is performed before overwrite. ### **SECURITY FREEZE** This operation does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. ### **MASTER PASSPHRASE SETUP, UPDATE, and CRYPTO ERASE** These operations are similar to the user passphrase enable and update. The only difference is that a different passphrase is used. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−setup−passphrase(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-setup-passphrase), [ndctl−remove−passphrase(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-remove-passphrase) # ndctl−wait−overwrite(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite#options)\ [THEORY OF OPERATION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite#theory-of-operation)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-wait-overwrite#see-also) ## NAME ndctl−wait−overwrite − wait for an overwrite operation to complete ## SYNOPSIS *ndctl wait−overwrite* \ \[\..\] \[\] ## DESCRIPTION The kernel provides a POLL(2) capable sysfs file (*security*) to indicate the state of overwrite. This command waits for a change in the state of this file across all specified dimms. ## OPTIONS A *nmemX* device name, or a dimm id number. The keyword *all* can be specified to carry out the operation on every dimm in the system, optionally filtered by bus id (see −−bus= option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. −v, −−verbose Emit debug messages. ## THEORY OF OPERATION The Intel Device Specific Methods (DSM) specification v1.7 and v1.8 \[1] introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel−managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key−ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key−encryption−key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* For testing purposes a user key with randomized payload can also be used as a *kek*. See \[2] for details. To perform any security operations, it is expected that the *kek* has been added to the kernel’s user keyring as shown in example below: ``` # keyctl show Session Keyring 736023423 --alswrv 0 0 keyring: _ses 675104189 --alswrv 0 65534 \_ keyring: _uid.0 680187394 --alswrv 0 0 \_ trusted: nvdimm-master ``` Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled. For the *overwrite* operation, in addition to the *regions*, the *dimm* also needs to be disabled. \[1] \ \[2] The following sub−sections describe specifics of each security feature. ### **UNLOCK** Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command (ndctl *load−keys*) is executed before the libnvdimm module is loaded by modprobe. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1\. Check the security state of the device and see if the DIMM is locked 2\. Request the associated encrypted passphrase from the kernel’s user key ring 3\. Use the *kek* to decrypt the passphrase 4\. Create the unlock DSM, copy the decrypted payload into the DSM 5\. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to revalidate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ### **SETUP USER PASSPHRASE** To setup the passphrase for a DIMM, it is expected that the *kek* to be used is present in the kernel’s user keyring. The *kek* encrypts the DIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. ### **UPDATE USER PASSPHRASE** The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update−passphrase*: 1\. Remove the encrypted passphrase from the kernel’s user keyring. 2\. Rename the passphrase blob to old. 3\. Load this old passphrase blob into the keyring with an "old" name. 4\. Create the new passphrase and encrypt with the *kek*. 5\. Send DSM with the old and new decrypted passphrases. 6\. Remove old passphrase and the passphrase blob from the keyring. ### **REMOVE USER PASSPHRASE** The *key−ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ### **CRYPTO (SECURE) ERASE** This operation is similar to remove−passphrase. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use ndctl’s sanitize−dimm command with the −−crypto−erase option to perform this operation. ### **OVERWRITE** This is invoked using −−overwrite option for ndctl *sanitize−dimm*. The overwrite operation wipes the entire NVDIMM. The operation can take a significant amount of time. NOTE: When the command returns successfully, it just means overwrite has been successfully started, and not that the overwrite is complete. Subsequently, 'ndctl wait−overwrite’can be used to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both −−crypto−erase and −−overwrite options are supplied, then crypto−erase is performed before overwrite. ### **SECURITY FREEZE** This operation does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. ### **MASTER PASSPHRASE SETUP, UPDATE, and CRYPTO ERASE** These operations are similar to the user passphrase enable and update. The only difference is that a different passphrase is used. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−sanitize−dimm(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-sanitize-dimm) # ndctl−wait−scrub(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2#description)\ [EXAMPLE](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2#example)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/untitled-2#see-also) ## NAME ndctl−wait−scrub − wait for an Address Range Scrub (ARS) operation to complete ## SYNOPSIS *ndctl wait−scrub* \[\ \ .. \] \[\] ## DESCRIPTION NVDIMM Address Range Scrub is a capability provided by platform firmware that allows for the discovery of memory errors by system software. It enables system software to pre−emptively avoid accesses that could lead to uncorrectable memory error handling events, and it otherwise allows memory errors to be enumerated. The kernel provides a POLL(2) capable sysfs file (*scrub*) to indicate the state of ARS. The *scrub* file maintains a running count of ARS runs that have taken place. While a current run is in progress a *+* character is emitted along with the current count. The *ndctl wait−scrub* operation waits for *scrub*, across all specified buses, to indicate not in−progress at least once. ## EXAMPLE Wait for scrub on all nvdimm buses in the system. The json listing report at the end only includes the buses that support ARS operations. ``` # ndctl wait-scrub [ { "provider":"nfit_test.1", "dev":"ndbus3", "scrub_state":"active" }, { "provider":"nfit_test.0", "dev":"ndbus2", "scrub_state":"active" } ] ``` When specifying an individual bus, or if there is only one bus in the system, the command reports whether ARS support is available. ``` # ndctl start−scrub e820 error starting scrub: Operation not supported ``` ## OPTIONS −v, −−verbose Emit debug messages for the ARS wait process ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−start−scrub(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-start-scrub), *ACPI* 6.2 Specification Section 9.20.7.2 Address Range Scrubbing (ARS) Overview" # ndctl−write−labels(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-write-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-write-labels#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-write-labels#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-write-labels#options)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-write-labels#see-also) ## NAME ndctl−write−labels − write data to the label area on a dimm ## SYNOPSIS *ndctl write−labels \ \[-i \]* ## DESCRIPTION The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to resolve aliasing between *pmem* and *blk* capacity by delineating namespace boundaries. Read data from the input filename, or stdin, and write it to the given device. Note that the device must not be active in any region, otherwise the kernel will not allow write access to the device’s label data area. ## OPTIONS One or more *nmemX* device names. The keyword *all* can be specified to operate on every dimm in the system, optionally filtered by bus id (see −−bus= option). −s, −−size= Limit the operation to the given number of bytes. A size of 0 indicates to operate over the entire label capacity. −O, −−offset= Begin the operation at the given offset into the label area. −b, −−bus= Limit operation to memory devices (dimms) that are on the given bus. Where *bus* can be a provider name or a bus id number. −v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). −i, −−input input file ## SEE ALSO *UEFI NVDIMM Label Protocol* # ndctl−zero−labels(1) [NAME](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels#description)\ [OPTIONS](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels#options)\ [COPYRIGHT](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-zero-labels#see-also) ## NAME ndctl−zero−labels − zero out the label area on a dimm or set of dimms ## SYNOPSIS *ndctl zero−labels* \ \[\..\] \[\] ## DESCRIPTION The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to resolve aliasing between *pmem* and *blk* capacity by delineating namespace boundaries. This command resets the device to its default state by deleting all labels. ## OPTIONS One or more *nmemX* device names. The keyword *all* can be specified to operate on every dimm in the system, optionally filtered by bus id (see −−bus= option). −s, −−size= Limit the operation to the given number of bytes. A size of 0 indicates to operate over the entire label capacity. −O, −−offset= Begin the operation at the given offset into the label area. −b, −−bus= Limit operation to memory devices (dimms) that are on the given bus. Where *bus* can be a provider name or a bus id number. −v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO *UEFI NVDIMM Label Protocol* # DAXCTL Man Pages This section links to the online man pages: * [daxctl](/ndctl-user-guide/v66/daxctl-man-pages/untitled-1) - Provides enumeration and provisioning commands for the Linux kernel Device−DAX facility * [daxctl-list](/ndctl-user-guide/v66/daxctl-man-pages/untitled-2) - dump the platform Device−DAX regions, devices, and attributes in json * [daxctl-migrate-device-model](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-migrate-device-model) - Opt−in to the /sys/bus/dax device−model, allow for alternative Device−DAX instance drivers * [daxctl-offline-memory](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory) - Offline the memory for a device that is in system−ram mode * [daxctl-online-memory](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3) - Online the memory for a device that is in system−ram mode * [daxctl-reconfigure-device](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device) - Reconfigure a dax device into a different mode # daxctl(1) [NAME](/ndctl-user-guide/v66/daxctl-man-pages/untitled-1#name)\ [SYNOPSIS](/ndctl-user-guide/v66/daxctl-man-pages/untitled-1#synopsis)\ [OPTIONS](/ndctl-user-guide/v66/daxctl-man-pages/untitled-1#options)\ [DESCRIPTION](/ndctl-user-guide/v66/daxctl-man-pages/untitled-1#description)\ [COPYRIGHT](/ndctl-user-guide/v66/daxctl-man-pages/untitled-1#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/daxctl-man-pages/untitled-1#see-also) ## NAME daxctl − Provides enumeration and provisioning commands for the Linux kernel Device−DAX facility ## SYNOPSIS *daxctl* \[−−version] \[−−help] COMMAND \[ARGS] ## OPTIONS −v, −−version Display daxctl version. −h, −−help Run daxctl help command. ## DESCRIPTION The daxctl utility provides enumeration and provisioning commands for the Linux kernel Device−DAX facility. This facility enables DAX mappings of performance / feature differentiated memory without need of a filesystem. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−create−namespace(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-create-namespace), [ndctl−list(1)](/ndctl-user-guide/v66/ndctl-man-pages/ndctl-list) # daxctl-list(1) [NAME](/ndctl-user-guide/v66/daxctl-man-pages/untitled-2#name)\ [SYNOPSIS](/ndctl-user-guide/v66/daxctl-man-pages/untitled-2#synopsis)\ [EXAMPLE](/ndctl-user-guide/v66/daxctl-man-pages/untitled-2#example)\ [OPTIONS](/ndctl-user-guide/v66/daxctl-man-pages/untitled-2#options)\ [COPYRIGHT](/ndctl-user-guide/v66/daxctl-man-pages/untitled-2#copyright) ## NAME daxctl−list − dump the platform Device−DAX regions, devices, and attributes in json. ## SYNOPSIS *daxctl list* \[\] Walk all the device−dax−regions in the system and list all device instances along with some of their major attributes. Options can be specified to limit the output to objects of a certain class. Where the classes are regions or devices. By default, *daxctl list* with no options is equivalent to: daxctl list −−devices ## EXAMPLE ``` # daxctl list --regions --devices { "id":1, "devices":[ { "chardev":"dax1.0", "size":3233808384 } ] } ``` ## OPTIONS −r, −−region= A device−dax region is a contiguous range of memory that hosts one or more /dev/daxX.Y devices, where X is the region id and Y is the device instance id. The keyword *all* can be specified to carry out the operation on every region in the system. −d, −−dev= Specify a dax device name, . tuple, or keyword *all* to filter the listing. For example to list the first device instance in region1: ``` # daxctl list --dev=1.0 { "chardev":"dax1.0", "size":3233808384 } ``` −D, −−devices Include device−dax instance info in the listing (default) −R, −−regions Include region info in the listing −i, −−idle Include idle (not enabled / zero−sized) devices in the listing −u, −−human By default *daxctl list* will output machine−friendly raw−integer data. Instead, with this flag, numbers representing storage size will be formatted as human readable strings with units, other fields are converted to hexadecimal strings. Example: ``` # daxctl list { "chardev":"dax1.0", "size":32828817408 } # daxctl list --human { "chardev":"dax1.0", "size":"30.57 GiB (32.83 GB)" } ``` ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. # daxctl−migrate−device−model(1) [NAME](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-migrate-device-model#name)\ [SYNOPSIS](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-migrate-device-model#synopsis)\ [COPYRIGHT](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-migrate-device-model#copyright) ## NAME daxctl−migrate−device−model − Opt−in to the /sys/bus/dax device−model, allow for alternative Device−DAX instance drivers. ## SYNOPSIS *daxctl migrate−device−model* Arrange for modprobe to disable the dax\_pmem\_compat, if present, and instead deploy the dax\_pmem module to convert to the /sys/bus/dax model. Kernel versions prior to v5.1 may not support /sys/bus/dax in which case the result of this command is a nop until the kernel is updated. The motivation for changing from /sys/class/dax to /sys/bus/dax is to allow for alternative drivers for Device−DAX instances, in particular the dax\_kmem driver. By default device−dax publishes a /dev/daxX.Y character device for userspace to directly map performance differentiated memory. This is fine if the memory is to be exclusively consumed / managed by userspace. Alternatively an administrator may want the kernel to manage the memory, make it available via malloc(), allow for over−provisioning, and / or apply kernel−based resource control schemes to the memory. In that case the memory fronted by a given Device−DAX instance can be assigned to the dax\_kmem driver which arranges for the core−kernel memory−management sub−system to assume management of the memory range. This behavior is opt−in for consideration of existing applications / scripts that may be hard coded to use /sys/class/dax. Fixes have been submitted to applications known to have these direct dependencies *FIO* *PMDK* , however, there may be others and a system−owner should be aware of the potential for regression of Device−DAX consuming scripts, applications, or older daxctl binaries. The modprobe policy established by this utility becomes effective after the next reboot, or after all DAX related modules have been removed and re−loaded with "udevadm trigger" ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. # daxctl−offline−memory(1) [NAME](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory#name)\ [SYNOPSIS](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory#synopsis)\ [EXAMPLES](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory#examples)\ [DESCRIPTION](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory#description)\ [OPTIONS](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory#options)\ [COPYRIGHT](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory#see-also) ## NAME daxctl−offline−memory − Offline the memory for a device that is in system−ram mode ## SYNOPSIS *daxctl offline−memory* \ \[\...\] \[\] ## EXAMPLES • Reconfigure dax0.0 to system−ram mode ``` # daxctl reconfigure-device --mode=system-ram --human dax0.0 { "chardev":"dax0.0", "size":"7.87 GiB (8.45 GB)", "target_node":2, "mode":"system-ram" } ``` • Offline the memory ``` # daxctl offline-memory dax0.0 dax0.0: 62 sections offlined offlined memory for 1 device ``` ## DESCRIPTION Offline the memory sections associated with a device that has been converted to the system−ram mode. If one or more blocks are already offline, attempt to offline the remaining blocks. If all blocks were already offline, print a message and return success without actually doing anything. This is complementary to the *daxctl−online−memory* command, and may be used when it is wished to offline the memory sections, but not convert the device back to *devdax* mode. ## OPTIONS −r, −−region= Restrict the operation to devices belonging to the specified region(s). A device−dax region is a contiguous range of memory that hosts one or more /dev/daxX.Y devices, where X is the region id and Y is the device instance id. −u, −−human By default the command will output machine−friendly raw−integer data. Instead, with this flag, numbers representing storage size will be formatted as human readable strings with units, other fields are converted to hexadecimal strings. −v, −−verbose Emit more debug messages ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [daxctl−reconfigure−device(1)](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device), [daxctl−online−memory(1)](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3) # daxctl−online−memory(1) [NAME](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3#name)\ [SYNOPSIS](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3#synopsis)\ [EXAMPLES](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3#examples)\ [DESCRIPTION](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3#description)\ [OPTIONS](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3#options)\ [COPYRIGHT](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/daxctl-man-pages/untitled-3#see-also) ## NAME daxctl−online−memory − Online the memory for a device that is in system−ram mode ## SYNOPSIS *daxctl online−memory* \ \[\...\] \[\] ## EXAMPLES • Reconfigure dax0.0 to system−ram mode, don’t online the memory ``` # daxctl reconfigure-device --mode=system-ram --no-online --human dax0.0 { "chardev":"dax0.0", "size":"7.87 GiB (8.45 GB)", "target_node":2, "mode":"system-ram" } ``` • Online the memory separately ``` # daxctl online-memory dax0.0 dax0.0: 62 new sections onlined onlined memory for 1 device ``` • Onlining memory when some sections were already online ``` # daxctl online-memory dax0.0 dax0.0: 1 section already online dax0.0: 61 new sections onlined onlined memory for 1 device ``` ## DESCRIPTION Online the memory sections associated with a device that has been converted to the system−ram mode. If one or more blocks are already online, print a message about them, and attempt to online the remaining blocks. This is complementary to the *daxctl−reconfigure−device* command, when used with the *−−no−online* option to skip onlining memory sections immediately after the reconfigure. In these scenarios, the memory can be onlined at a later time using *daxctl−online−memory*. ## OPTIONS −r, −−region= Restrict the operation to devices belonging to the specified region(s). A device−dax region is a contiguous range of memory that hosts one or more /dev/daxX.Y devices, where X is the region id and Y is the device instance id. −u, −−human By default the command will output machine−friendly raw−integer data. Instead, with this flag, numbers representing storage size will be formatted as human readable strings with units, other fields are converted to hexadecimal strings. −v, −−verbose Emit more debug messages ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [daxctl−reconfigure−device(1)](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device), [daxctl−offline−memory(1)](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-offline-memory) # daxctl−reconfigure−device(1) [NAME](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device#name)\ [SYNOPSIS](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device#synopsis)\ [EXAMPLES](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device#examples)\ [DESCRIPTION](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device#description)\ [OPTIONS](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device#options)\ [COPYRIGHT](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device#copyright)\ [SEE ALSO](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-reconfigure-device#see-also) ## NAME daxctl−reconfigure−device − Reconfigure a dax device into a different mode ## SYNOPSIS *daxctl reconfigure−device* \ \[\...\ \[\] ## EXAMPLES • Reconfigure dax0.0 to system−ram mode, don’t online the memory ``` # daxctl reconfigure-device --mode=system-ram --no-online dax0.0 [ { "chardev":"dax0.0", "size":16777216000, "target_node":2, "mode":"system-ram" } ] ``` • Reconfigure dax0.0 to devdax mode, attempt to offline the memory ``` # daxctl reconfigure-device --human --mode=devdax --force dax0.0 { "chardev":"dax0.0", "size":"15.63 GiB (16.78 GB)", "target_node":2, "mode":"devdax" } ``` • Reconfigure all dax devices on region0 to system−ram mode ``` # daxctl reconfigure-device --mode=system-ram --region=0 all [ { "chardev":"dax0.0", "size":16777216000, "target_node":2, "mode":"system-ram" }, { "chardev":"dax0.1", "size":16777216000, "target_node":3, "mode":"system-ram" } ] ``` • Run a process called *some−service* using numactl to restrict its cpu nodes to *0* and *1*, and memory allocations to node 2 (determined using daxctl\_dev\_get\_target\_node() or *daxctl list*) ``` # daxctl reconfigure-device --mode=system-ram dax0.0 [ { "chardev":"dax0.0", "size":16777216000, "target_node":2, "mode":"system-ram" } ] # numactl --cpunodebind=0-1 --membind=2 -- some-service --opt1 --opt2 ``` ## DESCRIPTION Reconfigure the operational mode of a dax device. This can be used to convert a regular *devdax* mode device to the *system−ram* mode which arranges for the dax range to be hot−plugged into the system as regular memory. **Note**\ This is a destructive operation. Any data on the dax device **will** be lost. **Note**\ Device reconfiguration depends on the dax−bus device model. See daxctl−migrate−device−model(1) for more information. If dax−class is in use (via the dax\_pmem\_compat driver), the reconfiguration will fail with an error such as the following: ``` # daxctl reconfigure-device --mode=system-ram --region=0 all libdaxctl: daxctl_dev_disable: dax3.0: error: device model is dax-class dax3.0: disable failed: Operation not supported error reconfiguring devices: Operation not supported reconfigured 0 devices ``` ## OPTIONS −r, −−region= Restrict the operation to devices belonging to the specified region(s). A device−dax region is a contiguous range of memory that hosts one or more /dev/daxX.Y devices, where X is the region id and Y is the device instance id. −m, −−mode= Specify the mode to which the dax device(s) should be reconfigured. • "system−ram": hotplug the device into system memory. • "devdax": switch to the normal "device dax" mode. This requires the kernel to support hot−unplugging *kmem* based memory. If this is not available, a reboot is the only way to switch back to *devdax* mode. −N, −−no−online By default, memory sections provided by system−ram devices will be brought online automatically and immediately with the *online\_movable* policy. Use this option to disable the automatic onlining behavior. NOTE: While this option prevents daxctl from automatically onlining\ the memory sections, there may be other agents, notably system udev\ rules, that online new memory sections as they appear. Coordinating\ with such rules is out of scope of this utility, and the system\ administrator is expected to remove them if they are undesirable.\ If such an agent races to online memory sections, daxctl is prepared\ to lose the race, and not fail the onlining operation as it only\ cares that the memory section was onlined, not that it was the one\ to do so. −f, −−force When converting from "system−ram" mode to "devdax", it is expected that all the memory sections are first made offline. By default, daxctl won’t touch online memory. However with this option, attempt to offline the memory on the NUMA node associated with the dax device before converting it back to "devdax" mode. −u, −−human By default the command will output machine−friendly raw−integer data. Instead, with this flag, numbers representing storage size will be formatted as human readable strings with units, other fields are converted to hexadecimal strings. −v, −−verbose Emit more debug messages ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [daxctl−list(1)](/ndctl-user-guide/v66/daxctl-man-pages/untitled-2), [daxctl−migrate−device−model(1)](/ndctl-user-guide/v66/daxctl-man-pages/daxctl-migrate-device-model) # Troubleshooting The `ndctl` utility is designed to be user friendly and informative. This section describes some of the common messages seen when using `ndctl`. ## "DAX unsupported by block device. Turning off DAX." **Issue:** During system boot or mounting of an EXT4 or XFS filesystem, the following message may be seen in dmesg: ``` [125.755367] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk [125.763736] XFS (pmem0): DAX unsupported by block device. Turning off DAX. ``` The filesystem should still mount, but it will not have the 'dax' (Direct Access) flag set as shown by the mount command: ``` $ mount | grep pmem0 /dev/pmem0 on /mnt/pmem0 type xfs (rw,relatime,attr2,inode64,noquota) ``` **Solution:** Check the namespace mode using `ndctl list -Nu`. In the following example the mode is 'raw': ``` # ndctl list -Nu { "namespaces":[ { "dev":"namespace0.0", "mode":"raw", "size":"704.00 MiB (738.20 MB)", "blockdev":"pmem0", "numa_node":0 } ] } ``` From the `ndctl-create-namespace` man page, 'raw' mode does not support DAX (Direct Access). ``` -m, --mode= · "raw": expose the namespace capacity directly with limitations. Neither a raw pmem namepace nor raw blk namespace support sector atomicity by default (see "sector" mode below). A raw pmem namespace may have limited to no dax support depending the kernel. In other words operations like direct-I/O targeting a dax buffer may fail for a pmem namespace in raw mode or indirect through a page-cache buffer. See "fsdax" and "devdax" mode for dax operation. ``` The namespace needs to be changed to either 'fsdax' or 'devdax' using the following: {% hint style="warning" %} **WARNING:** Changing the namespace mode will destroy any existing data. Backup all data before changing the mode. {% endhint %} ``` $ sudo ndctl create-namespace -f -e namespace0.0 --mode=fsdax { "dev":"namespace0.0", "mode":"memory", "size":"690.00 MiB (723.52 MB)", "uuid":"d1cc8945-7bf7-4b57-83b5-0a79565e9c15", "blockdev":"pmem0", "numa_node":0 } ``` The filesystem needs to be re-created, mounted with the 'dax' option, then any data restored: ``` $ sudo mkfs.xfs /dev/pmem0 $ sudo mount -o dax /dev/pmem0 /mnt/pmem0 $ mount | grep pmem0 /dev/pmem0 on /mnt/pmem0 type xfs (rw,relatime,attr2,dax,inode64,noquota) ``` ## "nmem0 is active, skipping..." **Issue:** When disabling an NVDIMM (nmem) device, a notice is displayed indicating it is active, eg: ``` # ndctl disable-dimm nmem0 nmem0 is active, skipping... disabled 0 nmem ``` **Cause:** The message indicates there's at least one active/enabled Region and/or Namespace using the NVDIMM. **Solution:** All active/enabled Regions and Namespaces must be destroyed an/or disabled prior to disabling the dimm. 1\) List the current configuration: ``` # ndctl list -NRD ``` 2\) Verify no fsdax or devdax namespaces are mounted or in-use by running applications. 3\) Destroy or disable the namespace(s). 4\) Disable the regions used by the NVDIMM (nmem) that needs to be disabled. 5\) Disable the NVDIMM (nmem). ## "ndctl: error while loading shared libraries: libjson-c.so.2: cannot open shared object file: No such file or directory" **Issue:** Executing the `ndctl` utility, with or without commands or options, yields the following missing library error: ``` # ndctl --version ndctl: error while loading shared libraries: libjson-c.so.2: cannot open shared object file: No such file or directory ``` **Cause:** The issue could be caused by one of the following issues: * The json-c package is not installed * A version mis-match between the `json-c` and `ndctl`. More recent versions of json-c deliver `libjson-c.so.4` rather than `libjson-c.so.2`. * The json-c library is installed in a non-default location and the LD\_LIBRARY\_PATH environment variable needs to be updated. Verify which libraries are missing from `ndctl` using the `ldd` utility and identifying libraries that are 'not found': ``` # ldd `which ndctl` linux-vdso.so.1 (0x00007ffc677c4000) libndctl.so.6 => /lib64/libndctl.so.6 (0x00007f31fca49000) libdaxctl.so.1 => /lib64/libdaxctl.so.1 (0x00007f31fc843000) libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f31fc63c000) libjson-c.so.2 => not found libc.so.6 => /lib64/libc.so.6 (0x00007f31fc27d000) libudev.so.1 => /lib64/libudev.so.1 (0x00007f31fc057000) [...] ``` **Solution:** Verify that the json-c package is installed. For example, on Fedora use `dnf info --installed json-c`. If it is installed, information about the package will be displayed, eg: ``` # sudo dnf info --installed json-c Installed Packages Name : json-c Version : 0.13.1 Release : 2.fc28 Arch : x86_64 Size : 65 k Source : json-c-0.13.1-2.fc28.src.rpm Repo : @System From repo : updates Summary : JSON implementation in C URL : https://github.com/json-c/json-c License : MIT Description : JSON-C implements a reference counting object model that allows you : to easily construct JSON objects in C, output them as JSON formatted : strings and parse JSON formatted strings back into the C representation : of JSON objects. It aims to conform to RFC 7159. ``` If the package is not installed, a message similar to the following will be returned: ``` # sudo dnf info --installed json-c Error: No matching Packages to list ``` To install the json-c package if it is missing, use ``` # sudo dnf install json-c ``` To query the json-c package to identify the library version use the following: ``` # dnf repoquery -l json-c | grep libjson-c.so Last metadata expiration check: 2:07:37 ago on Fri 06 Jul 2018 06:18:47 AM MDT. /usr/lib64/libjson-c.so.4 /usr/lib64/libjson-c.so.4.0.0 /usr/lib64/libjson-c.so.4 /usr/lib64/libjson-c.so.4.0.0 /usr/lib/libjson-c.so.4 /usr/lib/libjson-c.so.4.0.0 /usr/lib/libjson-c.so.4 /usr/lib/libjson-c.so.4.0.0 ``` Verify the LD\_LIBRARY\_PATH includes the location of libjson-c.so.\*. Note: `/usr/lib` and `/usr/lib64` are automatically included. ``` # echo $LD_LIBRARY_PATH /usr/local/lib:/usr/local/lib64 ``` If the package and LD\_LIBRARY\_PATH are correct, the version of ndctl will need to be updated. Using `ndctl --version` won't work and will simply return "*ndctl: error while loading shared libraries: libjson-c.so.2: cannot open shared object file: No such file or directory*". If the `ndctl` utility was installed using the ndctl package from the operating system's repository, update the package to the latest version. On Fedora: ``` # sudo dnf update -y ndctl ``` If the latest version within the package repository is old with no new versions available, download, compile, and install from source code. Detailed instructions can be found in the [Installing NDCTL](https://github.com/sscargal/pmem-docs-ndctl-user-guide/tree/bad2d0a9e4528c1cc9d2aa6271f8a427dad45420/getting-started-guide/installing-ndctl.md) chapter. If the `ndctl` utility was previously compiled and installed using source code, download the latest version from the [ndctl GitHub repository](https://github.com/pmem/ndctl), compile, and install. Detailed instructions can be found in the [Installing NDCTL](https://github.com/sscargal/pmem-docs-ndctl-user-guide/tree/bad2d0a9e4528c1cc9d2aa6271f8a427dad45420/getting-started-guide/installing-ndctl.md) chapter. ## Error: namespace0.0 is active, specify --force for re-configuration **Issue:** Destroying a namespace mode may fail with the following error: ``` # ndctl destroy-namespace namespace0.0 Error: namespace0.0 is active, specify --force for re-configuration ``` Changing a namespace mode may fail with the following error: ``` # ndctl create-namespace -e namespace0.0 -m fsdax Error: namespace0.0 is active, specify --force for re-configuration failed to reconfigure namespace: Device or resource busy ``` **Cause:** The error indicates the namespace is currently **active** and potentially mounted (FSDAX) or in use, so the operation is not permitted. **Solution:** 1\) Verify the current namespace mode using `ndctl list -N` to show all namespaces or filter the result by specifying namespace name using `ndctl list -N namespace{X.Y}`, eg: ``` # ndctl list -N { "dev":"namespace0.0", "mode":"raw", "size":134217728000, "uuid":"0a66b46c-5146-45bb-a0f1-320f28db4d40", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` If the namespace mode is 'fsdax', verify any associated filesystems are unmounted. If the namespace mode is 'devdax', verify any application using the device is stopped. {% hint style="info" %} The 'fuser' command on Linux can be used to identify any running processes with the /dev/dax\* or /dev/pmem\* devices open. {% endhint %} 2\) Either disable the namespace prior to deleting it or changing the mode or use the `-f`, `--force` options to override the active status checks. To change the mode of an active namespace without disabling it first, use the `-f`, `--force` option: ``` # ndctl create-namespace -f -e namespace0.0 -m fsdax { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"1cf25e33-8dfa-4ad2-b409-e236d598bc1e", "raw_uuid":"9368c235-26d3-4220-9048-e85f3ccd8469", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` To delete a namespace without disabling it first, use the `-f`, `--force` option: ``` # ndctl destroy-namespace -f namespace0.0 destroyed 1 namespace ``` Alternatively, disable the namespace then perform the destroy/mode change operation. Note: Changing a namespace mode automatically activates it. To change the namespace mode from 'raw' to 'fsdax': ``` # ndctl list -N { "dev":"namespace0.0", "mode":"raw", "size":134217728000, "uuid":"ce15f005-d3bd-4a94-8565-55be77bed6f6", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ndctl disable-namespace namespace0.0 disabled 1 namespace # ndctl create-namespace -e namespace0.0 -m fsdax { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"f45a6725-b80e-4174-a366-b07c91e66a58", "raw_uuid":"ee359fec-0a83-436a-91b6-44e3f45b04e0", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` To destroy a namespace: ``` # ndctl list -N { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":132118478848, "uuid":"f45a6725-b80e-4174-a366-b07c91e66a58", "raw_uuid":"ee359fec-0a83-436a-91b6-44e3f45b04e0", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ndctl disable-namespace namespace0.0 disabled 1 namespace # ndctl destroy-namespace namespace0.0 destroyed 1 namespace ``` ## "failed to create namespace: Resource temporarily unavailable" **Issue:** Creating a new namespace using 'ndctl create-namespace', may return the following error: ``` # ndctl create-namespace -r region0 failed to create namespace: Resource temporarily unavailable ``` **Cause:** There are many potential causes including: * There's no available capacity within the region because one or more namespaces exist and have consumed all the space. * The region is disabled. * There's an issue with the labels for the NVDIMMs belonging to the region. **Solution:** Use the `-v` option to print more information to help identify the cause. A debug version of `ndctl` may be required to get useful information. See [Installing NDCTL](https://github.com/sscargal/pmem-docs-ndctl-user-guide/tree/bad2d0a9e4528c1cc9d2aa6271f8a427dad45420/getting-started-guide/installing-ndctl.md) for instructions to build `ndctl` with debug options from source code. For a scenario where there's no space left within the region, a message similar to the following will be shown: ``` # ndctl create-namespace -r region0 -v ... namespace_create:772: region0: insufficient capacity size: 0 avail: 0 failed to create namespace: Resource temporarily unavailable ``` When a region is disabled, the following will be shown: ``` # ndctl create-namespace -r region0 -v ... validate_namespace_options:473: region0: disabled, skipping... failed to create namespace: Resource temporarily unavailable ``` Identifying label issues requires more investigation. Start with [Validating Labels](/ndctl-user-guide/v66/managing-label-storage-areas-lsa#checking-labels). The `Resource temporarily unavailable` message improvements is being handled by ## "man -k ndctl" returns "ndctl: nothing appropriate." **Issue:** When searching the short descriptions and manual page for a *keyword*, the man utility may return an empty set, for example: ``` # man -k ndctl ndctl: nothing appropriate. ``` **Cause:** If you built ndctl from source, the man page indexes may not have been automatically generated for you. Additionally, if you installed ndctl to a non-default location, your $MANPATH shell environment variable may not been updated to point to to the new man page locations. **Solution:** 1\) Verify the current system global search paths for manual pages ``` # manpath -g /usr/man:/usr/share/man:/usr/local/man:/usr/local/share/man:/usr/X11R6/man:/opt/man ``` 2\) Identify if these paths are overridden by a local MANPATH setting, for example the following shows no additional search paths ``` # echo $MANPATH # ``` 3\) If the install location of ndctl is not included in the above output, add the path to your MANPATH shell environment, otherwise skip this step: ``` # export MANPATH=$MANPATH: ``` 4\) Manually scan and build the man page indexes ``` # sudo mandb -c ``` This command may take a few minutes depending on the number of search paths and man pages it has to index. The output is verbose as it scans and indexes. Once complete, it will provide a summary: ``` ... 118 man subdirectories contained newer manual pages. 10550 manual pages were added. 0 stray cats were added. ``` 5\) If the ndctl man pages were scanned and indexed, `man -k ndctl` will now work as expected, eg: ``` # man -k ndctl ndctl (1) - Manage "libnvdimm" subsystem devices (Non-volatile Memory) ndctl-check-labels (1) - determine if the given dimms have a valid namespace index block ndctl-check-namespace (1) - check namespace metadata consistency ndctl-clear-errors (1) - clear all errors (badblocks) on the given namespace ... ``` # Glossary **BLK:** Block Mode: A set of one or more programmable memory mapped apertures provided by an NVDIMM to access its media. This indirection precludes the performance benefit of interleaving, but enables NVDIMM-bounded failure modes. **BTT:** Block Translation Table: Persistent memory is byte addressable. Existing software may have an expectation that the power-fail-atomicity of writes is at least one sector, 512 bytes. The BTT is an indirection table with atomic update semantics to front a PMEM/BLK block device driver and present arbitrary atomic sector sizes. **DAX:** Direct Access File system extensions to bypass the page cache and block layer to mmap persistent memory, from a PMEM block device, directly into a process address space. **DCR:** NVDIMM Control Region Structure: It defines a vendor-id, device-id, and interface format for a given DIMM. See [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) Section 5.2.25.5. **DPA:** DIMM Physical Address: is an NVDIMM-relative offset. With one NVDIMM in the system there would be a 1:1 system-physical-address:DPA association. Once more NVDIMMs are added a memory controller interleave must be decoded to determine the DPA associated with a given system-physical-address. BLK capacity always has a 1:1 relationship with a single-NVDIMM's DPA range. **DSM:** Device Specific Method: ACPI method to to control specific device - in this case the firmware. **LABEL:** Metadata stored on an NVDIMM device that partitions and identifies (persistently names) storage between PMEM and BLK. It also partitions BLK storage to host BTTs with different parameters per BLK-partition. Note that traditional partition tables, GPT/MBR, are layered on top of a BLK or PMEM device. **NVDIMM:** A Non-Volatile DIMM device (also called a module) installed in to a memory socket on the system board. Data is written to the device and stored persistently, meaning data is retained across power-cycles. **PMEM:** Persistent Memory Mode: A system-physical-address range where writes are persistent. A block device composed of PMEM is capable of DAX. A PMEM address range may span an interleave of several NVDIMMs. # Reference Material The following is a list of reference material: ## Documents See the latest documentation for the NVDIMM kernel sub-system here: ACPI v6.0: [http://www.uefi.org/sites/default/files/resources/ACPI\_6.0.pdf ](http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf) ACPI Specification (Latest): NVDIMM DSM Interface Specification (v1.8): NVDIMM DSM Interface Example: [http://pmem.io/documents/NVDIMM\_DSM\_Interface\_Example.pdf ](http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf) NVDIMM Driver Writer's Guide: NVDIMM Namespace Specification: [http://pmem.io/documents/NVDIMM\_Namespace\_Spec.pdf ](http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf) UEFI Specification (Latest): ## Source Code Repositories LIBNVDIMM: [https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git ](https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git) LIBNDCTL: [https://github.com/pmem/ndctl.git ](https://github.com/pmem/ndctl.git) NDCTL: Persistent Memory Development Kit (PMDK): # NDCTL User Guide ## Introduction `ndctl` is a utility for managing the Linux LIBNVDIMM Kernel subsystem. It is designed to work with various non-volatile memory devices (NVDIMMs) from different vendors. The LIBNVDIMM subsystem defines a kernel device model and control message interface for platform NVDIMM resources like those defined by the [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) NFIT (**N**VDIMM **F**irmware **I**nterface **T**able). The latest [ACPI ](http://www.uefi.org/specifications)and [UEFI ](http://www.uefi.org/specifications)specifications can be found at [uefi.org](http://www.uefi.org). Operations supported by `ndctl` include: * Provisioning capacity (namespaces) * Enumerating Devices * Enabling and Disabling NVDIMMs, Regions, and Namespaces * Managing NVDIMM Labels ## What's new in v67 This release incorporates functionality up to the 5.4 kernel, and adds a number of bug fixes, and improvements. Highlights include small changes for PowerPC compatibility, improvements to the dax.sh unit test to detect failures in mapping huge pages, support for the 'security frozen' attribute, user experience improvements for the daxctl-reconfigure-device command, including an option to specify movable vs. non-movable state for onlining memory, and an option to allow create-namespaces to create a maximal configuration until it exhausts all available region capacity. Commands: * [create-namespace](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-create-namespace): add --continue option * [daxctl-reconfigure-device](/ndctl-user-guide/v67/daxctl-man-pages/daxctl-reconfigure-device): add --no-movable option * [daxctl-reconfigure-device](/ndctl-user-guide/v67/daxctl-man-pages/daxctl-reconfigure-device): display movable state in listings * [daxctl-reconfigure-device](/ndctl-user-guide/v67/daxctl-man-pages/daxctl-reconfigure-device): detect races in memory onlining * [ndctl-freeze-security](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-freeze-security): support for 'security frozen' attribute APIs: * ndctl\_dimm\_security\_is\_frozen * daxctl\_memory\_is\_movable * daxctl\_memory\_online\_no\_movable # Installing NDCTL & DAXCTL The `ndctl` and `daxctl` utilities are used to manage the libnvdimm (non-volatile memory device) sub-system in the Linux Kernel. ndctl and daxctl are used to manage persistent memory devices and namespaces and they are required for certain [Persistent Memory Development Kit (PMDK)](https://pmem.io/pmdk) features. {% hint style="info" %} Microsoft Windows users should visit the [Understand and Deploy Persistent Memory](https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/deploy-pmem) documentation. {% endhint %} The `ndctl` and `daxctl` utilities can be installed on Linux using either of the two options: * [Install ndctl using packages](/ndctl-user-guide/v67/installing-ndctl/installing-ndctl-packages-on-linux) * [Install ndctl using the source code](/ndctl-user-guide/v67/installing-ndctl/installing-ndctl-from-source-on-linux) # Installing NDCTL and DAXCTL Packages on Linux The `ndctl` and `daxctl` utilities are available in many Linux distribution package repositories. This approach is the easiest to use and maintain, compared with [installing ndctl from source code](/ndctl-user-guide/v67/installing-ndctl/installing-ndctl-from-source-on-linux). The daxctl utility is bundled with the ndctl packages. {% tabs %} {% tab title="Fedora" %} 1\) Query the repository to confirm if the ndctl package is available: Fedora 21 or earlier ``` yum search ndctl ``` Fedora 22 or later ``` dnf repoquery ndctl ``` 2\) Install the ndctl utility Fedora 21 or earlier ``` yum install ndctl ``` Fedora 22 or later ``` dnf install ndctl ``` {% endtab %} {% tab title="RHEL & CentOS" %} The ndctl package is available on CentOS, RHEL, RHEL for SAP HANA 7.0 or later. 1\) Query the repository to identify if the ndctl package is available: ``` yum search ndctl ``` 2\) Install the ndctl package ``` yum install ndctl ``` {% endtab %} {% tab title="SLES & OpenSUSE" %} The ndctl package is available for OpenSUSE, SUSE Linux Enterprise Server (SLES), and SLES for SAP HANA v12.4 or later. 1\) Query the repository to identify if ndctl is delivered: ``` sudo zypper search ndctl ``` Example: ``` $ sudo zypper search ndctl S | Name | Summary | Type --+-----------+----------------------------------------------------------------------------+-------- | libndctl6 | Management library for "libnvdimm" subsystem devices (Non-volatile Memory) | package | ndctl | Manage "libnvdimm" subsystem devices (Non-volatile Memory) | package ``` 2\) Install the ndctl package ``` sudo zypper install ndctl ``` {% endtab %} {% tab title="Ubuntu" %} The ndctl package is available on Ubuntu 18.10 (Cosmic Cuttlefish) or later. 1\) Query the repository to identify if the ndctl package is available using either the aptitude, apt-cache, or apt utilities ``` aptitude search ndctl apt-cache search ndctl apt search ndctl ``` 2\) Verify if the ndctl package is currently installed and check the version ``` apt list --installed ndctl ``` 3\) Install the ndctl package or update an installed package ``` sudo apt-get install ndctl ``` {% endtab %} {% tab title="Debian" %} {% hint style="info" %} The ndctl package is available on Debian 10 (Buster) or later. See for up to date information. {% endhint %} 1\) Query available versions in configured repos: ``` apt policy ndctl ``` 2\) Query the repository to identify if ndctl is available using either the aptitude, apt-cache, or apt utilities ``` apt search ndctl ``` 3\) Verify if the ndctl package is currently installed and check the version ``` apt list --installed ndctl ``` 4\) Install the ndctl package or update an installed package ``` sudo apt install ndctl ``` {% endtab %} {% endtabs %} # Installing NDCTL and DAXCTL from Source on Linux These instructions provide a step-by-step guide for installing the `ndctl` and `daxctl` utilities from the GitHub project [master branch](https://github.com/pmem/ndctl). ### 1. Install the Prerequisites There are a number of packages required for the build steps that may not be installed by default. For information about the required packages, see the "BuildRequires:" lines in ndctl.spec.in. To successfully compile ndctl and daxctl from source with documentation, the following packages are required * autoconf * automake * asciidoc (asciidoctor) * bash-completion * doxygen * gcc * gcc-c++ (g++) * git * glib2 * glib2-devel * graphviz * json-c-devel * keyutils-libs-devel (libkeyutils-dev) * kmod * kmod-devel * libfabric * libfabric-devel * libtool * libudev-devel * libuuid-devel * ncurses * pandoc * pkg-config * rubygem-asciidoctor (asciidoctor) * xmlto To install these prerequisites, use: {% tabs %} {% tab title="Fedora" %} Fedora 21 or earlier ``` sudo yum install git gcc gcc-c++ autoconf automake asciidoc asciidoctor xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel ``` Fedora 22 or later ``` sudo dnf install git gcc gcc-c++ autoconf automake asciidoc asciidoctor xmlto libtool pkg-config glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel ``` {% endtab %} {% tab title="RHEL & CentOS" %} These instructions apply to RHEL, CentOS, and RHEL for SAP HANA v7.6 or later. Some of the required packages can be found in the EPEL repository (Extra Packages for Enterprise Linux). Verify the EPEL repository is available and active: ``` yum repolist ``` Example: ``` $ yum repolist repo id repo name status epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 13,217 ``` If the EPEL repository is not listed, install and activate it using: ``` sudo yum install epel-release ``` Install the required packages ``` sudo yum install git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkgconfig glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel rubygem-asciidoctor keyutils-libs-devel ``` {% endtab %} {% tab title="Ubuntu & Debian" %} **For Ununtu 18.04 (Bionic) and Debian 9 (Stretch) or later:** ``` sudo apt install -y git gcc g++ autoconf automake asciidoc asciidoctor bash-completion xmlto libtool pkg-config libglib2.0-0 libglib2.0-dev libfabric1 libfabric-dev doxygen graphviz pandoc libncurses5 libkmod2 libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev ``` **For Ubuntu 16.04 (Xenial) and Debian 8 (Jessie):** {% hint style="info" %} Earlier releases of Ubuntu and Debian do not have libfabric1 or libfabric-dev available in the repository. If these libraries are required, you should compile them yourself. See {% endhint %} ``` sudo apt-get install -y git gcc g++ autoconf automake asciidoc asciidoctor bash-completion xmlto libtool pkg-config libglib2.0-0 libglib2.0-dev doxygen graphviz pandoc libncurses5 libkmod2 libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev ``` {% endtab %} {% endtabs %} ### 2. Clone the GitHub Repository 2.1) If you're behind a company proxy, configure git to work with your proxy server first. The following configures a HTTP and HTTPS proxy for all users. Refer to the [git-config documentation](https://git-scm.com/docs/git-config) for more options and information. ``` git config --global http.proxy http://proxyUsername:proxyPassword@proxy.server.com:port git config --global https.proxy https://proxyUsername:proxyPassword@proxy.server.com:port ``` 2.2) Create a working directory to clone the ndctl GitHub repository to, eg: 'downloads' ``` mkdir ~/downloads ``` 2.3) Clone the repository: ``` cd ~/downloads git clone https://github.com/pmem/ndctl cd ndctl ``` ### 3. Build The following configures ndctl to be installed in to the /usr/local directory. ``` ./autogen.sh ./configure CFLAGS='-g -O2' --prefix=/usr/local --sysconfdir=/etc --libdir=/usr/local/lib64 make ``` #### **Build using an alternative compiler** **Note:** If you want to compile with a different compiler other than gcc, you have to provide the CC and CXX environment variables. For example: ``` make CC=clang CXX=clang++ ``` These variables are independent and setting CC=clang does not set CXX=clang++. #### Build a Debug Version To compile ndctl with debugging, use the `--enable-debug` option: ``` ./autogen.sh ./configure CFLAGS='-g -O2' --enable-debug --prefix=/usr/local --sysconfdir=/etc --libdir=/usr/local/lib64 make ``` For a full list of configure options use: ``` ./configure --help ``` ### 4. Install Once ndctl has successfully been compiled, it can be installed using the following: ``` sudo make install ``` ### 5. Build and Run Unit Tests (Optional) The unit tests run by `make check` require the nfit\_test.ko module to be loaded. To build and install nfit\_test.ko: 1. Obtain the kernel source. For example, `git clone -b libnvdimm-for-next git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git` 2. Skip to step 3 if the kernel version is >= v4.8. Otherwise, for kernel versions < v4.8, configure the kernel to make some memory available to CMA (contiguous memory allocator). This will be used to emulate DAX. `CONFIG_DMA_CMA=y` `CONFIG_CMA_SIZE_MBYTES=200` **or** `cma=200M` on the kernel command line. 3. Compile the libnvdimm sub-system as a module, make sure "zone device" memory is enabled, and enable the btt, pfn, and dax features of the sub-system: `CONFIG_X86_PMEM_LEGACY=m` `CONFIG_ZONE_DEVICE=y` `CONFIG_LIBNVDIMM=m` `CONFIG_BLK_DEV_PMEM=m` `CONFIG_ND_BLK=m` `CONFIG_BTT=y` `CONFIG_NVDIMM_PFN=y` `CONFIG_NVDIMM_DAX=y` `CONFIG_DEV_DAX_PMEM=m` 4. Build and install the unit test enabled libnvdimm modules in the following order. The unit test modules need to be in place prior to the `depmod` that runs during the final `modules_install` `make M=tools/testing/nvdimm` `sudo make M=tools/testing/nvdimm modules_install` `sudo make modules_install` 5. Now run `make check` in the ndctl source directory, or `ndctl test`, if ndctl was built with `--enable-test`. ### Troubleshooting The unit tests will validate that the environment is set up correctly before they try to run. If the platform is misconfigured, i.e. the unit test modules are not available, or the test versions of the modules are superseded by the "in-tree/production" version of the modules `make check` will skip tests and report a message like the following in test/test-suite.log: ``` SKIP: libndctl ============== test/init: nfit_test_init: nfit.ko: appears to be production version: /lib/modules/4.8.8-200.fc24.x86_64/kernel/drivers/acpi/nfit/nfit.ko.xz __ndctl_test_skip: explicit skip test_libndctl:2684 nfit_test unavailable skipping tests ``` If the unit test modules are indeed available in the modules 'extra' directory the default depmod policy can be overridden by adding a file to /etc/depmod.d with the following contents: ``` override nfit * extra override device_dax * extra override dax_pmem * extra override libnvdimm * extra override nd_blk * extra override nd_btt * extra override nd_e820 * extra override nd_pmem * extra ``` The nfit\_test module emulates pmem with memory allocated via vmalloc(). One of the side effects is that this breaks 'physically contiguous' assumptions in the driver. Use the `--align=4K` option to `ndctl create-namespace` to avoid these corner case scenarios. # Quick Start The `ndctl` command is designed to be user friendly. Once [installed](/ndctl-user-guide/v67/installing-ndctl), a list of commands can be shown using any of the following: 1\) With no arguments or options, `ndctl` and `daxctl` show a simple usage message: ``` # ndctl usage: ndctl [--version] [--help] COMMAND [ARGS] See 'ndctl help COMMAND' for more information on a specific command. ndctl --list-cmds to see all available commands ``` ``` # daxctl usage: daxctl [--version] [--help] COMMAND [ARGS] See 'daxctl help COMMAND' for more information on a specific command. daxctl --list-cmds to see all available commands ``` 2\) Using `ndctl help` and `daxctl help` displays basic help and syntax: ``` # ndctl help usage: ndctl [--version] [--help] COMMAND [ARGS] See 'ndctl help COMMAND' for more information on a specific command. ndctl --list-cmds to see all available commands ``` ``` # daxctl help usage: daxctl [--version] [--help] COMMAND [ARGS] See 'daxctl help COMMAND' for more information on a specific command. daxctl --list-cmds to see all available commands ``` Below is an example of using the `ndctl help` command to launch the `create-namespace` man page: ``` # ndctl help create-namespace ``` 3\) Using `ndctl --list-cmds` and `daxctl --list-cmds`lists all commands as a single list. ``` # ndctl --list-cmds version enable-namespace disable-namespace create-namespace destroy-namespace check-namespace clear-errors enable-region disable-region enable-dimm disable-dimm zero-labels read-labels write-labels init-labels check-labels inject-error update-firmware inject-smart wait-scrub start-scrub setup-passphrase update-passphrase remove-passphrase freeze-security sanitize-dimm load-keys wait-overwrite list monitor help ``` ``` # daxctl --list-cmds version list help migrate-device-model reconfigure-device online-memory offline-memory ``` An alternative method for listing commands uses the TAB key completion feature of `ndctl` and `daxctl`. By typing`ndctl ` or `daxctl ` we can list the available commands, eg: ``` # ndctl check-labels freeze-security setup-passphrase check-namespace help start-scrub clear-errors init-labels update-firmware create-namespace inject-error update-passphrase destroy-namespace inject-smart version disable-dimm list wait-overwrite disable-namespace load-keys wait-scrub disable-region monitor write-labels enable-dimm read-labels zero-labels enable-namespace remove-passphrase enable-region sanitize-dimm ``` ``` # daxctl version list help migrate-device-model reconfigure-device online-memory offline-memory ``` ## TAB Command and Argument Completion `ndctl` and `daxctl`supports command completion using the TAB key. For example, typing `ndctl enable-` lists all commands beginning with 'enable', eg: ``` # ndctl enable- enable-dimm enable-namespace enable-region ``` TAB completion also works with command arguments. For example, typing `ndctl enable-dimm ` will show all available command arguments. For example, the 'enable-dimm' command can enable one, more than one, or all NVDIMMs. It will list all available NVDIMMs (nmem) devices when using the TAB completion, eg: ``` # ndctl enable-dimm all nmem0 ``` ## Getting Help NDCTL ships with a man page for each command. Each man page describes the required arguments and features in detail. Man pages can be found and accessed using the `man` or `ndctl` utilities. The following `man -k ndctl` searches for any man page containing the "ndctl" keyword: ``` # man -k ndctl ndctl (1) - Manage "libnvdimm" subsystem devices (Non-volatile Memory) ndctl-check-labels (1) - determine if the given dimms have a valid namespace index block ndctl-check-namespace (1) - check namespace metadata consistency ndctl-clear-errors (1) - clear all errors (badblocks) on the given namespace ndctl-create-namespace (1) - provision or reconfigure a namespace ndctl-destroy-namespace (1) - destroy the given namespace(s) ndctl-disable-dimm (1) - disable one or more idle dimms ndctl-disable-namespace (1) - disable the given namespace(s) ndctl-disable-region (1) - disable the given region(s) and all descendant namespaces ndctl-enable-dimm (1) - enable one more dimms ndctl-enable-namespace (1) - enable the given namespace(s) ndctl-enable-region (1) - enable the given region(s) and all descendant namespaces ndctl-freeze-security (1) - Set the given DIMM(s) to reject future security operations ndctl-init-labels (1) - initialize the label data area on a dimm or set of dimms ndctl-inject-error (1) - inject media errors at a namespace offset ndctl-inject-smart (1) - perform smart threshold/injection operations on a DIMM ndctl-list (1) - dump the platform nvdimm device topology and attributes in json ndctl-load-keys (1) - load the kek and encrypted passphrases into the keyring ndctl-monitor (1) - Monitor the smart events of nvdimm objects ndctl-read-labels (1) - read out the label area on a dimm or set of dimms ndctl-remove-passphrase (1) - Stop a DIMM from locking at power-loss and requiring a passphrase to access media ndctl-sanitize-dimm (1) - Perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s) ndctl-setup-passphrase (1) - setup and enable the security passphrase for an NVDIMM ndctl-start-scrub (1) - start an Address Range Scrub (ARS) operation ndctl-update-firmware (1) - provides for updating the firmware on an NVDIMM ndctl-update-passphrase (1) - update the security passphrase for an NVDIMM ndctl-wait-overwrite (1) - wait for an overwrite operation to complete ndctl-wait-scrub (1) - wait for an Address Range Scrub (ARS) operation to complete ndctl-write-labels (1) - write data to the label area on a dimm ndctl-zero-labels (1) - zero out the label area on a dimm or set of dimms ``` ``` # man -k daxctl daxctl (1) - Provides enumeration and provisioning commands for the Linux kernel Device-DAX facility daxctl-list (1) - dump the platform Device-DAX regions, devices, and attributes in json. daxctl-migrate-device-model (1) - Opt-in to the /sys/bus/dax device-model, allow for alternative Device-DAX instance drivers. daxctl-offline-memory (1) - Offline the memory for a device that is in system-ram mode daxctl-online-memory (1) - Online the memory for a device that is in system-ram mode daxctl-reconfigure-device (1) - Reconfigure a dax device into a different mode ``` {% hint style="info" %} Note: If `man -k ndctl` returns "ndctl: nothing appropriate." or similar, see the [Troubleshooting](/ndctl-user-guide/v67/troubleshooting) section to manually build the indexes. {% endhint %} Additionally, executing `ndctl help ` can be used to display the man page for the command, eg: ``` # ndctl help enable-dimm ``` A list of ndctl and daxctl man pages are available online. See [NDCTL Man Pages](/ndctl-user-guide/v67/ndctl-man-pages) and [DAXCTL Man Pages](/ndctl-user-guide/v67/daxctl-man-pages) for a complete list. ## Displaying Bus, NVDIMM, Region, and Namespace Information The `ndctl list` command is a very powerful and feature rich command. A list of options is shown below: ``` # ndctl list -? Error: unknown switch `?' usage: ndctl list [] -b, --bus filter by bus -r, --region filter by region -d, --dimm filter by dimm -n, --namespace filter by namespace id -m, --mode filter by namespace mode -t, --type filter by region-type -U, --numa-node filter by numa node -B, --buses include bus info -D, --dimms include dimm info -F, --firmware include firmware info -H, --health include dimm health -R, --regions include region info -N, --namespaces include namespace info (default) -X, --device-dax include device-dax info -C, --capabilities include region capability info -i, --idle include idle devices -M, --media-errors include media errors -u, --human use human friendly number formats -v, --verbose increase output detail ``` Using the filters is a powerful way to limit the output. ### Examples To list all active/enabled namespaces: ``` # ndctl list -N ``` To list all active/enabled regions: ``` # ndctl list -R ``` To list all active/enabled NVDIMMs: ``` # ndctl list -D ``` To list all active/enabled NVDIMMs, Regions, and Namespaces: ``` # ndctl list -DRN ``` To list all active/enabled and disabled/inactive (idle) NVDIMMs, Regions, and Namespaces: ``` # ndctl list -DRNi ``` To list all active/enabled and disabled/inactive (idle) NVDIMMs, Regions, and Namespaces with human readable values: ``` # ndctl list -iNuRD ``` ## Increasing output verbosity The `-v, --verbose` option increases the output verbosity of the command. Using `-vv` or -`-vvv` further increases the output and verbosity. # Concepts ## Persistent Memory Concepts This section describes the basic concepts when configuring and managing NVDIMMs. The following describes terms used throughout this chapter. **Interleave Sets:** two or more NVDIMMs create an N-Way interleave set to provide striped read/write operations for increased throughput. Commonly 2-Way or 4-Way interleaving. **Region:** a grouping of one or more NVDIMMs, or an interleaved set, that can be divided up into one or more Namespaces. Regions are created within interleaved sets. **Label Storage Area (LSA):** Namespaces are defined by Labels which are stored in the Label Storage Area(s). **Namespace:** defines a contiguously-addressed range of Non-Volatile Memory conceptually similar to a hard disk partition, SCSI Logical Unit (LUN), or an NVM Express namespace. It is the unit of persistent memory storage that appears in /dev as a device usable for I/O. **Type:** defines the way in which the persistent memory associated with a Namespace or Region can be accessed. Valid Types are PMEM and BLK: * **PMEM:** Direct access to the media via load/store operations. DAX Supported. * **BLK:** Direct access to the media via Apertures (sliding memory windows). DAX is not supported. **Mode:** defines which NVDIMM software features are enabled for a given Namespace. Namespace Modes include raw, sector, fsdax(memory), and devdax(dax). Sibling Namespaces of the same parent Region will always have the same Type, but might be configured to have different Modes. ## Configuration Options This section discusses some of the common configuration options but does not cover all possible supported config. Using one or more physical NVDIMMs, the available capacity can be configure in many different ways. Figure 1 below shows a typical Filesystem-DAX (FSDAX) configuration with three physical NVDIMMs that have been interleaved. All available capacity has been used to create a single region (Region0), namespace (Namespace0.0), and dax enabled filesystem created. Namespace naming convention is commonly X.Y where X is the region number and Y is the namespace. The persistent memory pool(s) are memory mapped to the application, thus allowing direct load/store access to the NVDIMMs. ![](https://1414645753-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlBtYw13PknzW_5Tx%2Fdraw.io-gitbook-interleaved-dimms-fsdax.jpg?alt=media\&token=5f2e188f-5683-4441-8e5d-2e2c75310589) Figure 2 shows a configuration where the region has been partitioned into two namespaces. Each namespace has a single DAX Filesystem. ![](https://1414645753-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlFmTfOvpD8GUY9eA%2Fdraw.io-gitbook-interleaved-dimms-with-2-namespaces-fsdax%20\(1\).jpg?alt=media\&token=05d9d599-9773-470b-ba61-4bac6491b4d5) Figure 3 shows a configuration where NVDIMM interleaving has been disabled. In this mode, each NVDIMM becomes addressable with single regions per device. Regions and Namespaces cannot be striped or interleaved across NVDIMMs. It is possible to use software RAID such as Linux's Device Mapper to implement mirroring, striping, or concatenation of the /dev/pmem{N} devices. ![](https://1414645753-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlSA1qHvOy6St0XnM%2Fdraw.io-gitbook-non-interleaved-dimm-fsdax.jpg?alt=media\&token=2ca8c490-07d3-4fd0-84c6-49cf2f5d3457) The following sections describe each layer in more detail. # NVDIMM Devices A generic DIMM device object, named /dev/nmemX, is registered for each physical memory device indicated in the ACPI NFIT table, or other platform NVDIMM resource discovery mechanism. Physical NVDIMMs may have configuration options available via the BIOS. The features and available configuration options will be dependant on the NVDIMM device and BIOS. The Linux LIBNVDIMM core provides a built-in driver for these DIMM devices. The driver is responsible for determining if the DIMM implements a namespace label area, and initializing the kernel's in-memory copy of that label data. The kernel performs runtime modifications of that data when namespace provisioning actions are taken, and actively blocks user-space from initiating label data changes while the DIMM is active in any region. Disabling a DIMM, after all the regions it is a member of have been disabled, allows userspace to manually update the label data to be consumed when the DIMM is next enabled. Refer to [Managing NVDIMMs](/ndctl-user-guide/v67/managing-nvdimms) for more information on administering NVDIMMs. # PMEM and BLK Modes The NFIT (**N**VDIMM **F**irmware **I**nterface **T**able) specification defined by the [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) standardizes not only the description of Persistent Memory (PMEM) and Block (BLK) modes, but also the platform message-passing entry points for control and configuration. ## LIBNVDIMM Sub-System The Linux LIBNVDIMM subsystem provides support for three types of NVDIMMs; PMEM, BLK, and NVDIMM. PMEM (Persistent Memory) devices allow byte addressable access. BLK (block) devices allow sector atomicity like traditional storage devices. NVDIMM devices can simultaneously support both PMEM and BLK mode access. These three modes of operation are described by the "NVDIMM Firmware Interface Table" (NFIT) in [ACPI v6.0](http://www.uefi.org/sites/default/files/resources/ACPI_6_0_Errata_A.PDF) or later. While the LIBNVDIMM implementation is generic and supports pre-NFIT platforms, it was guided by the superset of capabilities need to support this ACPI v6.0 definition for NVDIMM resources. The bulk of the kernel implementation is in place to handle the case where the DIMM Physical Address (DPA) accessible via PMEM is aliased with DPA accessible via BLK. When that occurs a LABEL is needed to reserve DPA for exclusive access via one mode a time. For each NVDIMM access method (PMEM, BLK), LIBNVDIMM provides a block device driver: 1. **PMEM (nd\_pmem.ko):** Drives a system-physical-address range. This range is contiguous in system memory and may be interleaved (hardware memory controller striped) across multiple DIMMs. When interleaved the platform may optionally provide details of which DIMMs are participating in the interleave. Note that while LIBNVDIMM describes system-physical-address ranges that may alias with BLK access as ND\_NAMESPACE\_PMEM ranges and those without alias as ND\_NAMESPACE\_IO ranges, to the nd\_pmem driver there is no distinction. The different device-types are an implementation detail that userspace can exploit to implement policies like "only interface with address ranges from certain DIMMs". It is worth noting that when aliasing is present and a DIMM lacks a label, then no block device can be created by default as userspace needs to do at least one allocation of DPA to the PMEM range. In contrast ND\_NAMESPACE\_IO ranges, once registered, can be immediately attached to nd\_pmem. 2. **BLK (nd\_blk.ko):** This driver performs I/O using a set of platform defined apertures. A set of apertures will access just one DIMM. Multiple windows (apertures) allow multiple concurrent accesses, much like tagged-command-queuing, and would likely be used by different threads or different CPUs. The NFIT specification defines a standard format for a BLK-aperture, but the spec also allows for vendor specific layouts, and non-NFIT BLK implementations may have other designs for BLK I/O. For this reason "nd\_blk" calls back into platform-specific code to perform the I/O. One such implementation is defined in the "Driver Writer's Guide" and "DSM Interface Example". ## Differences between PMEM and BLK Modes While PMEM provides direct byte-addressable CPU-load/store access to NVDIMM storage, it does not provide the best system RAS (recovery, availability, and serviceability) model. An access to a corrupted system-physical-address address causes a CPU exception while an access to a corrupted address through an BLK-aperture causes that block window to raise an error status in a register. The latter is more aligned with the standard error model that host-bus-adapter attached disks present. Also, if an administrator ever wants to replace a memory it is easier to service a system at DIMM module boundaries. Compare this to PMEM where data could be interleaved in an opaque hardware specific manner across several DIMMs. BLK-apertures solve these RAS problems, but their presence is also the major contributing factor to the complexity of the ND subsystem. They complicate the implementation because PMEM and BLK alias in DPA space. Any given DIMM's DPA-range may contribute to one or more system-physical-address sets of interleaved DIMMs, and may also be accessed in its entirety through its BLK-aperture. Accessing a DPA through a system-physical-address while simultaneously accessing the same DPA through a BLK-aperture has undefined results. For this reason, DIMMs with this dual interface configuration include a DSM function to store/retrieve a LABEL. The LABEL effectively partitions the DPA-space into exclusive system-physical-address and BLK-aperture accessible regions. For simplicity a DIMM is allowed a PMEM "region" per each interleave set in which it is a member. The remaining DPA space can be carved into an arbitrary number of BLK devices with discontiguous extents. # Regions A generic REGION device is registered for each PMEM range or BLK-aperture set. Regions represent the physical persistent memory devices in an interleaved or non-interleaved set. LIBNVDIMM provides a built-in driver for these REGION devices. This driver is responsible for reconciling the aliased DPA mappings across all regions, parsing the LABEL, if present, and then emitting NAMESPACE devices with the resolved/exclusive DPA-boundaries for the `nd_pmem` or `nd_blk` device driver to consume. Refer to [Managing Regions](/ndctl-user-guide/v67/managing-regions) for more information. # Regions, Atomic Sectors, and DAX One of the few reasons to allow multiple BLK namespaces per REGION is so that each BLK-namespace can be configured with a Block Translation Table (BTT) with unique atomic sector sizes. While a PMEM device can host a BTT, the LABEL specification does not provide for a sector size to be specified for a PMEM namespace. This is due to the expectation that the primary usage model for PMEM is via DAX, and the BTT is incompatible with DAX. However, for the cases where an application or filesystem still needs atomic sector update guarantees, it can register a BTT on a PMEM device or partition. ## Example NVDIMM Platform Figure 1 shows an example platform with four NVDIMMs, two integrated memory controllers (IMC), and a single CPU . ``` (a) (b) DIMM BLK-REGION +-------------------+--------+--------+--------+ +------+ | pm0.0 | blk2.0 | pm1.0 | blk2.1 | 0 region2 | imc0 +--+- - - region0- - - +--------+ +--------+ +--+---+ | pm0.0 | blk3.0 | pm1.0 | blk3.1 | 1 region3 | +-------------------+--------v v--------+ +--+---+ | | | cpu0 | region1 +--+---+ | | | +----------------------------^ ^--------+ +--+---+ | blk4.0 | pm1.0 | blk4.0 | 2 region4 | imc1 +--+----------------------------| +--------+ +------+ | blk5.0 | pm1.0 | blk5.0 | 3 region5 +----------------------------+--------+--------+ ``` *Figure 1: Example sysfs layout* Each unique interface (BLK or PMEM) to DPA space is identified by a region device with a dynamically assigned id (REGION0 - REGION5). 1. The first portion of DIMM0 and DIMM1 are interleaved as REGION0. A single PMEM namespace is created in the REGION0-SPA-range that spans most of DIMM0 and DIMM1 with a user-specified name of "pm0.0". Some of that interleaved system-physical-address range is reclaimed as BLK-aperture accessed space starting at DPA-offset (a) into each DIMM. In that reclaimed space we create two BLK-aperture "namespaces" from REGION2 and REGION3 where "blk2.0" and "blk3.0" are just human readable names that could be set to any user-desired name in the LABEL. 2. In the last portion of DIMM0 and DIMM1 we have an interleaved ``` system-physical-address range, REGION1, that spans those two DIMMs as ``` well as DIMM2 and DIMM3. Some of REGION1 is allocated to a PMEM namespace named "pm1.0", the rest is reclaimed in 4 BLK-aperture namespaces (for each DIMM in the interleave set), "blk2.1", "blk3.1", "blk4.0", and "blk5.0". 3. The portion of DIMM2 and DIMM3 that do not participate in the REGION1 interleaved system-physical-address range (i.e. the DPA address past offset (b) are also included in the "blk4.0" and "blk5.0" namespaces. Note, that this example shows that BLK-aperture namespaces don't need to be contiguous in DPA-space. This bus is provided by the kernel under the device `/sys/devices/platform/nfit_test.0` when CONFIG\_NFIT\_TEST is enabled and the nfit\_test.ko module is loaded. This not only test LIBNVDIMM but the acpi\_nfit.ko driver as well. # Label Storage Area (LSA) The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to store the definition of NVDIMM *namespaces.* # Namespaces The capacity of an NVDIMM REGION (contiguous span of persistent memory) is accessed via one or more NAMESPACE devices. A REGION is the Linux term for what ACPI and UEFI call a DIMM-interleave-set, or a system-physical-address-range that is striped (by the memory controller) across one or more memory modules. The UEFI specification defines the *NVDIMM Label Protocol* as the combination of label area access methods and a data format for provisioning one or more NAMESPACE objects from a REGION. Note that label support is optional and if Linux does not detect the label capability it will automatically instantiate a "label-less" namespace per region. Examples of label-less namespaces are the ones created by the kernel's memmap option, or NVDIMMs without a valid *namespace index* in their label area. A namespace can be provisioned to operate in one of 4 modes, fsdax, devdax, sector, and raw: * **fsdax:** Filesystem-DAX mode is the default mode of a namespace when specifying `ndctl create-namespace` with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working-sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. * **devdax:** Device-DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem-DAX. Instead of a block-device that can support a DAX enabled filesystem, this mode provides a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual-machine, register persistent memory for RDMA, or when gigantic mappings are needed. * **sector:** Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. * **raw:** Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux fsdax or devdax mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. Figure 1 below shows a typical Filesystem-DAX (FSDAX) configuration with three physical NVDIMMs that have been interleaved. All available capacity has been used to create a single region (Region0), namespace (Namespace0.0), and dax enabled filesystem created. Namespace naming convention is commonly X.Y where X is the region number and Y is the namespace ![](https://1414645753-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LvwSH5QpH0_ER83uxYK%2F-LvwkomCR1BL0VFpLSgN%2F-LvwlBtYw13PknzW_5Tx%2Fdraw.io-gitbook-interleaved-dimms-fsdax.jpg?alt=media\&token=5f2e188f-5683-4441-8e5d-2e2c75310589) Refer to [Managing Namespaces](/ndctl-user-guide/v67/managing-namespaces) for information on how to create, delete, edit, and manage the namespace configuration. # Managing NVDIMMs Managing physical or emulated NVDIMMs using `ndctl` has no functional difference. Physical NVDIMM features and options may be controlled through the system BIOS. The BIOS cannot see emulated NVDIMMs. Observe the following restrictions when managing NVDIMMs * DO NOT change the memory slot of physical NVDIMMs when they are part of an Interleave Set. Doing so changes the order of interleaving so data access will be compromised or corrupted. If interleaving is disabled, moving NVDIMMs to different slot locations is okay, but not recommended. * DO NOT disable NVDIMMs when they are part of an active Region and/or Namespace as this will prevent data access and may corrupt data. * ALWAYS backup the data and make a copy of the configuration layout prior to any changes. ## Listing active/enabled NVDIMMs The `ndct list -D` , or equivalent `ndct list --dimm` , can be used to show active/enabled NVDIMM devices on the system, eg: ``` # ndctl list -D { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29 } ``` ## Listing disabled/inactive NVDIMMs By default, `ndctl` only lists enabled/active dimms, regions, and namespaces. To include previously disabled (inactive) NVDIMMs, include the `-i` flag to show both enabled and disabled devices, eg: ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":0, "phys_id":29 } { "dev":"nmem1", "id":"9759-b5-1459-00000502", "handle":1, "phys_id":30 "state":"disabled" } ... ``` {% hint style="info" %} NVDIMM vendor specific tools can be used to display more information about the NVDIMMs from the operating system layer. For example, Intel Optane DC Persistent Memory Modules can be managed using the [ipmctl](https://github.com/intel/ipmctl) utility. Refer to the [IPMCTL User Guide](https://docs.pmem.io/ipmctl-user-guide/) for more information. For other persistent memory products, refer to the vendor specific documentation. {% endhint %} ## Disabling NVDIMMs NVDIMMs can only be disabled if they have no active Regions or Namespaces. If an active/enabled namespace and/or region exists, a message is displayed: ``` # ndctl disable-dimm nmem0 nmem0 is active, skipping... disabled 0 nmem ``` 1\) List the current active/enabled configuration ``` # ndctl list -NRD ``` 2\) Verify no fsdax or devdax namespaces are mounted or in-use by running applications 3\) Destroy or disable any active/enabled namespace(s). ``` # ndctl disable-namespace - or - # ndctl destroy-namespace ``` See [Destroying Namespaces](/ndctl-user-guide/v67/managing-namespaces#destroying-namespaces) or [Disabling Namespaces](/ndctl-user-guide/v67/managing-namespaces#disabling-namespaces) for more information. 4\) Disable the regions used by the NVDIMM (nmem) that needs to be disabled ``` # ndctl disable-region ``` See [Disabling Regions](/ndctl-user-guide/v67/managing-regions#disabling-regions) for more information. 5\) Disable a single, subset, or all NVDIMM (nmem) devices To disable a single NVDIMM, use: ``` # ndctl disable-dimm nmem0 disabled 1 nmem ``` To disable a subset or specific list or NVDIMMs, use: ``` # ndctl disable-dimm nmem0 nmem1 nmem5 disabled 3 nmem ``` To disable all NVDIMMs, use: ``` # ndctl disable-dimm all disabled 12 nmem ``` 6\) Verify the NVDIMM(s) are disabled by listing inactive dimms and verifying the 'state': ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29, "state":"disabled" } ... ``` ## Enabling NVDIMMs 1\) Verify the nmem device, or list of nmem devices, that need to be enabled using the `ndctl list -Di` command: ``` # ndctl list -Di { "dev":"nmem0", "id":"8089-a2-1809-00000107", "handle":1, "phys_id":29, "state":"disabled" } ``` 2\) Enable the NVDIMM(s) To enable a single NVDIMM (nmemX) device, use: ``` # ndctl enable-dimm nmem0 enabled 1 nmem ``` To enable a subset of disabled NVDIMMs, use: ``` # ndctl enable-dimm nmem0 nmem1 nmem5 enabled 3 nmem ``` To enable all disabled NVDIMMs, use: ``` # ndctl enable-dimm all enabled 12 nmem ``` 3\) Verify the state by listing all NVDIMMs ``` # ndctl list -Di ``` A filtered list of NVDIMMs can shown using the `-d ` or `-dimm ` option, eg: ``` # ndctl list -Di -d nmem0 - or - # ndctl list -Di -dimm nmem0 ``` ## Sanitize NVDIMMs Sanitizing one or more NVDIMMs perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s). The sanitize-dimm command performs a cryptographic destruction of the contents of the given NVDIMM. It scrambles the data, and any metadata or info-blocks, but it doesn’t modify namespace labels. Therefore, any namespaces on regions associated with the given NVDIMM will be retained, but they will end up in the raw mode. Additionally, after completion of this command, the security and passphrase for the given NVDIMM will be disabled, and the passphrase and any key material will also be removed from the keyring and the ndctl keys directory at /etc/ndctl/keys. The command supports two different methods of performing the cryptographic erase. The default is crypto-erase, but additionally, an overwrite option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. {% hint style="info" %} Sanitizing NVDIMMs requires the region to be disabled first. If there are existing namespaces, those too must be destroyed or disabled. See the [Managing Regions](/ndctl-user-guide/v67/managing-regions) section for more information. {% endhint %} ### Examples Sanitize a single NVDIMM. This requires the region is disabled. The following destroys all namespaces, disables the region, the sanitizes the DIMMs within the region. 1\) Stop all applications that use a namespace associated with the NVDIMMs that need to be sanitized. 2\) Unmount any mounted file systems ``` $ sudo umount ``` 3\) Destroy the namespaces ``` $ sudo ndctl destroy-namespace --force [all | --region X] ``` 4\) Identify the NVDIMMs associated with the region using the `ndctl list -Rvv` or `ndctl list -DR` command. `-D` displays DIMMs, `-R` displays regions. The JSON formatted output shows the hierarchy. ``` $ sudo ndctl list -DR { ... "regions":[ { "dev":"region1", "size":1623497637888, "available_size":0, "max_available_extent":0, "type":"pmem", "iset_id":-2506113243053544244, "mappings":[ { "dimm":"nmem11", "offset":268435456, "length":270582939648, "position":5 }, { "dimm":"nmem10", "offset":268435456, "length":270582939648, "position":1 }, { "dimm":"nmem9", "offset":268435456, "length":270582939648, "position":3 }, { "dimm":"nmem8", "offset":268435456, "length":270582939648, "position":2 }, { "dimm":"nmem7", "offset":268435456, "length":270582939648, "position":4 }, { "dimm":"nmem6", "offset":268435456, "length":270582939648, "position":0 } ], "persistence_domain":"memory_controller" }, { "dev":"region0", "size":1623497637888, "available_size":1597727834112, "max_available_extent":1597727834112, "type":"pmem", "iset_id":3259620181632232652, "mappings":[ { "dimm":"nmem5", "offset":268435456, "length":270582939648, "position":5 }, { "dimm":"nmem4", "offset":268435456, "length":270582939648, "position":1 }, { "dimm":"nmem3", "offset":268435456, "length":270582939648, "position":3 }, { "dimm":"nmem2", "offset":268435456, "length":270582939648, "position":2 }, { "dimm":"nmem1", "offset":268435456, "length":270582939648, "position":4 }, { "dimm":"nmem0", "offset":268435456, "length":270582939648, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 5\) Disable the region(s) ``` $ sudo ndctl disable-region 1 disabled 1 region ``` 6\) Sanitize the DIMM(s) associated with the region(s) using the overwrite (`-o`) procedure. The -`c` (cryptographic) option works on DIMMs that were previously encrypted. ``` $ sudo ndctl sanitize-dimm -o nmem6 nmem7 nmem8 nmem9 nmem10 nmem11 overwrite issued for 6 nmems. ``` {% hint style="info" %} Note: If the DIMMs were not encrypted with a passphrase, you may see harmless messages such as the following for each DIMM ``` failed to open file /etc/ndctl/keys/nvdimm_8089-a2-1837-00000e88_{hostname}.blob: No such file or directory ``` {% endhint %} 7\) You may need to re-provision the NVDIMMs using the vendors utility. For example, use [ipmctl ](https://docs.pmem.io/ipmctl-user-guide/)for Intel Optane DC Persistent Memory to recreate the goal. Then use `ndctl` to [create the namespace configuration](/ndctl-user-guide/v67/managing-namespaces). # Managing Regions A region is a grouping of one or more NVDIMMs, or an interleaved set, that can be divided up into one or more Namespaces. Regions are of type PMEM or BLK. See [PMEM or BLK](/ndctl-user-guide/v67/concepts/libnvdimm-pmem-and-blk-modes) modes for more information. Regions can only be created, destroye, and configured using the vendor specific NVDIMM utility that manages the NVDIMMs or BIOS options, if available. ## Disabling Regions 1\) List all active/enabled regions: ``` # ndctl list -R { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of active/enabled regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -R -r region0 - or - # ndctl list -R -region region0 ``` 2\) Disable the region ``` # ndctl disable-region region0 disabled 1 region ``` 3\) Verify the region is disabled by including the `-i` option: ``` # ndctl list -Ri { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of disabled/inactive regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -Ri -r region0 - or - # ndctl list -Ri -region region0 ``` ## Enabling Regions 1\) List disabled/inactive regions ``` # ndctl list -Ri { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "state":"disabled", "persistence_domain":"memory_controller" } ``` A filtered list of active/enabled regions can be displayed using the `-r ` or `--region ` option, eg: ``` # ndctl list -R -r region0 - or - # ndctl list -R -region region0 ``` 2\) Enable the region ``` # ndctl enable-region region0 enabled 1 region ``` 3\) Verify the region is enabled using `ndctl list -R`. Note the 'state' field is not displayed for enabled regions. ``` # ndctl list -R { "dev":"region0", "size":134217728000, "available_size":134217728000, "type":"pmem", "numa_node":0, "iset_id":-7501067817058727390, "persistence_domain":"memory_controller" } ``` # Managing Label Storage Areas (LSA) The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to store the definition of any NVDIMM namespaces. The labels on the NVDIMMs cannot be edited directly. Changing the configuration can be achieved using the ndctl **create** or **destroy** commands. See [Managing Namespaces](/ndctl-user-guide/v67/managing-namespaces) and [Managing Regions](/ndctl-user-guide/v67/managing-regions). `ndctl` provides several label related commands shown below: * [check-labels](/ndctl-user-guide/v67/managing-label-storage-areas-lsa#checking-labels) - determine if the given dimm(s) have a valid namespace index block * [init-labels](/ndctl-user-guide/v67/managing-label-storage-areas-lsa#initializing-labels) - initialize the label data area on a dimm or set of dimms * [read-labels](/ndctl-user-guide/v67/managing-label-storage-areas-lsa#reading-labels) - read out the label area on a dimm or set of dimms * [write-labels](/ndctl-user-guide/v67/managing-label-storage-areas-lsa#writing-labels) - write data to the label area on a dimm * [zero-labels](/ndctl-user-guide/v67/managing-label-storage-areas-lsa#erasing-zeroing-labels) - zero out the label area on a dimm or set of dimms ## Checking Labels The `check-labels` command validates the index block within the label storage area on the specified NVDIMMs. If issues are found, running the command in verbose mode using the `-v` option, reports the reason the index block is deemed invalid. ``` usage: ndctl check-labels [..] [] -b, --bus must be on a bus with an id/provider of -v, --verbose turn on debug ``` 1\) Identify the NVDIMMs (nmem's) in the host using the `ndctl list -D` command: ``` # ndctl list -D [ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ] ``` 2\) Check the labels of one or more devices. To check the labels on a single device: ``` # ndctl check-labels nmem3 ``` To check the labels on multiple devices: ``` # ndctl check-labels nmem3 nmem0 nmem2 ``` To check the labels on all devices: ``` # ndctl check-labels all ``` To display more information, if available, use the `-v` option: ``` # ndctl check-labels nmem3 -v ``` {% hint style="info" %} Note: The `-v` option may not produce any additional information if the labels are valid or ndctl was not built with logging or debug options. {% endhint %} ## Initializing Labels This command can be used to initialize the namespace index block if it is missing or reinitialize it if it is damaged or was previously zero'd. Note that reinitialization effectively destroys all existing namespace labels on the DIMM. {% hint style="danger" %} This is an expert mode operation. Overwriting labels will cause data loss. Backup all data prior to overwriting labels. {% endhint %} ``` usage: ndctl init-labels [..] [] -b, --bus must be on a bus with an id/provider of -v, --verbose turn on debug -f, --force force initialization even if existing index-block present -V, --label-version namespace label specification version (default: 1.1) ``` Before initializing the labels, all namespaces and regions must be destroyed or disabled. A warning message is displayed when attempting to initialize labels on active regions. ``` # ndctl init-labels nmem3 nmem2 nmem3: regions active, abort label write nmem2: regions active, abort label write ``` 1\) Identify the active/enabled NVDIMM devices, Regions, and Namespaces ``` # ndctl list -DRN { "dimms":[ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ], "regions":[ { "dev":"region1", "size":534723428352, "available_size":0, "type":"pmem", "iset_id":3026489321642266624, "mappings":[ { "dimm":"nmem3", "offset":268435456, "length":267361714176, "position":1 }, { "dimm":"nmem2", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":526366277632, "uuid":"76fc4dd1-e793-4eee-81ed-2f31275eb820", "blockdev":"pmem1" } ] }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "mappings":[ { "dimm":"nmem0", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 2\) Destroy or disable the namespace(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl destroy-namespace -f namespace1.0 - or - # ndctl disable-namespace namespace1.0 # ndctl destroy-namespace namespace1.0 ``` 3\) Disable the region(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl disable-region region1 ``` 4\) Initialize the labels ``` # ndctl init-labels nmem2 nmem3 ``` If the `init-labels` command returns "error: labels already initialized", use the `-f`, `--force` option. 5\) Enable the region(s). After initializing the labels, the region(s) may not be brought online automatically. List the active and disabled regions: ``` # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "state":"disabled", "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` Activate any disabled regions ``` # ndctl enable-region region1 # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` 6\) New namespaces can now be created on the regions. See [Managing Namespaces](/ndctl-user-guide/v67/managing-namespaces) for more information. ## Reading Labels The `read-labels` command dumps the data in a dimm’s label area to stdout or a file in raw or JSON format. When multiple dimms (nmem's) are specified, the data is concatenated. Usage: ``` ndctl read-labels [..] [] OPTIONS One or more nmemX device names. The keyword all can be specified to operate on every dimm in the system, optionally filtered by bus id (see --bus= option). -b, --bus= Limit operation to memory devices (dimms) that are on the given bus. Where bus can be a provider name or a bus id number. -v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). -o, --output output file -j, --json parse the label data into json assuming the NVDIMM Namespace Specification format. ``` The following shows a label from nvdimm0 (nmem0) printed to stdout in JSON format. ``` # ndctl read-labels -j nmem0 { "dev":"nmem0", "index":[ { "signature":"NAMESPACE_INDEX", "major":1, "minor":2, "labelsize":256, "seq":2, "nslot":510 }, { "signature":"NAMESPACE_INDEX", "major":1, "minor":2, "labelsize":256, "seq":1, "nslot":510 } ], "label":[ { "uuid":"9ed56df2-c0a9-4e17-b82d-28f642f95a22", "name":"", "slot":0, "position":0, "nlabel":1, "isetcookie":710683120631319073, "lbasize":512, "dpa":268435456, "rawsize":267361714176, "type_guid":"79d3f066-f3b4-7440-ac43-0d3318b78cdb", "abstraction_guid":"ba006426-9ffb-7746-bcb0-968f11d0d225" } ] } read 1 nmem ``` To backup an NVDIMM label using the raw format, use: ``` # ndctl read-labels nmem0 > nmem0.lsa ``` ## Writing Labels {% hint style="danger" %} This is an expert mode operation. Overwriting labels will cause data loss. Backup all data prior to overwriting labels. {% endhint %} Writing labels overwrites the current label with the contents of the specified file, or stdin, collected from a [read-labels](/ndctl-user-guide/v67/managing-label-storage-areas-lsa#reading-labels) operation from a different label, or a previous label backup. Read data from the input filename, or stdin, and write it to the given device. The device being written to must not be active in any region, otherwise the kernel will not allow write access to the device’s label data area. The following shows how a raw label from an active nvdimm can be written to an inactive nvdimm. ``` $ ndctl read-labels nmem0 -o nmem0.label.backup.raw $ ndctl write-labels nmem1 -i nmem0.label.backup.raw ``` The above read-label raw output can be piped directly to the write-label command using: ``` $ ndctl read-labels nmem0 | ndctl write-labels nmem1 ``` ## Erasing/Zeroing Labels The `zero-labels` command resets the device to its default state by deleting all labels. {% hint style="danger" %} This operation cannot be undone unless a previous backup of the labels was created using the [read-labels](/ndctl-user-guide/v67/managing-label-storage-areas-lsa#reading-labels) command. All data on the device will be destroyed. Create a backup before proceeding. {% endhint %} Before erasing/zeroing the labels, all namespaces and regions must be destroyed or disabled. 1\) Identify the active/enabled NVDIMM devices, Regions, and Namespaces ``` # ndctl list -DRN { "dimms":[ { "dev":"nmem3", "id":"8089-a1-1811-0000005d", "handle":4353, "phys_id":68 }, { "dev":"nmem0", "id":"8089-a1-1811-00000058", "handle":1, "phys_id":32 }, { "dev":"nmem2", "id":"8089-a1-1811-00000068", "handle":4097, "phys_id":56 } ], "regions":[ { "dev":"region1", "size":534723428352, "available_size":0, "type":"pmem", "iset_id":3026489321642266624, "mappings":[ { "dimm":"nmem3", "offset":268435456, "length":267361714176, "position":1 }, { "dimm":"nmem2", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":526366277632, "uuid":"76fc4dd1-e793-4eee-81ed-2f31275eb820", "blockdev":"pmem1" } ] }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "mappings":[ { "dimm":"nmem0", "offset":268435456, "length":267361714176, "position":0 } ], "persistence_domain":"memory_controller" } ] } ``` 2\) Destroy or disable the namespace(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl destroy-namespace -f namespace1.0 - or - # ndctl disable-namespace namespace1.0 # ndctl destroy-namespace namespace1.0 ``` 3\) Disable the region(s) belonging to the NVDIMMs (nmem's) that require re-initializing. ``` # ndctl disable-region region1 ``` 4\) To erase the labels on a single 'nmem0' device. ``` $ ndctl zero-labels nmem0 ``` To erase labels on multiple devices, use: ``` $ ndctl zero-labels nmem0 nmem2 nmem3 ``` 5\) Enable the region(s). After initializing the labels, the region(s) may not be brought online automatically. List the active and disabled regions: ``` # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "state":"disabled", "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` Activate any 'disabled' regions ``` # ndctl enable-region region1 # ndctl list -iR [ { "dev":"region1", "size":534723428352, "available_size":534723428352, "type":"pmem", "iset_id":3026489321642266624, "persistence_domain":"memory_controller" }, { "dev":"region0", "size":267361714176, "available_size":0, "max_available_extent":0, "type":"pmem", "persistence_domain":"memory_controller" } ] ``` 6\) New namespaces can now be created on the regions. See [Managing Namespaces](/ndctl-user-guide/v67/managing-namespaces) for more information. # Managing Namespaces Regions can be partitioned in to one or more namespaces. Namespace operations include listing, creating, destroying, enabling, disabling, and checking (validating) using the following commands: * [list ](/ndctl-user-guide/v67/managing-namespaces#listing-namespaces)- dump the platform nvdimm device topology and attributes in json format * [create-namespace](/ndctl-user-guide/v67/managing-namespaces#creating-namespaces) - provision or reconfigure a namespace * [destroy-namespace](/ndctl-user-guide/v67/managing-namespaces#destroying-namespaces) - destroy the given namespace(s) * [disable-namespace](/ndctl-user-guide/v67/managing-namespaces#disabling-namespaces) - disable the given namespace(s) * [enable-namespace](/ndctl-user-guide/v67/managing-namespaces#enabling-namespaces) - enable the given namespace(s) * [check-namespace](/ndctl-user-guide/v67/managing-namespaces#checking-sector-namespaces) - check namespace metadata consistency ## Listing Namespaces #### Displaying Active/Enabled Namespaces The `ndctl list -N` command is used to display namespaces, for example: ``` # ndctl list -N { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":4294967296, "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` The output can be filtered using any combination of the following options: ``` -n, --namespace= An namespaceX.Y device name, or namespace region plus id tuple X.Y. Limit the namespace list to the single identified device if present. -m, --mode= Filter listing by the mode (raw, fsdax, sector or devdax) of the namespace(s). ``` **Examples:** List all active namespaces ``` # ndctl list -N ``` List all active namespaces and print human readable values (MB, MiB, GB, GiB, etc) ``` # ndctl list -Nu { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":"4.00 GiB (4.29 GB)", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` List a single namespace (namespace0.0) ``` # ndctl list -n namespace0.0 ``` List all active 'fsdax' mode namespaces ``` # ndctl list -m fsdax ``` #### Displaying Disabled/Inactive Namespaces By default, `ndctl list -N` lists only active/enabled namespaces. In the following example no active/enabled namespaces exist so no output is displayed: ``` # ndctl list -N # ``` Adding the `-i` flag includes both active/enabled and inactive/destroyed/disabled namespaces. The following example shows two previously destroyed namespaces. The `size` and `uuid` are clear indications that no information exists about the namespace. The `ndctl destroy` command simply NULLs the existing namespace entry and does not completely erase it from the metadata. For this reason, it may be possible to see more than one namespace using `ndctl list -Ni` ``` # ndctl list -Ni [ { "dev":"namespace0.0", "mode":"raw", "size":0, "uuid":"00000000-0000-0000-0000-000000000000", "sector_size":512, "state":"disabled", "numa_node":0 } ] ``` The `-i` option can be used in coordination with any other valid ndctl option. **Examples:** List all enabled and disabled namespaces ``` # ndctl list -Ni ``` List a single disabled namespace (namespace0.0) ``` # ndctl list -in namespace0.0 ``` List all enabled and disabled 'fsdax' mode namespaces ``` # ndctl list -im fsdax ``` ## Creating Namespaces The create-namespace command has a lot of options summarized below. The `ndctl-create-namespace` man page contains further details of each option. * **-t, --type:** Set the type of the namespace to either "pmem" or "blk" * **-m, mode:** Define the namespace mode - fsdax, devdax, sector, and raw * **-s, --size:** Specify the capacity of the namespace * **-a, --align:** Specify the alignments size, eg 4K or 2M * **-e, --reconfigure:** Reconfigure existing namespace configuration * **-u, --uuid:** Used for recovery purposes only * **-n, --name:** Specify a friendly name for namespaces that support labels * **-l, --sector-size:** Override the default sector size for block based namespaces * **-M, --map:** For 'fsdax' or 'devdax' namespaces, define whether metadata is stored in volatile memory (mem) or persistent storage (dev) * **-c, --continue:** Do not stop after creating one namespace. Instead, create as many namespaces as possible within the given --bus and --region filter restrictions. * **-f, --force:** Allow the operation to continue on enabled namespaces * **-L, --autolabel, --no-autolabel:** Manage labels for legacy NVDIMMs that do not support labels * **-v, --verbose:** Emit verbose messages during the creation process * **-r, --region:** Limit the operation to a specific region * **-b, --bus:** Limit the operation to a specific bus The `mode` is the most important feature to get correct. The four modes available are defined as: * **fsdax:** Filesystem-DAX mode is the default mode of a namespace when specifying `ndctl create-namespace` with no options. It creates a block device (/dev/pmemX\[.Y]) that supports the DAX capabilities of Linux filesystems (xfs and ext4 to date). DAX removes the page cache from the I/O path and allows mmap(2) to establish direct mappings to persistent memory media. The DAX capability enables workloads / working-sets that would exceed the capacity of the page cache to scale up to the capacity of persistent memory. Workloads that fit in page cache or perform bulk data transfers may not see benefit from DAX. When in doubt, pick this mode. * **devdax:** Device-DAX mode enables similar mmap(2) DAX mapping capabilities as Filesystem-DAX. However, instead of a block-device that can support a DAX-enabled filesystem, this mode emits a single character device file (/dev/daxX.Y). Use this mode to assign persistent memory to a virtual-machine, register persistent memory for RDMA, or when gigantic mappings are needed. * **sector:** Use this mode to host legacy filesystems that do not checksum metadata or applications that are not prepared for torn sectors after a crash. Expected usage for this mode is for small boot volumes. This mode is compatible with other operating systems. * **raw:** Raw mode is effectively just a memory disk that does not support DAX. Typically this indicates a namespace that was created by tooling or another operating system that did not know how to create a Linux fsdax or devdax mode namespace. This mode is compatible with other operating systems, but again, does not support DAX operation. {% hint style="info" %} If the *-s, --size* option is used, the value provided to ndctl is used to create the namespace. Depending on the *mode*, the available capacity may be smaller than the specified size due to the calculated spared required for metadata. This is shown below in the examples. {% endhint %} ### FSDAX and DEVDAX Capacity Considerations The namespace configuration (label metadata) is stored at the beginning of the persistent memory address range. See the [Label Storage Area (LSA)](/ndctl-user-guide/v67/concepts/label-area) and [Managing Label Storage Area (LSA)](/ndctl-user-guide/v67/managing-label-storage-areas-lsa) sections for more information. In addition to the namespace label metadata, both fsdax and devdax modes require space to store the Page Frame Number (PFN) metadata (also known as “struct page” metadata) for each page within the namespace. A PFN is simply in index within physical memory, or on the NVDIMM, that is counted in page-sized units. For modes with PFN metadata, shown in Table 1 below, the requirement is 64 bytes per 4 KiB of persistent memory. To calculate the estimated space for metadata, use: $$ metadata \_{(bytes)} = (size \_{(bytes)}/4096)\*64 $$ **Example 1:** A 256GiB namespace requires \~3.93GiB of metadata (263182090240/4096)\*64 = 4112220160 bytes == 3.93GB **Example 2:** A 1TiB namespace requires \~17.18GiB of metadata (1099511627776/4096)\*64 = 17179869184 bytes == 17.18GiB The location of PFN metadata can either be in volatile DDR from the NVDIMM. The default is to keep metadata on the NVDIMM. The location is defined per namespace by the `-M, --map` configuration option. The following provides guidance for PFN metadata location: * **map=mem:** PFN metadata is stored in regular system memory (DDR) * Adequate for small persistent memory capacities * Frequent or high performance PFN access is required. For example, when time sensitive, or frequent, registration and deregistration of RDMA memory regions is important. Normal RDMA memory \[de]registration can be achieved with 'map=dev', but there are situations where \[de]registration needs to be very fast. This assumes the metadata fits in DDR memory. * **map=dev:** PFN metadata is stored on persistent memory (NVDIMM) * Intended for large persistent memory capacities. There might not be enough regular memory in the system if the ratio of persistent memory to DDR is high (>8:1) The persistence of the PFN metadata is not important as it's re-mapped at system boot time or when new namespaces are created/destroyed. It is convenient to store the metadata on the NVDIMM because it scales with the persistent memory capacity. The space for metadata is calculated and reserved at namespace creation time, meaning the usable capacity is lower than the physical capacity depending on the mode and map values. **Table 1: Features for each namespace mode** | Mode | Description | Device Path | Label Metadata | Atomicity | Filesystems | DAX | PFN Metadata | | ------ | -------------- | ----------- | -------------- | --------- | ----------- | --- | ------------ | | raw | raw | /dev/pmemN | no | no | yes | no | no | | sector | sector atomic | /dev/pmemNs | yes | yes | yes | no | no | | fsdax | filesystem DAX | /dev/pmemN | yes | no | yes | yes | yes | | devdax | device DAX | /dev/daxN.M | yes | no | no | yes | yes | **PFN Metadata Usage Example** The following demonstrates the difference between `--map=dev` and `--map=mem`. The example uses a \~250GiB region to create an fsdax namespace. A devdax namespace has identical behavior. When a namespace using the default `--map=dev` is created, the usable 'size' is \~4GiB smaller than the 249GiB region size due to the space reserved for PFN metdata. ``` # ndctl create-namespace -r region0 # ndctl list -RuN -r region0 { "regions":[ { "dev":"region0", "size":"249.00 GiB (267.36 GB)", "available_size":0, "type":"pmem", "iset_id":"0xb000000058000000", "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"245.11 GiB (263.18 GB)", "uuid":"19cccf5d-d227-42fe-b338-cfa882a6b6ce", "blockdev":"pmem0" } ] } ] } ``` When using `--map=mem`, the usable capacity on the namespace increases, but the amount of 'used' memory as shown by `free` has increased by \~4GiB. ``` # free total used free shared buff/cache available Mem: 362926556 5747588 356160304 2608 1018664 355200308 Swap: 32882684 0 32882684 # ndctl create-namespace -r region0 --map=mem # ndctl list -RuN -r region0 { "regions":[ { "dev":"region0", "size":"249.00 GiB (267.36 GB)", "available_size":0, "type":"pmem", "iset_id":"0xb000000058000000", "persistence_domain":"memory_controller", "namespaces":[ { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":"249.00 GiB (267.36 GB)", "uuid":"ff626073-eef9-422c-9b09-814fc1498c1b", "blockdev":"pmem0" } ] } ] } # free total used free shared buff/cache available Mem: 362926556 9827632 352080120 2612 1018804 351120192 Swap: 32882684 0 32882684 ``` #### **Creating Namespace Examples** This example section has been split in to four, one for each mode described above: * [FSDAX Mode Examples](/ndctl-user-guide/v67/managing-namespaces#fsdax-mode-examples) * [DEVDAX Mode Examples](/ndctl-user-guide/v67/managing-namespaces#devdax-mode-examples) * [SECTOR Mode Examples](/ndctl-user-guide/v67/managing-namespaces#sector-mode-examples) * [RAW Mode Examples](/ndctl-user-guide/v67/managing-namespaces#raw-mode-examples) #### **FSDAX Mode Examples** **Example 1:** Create an 'fsdax' mode namespace using all available capacity (default behavior). The `ndctl create-namespace` command shown below is the equivalent of `ndctl create-namespace -m fsdax`. A new `/dev/pmem{X[.Y]}` device is created. This can be used to create a new DAX enabled filesystem such as XFS or EXT4. The 'X' value represents the region on which the namespace is created. This defaults to zero (0). If more than one namespace is created within a region, the naming convention of pmem{X.Y} is used, where Y represents a sequentially increasing integer value for the new namespace. If multiple namespaces within a region are created, the first namespace is always created as 'pmem0', it does not get re-enumerated to 'pmem0.0' like subsequent namespaces. ``` # ndctl create-namespace { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"a60e6a4f-274d-4cd5-8d39-c8dd263345e2", "raw_uuid":"8d28948f-5434-4c8d-8efa-581eacad265a", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ls -l /dev/pmem* brw-rw----. 1 root disk 259, 0 Jul 9 10:42 /dev/pmem0 ``` **Example 2:** Create a 50GiB 'fsdax' mode namespace. The 'size' provided to ndctl includes space required for metadata so the resulting available capacity for a filesystem will be smaller as the example below shows: ``` # ndctl create-namespace -m fsdax -s 50G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"638d67f3-4c18-4b2e-a6f2-a044bdc82253", "raw_uuid":"6c6dfdad-e12f-418d-9fb0-3a75032dd9de", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` **Note:** If the remaining capacity needs to be assigned to another namespace using the same or different mode, the remaining capacity can be assigned to the new namespace without specifying the '-s \' option. For example, executing `ndctl create-namespace` after the command above creates a new namespace in the same region using all remaining capacity (\~78GiB): ``` # ndctl create-namespace { "dev":"namespace0.1", "mode":"fsdax", "map":"dev", "size":"73.83 GiB (79.27 GB)", "uuid":"d7f9473e-97aa-48cf-aefa-128797c83e88", "raw_uuid":"6c116e57-19dd-43d8-ae03-039f1588a23a", "sector_size":512, "blockdev":"pmem0.1", "numa_node":0 } ``` **Example 3:** Create a namespace with a friendly name (tag) Tagging the namespace with a friendly name or description using the '-n, --name' option can be useful to know what a namespace is used for. This is particularly useful when provisioning multiple end-users, applications, or to identify a namespace with production or test/dev environments. Creating a namespace with a tag/name/description can be done using: ``` # ndctl create-namespace -n "PROD Web DB 1" { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"6a0abb59-5279-4731-921a-0099101e17f2", "raw_uuid":"03b40e23-56e1-407a-b5d1-f1ec929645c1", "sector_size":512, "blockdev":"pmem0", "name":"PROD Web DB 1", "numa_node":0 } ``` **Example 4:** Automatically create as many namespaces as possible By default, the `create-namespace` operation creates a single namespace. On systems with more than one region (usually defined by the number of CPU sockets), this requires executing `create-namespace` for each region. On high CPU socket systems, this can be time consuming. Version 67 of ndctl introduced the `-c, --continue` option that will automatically create as many namespaces as it can. This is similar to using a FOR loop around `create-namespace`. This operation will abort if any creation attempt results in an error unless --force is also supplied. The following example shows how to automatically create two fsdax namespaces on empty regions. The size of the namespaces will use the full capacity of each region. ``` # ndctl create-namespace -c { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":"1488.37 GiB (1598.13 GB)", "uuid":"4eab54fb-452a-42a1-b6fb-172a14dfd0b4", "sector_size":512, "align":2097152, "blockdev":"pmem1" } { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"1488.37 GiB (1598.13 GB)", "uuid":"303e1f77-84c1-4e45-a631-61a4551bdf3d", "sector_size":512, "align":2097152, "blockdev":"pmem0" } created 2 namespaces ``` #### **DEVDAX Mode Examples** Create a pmem namespace with devdax mode using all available capacity ``` # ndctl create-namespace -m devdax { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":"123.04 GiB (132.12 GB)", "uuid":"e1c1d031-71a3-4a6b-b3d7-a4037a5256c3", "raw_uuid":"2a15b77f-9f31-45e5-84a3-4ad06817302a", "daxregion":{ "id":0, "size":"123.04 GiB (132.12 GB)", "align":2097152, "devices":[ { "chardev":"dax0.0", "size":"123.04 GiB (132.12 GB)" } ] }, "numa_node":0 } # ls -l /dev/dax* crw-------. 1 root root 250, 1 Jul 9 10:39 /dev/dax0.0 ``` Create a 50GiB 'devdax' namespace ``` # ndctl create-namespace -s 50G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"b49087fc-9786-48d1-899b-6166c7c586f9", "raw_uuid":"f441a441-77ad-4286-9585-ac9511d5381a", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } ``` #### **Sector Mode Examples** Create a sector mode namespace using all available capacity and a default 4K sector size ``` # ndctl create-namespace -m sector { "dev":"namespace0.0", "mode":"sector", "size":"124.88 GiB (134.09 GB)", "uuid":"08f4e273-bbdd-4d1d-85e8-cf1f847e1df7", "raw_uuid":"f3bd03ad-4225-4919-a370-bb6293180e4d", "sector_size":4096, "blockdev":"pmem0s", "numa_node":0 } # ls -l /dev/pmem0 brw-rw----. 1 root disk 259, 0 Jul 9 10:47 /dev/pmem0s ``` #### RAW Mode Examples Create a 'raw' mode namespace using all available capacity ``` # ndctl create-namespace -m raw { "dev":"namespace0.0", "mode":"raw", "size":"125.00 GiB (134.22 GB)", "uuid":"24c1d696-b197-4c89-96b2-f91b60f4f295", "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ls -l /dev/pmem0 brw-rw----. 1 root disk 259, 0 Jul 9 10:52 /dev/pmem0 ``` ## Editing Namespace Properties Changing properties of existing namespaces can be done online using the `ndctl create-namespace -fe ` command. The `-e, --reconfigure` flag edits existing namespaces. Using the `-f` flag does not require that the namespace be manually disabled. The command may still fail if the namespace is currently being used. Supported options are listed above in the [Creating Namespaces](/ndctl-user-guide/v67/managing-namespaces#creating-namespaces) section, or review the ndctl-create-namespace [man page](/ndctl-user-guide/v67/ndctl-man-pages). Some namespace properties are read-only and cannot be changed using the `ndctl` utility. [Resizing Namespaces](/ndctl-user-guide/v67/managing-namespaces#resizing-namespaces) or [Changing Namespace Modes](/ndctl-user-guide/v67/managing-namespaces#changing-namespace-modes) can be achieved by changing existing namespace properties. They are discussed below under their own headings to provide additional detail. #### Examples Add or Change a namespace name (also referred to as a tag or description): ``` # ndctl create-namespace -fe namespace0.0 -n "prod_web_db1" { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"49.22 GiB (52.85 GB)", "uuid":"086d4800-4dec-41f4-a3b8-bb3e53974c54", "raw_uuid":"411b37d2-284e-4626-8724-efd0855e3c39", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` ## Resizing Namespaces {% hint style="danger" %} WARNING: Data will be lost or corrupted if the new size is smaller than the existing size or extending the namespace and growing the filesystem(s). Backup all data before resizing namespaces. {% endhint %} The following example resizes an existing 'fsdax' namespace (namespace0.0) to 20GiB: ``` # ndctl create-namespace -fe namespace0.0 -s 20G { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":"19.69 GiB (21.14 GB)", "uuid":"75cac06f-fc80-4c51-8f0b-2d73e2d2e31b", "raw_uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` If there is an existing EXT4 or XFS filesystem on this device, the partition table may need to be manually changed to accommodate the new device capacity before the filesystem can be resized. Due to the many configuration options available when using fsdax devices, such as whether it's being managed by the Linux device-mapper, LVM (Linux Volume Manager), or just an XFS/EXT4 filesystem on top, refer to the relevant Linux distribution documentation for specific procedures to handle your configuration. ## Changing Namespace Modes The namespace mode can be changed on any enabled or disabled namespace. Valid modes are raw, sector, fsdax, and devdax. Refer to the `ndctl-create-namespace` man page or [Namespaces](/ndctl-user-guide/v67/concepts/nvdimm-namespaces) chapter for more information on each mode. {% hint style="warning" %} {% endhint %} * Changing the namespace mode will destroy all current data. Backup all data before proceeding. * If a filesystem using an fsdax or sector namespace is mounted, unmount it before changing the mode. * If a devdax or raw namespace is currently being used by a running application, stop the application before changing modes. Switching modes of an enabled/active namespace can be done using `ndctl create-namespace -fe --mode=`. For example, to switch an enabled fsdax namespace to devdax mode, use: ``` # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"fsdax", "map":"mem", "size":4294967296, "sector_size":512, "blockdev":"pmem0", "numa_node":0 } # ndctl create-namespace -fe namespace0.0 --mode=devdax { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":"3.94 GiB (4.23 GB)", "uuid":"d71504d1-0286-474d-839a-52afe7d0da5f", "daxregion":{ "id":0, "size":"3.94 GiB (4.23 GB)", "align":2097152, "devices":[ { "chardev":"dax0.0", "size":"3.94 GiB (4.23 GB)" } ] }, "numa_node":0 } # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"devdax", "map":"dev", "size":4225761280, "uuid":"d71504d1-0286-474d-839a-52afe7d0da5f", "chardev":"dax0.0", "numa_node":0 } ``` If the namespace is currently disabled, the `-f` flag is not required. ## Destroying Namespaces Namespaces should be inactive/disabled prior to deletion otherwise a warning message is displayed. The following shows an attempt to destroy an active namespace ``` # ndctl destroy-namespace namespace0.0 Error: namespace0.0 is active, specify --force for re-configuration error destroying namespaces: Device or resource busy ``` First disable the namespace, then destroy it ``` # ndctl disable-namespace namespace0.0 disabled 1 namespace # ndctl destroy-namespace namespace0.0 destroyed 1 namespace ``` Alternatively using the `-f` flag to the command forcibly destroys the namespace without doing any validation checks and does not require that the namespace be disabled first, eg: ``` # ndctl destroy-namespace -f namespace0.0 destroyed 1 namespace ``` ## Disabling Namespaces {% hint style="warning" %} **Warning:** Disabling namespaces while the namespace is mounted or in-use will result in undefined behavior by the application(s) using the namespace. Always stop the application and unmount filesystems (fsdax mode) before disabling namespaces. {% endhint %} 1\) Disable the namespace (namespace0.0) using: ``` # ndctl disable-namespace namespace0.0 disabled 1 namespace ``` 2\) Verify the namespace is *disabled*. The `-i` flag is required to display inactive/disabled namespaces otherwise only active/enabled namespaces are shown. The example below filters the output to a specific namespace (namespace0.0) to avoid potentially large volumes of output being shown: ``` # ndctl list -Ni -n namespace0.0 { "dev":"namespace0.0", "mode":"raw", "size":21474836480, "uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "state":"disabled", "name":"prod_web_db1", "numa_node":0 } ``` ## Enabling Namespaces 1\) Enable the namespace (namespace0.0) using: ``` # ndctl enable-namespace namespace0.0 enabled 1 namespace ``` 2\) Verify the namespace is enabled. The state is only displayed for disabled namespaces. If the state field is not listed, the namespace is assumed to be enabled. ``` # ndctl list -N -n namespace0.0 { "dev":"namespace0.0", "mode":"fsdax", "map":"dev", "size":21137195008, "uuid":"75cac06f-fc80-4c51-8f0b-2d73e2d2e31b", "raw_uuid":"89b14037-0299-4887-8acb-7c31be3ae31b", "sector_size":512, "blockdev":"pmem0", "name":"prod_web_db1", "numa_node":0 } ``` ## Checking Sector Namespaces The `check-namespace` command only works on namespaces in *sector mode*. Attempting to check namespaces in any other mode will yield an error similar to the following which attempts to check an fsdax mode namespace: ``` # ndctl check-namespace namespace0.0 namespace0.0: namespace_check: namespace0.0: check aborted, namespace online error checking namespaces: Device or resource busy ``` A namespace in the sector mode will have metadata on it to describe the Kernel BTT (Block Translation Table). The check-namespace command can be used to check the consistency of this metadata, and optionally, also attempt to repair it, if it has enough information to do so. The command usage is: ``` ndctl check-namespace [] ``` Where the available options include: ``` -b, --bus limit namespace to a bus with an id or provider of -r, --region limit namespace to a region with an id or name of -v, --verbose emit extra debug messages to stderr -R, --repair perform metadata repairs -L, --rewrite-log regenerate the log -f, --force check namespace even if currently active ``` **Examples** The namespace being checked has to be disabled before initiating a check on it as a precautionary measure. ``` # ndctl disable-namespace namespace0.0 # ndctl check-namespace namespace0.0 ``` The `-f, --force` option can override the need to manually disable the namespace first, although it is not recommended to do so: ``` # ndctl check-namespace -f namespace0.0 checked 1 namespace ``` Adding the `-v, --verbose` option yields more information, eg: ``` # ndctl check-namespace -f namespace0.0 -v namespace0.0: namespace_check: checking namespace0.0 namespace0.0: btt_discover_arenas: found 1 BTT arena namespace0.0: log_set_indices: arena[0]: log_index_0 = 0 namespace0.0: log_set_indices: arena[0]: log_index_1 = 1 namespace0.0: btt_check_arenas: checking arena 0 checked 1 namespace ``` Check a namespace and perform repairs if possible or required can be done using: ``` # ndctl disable-namespace namespace0.0 # ndctl check-namespace --repair namespace0.0 ``` # Managing NVDIMM Security {% hint style="info" %} Note: These security features require ndctl v64.0 or later and requires Linux Kernel 5.0 or later. Vendor specific utilities may also provide the same functionality on earlier Linux Kernel versions. For Intel(R) Optane(TM) DC Persistent Memory Modules, see the '[ipmctl](https://github.com/intel/ipmctl)' utility. {% endhint %} NVDIMMs that provide additional security mechanisms, such as the Inte(R) Optane(TM) DC Persistent Memory Modules, can be managed using the `ndctl` utility and the following sub-commands: * ndctl-load-keys * ndctl-freeze-security * ndctl-remove-passphrase * ndctl-sanitize-dimm * ndctl-setup-passphrase * ndctl-update-passphrase {% hint style="warning" %} **Note:** Before performing any of the security operations, all the regions associated with the DIMM in question need to be disabled, see '[Disabling Regions](/ndctl-user-guide/v67/managing-regions#disabling-regions)'. For the *overwrite* operation, in addition to disabling the *regions*, the *nvdimms* also need to be disabled, see '[Disabling NVDIMMs](/ndctl-user-guide/v67/managing-nvdimms)'. {% endhint %} The Intel(R) Device Specific Methods (DSM) specification [v1.7](http://pmem.io/documents/NVDIMM_DSM_Interface-V1.7.pdf) and [v1.8](http://pmem.io/documents/NVDIMM_DSM_Interface-V1.8.pdf) introduced the following security management operations: enable passhprase, update passphrase, unlock DIMM, disable security, freeze security, secure (crypto) erase, overwrite, master passphrase enable, master passphrase update, and master passphrase secure erase. The security management for NVDIMMs is comprised of two parts. The front end uses the Linux key management framework (trusted and encrypted keys \[2]) to store the encrypted passphrases in the kernel-managed keyring. The interface for this is the *keyutils* utility which uses the key management APIs in the Linux kernel. The back end takes the decrypted payload (which is the DIMM passphrase) and passes it to the DIMM. Unlike other DSMs which are composed by libndctl and sent to the kernel via an ioctl, the security DSMs are managed through the *security* sysfs attribute under the *dimm* device. A *key-ID* is written to the *security* attribute and the kernel pulls the associated key material from the user keyring that is maintained by the kernel. The security process begins with the generation of a *master key* that is used to seal (encrypt) the passphrase for the DIMM. There can either be one common *master key* that is used to encrypt every DIMM’s passphrase, or a separate key can be generated for each DIMM. The *master key* is also referred to as the *key-encryption-key* (kek). The *kek* can either be generated by the TPM (Trusted Platform Module) on the system, or alternatively, the *System Master Key* can also be used as the *kek* ## Setup a User Passphrase The `ndctl setup-passphrase` is used to setup and enable a security passphrase for one or more NVDIMMs. To setup the passphrase for a NVDIMM or interleaved set of NVDIMMs, it is expected that the master key has previously been loaded into the user keyring. More information on how this can be done can be found in the kernel documentation at: . The *kek* encrypts the NVDIMM passphrase using the *enc32* key format. The plaintext passphrase is never provided by or made visible to the user. It is instead randomly generated by the kernel and userspace does not have access to it. Upon encryption, a binary blob of the passphrase is written to the passphrase blob storage directory (/etc/ndctl/keys). The user is responsible for backing up the passphrase blobs to a secure location. The command will fail if the passphrase is already in the user keyring or if a passphrase blob already exists in /etc/ndctl/keys. ## Update a User Passphrase The `ndctl update-passphrase` command updates the security passphrase for one or more NVDIMMs that have previously been setup using the `ndctl setup-passphrase` command. Prerequisites for this command to succeed: 1. The kek has been loaded into the kernel’s user keyring 2. `setup-passphrase` has successfully been executed on the NVDIMM(s) and the NVDIMM is unlocked The updated key blobs will be created in the /etc/ndctl/keys directory with a file name format of "nvdimm.blob". The update user passphrase operation uses the same DSM command as enable user passphrase. Most of the work is done on the key management side. The user has the option of providing a new *kek* for the new passphrase, but continuing to use the existing *kek* is also acceptable. The following operations are performed for *update-passphrase*: 1. Remove the encrypted passphrase from the kernel’s user keyring. 2. Rename the passphrase blob to old. 3. Load this old passphrase blob into the keyring with an “old” name. 4. Create the new passphrase and encrypt with the *kek*. 5. Send DSM with the old and new decrypted passphrases. 6. Remove old passphrase and the passphrase blob from the keyring. ## Remove a User Passphrase The `ndctl remove-passphrase` command prevents a DIMM from locking at power-loss and requiring a passphrase to access media. The *key-ID* for the passphrase to be removed is written to sysfs. The kernel then sends the DSM to disable security, and the passphrase is then removed from the keyring, and the associated passphrase blob is deleted. ## Crypto (Secure) Erase NVDIMMs This operation is similar to `ndctl remove-passphrase`. The kernel issues a WBINVD instruction before and after the operation to ensure no data corruption from a stale CPU cache. Use the `ndctl sanitize-dimm` command with the `--crypto-erase` option to perform this operation. The `ndctl sanitize-dimm` command performs a cryptographic destruction of the contents of the given NVDIMM. It scrambles the data, and any metadata or info-blocks, but it doesn’t modify namespace labels. Therefore, any namespaces on regions associated with the given NVDIMM will be retained, but they will end up in the raw mode. Additionally, after completion of this command, the security and passphrase for the given NVDIMM will be disabled, and the passphrase and any key material will also be removed from the keyring and the ndctl keys directory at /etc/ndctl/keys The command supports two different methods of performing the cryptographic erase. The default is `crypto-erase`, but additionally, an overwrite option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. ## Overwriting (Sanitizing) NVDIMMs One or more NVDIMMs can be sanitized using `ndctl sanitize-dimm [--crypto-erase][--overwrite]`. The command supports two different methods of performing the cryptographic erase. The default is `crypto-erase`, but additionally, an `overwrite` option is available which overwrites not only the data area, but also the label area, thus losing record of any namespaces the given NVDIMM participates in. The overwrite operation wipes the entire NVDIMM(s) and can take a significant amount of time depending on the size of the device(s) and number of devices. When the command returns successfully, it just means overwrite has been successfully started, the overwrite operation continues asynchronously in the background. Use the `ndctl wait-overwrite` command to wait for the NVDIMMs that are performing overwrite. Upon successful completion of an overwrite, the WBINVD instruction is issued by the kernel. If both `–crypto-erase` and `–overwrite` options are supplied, then crypto-erase is performed before overwrite. ## Security Freeze The `ndctl freeze-security` command does not require a passphrase. This will cause any security command other than a status query to be locked out until the next boot. This is used in scenarios where the administrator has taken all expected security actions for the current boot and wants the NVDIMMs to enforce/lock the current state until system reboot. To check the current security status of a single NVDIMM (nmem{X} device): ``` $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"unlocked" } ] ``` To freeze a single NVDIMM: ``` $ ndctl freeze-security nmem0 security freezed 1 nmem. $ ndctl list -d nmem0 [ { "dev":"nmem0", "id":"cdab-0a-07e0-ffffffff", "handle":0, "phys_id":0, "security":"frozen" } ] ``` ## Unlock NVDIMMs Unlock is performed by the kernel, however a preparation step must happen before the unlock DSM can be issued by the kernel. It is expected that from the initramfs, a setup command `ndctl load-keys` is executed before the libnvdimm module is loaded by modprobe. See the 'Loading Keys' section. This command will inject the *kek* and the encrypted passphrases into the kernel’s user keyring. During the *probe* of the libnvdimm driver, it will: 1. Check the security state of the device and see if the DIMM is locked 2. Request the associated encrypted passphrase from the kernel’s user key ring 3. Use the *kek* to decrypt the passphrase 4. Create the unlock DSM, copy the decrypted payload into the DSM 5. Issue the DSM to unlock the DIMM If the DIMM is already unlocked, the kernel will attempt to re-validate the passphrase. If we fail to revalidate the passphrase, the kernel will freeze the security and disallow any further security configuration changes. A kernel module parameter is available to override this behavior. ## Loading Keys {% hint style="info" %} **Note#1:** All key files are expected to be in the format: nvdimm*\_hostname The '*' character is used to delimit the different components in the file name. Within the hostname, the '\_' character is allowed since it is the last component of the file name. **Note#2:** This command is typically never called directly by a user. {% endhint %} The `load-keys` command loads the master key (kek) and the encrypted passphrases for all NVDIMMs into the user keyring maintained by the kernel. The command is expected to be called during initialization and before the libnvdimm kernel module is loaded, typically from an initrd. This is typically set up using a modprobe config that calls the command before module load. See for more information on the Kernel implementation. ## Master Passphrase - Setup, Update, and Crypto Erase These operations are similar to creating and updating the user passphrase. The only difference is that a different passphrase is used. Both `setup-passphrase` and `update-passphrase` support the `-m` flag to differentiate between user and master passphrase operations. The master passphrase has no relation to the master key (*kek*) which is used for encryption of either passphrase. More information about the master key can be found in the kernel documentation at: . # NDCTL Man Pages This section links to the online man pages: * [ndctl ](/ndctl-user-guide/v67/ndctl-man-pages/untitled)- Manage “libnvdimm” subsystem devices (Non-volatile Memory) * [ndctl-check-labels](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels) - determine if the given dimms have a valid namespace index block * [ndctl-check-namespace](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace) - check namespace metadata consistency * [ndctl-create-namespace](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-create-namespace) - provision or reconfigure a namespace * [ndctl-destroy-namespace](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-destroy-namespace) - destroy the given namespace(s) * [ndctl-disable-dimm](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-disable-dimm) - disable one or more idle dimms * [ndctl-disable-namespace](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-disable-namespace) - disable the given namespace(s) * [ndctl-disable-region](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-disable-region) - disable the given region(s) and all descendant namespaces * [ndctl-enable-dimm](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-enable-dimm) - enable one more dimms * [ndctl-enable-namespace](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-enable-namespace) - enable the given namespace(s) * [ndctl-enable-region](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-enable-region) - enable the given region(s) and all descendant namespaces * [ndctl-freeze-security](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-freeze-security) - Set the given DIMM(s) to reject future security operations * [ndctl-init-labels](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-init-labels) - initialize the label data area on a dimm or set of dimms * [ndctl-inject-error](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-inject-error) - inject media errors at a namespace offset * [ndctl-inject-smart](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-inject-smart) - perform smart threshold/injection operations on an NVDIMM * [ndctl-list](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-list) - print the platform nvdimm device topology and attributes * [ndctl-load-keys](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-load-keys) - load the master key (kek) and encrypted passphrases into the keyring * [ndctl-monitor](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-monitor) - Monitor the SMART events from NVDIMMs * [ndctl-read-labels](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-read-labels) - read out the label area on a NVDIMM or set of NVDIMMs * [ndctl-remove-passphrase](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-remove-passphrase) - Stop an NVDIMM from locking at power-loss and requiring a passphrase to access media * [ndctl-sanitize-dimm](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-sanitize-dimm) - Perform a cryptographic destruction or overwrite of the contents of the given NVDIMM(s) * [ndctl-setup-passphrase](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-setup-passphrase) - setup and enable the security passphrase for one or more NVDIMMs * [ndctl-start-scrub](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-start-scrub) - start an Address Range Scrub (ARS) operation * [ndctl-update-firmware](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-update-firmware) - provides for updating the firmware on an NVDIMM * [ndctl-update-passphrase](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-update-passphrase) - update the security passphrase for one or more NVDIMMs * [ndctl-wait-overwrite](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-wait-overwrite) - wait for an overwrite operation to complete * [ndctl-wait-scrub](/ndctl-user-guide/v67/ndctl-man-pages/untitled-2) - wait for an Address Range Scrub (ARS) operation to complete * [ndctl-write-labels](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-write-labels) - write data to the label area on a dimm * [ndctl-zero-labels](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-zero-labels) - zero out the label area on a dimm or set of dimms # ndctl(1) [NAME](/ndctl-user-guide/v67/ndctl-man-pages/untitled#name)\ [SYNOPSIS](/ndctl-user-guide/v67/ndctl-man-pages/untitled#synopsis)\ [OPTIONS](/ndctl-user-guide/v67/ndctl-man-pages/untitled#options)\ [DESCRIPTION](/ndctl-user-guide/v67/ndctl-man-pages/untitled#description)\ [COPYRIGHT](/ndctl-user-guide/v67/ndctl-man-pages/untitled#copyright)\ [SEE ALSO](/ndctl-user-guide/v67/ndctl-man-pages/untitled#see-also) ## NAME ndctl − Manage "libnvdimm" subsystem devices (Non−volatile Memory) ## SYNOPSIS *ndctl* \[−−version] \[−−help] \[OPTIONS] COMMAND \[ARGS] ## OPTIONS −v, −−version Display ndctl version. −h, −−help Run ndctl help command. ## DESCRIPTION ndctl is utility for managing the "libnvdimm" kernel subsystem. The "libnvdimm" subsystem defines a kernel device model and control message interface for platform NVDIMM resources like those defined by the ACPI 6.0 NFIT (NVDIMM Firmware Interface Table). Operations supported by the tool include, provisioning capacity (namespaces), as well as enumerating/enabling/disabling the devices (dimms, regions, namespaces) associated with an NVDIMM bus. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−create−namespace(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-create-namespace), [ndctl−destroy−namespace(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-destroy-namespace), [ndctl−check−namespace(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace), [ndctl−enable−region(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-enable-region), [ndctl−disable−region(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-disable-region), [ndctl−enable−dimm(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-enable-dimm), [ndctl−disable−dimm(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-disable-dimm), [ndctl−enable−namespace(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-enable-namespace), [ndctl−disable−namespace(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-disable-namespace), [ndctl−zero−labels(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-zero-labels), [ndctl−read−labels(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-read-labels), [ndctl−inject−error(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-inject-error), [ndctl−list(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-list), *LIBNVDIMM* Overview" , *NVDIMM Driver* Writer’s Guide" # ndctl−check−labels(1) [NAME](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels#description)\ [OPTIONS](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels#options)\ [COPYRIGHT](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels#copyright)\ [SEE ALSO](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels#see-also) ## NAME ndctl−check−labels − determine if the given dimms have a valid namespace index block ## SYNOPSIS *ndctl check−labels* \ \[\..\] \[\] ## DESCRIPTION The namespace label area is a small persistent partition of capacity available on some NVDIMM devices. The label area is used to resolve aliasing between *pmem* and *blk* capacity by delineating namespace boundaries. In addition to checking if a label area has a valid index block, running this command in verbose mode reports the reason the index block is deemed invalid. ## OPTIONS \ One or more *nmemX* device names. The keyword *all* can be specified to operate on every dimm in the system, optionally filtered by bus id (see −−bus= option). −s, −−size= Limit the operation to the given number of bytes. A size of 0 indicates to operate over the entire label capacity. −O, −−offset= Begin the operation at the given offset into the label area. −b, −−bus= Limit operation to memory devices (dimms) that are on the given bus. Where *bus* can be a provider name or a bus id number. −v Turn on verbose debug messages in the library (if ndctl was built with logging and debug enabled). ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO *UEFI NVDIMM Label Protocol* # ndctl−check−namespace(1) [NAME](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-labels#name)\ [SYNOPSIS](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace#synopsis)\ [DESCRIPTION](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace#description)\ [EXAMPLES](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace#examples)\ [OPTIONS](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace#options)\ [COPYRIGHT](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace#copyright)\ [SEE ALSO](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-check-namespace#see-also) ## NAME ndctl−check−namespace − check namespace metadata consistency ## SYNOPSIS *ndctl check−namespace* \ \[\] ## DESCRIPTION A namespace in the *sector* mode will have metadata on it to describe the kernel BTT (Block Translation Table). The check−namespace command can be used to check the consistency of this metadata, and optionally, also attempt to repair it, if it has enough information to do so. The namespace being checked has to be disabled before initiating a check on it as a precautionary measure. The `−−force` option can override this. ## EXAMPLES Check a namespace (only report errors) ``` ndctl disable−namespace namespace0.0 ndctl check−namespace namespace0.0 ``` Check a namespace, and perform repairs if possible ``` ndctl disable−namespace namespace0.0 ndctl check−namespace −−repair namespace0.0 ``` ## OPTIONS −R, −−repair Perform metadata repairs if possible. Without this option, the raw namespace contents will not be touched. −L, −−rewrite−log Regenerate the BTT log and write it to media. This can be used to convert from the old (pre 4.15) padding format that was incompatible with other BTT implementations to the updated format. This requires the `−−repair` option to be provided. WARNING: Do not interrupt this operation as it can potentially cause unrecoverable metadata corruption. It is highly recommended to create a backup of the raw namespace before attempting this. −f, −−force Unless this option is specified, a check−namespace operation will fail if the namespace is presently active. Specifying `−−force` causes the namespace to be disabled before checking. −v, −−verbose Emit debug messages for the namespace check process. −r, −−region= A *regionX* device name, or a region id number. The keyword *all* can be specified to carry out the operation on every region in the system, optionally filtered by bus id (see `−−bus=` option). −b, −−bus= Enforce that the operation only be carried on devices that are attached to the given bus. Where *bus* can be a provider name or a bus id number. ## COPYRIGHT Copyright (c) 2016 − 2019, Intel Corporation. License GPLv2: GNU GPL version 2 . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ## SEE ALSO [ndctl−disable−namespace(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-disable-namespace), [ndctl−enable−namespace(1)](/ndctl-user-guide/v67/ndctl-man-pages/ndctl-enable-namespace), *UEFI NVDIMM Label Protocol* --- [Next Page](https://docs.pmem.io/ndctl-user-guide/llms-full.txt/1)