Difference between revisions of "UBIFS (Unsorted Block Image File System)"

From Bauman National Library
This page was last modified on 26 May 2017, at 12:23.
(Mounting empty volumes)
 
Line 209: Line 209:
 
An example of a video on Debian Jessie of mounting an empty ubi volume on an mtd device. Important note: Sometimes it is impossible to mount device to the folder, because its size is too small. As shown in the example, you need to allocate a partition with a large amount of memory.
 
An example of a video on Debian Jessie of mounting an empty ubi volume on an mtd device. Important note: Sometimes it is impossible to mount device to the folder, because its size is too small. As shown in the example, you need to allocate a partition with a large amount of memory.
  
<center><youtube>Link</youtube></center>
+
<center><youtube>X-iewPtiMyw</youtube></center>
  
 
==Why we need to use ubiformat==
 
==Why we need to use ubiformat==

Latest revision as of 12:23, 26 May 2017

UBIFS (Unsorted Block Image File System)
Developer(s) Nokia with help of University of Szeged
Full name Unsorted Block Image File System
Introduced 2008 with Linux kernel 2.6.27
Structures
Directory contents B+ trees
Limits
Allowed characters in filenames Any byte except NUL and forward slash "/"Template:Fix/category[citation needed]]]
Features
Forks Yes
Attributes Yes
File system permissions POSIX
Other
Supported operating systems Linux

Unsorted Block Image File System (UBIFS) — a new flash file system developed by Nokia engineers with help of the University of Szeged[1]. A file system for flash which works on top of UBI[2] over MTD[3]. Considered as the next generation of JFFS2. Mainlined since kernel version 2.6.27, 17 July 2008.[4]

Two major differences between UBIFS and JFFS2 are that UBIFS supports write caching,[5] and UBIFS errs on the pessimistic side of free space calculation.[6] UBIFS tends to perform better than JFFS2 for large NAND FLASH devices.[7] This is a consequence of the UBIFS design goals:[8] faster mounting, quicker access to large files, and improved write speeds. UBIFS also preserves or improves upon JFFS2's on-the-fly compression, recoverability and power fail tolerance.[8] UBIFS's on-the-fly data compression allows zlib (deflate algorithm) or LZO.[source 1]

JFFS2 stores filesystem indexes in memory whereas UBIFS stores indexes in flash.[9] This directly impacts the scalability of JFFS2 as the tables must be rebuilt every time the volume is mounted. Also, the JFFS2 tables may consume enough system RAM that some images may be unusable.

UBI

Properties of different file systems

UBI (Unsorted Block Images) is an erase block management layer for flash memory devices. UBI serves two purposes, tracking NAND flash bad blocks and providing wear leveling. Wear leveling spreads the erases and writes across the entire flash device. UBI presents logical erase blocks to higher layers and maps these to physical flash erase blocks. UBI was written specifically for UBIFS so that it does not have to deal with wear leveling and bad blocks. However, UBI may also be useful with squashfs and NAND flash; squashfs is not aware of NAND flash bad blocks.[source 1]

UBI's documentation explains that it is not a complete flash translation layer (FTL). Although a FTL also handles bad blocks and wear leveling, the interface a FTL provides is a block device with small (typically 512 byte) sectors that can be written completely independently. In contrast, UBI's interface directly exposes erase blocks and programmable pages (which are different sizes, and much larger than typical block device sectors), and filesystems that use UBI must be aware of the sizes and restrictions on how blocks must be erased before being written.[source 1]

Fastmap

UBI was augmented in Linux 3.7 with fastmap support.[10][11] Fastmap maintains an on-disk version of information previously created in memory by scanning the entire flash device. The code falls back to the previous mechanism of a full scan on failures and older UBI systems will simply ignore the fastmap information.[source 1]


Features

  • scalability - UBIFS scales well with respect to flash size; namely, mount time, memory consumption and I/O speed does not depend on flash size (currently it is not 100% true for memory consumption, but the dependency is very weak, and this may be fixed); UBIFS (not UBI!) should work fine for hundreds of GiB flashes; however, UBIFS depends on UBI which has scalability limitations (see here); nonetheless, the UBI/UBIFS stack scales much better than JFFS2, and if UBI becomes a bottleneck, it is always possible to implement UBI2 without changing UBIFS;
  • fast mount - unlike JFFS2, UBIFS does not have to scan whole media when mounting, it takes milliseconds for UBIFS to mount the media, and this does not depend on flash size; however, UBI initialization time depends on flash size and has to be taken into account (see here for more details);
  • write-back support - this dramatically improves the throughput of the file system in many workloads, comparing to JFFS2, which is write-through; see here for more details;
  • tolerance to unclean reboots - UBIFS is a journaling file system and it tolerates sudden crashes and unclean reboots; UBIFS just replays the journal and recovers from the unclean reboot; mount time is a little bit slower in this case, because of the need to replay the journal, but UBIFS does not need to scan whole media, so it anyway takes fractions of a second to mount UBIFS; note, authors payed special attention to this UBIFS aspect, see here;
  • fast I/O - even with write-back disabled (e.g., if UBIFS is mounted with the "-o sync" mount option) UBIFS shows good performance which is close to JFFS2 performance; bear in mind, it is extremely difficult to compete with JFFS2 in synchronous I/O, because JFFS2 does not maintain indexing data structures on flash, so it does not have the maintenance overhead, while UBIFS does have it; but UBIFS is still fast because of the way UBIFS commits the journal - it does not move the data physically from one place to another but instead, it just adds corresponding information to the file system index and picks different eraseblocks for the new journal (i.e., UBIFS has sort of "wandering" journal which constantly changes the position); there are other tricks like multi-headed journal which make UBIFS perform well;
  • on-the-flight compression - the data are stored in compressed form on the flash media, which makes it possible to put considerably more data to the flash than if the data were not compressed; this is very similar to what JFFS2 has; UBIFS also allows to switch the compression on/off on per-inode basis, which is very flexible; for example, one may switch the compression off by default and enable it only for certain files which are supposed to compress well; or one may switch compression on by default but disable it for supposedly uncompressible data like multimedia files; at the moment UBIFS supports only zlib and LZO compressors and it is not difficult to add more; see this section for more information.
  • recoverability - UBIFS may be fully recovered if the indexing information gets corrupted; each piece of information in UBIFS has a header which describes this piece of information, and it is possible to fully reconstruct the file system index by scanning the flash media; this is very similar to JFFS2; to make it more clear, imaging you have wiped out the FAT table on your FAT file system; for FAT FS this would be fatal; but if you similarly wipe out UBIFS index, you still may re-construct it, although a special user-space tool would be required to do this (this utility is not implemented at the moment, though);
  • integrity - UBIFS (as well as UBI) checksums everything it writes to the flash media to guarantee data integrity, UBIFS does not leave data or meta-data corruptions unnoticed (JFFS2 is doing the same); By default, UBIFS checks only meta-data CRC when it reads from the media, but not the data CRC; however, you may force CRC checking for data using one of the UBIFS mount options

Setting

To begin need enable UBI and UBIFS:

In the Linux configuration menu, go to "Device Drivers" -> "Memory Technology Devices (MTD)" -> "UBI - Unsorted block images", and mark the "Enable UBI" check-box. UBI may be either compiled into the kernel or be built as a kernel module.

Then in the Linux configuration menu, go to "File systems" -> "Miscellaneous filesystems", and mark the "UBIFS file system support" check-box. UBIFS may be either compiled into the kernel or be built as a kernel module.[source 2]

Lets apt-get update packages:

sudo apt-get update 

Then we need to download and install mtd-utils:

sudo apt-get install mtd-utils 

Operations UBIFS

Mounting UBIFS

For mounting UBIFS on UBI volume:

sudo mount -t ubifs /dev/ubi0_0 /mnt/ubifs 

For example:

sudo mount -t ubifs ubi1_0 /mnt/ubifs 

mounts volume 0 on UBI device 1 to /mnt/ubifs, and

sudo mount -t ubifs ubi0:rootfs /mnt/ubifs 

mounts "rootfs" volume of UBI device 0 to /mnt/ubifs ("rootfs" is volume name). This method of specifying UBI volume is more preferable because it does not depend on volume number.

Creating UBIFS image

Creating UBIFS images might be a little trickier than creating JFFS2 images. First of all, you have to understand that UBIFS works on top of UBI which works on top or MTD which basically represents your raw flash. This means, that if you need to create an image which should be flashed to the raw flash, you should first create an UBIFS image, then UBI image. In other words, the process has 2 steps.[source 2]

So, there are 2 utilities:

  • mkfs.ubifs which creates UBIFS images;
  • ubinize which creates UBI images out of UBIFS images.

For example:

sudo mkfs.ubifs -q -r root-fs -m 2048 -e 129024 -c 2047 -o ubifs.img 
sudo ubinize -o ubi.img -m 2048 -p 128KiB -s 512 ubinize.cfg 

where ubinize.cfg contains:

[ubifs]

mode=ubi

image=ubifs.img

vol_id=0

vol_size=200MiB

vol_type=dynamic

vol_name=rootfs

vol_flags=autoresize


Some comments about what the options mean:

  • -r root-fs: tells mkfs.ubifs to create an UBIFS image which would have identical contents as the local root-fs directory;
  • -m 2048: tells mkfs.ubifs that the minimum input/output unit size of the flash this UBIFS image is created for is 2048 bytes (NAND page in this case);
  • -e 129024: logical eraseblock size of the UBI volume this image is created for;
  • -c 2047: specifies maximum file-system size in logical eraseblocks; this means that it will be possible to use the resulting file-system on volumes up to this size (less or equivalent); so in this particular case, the resulting FS may be put on volumes up to about 251MiB (129024 multiplied by 2047); See this section for more details.
  • -p 128KiB: tells ubinize that physical eraseblock size of the flash chip the UBI image is created for is 128KiB (128 * 1024 bytes);
  • -s 512: tells ubinize that the flash supports sub-pages and sub-page size is 512 bytes; ubinize will take this into account and put the VID header to the same NAND page as the EC header.

Mounting empty volumes

For mounting empty UBI volumes on MTD device we need to wipe it out. We could use flash_eraseall, but we do not want to lose erase counters:

sudo ubiformat /dev/mtd0 

Load UBI module:

sudo modprobe ubi 

Attach mtd0 to UBI - UBI will detect that the MTD device is empty and automatically format it. This command will also create UBI device 0 and udev should create /dev/ubi0 node:

sudo ubiattach /dev/ubi_ctrl -m 0 

Create an UBI volume - the created volume will be empty:

sudo ubimkvol /dev/ubi0 -N test_volume -s 10MiB 

Mount UBIFS - it will automatically format the empty volume:

sudo mount -t ubifs ubi0:0 /mnt/ubifs 

It is also possible to wipe out an existing UBIFS volume represented by /dev/ubi0_0 using the following command:

sudo ubiupdatevol /dev/ubi0_0 -t 

An example of a video on Debian Jessie of mounting an empty ubi volume on an mtd device. Important note: Sometimes it is impossible to mount device to the folder, because its size is too small. As shown in the example, you need to allocate a partition with a large amount of memory.

Why we need to use ubiformat

The first obvious reason is that ubiformat preserves erase counters, so you do not lose your wear-leveling information when flashing new images.[source 2]

The other reason is more subtle, and specific to NAND flashes which have ECC calculation algorithm which produces ECC code not equivalent to all 0xFF bytes if the NAND page contains only 0xFF bytes. Consider an example.

  • We erase whole flash, so everything is 0xFF'ed now.
  • We write an UBI/UBIFS image to flash using nandwrite.
  • Some eraseblocks in the UBIFS image may contain several empty NAND pages at the end, and UBIFS will write to them when it is run.
  • The nandwrite utility writes whole image, and it explicitly writes 0xFF bytes to those NAND pages.
  • The ECC checksums are calculated for these 0xFF'ed NAND pages and are stored in the OOB area. The ECC codes are not 0xFF'ed. This is often the case for HW ECC calculation engines, and it is difficult to fix this. Normally, ECC codes should be 0xFF'ed for such pages.
  • When later UBIFS runs, it writes data to these NAND pages, which means that a new ECC code is calculated, and written on top of the existing one (unsuccessfully, of course). This may trigger an error straight away, but usually at this point no error is triggered.
  • At some point UBIFS is trying to read from these pages, and gets and an ECC error (-EBADMSG = -74).

In fewer words, ubiformat makes sure that every NAND page is written once and only once after the erasure. If you use nandwrite, some pages are written twice - once by nandwrite, and once by UBIFS.

How to compile mkfs.ubifs

The mkfs.ubifs utility requires zlib, lzo and uuid libraries. The former two are used for compressing the data, and the latter one is used for generating universally unique ID number for the file-system. In Fedora install zlib-devel, lzo-devel, and libuuid-devel. Old Fedora distributions (Fedora 11 and earlier) had the uuid library in the e2fsprogs-devel package. In Debian install zlib1g-dev, liblzo2-dev and uuid-dev packages.[source 2]

Does UBIFS become slower when it is full?

Yes, UBIFS writes (but not reads) become slower when it is full or close to be full. There are 2 main reasons for this:

  • Garbage Collection becomes slower, because the small pieces of dirty space are distributed all over the media, and UBIFS Garbage Collector has to move many eraseblocks to produce one free eraseblock; this is a fundamental reason and it exists in JFFS2 as well;
  • The file-system becomes more synchronous; UBIFS buffers dirty data, but due to compression and some other factors like wasting small pieces of space at the end of eraseblocks, UBIFS does not exactly know how much space the buffered dirty data would take on the flash media, so it uses pessimistic calculations and assumes that the data are uncompressible. In many cases this is not true, but UBIFS has to assume worst-case scenario. So when all free space on the file-system is reserved for the buffered dirty data, but users want to write more, UBIFS forces write-back to actually write the buffered dirty data and see how much space will be available after that. Then UBIFS may allow new data to be written. Thus, it is obvious that the less free flash space is available, the less dirty data may be buffered, and the more synchronous the file-system becomes.

How to extract files from an UBI/UBIFS image

Unfortunately, at the moment there are no user-space tools which can unwrap UBI and UBIFS images. UBIFS cannot be loop-back mounted either, because it does not work with block devices.

However, kernel modules exist that allow you to create a virtual MTD onto which UBIFS can be mounted. You have two options:

  • nandsim, which can simulate various NAND devices. You can find an incomplete list of those devices here[12]. To emulate a given device, pass its ID bytes via the module parameters first_id_byte, second_id_byte, third_id_byte, and fourth_id_byte.
  • mtdram, which simulates a generic MTD. Use the module parameter total_size to specify the device size in KiB and erase_size to specify the erase block size in KiB. This module is useful for mounting UBIFS images made for NOR memory, which usually have a minimum I/O unit of 1 byte, as opposed to NAND devices which have a minimum unit of at least 512 bytes. See more information here[13].

References

Cite error: Invalid <references> tag; parameter "group" is allowed only.

Use <references />, or <references group="..." />

Sources

Cite error: Invalid <references> tag; parameter "group" is allowed only.

Use <references />, or <references group="..." />

Links

  • Artem Bityutskiy (2008-03-27). "UBIFS - new flash file system". Retrieved 2016-12-18. 
  • "UBIFS - UBI File-System: Big red note". 2015-01-22. Retrieved 2016-12-18. It does not work on top of block devices. UBIFS was designed to work on top of raw flash. 
  • "UBIFS - UBI File-System: Overview". 2015-01-22. Retrieved 2016-12-18. 
  • Artem Bityutskiy (2008-03-27). "UBIFS - new flash file system". Retrieved 2016-12-18. 
  • "UBIFS - UBI File-System: Write-back support". 2015-01-22. Retrieved 2016-12-18. 
  • "UBIFS FAQ and HOWTO: Why does df report too little free space?". 2015-12-18. Retrieved 2016-12-18. 
  • "UBIFS - UBI File-System: Scalability". 2015-01-22. Retrieved 2016-12-18. 
  • 8.0 8.1 Bityutskiy, Artem; Hunter, Adrian (2008-09-24). "UBIFS File System" (PDF). p. 9. 
  • Adrian Hunter (2008-03-27). "A Brief Introduction to the Design of UBIFS" (PDF). 
  • Thomas Petazzoni (2012-10-03). "UBI fastmap making its way to mainline". Archived from the original on 2012-11-15. Retrieved 2016-12-18. 
  • Richard Weinberger (2012-09-24). "UBI: Fastmap request for inclusion (v18)". Retrieved 2016-12-18. 
  • "NAND Flash Support Table". 
  • "Minimum flash input/output unit". 

  • Cite error: <ref> tags exist for a group named "source", but no corresponding <references group="source"/> tag was found, or a closing </ref> is missing