Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LittleFS corruption problem #16

Closed
arturkow2000 opened this issue Apr 5, 2022 · 0 comments · Fixed by #24 or #29
Closed

LittleFS corruption problem #16

arturkow2000 opened this issue Apr 5, 2022 · 0 comments · Fixed by #24 or #29

Comments

@arturkow2000
Copy link

Hello,
I've got problem with LittleFS corrupting itself when creating directories.
I want to create following directory structure

/
│
├── fobnail_client
│   └── dat
│       └── cert
│           └── Fobnail
│               └── 7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375
└── trussed

7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375 is a file
with 855 bytes of data. It has one xattr attached: ID 0, data 0x01 (1 byte).
At that point creating trussed directory will corrupt filesystem.

Test described below have been done on latest LittleFS release (v2.4.2,
bindings updated, trussed-dev/littlefs2-sys#4), however this
problem is present with latest binding release too. I was unable to reproduce
this issue when using raw C API.

Reproduction:

  • take lfs tool
    (from littlefs_fix branch for latest LittleFS version)

  • build it, doesn't make difference whether it is debug or release

  • run the following commands

    dd if=/dev/zero bs=4096 count=32 of=flash.bin
lfs -f flash.bin --format install-certificate bad_cert.pem --trusted
lfs -f flash.bin mkdir /trussed <-- thats where filesystem becomes corrupted
                                    further mount attempts will fail
lfs -f flash.bin dir /

  • whether corruption happens depends certificate used - it is 100% reproducible
    for bad_cert.pem, so far never happened with good_cert.pem. Both
    certificates have the same size, but contents differ which affects name of
    created file (hash of file contents).

  • for the problem to occur certificate installation must be done in one run
    which suggests corruption of in-memory state. If steps are split each into
    separate commands it works:

    lfs -f flash.bin --format mkdir /fobnail_client
lfs -f flash.bin mkdir /fobnail_client/dat
lfs -f flash.bin mkdir /fobnail_client/dat/cert
lfs -f flash.bin mkdir /fobnail_client/dat/cert/Fobnail
lfs -f flash.bin copy-to ../bad_cert.der /fobnail_client/dat/cert/Fobnail/7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375
lfs -f flash.bin set-attr /fobnail_client/dat/cert/Fobnail/7415985b61e40e9ef347b1617e4c52161b657cdbbc15cbd4566f4f18f3911375 0 1
lfs -f flash.bin mkdir /trussed
lfs -f flash.bin dir /

  • Problem does not occur if --trusted flag is omitted from
    install-certificate command, even for bad certificate. Removing --trusted
    flag skips set-attr stage. Looks like the problem is triggered by writing to
    a file and immediatelly setting custom attribute on that file.

  • This is the dump of flash_bin_cert_rust.bin (using readtree.py script from
    littlefs repo) with bad certificate installed, and marked as trusted.

    littlefs v2.0                                  data (truncated, if it fits)
gstate 0x000000000000000000000000
dir "/"
mdir {0x1, 0x0} rev 2 (was 1) -> {0x1e, 0x1f}
  off       tag       type            id  len  data (truncated)
  00000008: 0ff00008  superblock       0    8  6c 69 74 74 6c 65 66 73  littlefs
  00000014: 20100018  inlinestruct     0   24  00 00 02 00 00 10 00 00  ........
  0000003c: 0020040e  dir              1   14  66 6f 62 6e 61 69 6c 5f  fobnail_
  0000004e: 20000408  dirstruct        1    8  1e 00 00 00 1f 00 00 00  ........
  0000005a: 600ffc08  softtail         .    8  1e 00 00 00 1f 00 00 00  ........
dir "/fobnail_client"
mdir {0x1e, 0x1f} rev 1 (was 0) -> {0x2, 0x3}
  off       tag       type            id  len  data (truncated)
  00000014: 00200003  dir              0    3  64 61 74                 dat
  0000001b: 20000008  dirstruct        0    8  02 00 00 00 03 00 00 00  ........
  00000027: 600ffc08  softtail         .    8  02 00 00 00 03 00 00 00  ........
dir "/fobnail_client/dat"
mdir {0x2, 0x3} rev 1 (was 0) -> {0x4, 0x5}
  off       tag       type            id  len  data (truncated)
  00000014: 00200004  dir              0    4  63 65 72 74              cert
  0000001c: 20000008  dirstruct        0    8  04 00 00 00 05 00 00 00  ........
  00000028: 600ffc08  softtail         .    8  04 00 00 00 05 00 00 00  ........
dir "/fobnail_client/dat/cert"
mdir {0x4, 0x5} rev 1 (was 0) -> {0x6, 0x7}
  off       tag       type            id  len  data (truncated)
  00000014: 00200007  dir              0    7  46 6f 62 6e 61 69 6c     Fobnail
  0000001f: 20000008  dirstruct        0    8  06 00 00 00 07 00 00 00  ........
  0000002b: 600ffc08  softtail         .    8  06 00 00 00 07 00 00 00  ........
dir "/fobnail_client/dat/cert/Fobnail"
mdir {0x6, 0x7} rev 1 (was 0)
  off       tag       type            id  len  data (truncated)
  00000014: 00100040  reg              0   64  37 34 31 35 39 38 35 62  7415985b
  00000064: 20200008  ctzstruct        0    8  08 00 00 00 57 03 00 00  ....W...
  00000078: 30000001  userattr 0x0     0    1  01                       .

  • This is what happens after creating trussed directory.

    littlefs v2.0                                  data (truncated, if it fits)
gstate 0x000000000000000000000000
dir "/"
mdir {0x1, 0x0} rev 2 (was 1) -> {0x1f, 0x2}
  off       tag       type            id  len  data (truncated)
  00000008: 0ff00008  superblock       0    8  6c 69 74 74 6c 65 66 73  littlefs
  00000014: 20100018  inlinestruct     0   24  00 00 02 00 00 10 00 00  ........
  0000003c: 0020040e  dir              1   14  66 6f 62 6e 61 69 6c 5f  fobnail_
  0000004e: 20000408  dirstruct        1    8  1e 00 00 00 1f 00 00 00  ........
  00000074: 00200807  dir              2    7  74 72 75 73 73 65 64     trussed
  0000007f: 20000808  dirstruct        2    8  1f 00 00 00 02 00 00 00  ........
  0000008b: 600ffc08  softtail         .    8  1f 00 00 00 02 00 00 00  ........
dir "/trussed"
mdir {0x1f, 0x2} rev 2 (was 1) -> {0x1e, 0x1f}
  off       tag       type            id  len  data (truncated)
  00000008: 600ffc08  softtail         .    8  1e 00 00 00 1f 00 00 00  ........
dir "/fobnail_client"
mdir {0x1f, 0x1e} rev 2 (was 1) -> {0x1e, 0x1f}
  off       tag       type            id  len  data (truncated)
  00000008: 600ffc08  softtail         .    8  1e 00 00 00 1f 00 00 00  ........
*** cycle detected {0x1f, 0x1e}! ***

  • This problem occurs only when marking certificate as trusted and this is the
    difference beetween image with trusted certificate (corrupted) and untrusted
    certificate

    --- hex.bad_untrusted	2022-04-05 15:11:02.089691364 +0200
+++ hex.bad	2022-04-05 15:10:52.585639558 +0200
@@ -1541,8 +1541,8 @@
 00006040  63 62 64 34 35 36 36 66  34 66 31 38 66 33 39 31  |cbd4566f4f18f391|
 00006050  31 33 37 35 20 00 00 40  70 1f fc 04 70 aa e6 c3  |1375 ..@p...p...|
 00006060  70 2f fc 0c 08 00 00 00  57 03 00 00 70 2f fc 0c  |p/......W...p/..|
-00006070  a6 5b aa 48 ff ff ff ff  ff ff ff ff ff ff ff ff  |.[.H............|
-00006080  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
+00006070  a6 5b aa 48 60 0f fc 05  01 60 0f fc 06 4d 2b d6  |.[.H`....`...M+.|
+00006080  33 ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |3...............|
 00006090  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
 000060a0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
 000060b0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

  • same thing from readtree.py perspective

    --- untrusted	2022-04-05 15:26:12.690029226 +0200
+++ trusted	2022-04-05 15:26:19.502059088 +0200
@@ -76,10 +76,12 @@
   0000005c: 500ffc04  crc 0x0          .    4  70 aa e6 c3              p...
   00000064: 20200008  ctzstruct        0    8  08 00 00 00 57 03 00 00  ....W...
   00000070: 500ffc04  crc 0x0          .    4  a6 5b aa 48              .[.H
-  00000078: aff003fb  struct 0xff      0 1019  ff ff ff ff ff ff ff ff  ........
-  00000477: 500ffc04  crc (bad)        .    4  ff ff ff ff              ....
-  0000047f: aff003fb  struct 0xff      0 1019  ff ff ff ff ff ff ff ff  ........
-  0000087e: 500ffc04  crc (bad)        .    4  ff ff ff ff              ....
-  00000886: aff003fb  struct 0xff      0 1019  ff ff ff ff ff ff ff ff  ........
-  00000c85: 500ffc04  crc (bad)        .    4  ff ff ff ff              ....
-  00000c8d: aff003fb  struct 0xff      0 1019  ff ff ff ff ff ff ff ff  ........
+  00000078: 30000001  userattr 0x0     0    1  01                       .
+  0000007d: 500ffc07  crc 0x0          .    7  4d 2b d6 33 ff ff ff     M+.3...
+  00000088: aff003f8  struct 0xff      0 1016  ff ff ff ff ff ff ff ff  ........
+  00000484: 500ffc07  crc (bad)        .    7  ff ff ff ff ff ff ff     .......
+  0000048f: aff003f8  struct 0xff      0 1016  ff ff ff ff ff ff ff ff  ........
+  0000088b: 500ffc07  crc (bad)        .    7  ff ff ff ff ff ff ff     .......
+  00000896: aff003f8  struct 0xff      0 1016  ff ff ff ff ff ff ff ff  ........
+  00000c92: 500ffc07  crc (bad)        .    7  ff ff ff ff ff ff ff     .......
+  00000c9d: aff003f8  struct 0xff      0 1016  ff ff ff ff ff ff ff ff  ........

Looks like presence of user attribute causes mkdir to corrupt filesystem, what
puzzles me more is the connection with file contents (corruption doesn't happen
with "good" certificate). Also there are some differences in behaviour between C
and Rust - Rust code starts block allocation from the last 2 blocks (except root
directory which is always at {1, 0}), fobnail_client directory is allocated at
{30, 31} in case of Rust and {10, 11} in case of C. Shouldn't LittleFS allocate
the same block?

I upload here both good and bad certificates as well as various images: with
trusted and untrusted bad certificate, example of corrupted image and C code
(which does the same but without corrupting fs).
lfs_c.zip
images.zip
@arturkow2000 arturkow2000 mentioned this issue Jun 30, 2022
Closed
robin-nitrokey added a commit to Nitrokey/littlefs2 that referenced this issue Jan 26, 2023
@robin-nitrokey
Fix lookahead buffer size reported to littlefs2-sys
1a0b649 
Previously, we reported the lookahead buffer size in bytes but
littlefs2-sys expects the lookahead buffer size as a multiple of 8
bytes.  This could lead to a buffer overflow causing filesystem
corruption.  This patch fixes the reported lookahead buffer size.

Note that Storage::LOOKAHEAD_WORDS_SIZE allows users to set invalid
values (as it is measured in 4 bytes, not in 8 bytes).  Invalid values
that were previously accepted because of the wrong buffer size
calculation can now be rejected by littlefs2-sys.

This is a combination of two previous patches:
	trussed-dev#19
	#1

Fixes: trussed-dev#16
@robin-nitrokey robin-nitrokey mentioned this issue Jan 26, 2023
Merged
robin-nitrokey added a commit to Nitrokey/littlefs2 that referenced this issue Jan 27, 2023
@robin-nitrokey
Fix lookahead buffer size reported to littlefs2-sys
694c6a8 
Previously, we reported the lookahead buffer size in bytes but
littlefs2-sys expects the lookahead buffer size as a multiple of 8
bytes.  This could lead to a buffer overflow causing filesystem
corruption.  This patch fixes the reported lookahead buffer size.

Note that Storage::LOOKAHEAD_WORDS_SIZE allows users to set invalid
values (as it is measured in 4 bytes, not in 8 bytes).  Invalid values
that were previously accepted because of the wrong buffer size
calculation can now be rejected by littlefs2-sys.

This is a combination of two previous patches:
	trussed-dev#19
	#1

Fixes: trussed-dev#16
robin-nitrokey added a commit to Nitrokey/littlefs2 that referenced this issue Feb 3, 2023
@robin-nitrokey
Fix lookahead buffer size reported to littlefs2-sys
cc79291 
Previously, we reported the lookahead buffer size in bytes but
littlefs2-sys expects the lookahead buffer size as a multiple of 8
bytes.  This could lead to a buffer overflow causing filesystem
corruption.  This patch fixes the reported lookahead buffer size.

Note that Storage::LOOKAHEAD_WORDS_SIZE allows users to set invalid
values (as it is measured in 4 bytes, not in 8 bytes).  Invalid values
that were previously accepted because of the wrong buffer size
calculation can now be rejected by littlefs2-sys.

This is a combination of two previous patches:
	trussed-dev#19
	#1

Fixes: trussed-dev#16
robin-nitrokey added a commit to Nitrokey/littlefs2 that referenced this issue Feb 7, 2023
@robin-nitrokey
Fix lookahead buffer size reported to littlefs2-sys
2c7bbad 
Previously, we reported the lookahead buffer size in bytes but
littlefs2-sys expects the lookahead buffer size as a multiple of 8
bytes.  This could lead to a buffer overflow causing filesystem
corruption.  This patch fixes the reported lookahead buffer size.

Note that Storage::LOOKAHEAD_WORDS_SIZE allows users to set invalid
values (as it is measured in 4 bytes, not in 8 bytes).  Invalid values
that were previously accepted because of the wrong buffer size
calculation can now be rejected by littlefs2-sys.

This is a combination of two previous patches:
	trussed-dev#19
	#1

Fixes: trussed-dev#16
robin-nitrokey added a commit to Nitrokey/littlefs2 that referenced this issue Feb 7, 2023
@robin-nitrokey
Fix lookahead buffer size reported to littlefs2-sys
82ac3eb 
Previously, we reported the lookahead buffer size in bytes but
littlefs2-sys expects the lookahead buffer size as a multiple of 8
bytes.  This could lead to a buffer overflow causing filesystem
corruption.  This patch fixes the reported lookahead buffer size.

Note that Storage::LOOKAHEAD_WORDS_SIZE allows users to set invalid
values (as it is measured in 4 bytes, not in 8 bytes).  Invalid values
that were previously accepted because of the wrong buffer size
calculation can now be rejected by littlefs2-sys.

This is a combination of two previous patches:
	trussed-dev#19
	#1

Fixes: trussed-dev#16
robin-nitrokey added a commit that referenced this issue Feb 7, 2023
@robin-nitrokey
Fix lookahead buffer size reported to littlefs2-sys
aafa02e 
Previously, we reported the lookahead buffer size in bytes but
littlefs2-sys expects the lookahead buffer size as a multiple of 8
bytes.  This could lead to a buffer overflow causing filesystem
corruption.  This patch fixes the reported lookahead buffer size.

Note that Storage::LOOKAHEAD_WORDS_SIZE allows users to set invalid
values (as it is measured in 4 bytes, not in 8 bytes).  Invalid values
that were previously accepted because of the wrong buffer size
calculation can now be rejected by littlefs2-sys.

This is a combination of two previous patches:
	#19
	Nitrokey#1

Fixes: #16
@robin-nitrokey robin-nitrokey mentioned this issue Feb 7, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@arturkow2000