How to use the search command?

[chmac]

I have APFS on an external USB HDD. It seems like maybe APFS is only suitable for SSDs, so maybe that was our first mistake. The data container won't mount, but the decryption does work.

I'm trying to understand how to explore the disk to see if the data is still readable.

drat inspect output

sudo drat inspect /dev/disk2s3
Opening file at `/dev/disk2s3` in read-only mode ... OK.
Simulating a mount of the APFS container.
Reading container superblock at address 0x0, assuming default block size of 4096 bytes ... validating checksum ... OK.

Details of block 0x0:
--------------------------------------------------------------------------------
Stored checksum:    0xcfad85069e26a479
OID:                0x1
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               Container superblock
Subtype:            (invalid type / no subtype)
Keybag location: first block 0x5e0080, spans 1 (0x1) blocks (last block 0x5e0080)
Media keybag location: none (spans 0 blocks)
Magic string:       NXSB
Latest version of Apple APFS software that mounted this container: 1525.339.742.117.64
Block size:         4096 bytes
Block count:        683824764 (last block 0x28c2567b)
Supported features:
- No flags.
Supported read-only compatible features:
- No flags.
Backward-incompatible features:
- This container uses APFS version 2, as implemented in macOS 10.13 and iOS 10.3.
UUID:       6B329396-08A9-41B9-8D66-536B805AF3FC
Next OID:                       0xfd1
Next XID:                       0x904
Space manager Ephemeral OID:    0x400
Object map Physical OID:        0x2ec48d
Reaper Ephemeral OID:           0x401
Other flags:
- This container uses software cryptography.
--------------------------------------------------------------------------------

Locating the checkpoint descriptor area:
- Its length is 284 blocks.
- It is contiguous.
- The address of its first block is 0x1.
Loading the checkpoint descriptor area into memory ... OK.
Locating the most recent well-formed container superblock in the checkpoint descriptor area:
- It lies at index 69 within the checkpoint descriptor area.

Details of this container superblock:
--------------------------------------------------------------------------------
Stored checksum:    0xd0d4bca29cff205f
OID:                0x1
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               Container superblock
Subtype:            (invalid type / no subtype)
Keybag location: first block 0x5e0080, spans 1 (0x1) blocks (last block 0x5e0080)
Media keybag location: none (spans 0 blocks)
Magic string:       NXSB
Latest version of Apple APFS software that mounted this container: 1525.339.742.117.64
Block size:         4096 bytes
Block count:        683824764 (last block 0x28c2567b)
Supported features:
- No flags.
Supported read-only compatible features:
- No flags.
Backward-incompatible features:
- This container uses APFS version 2, as implemented in macOS 10.13 and iOS 10.3.
UUID:       6B329396-08A9-41B9-8D66-536B805AF3FC
Next OID:                       0xfd1
Next XID:                       0x904
Space manager Ephemeral OID:    0x400
Object map Physical OID:        0x2ec48d
Reaper Ephemeral OID:           0x401
Other flags:
- This container uses software cryptography.
--------------------------------------------------------------------------------
- The corresponding checkpoint starts at index 68 within the checkpoint descriptor area, and spans 2 blocks.

Loading the corresponding checkpoint ... OK.

Details of each block in this checkpoint:
--------------------------------------------------------------------------------
Stored checksum:    0x8c0abd1333f216d2
OID:                0x45
XID:                0x903
Storage type:       Physical
Type flags:         (no flags)
Type:               Checkpoint map
Subtype:            (invalid type / no subtype)
Flags:
- Last checkpoint-mapping block in the corresponding checkpoint.
Number of mappings: 8
--------------------------------------------------------------------------------
Stored checksum:    0xd0d4bca29cff205f
OID:                0x1
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               Container superblock
Subtype:            (invalid type / no subtype)
Keybag location: first block 0x5e0080, spans 1 (0x1) blocks (last block 0x5e0080)
Media keybag location: none (spans 0 blocks)
Magic string:       NXSB
Latest version of Apple APFS software that mounted this container: 1525.339.742.117.64
Block size:         4096 bytes
Block count:        683824764 (last block 0x28c2567b)
Supported features:
- No flags.
Supported read-only compatible features:
- No flags.
Backward-incompatible features:
- This container uses APFS version 2, as implemented in macOS 10.13 and iOS 10.3.
UUID:       6B329396-08A9-41B9-8D66-536B805AF3FC
Next OID:                       0xfd1
Next XID:                       0x904
Space manager Ephemeral OID:    0x400
Object map Physical OID:        0x2ec48d
Reaper Ephemeral OID:           0x401
Other flags:
- This container uses software cryptography.
--------------------------------------------------------------------------------

Details of each checkpoint-mapping in this checkpoint:
--------------------------------------------------------------------------------
Ephemeral OID:                      0x400
Logical block address on disk:      0x4d4e
Object type:                        Space manager
Object subtype:                     (invalid type / no subtype)
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
Ephemeral OID:                      0x401
Logical block address on disk:      0x4d4f
Object type:                        Container reaper
Object subtype:                     (invalid type / no subtype)
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
Ephemeral OID:                      0x403
Logical block address on disk:      0x4d50
Object type:                        B-tree (root node)
Object subtype:                     Space manager free-space queue
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
Ephemeral OID:                      0x405
Logical block address on disk:      0x4d51
Object type:                        B-tree (root node)
Object subtype:                     Space manager free-space queue
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
Ephemeral OID:                      0x411
Logical block address on disk:      0x4d52
Object type:                        B-tree (non-root) node
Object subtype:                     Space manager free-space queue
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
Ephemeral OID:                      0x412
Logical block address on disk:      0x4d53
Object type:                        Container reaper list
Object subtype:                     (invalid type / no subtype)
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
Ephemeral OID:                      0xfcf
Logical block address on disk:      0x4d54
Object type:                        B-tree (non-root) node
Object subtype:                     Space manager free-space queue
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
Ephemeral OID:                      0xfd0
Logical block address on disk:      0x4d55
Object type:                        B-tree (non-root) node
Object subtype:                     Space manager free-space queue
Object size:                        4096 bytes
Associated volume OID (virtual):    0x0
--------------------------------------------------------------------------------
- There are 8 checkpoint-mappings in this checkpoint.

Reading the Ephemeral objects used by this checkpoint ... OK.
Validating the Ephemeral objects ... OK.

Details of the Ephemeral objects:
--------------------------------------------------------------------------------
Stored checksum:    0xa33332678bf345e6
OID:                0x400
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               Space manager
Subtype:            (invalid type / no subtype)
--------------------------------------------------------------------------------
Stored checksum:    0x00d040c27f2f8a52
OID:                0x401
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               Container reaper
Subtype:            (invalid type / no subtype)
--------------------------------------------------------------------------------
Stored checksum:    0x642cd492eacf1291
OID:                0x403
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               B-tree (root node)
Subtype:            Space manager free-space queue
--------------------------------------------------------------------------------
Stored checksum:    0xe3b6a79087d2099a
OID:                0x405
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               B-tree (root node)
Subtype:            Space manager free-space queue
--------------------------------------------------------------------------------
Stored checksum:    0x40d93f02fbdfd16d
OID:                0x411
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               B-tree (non-root) node
Subtype:            Space manager free-space queue
--------------------------------------------------------------------------------
Stored checksum:    0x1d7d1f06a1f6262f
OID:                0x412
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               Container reaper list
Subtype:            (invalid type / no subtype)
--------------------------------------------------------------------------------
Stored checksum:    0x148ae3733d67dc30
OID:                0xfcf
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               B-tree (non-root) node
Subtype:            Space manager free-space queue
--------------------------------------------------------------------------------
Stored checksum:    0xc2f7a8c299cfaa6c
OID:                0xfd0
XID:                0x903
Storage type:       Ephemeral
Type flags:         (no flags)
Type:               B-tree (non-root) node
Subtype:            Space manager free-space queue
--------------------------------------------------------------------------------

The container superblock states that the container object map has Physical OID 0x2ec48d.
Loading the container object map ... OK.
Validating the container object map ... OK.

Details of the container object map:
--------------------------------------------------------------------------------
Stored checksum:    0xb3b94f408be91eb2
OID:                0x2ec48d
XID:                0x8e1
Storage type:       Physical
Type flags:         (no flags)
Type:               Object map
Subtype:            (invalid type / no subtype)
Flags:
- No snapshot support
Object mappings tree:
- Storage type:         Physical
- Type flags:           (no flags)
- Type:                 B-tree (root node)
- Object ID:            0x2ec48e
Snapshots tree:
- Storage type:         Physical
- Type flags:           (no flags)
- Type:                 B-tree (root node)
- Object ID:            0x0
- Number of snapshots:  0 snapshots
- Latest snapshot XID:  0x0
In-progress revert:
- Minimum XID:          0x0
- Maximum XID:          0x0
--------------------------------------------------------------------------------

Reading the root node of the container object map B-tree ... OK.
Validating the root node of the container object map B-tree ... OK.

Details of the container object map B-tree:
--------------------------------------------------------------------------------
Stored checksum:    0x5215d8735d9b8a83
OID:                0x2ec48e
XID:                0x8e1
Storage type:       Physical
Type flags:         (no flags)
Type:               B-tree (root node)
Subtype:            Object map
Flags:
- Root node
- Leaf node
- Fixed size for keys and values
Number of child levels:         0
Number of keys in this node:    1
Location of table of contents:
- Offset from start of node data area:  0x0 = 0
- Length (bytes):                       0x1c0 = 448
Location of key–value shared free space:
- Offset from start of keys area:       0x20 = 32
- Length (bytes):                       0xda0 = 3488

Info relating to the entire B-tree:
- Flags:
- This B-tree is currently undergoing a series of sequential inserts --- optimise operations if possible
- Child nodes are referred to using Physical OIDs
- Node size:                4096 bytes
- Key size:                 16 bytes
- Value size:               16 bytes

- Length of longest key:    16 bytes
- Length of longest value:  16 bytes
- Number of keys:           1
- Number of nodes:          1
--------------------------------------------------------------------------------

The container superblock lists 1 APFS volumes, whose superblocks have the following Virtual OIDs:
- 0x413

Reading the APFS volume superblocks ... OK.
Validating the APFS volume superblocks ... OK.

Details of these volume superblocks:
--------------------------------------------------------------------------------
Stored checksum:    0x20becaf57b87fe5c
OID:                0x413
XID:                0x8e1
Storage type:       Virtual
Type flags:         (no flags)
Type:               APFS volume
Subtype:            (invalid type / no subtype)

Magic string:                           APSB
Index within container volume array:    0

Volume name:        ### cosi private ###
Role:               (no role)
Flags:
- Single VEK (volume encryption key) for all files in this volume.
Supported features:
- This volume has hardlink map records.
Supported read-only compatible features:
- No flags.
Backward-incompatible features:
- Filenames on this volume are case-insensitive.

Last unmount time:                  Wed Jul 21 14:13:10 2021
Last modification time:             Thu Jul 22 10:44:04 2021

Reserved blocks:                    0 blocks
Block quota:                        0 blocks
Allocated blocks:                   31499097 blocks

Volume object map Physical OID:     0x2ec487

Root tree info:
- OID:              0x414
- Storage type:     Virtual
- Type flags:       (no flags)
- Object type:      B-tree (root node)

Extent-reference tree info:
- OID:              0x2ec287
- Storage type:     Physical
- Type flags:       (no flags)
- Object type:      B-tree (root node)

Snapshot metadata tree info:
- OID:              0x2ec25c
- Storage type:     Physical
- Type flags:       (no flags)
- Object type:      B-tree (root node)

On next mount, revert to:
- snapshot with this XID:                           0x0
- APFS volume superblock with this Physical OID:    0x0

Next file-system object ID that will be assigned:   0x87f9
Next document ID that will be assigned:             0x3

Number of:

- regular files:                30460
- directories:                  1875
- symbolic links:               0
- other file-system objects:    0

- snapshots:                    0
- block allocations ever made:  31546170
- block liberations ever made:  50159

UUID:   E8C05B7D-CE69-4E46-B5E2-50CD530A2EEF
Formatted by:
- ID: diskmanagementd (1677.81.1)
- Timestamp: Fri Feb 19 22:57:12 2021
- Last XID: 25

Last modified by:
- ID: apfs_kext (1677.120.9)
- Timestamp: Wed Jul 21 14:13:10 2021
- Last XID: 8de
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------
--------------------------------------------------------------------------------

Simulating a mount of volume 0 (cosi private).

The volume object map has Physical OID 0x2ec487.
Reading the volume object map ... OK.
Validating the volume object map ... OK.

Details of the volume object map:
--------------------------------------------------------------------------------
Stored checksum:    0xb3b91f908be94e6e
OID:                0x2ec487
XID:                0x8e1
Storage type:       Physical
Type flags:         (no flags)
Type:               Object map
Subtype:            (invalid type / no subtype)
Flags:
- No flags.
Object mappings tree:
- Storage type:         Physical
- Type flags:           (no flags)
- Type:                 B-tree (root node)
- Object ID:            0x2ec489
Snapshots tree:
- Storage type:         Physical
- Type flags:           (no flags)
- Type:                 B-tree (root node)
- Object ID:            0x0
- Number of snapshots:  0 snapshots
- Latest snapshot XID:  0x0
In-progress revert:
- Minimum XID:          0x0
- Maximum XID:          0x0
--------------------------------------------------------------------------------

Reading the root node of the volume object map B-tree ... OK.
Validating the root node of the volume object map B-tree ... OK.

Details of the volume object map B-tree:
--------------------------------------------------------------------------------
Stored checksum:    0xbb05ca8ae7df3dae
OID:                0x2ec489
XID:                0x8e1
Storage type:       Physical
Type flags:         (no flags)
Type:               B-tree (root node)
Subtype:            Object map
Flags:
- Root node
- Fixed size for keys and values
Number of child levels:         1
Number of keys in this node:    26
Location of table of contents:
- Offset from start of node data area:  0x0 = 0
- Length (bytes):                       0x240 = 576
Location of key–value shared free space:
- Offset from start of keys area:       0x1a0 = 416
- Length (bytes):                       0xaf0 = 2800

Info relating to the entire B-tree:
- Flags:
- This B-tree is currently undergoing a series of sequential inserts --- optimise operations if possible
- Child nodes are referred to using Physical OIDs
- Node size:                4096 bytes
- Key size:                 16 bytes
- Value size:               16 bytes

- Length of longest key:    16 bytes
- Length of longest value:  16 bytes
- Number of keys:           2778
- Number of nodes:          27
--------------------------------------------------------------------------------

The file-system tree root for this volume has Virtual OID 0x414.
Looking up this Virtual OID in the volume object map ... corresponding block address is 0x1651f.
Reading ... validating ... FAILED.
Going back to look at the previous checkpoint instead.
END: Handling of this case has not yet been implemented.

I get the following error trying to run First Aid on the disk:

Checking the extent ref tree.
error: (oid 0x2ec287) apfs_extentref: bt: invalid o_type (0x40000003, expected 0x40000002)
Extent ref tree is invalid.

I'm trying to understand how to browse the container with drat, but I'm struggling to put it all together. The docs refer to explore-fs but running drat lists no such command, maybe it's renamed to list?

How does the search command work? I've tried drat search /dev/disk2s3 --oid 0x404 but that says "Incorrect number of arguments". I can't figure out what arguments I'm supposed to use.

Bottom line, is there a beginners guide to APFS somewhere? Or a "if your disk is broken, try these steps" guide?

Also, thanks a lot for this software, it's a relief to know that the container is at least working and most of the data is still there even if macOS refuses to mount it.

[jivanpal]

Hi, please see #22; v0.1.3 does not have search, and the documentation is for v0.2.0, which is currently in development.

Bottom line, is there a beginners guide to APFS somewhere? Or a "if your disk is broken, try these steps" guide?

I will try to provide something of this nature in the documentation once v0.2.0 is released, and perhaps add a guided recovery process into the software itself in future.

[chmac]

@jivanpal If I run drat --version I see 0.1.3 and if I run drat I get search as one of the available commands. But maybe what you mean is that it's listed but it's not implemented or something? Anyway, as I'm stuck on the encryption topic, I guess that I don't need to figure out how search works any more!

Thanks for sharing this code btw, it seems like a real asset to anyone with APFS corruption issues.

[jivanpal]

Ah, that's there because I implemented some rudimentary searching for my own use, but you'd have to mess with the source code to set search criteria — there's no command-line options as detailed in the docs.