Just after the month of releasing macOS High Sierra and new file system (APFS), Mac users experienced a bug that exposed the encrypted APFS passwords via the password hint feature. The issue affected Macs with APFS file-system and users who had entered a password hint. Apple soon issued an emergency patch to fix the Password Leak Disk Encryption Utility.
Earlier this year, Apple tackled a bug – a specific character in the Indian language Telugu – which caused crashing of apps and system in iPhone, iPad, and Mac. Apple released updates for all its devices (iPhones, iPads, Apple Watch, Apple TV & Mac) to fix the ‘Telugu' bug.
Unlike these instances of bug reports, Apple suffered from an APFS bug vulnerability that leads to data loss in macOS High Sierra. Mike Bombich, the creator of Carbon Copy Cloner (a Mac backup application) uncovered the ‘'sparsebundle disk images'' flaw in APFS-formatted Mac.
A Sparsebundle is a disk image format introduced by Apple in OSX 10.5 as a method of effectively using space for backups on a disk. This is an expandable/retractable disk that will only take up as much room as the data inside of it up to a predefined limit (in the case of TM, the size of the partition). It changes the file's extension from sparsebundle, a disk image format that can be mounted and browsed like a physical volume, to backupbundle. This isn't documented on Apple's site, and it.
Data Writing Flaw in APFS
Disk images are useful to take necessary backup of sensitive files to network volumes and for disk cloning operations. Sparse disk image is basically a disk image file that contains the structure of a disk volume which mounts on your Mac and is treated as a physically connected drive.
Sparse disk image shows available space which permits you to utilize the free space available. However, there is a mismatch in the values of APFS sparse disk image ‘free disk space' and the ‘actual free disk space' which results in data loss.
The bug came to light when Bombich reported that his sparsebundle disk image showed sufficient free space and when he copied a video file to disk image volume it got corrupt. Here's what he experienced:
'Earlier this week I noticed that an APFS-formatted sparsebundle disk image volume showed ample free space, despite that the underlying disk was completely full. Curious, I copied a video file to the disk image volume to see what would happen. The whole file copied without error! I opened the file, verified that the video played back start to finish, checksummed the file – as far as I could tell, the file was intact and whole on the disk image. When I unmounted and remounted the disk image, however, the video was corrupted. If you've ever lost data, you know the kick-in-the-gut feeling that would have ensued. Thankfully, I was just running some tests and the file that disappeared was just test data. Taking a closer look, I discovered two bugs in macOS's 'diskimages-helper' service that lead to this result.'
Source:https://bombich.com/blog/2018/02/15/macos-may-lose-data-on-apfs-formatted-disk-images
The following example illustrates data loss due to file system bug:
- Suppose you create a 100 GB APFS-formatted sparse disk image
- You copy 80 GB data to the sparse disk image
- The APFS-formatted sparsebundle disk image file is not updated and continues to show 100 GB of free space
- You copy another 50 GB of files to the sparse disk image
- You even access the files after transferring the files
- Remount the disk image, transferred files become corrupt and 30 GB of data disappears
The file system bug affects only the APFS-formatted sparsebundle disk image and not the startup volume. Bombich further identified two issues:
- The APFS-formatted sparse disk image doesn't update the free space available
- The user does not receive any ‘no more space' notification
Stellar Data Recovery Professional for Mac : your trusted recovery tool
Although Bombich lost a test file, you might lose important files and folders due to the file system bug. As of now, Carbon Copy Cloner has dropped the support for APFS-formatted sparse disk image until the bug is resolved; however, if you lose your files, you can recover it back using Stellar Data Recovery Professional for Mac software.
The software recovers data lost due to file system bug as it supports Apple Time Capsule on sparse bundle disk images. Also, as you're concerned about the lost data, its intuitive user-interface and easy layout provide relief to you and delivers favorable outcomes.
Stellar Data Recovery Professional for Mac is easy to understand and easy to run. You do not need to spend time getting familiar with its functionality, simply make your selection and recover data efficiently and effortlessly.
The recovery steps are as follows:
- Run Stellar Data Recovery Professional for Mac and toggle-on ‘Recover Everything' tab. Click Next (You can even customize your selection)
- Select the sparsebundle disk image and click Scan
- The software scans the disk image for recoverable files
- Once the scanning is complete, all the recoverable files are listed hierarchically
- Preview the files before saving them
- Select the files, click Recover, specify the destination folder and click Save
Conclusion
Apple is known for challenging status quo by producing magical experiences through its exemplary products. With every update, Apple introduces ground breaking innovations and steps up the game. The same pattern is followed with the new file system (APFS), Mac users get a good deal of reasons to sink their teeth into this.
Although there will be some performance hiccups and random bugs but the Cupertino-based tech giant always resolves the issue at the earliest. But when your data is at stake, you should take an extra step in protecting it as the case with the sparse disk image bug. This is when Stellar Data Recovery Professional for Mac proves worthwhile. Until there is a macOS update to resolve the bug, you can count on the software to recover data after the bug attack and continue with the magical Mac experience.
Having a virtual machine running macOS opens up a lot of opportunities for learning. If you are into security, you can set up a VM for your security lab. Or if you want to learn networking or kernel debugging, it is also helpful to use a VM. The other option is risk breaking your work machine in your experiments (not fun). To build our VM, we need to use Disk Images, another topic that is useful in other areas.
Both topics are interesting, and we can combine them. In this post, I'll show you the basics of working with Disk Images and also as an example we are going to create an ISO image to build our macOS Catalina VM using VirtualBox.
There's a lot of information so let's get started.
NOTE: We need to download the macOS Catalina installer from the AppStore, it's around 9Gb, so It'll take some time to download. Better start now while we learn about Disk Images.
Table of Contents
We'll learn by making, so let's start by creating a folder that we'll use as the base for our image.
Let's add some content to our folder:
Ok, this is going to be our simple image structure:
We are going to use hdiutil. This tool uses the DiskImages.framework for.. well, manage disk images. Or at least that is what its man page says. But I couldn't find any documentation on the DiskImages.framework so I assume it's a private framework. If you know anything about it, let me know. In the meantime, I encourage you to read the man page hdiutil(1) for more information.
Alright, that should have created miapplicacion.dmg image in your desktop. We can now delete the ~/Desktop/myapp, we don't need it anymore.
We can now mount the image. We can do that by clicking it using the Finder, or by using hdiutil again. I'll use hdiutil so we get more comfortable using it.
That will mount our image in /Volumes/myapp. I purposefully gave different names to the dmg and the image so you can see which one will be used when mounted.
If you check the newly mounted image, you'll see our hello.txt with the content Hola, mundo!.
Now you can eject it from the Finder or use hdiutil again. As you probably imagined, I'll show you how to do it with hdiutil.
You can use the name of the volume or the device, in my case:
We create the image using a source folder. We could also have created an empty image and then add content to it. Let's see how to do that.
That create s new image of ten megabytes of space. To work with that image, we need to mount it. We are going to mount it but without including its filesystem.
DANGER, WILL ROBINSON! PLEASE MAKE SURE YOU ARE POINTING TO THE CORRECT DEVICE, ELSE YOU CAN LOSE DATA! I WON'T BE RESPONSIBLE FOR ANYTHING YOU DO WITH THE INSTRUCTIONS I POST HERE, TREAD CAREFULLY.
In my computer the image is mounted in /dev/disk3 it might use a different one on your computer so make sure you refer to the correct one or BAD THINGS WILL HAPPEN.
Why did we mount our image this way? Well, we did it so we can format the volume using diskutil. Let's do that.
Now we can add some content to that Image.
Now we can eject the volume.
And now you can mount it again, and you'll see your howdy.txt file. The images we've created are Read/Write. We can also convert it to a read-only image:
You can see all the supported formats in the man page. Here is the list in mine:
Now if we mount our new image and try to write to it we'll get an error:
Alright, that should give us enough background on Disk Images (.dmg) to build our ISO image for Catalina. Let's do that now.
Since Mac OS X Lion (10.7 - released in 2011), Apple decided to no longer distribute an installation DVD, it provides the installer as an application that can be downloaded via the AppStore. This installer application contains the base system for installation and also a Recovery image that includes a basic OS implementation that is used at boot time. When you boot to the Recovery volume, you can use it to restore and/or install the operating system. You can read more on This detailed article on AFP548.
The idea is to extract all the parts from the Install macOS Catalina.app application bundle to build our ISO. I'm going to use Jeff Geerling's macOS VirtualBox VM script as a basis. It is similar to the instructions in Kedy Liu's article on macOS Kernel Debugging.
We are going to mainly focus on the InstallESD.dmg and BaseSystem.dmg images that come inside the application installer to generate an ISO image.
If you are not familiar with the ISO format, an ISO image is the image format used on an optical disk. It is the format used by DVD, CD, or Blue-ray Disc. In the past, it was common to use CDs as the media to install the operating systems. VirtualBox supports that format. We plan to create an ISO image from the installer.
Let's begin.
After the AppStore download is complete, you'll be able to find the Install macOS Catalina.app inside your /Applications/ directory.
I'll be using macOS Catalina as an example, but as long as you have the installer of any OS newer than 10.7, the process should be similar.
Navigate to the /Applications/Install macOS Catalina.app/Contents/SharedSupport/ directory, and list the files:
Excellent, we have the BaseSystem.dmg and the InstallESD.dmg. Inside the InstallESD.dmg we'll find the installer that uses the BaseSystem image to install the OS.
Sparse Image Mac
To extract the installer we'll need to mount the InstallESD.dmg image.
The BaseSystem.dmg has an image formatted in HFS. We can create an empty image as we did in the last example of the previous section. Then we can 'restore' the BaseSystem image in the newly created image. It sounds confusing, but you'll understand it better when we begin doing it.
Let's begin by creating the destination image. I'll show you the command first and after I'll explain what it does.
We are creating a sparse image. The sparse image type means that the image we create will only take as much space as the content we put on it, with the limit of 9 Gigabytes that we set. The layout indicates the partition table's layout for the created volume. We are stating that we want a single entry. The partition table layout can be split in multiple /dev entries. For example, we could have one for the GUID Partition Table (GPT), and the other partitions have the data. With the SPUD layout, we will only have one /dev entry. If you want to learn more visit the following links about Apple partition Map and GUID Partition Table.
We also define the file system type to be HFS+J. The BaseSystem.dmg disk image is also in HFS, so that is why we choose it (also because hdiutil can only resize HFS).
Ok, now you can run the command to create the sparse image. I'm showing you the command again:
Apple keyboard german layout. Let's mount it so we can work on it.
Now we are going to copy the BaseSystem.dmg image into our /Volume/install_build image.
To copy the image we are going to use the asr command (you can check the man page asr(8) for more details).
Alright, this will copy the Base System inside our volume /Volumes/install_build, so It will erase our /Volumes/install_build and replace it with /Volumes/macOS Base System/. If you check our /Volumes/ you'll see it there.
Apple Sparse Bundle Disk Image Media System
The BaseSystem.dmg includes symbolic links to the Packages directory. When we are installing the OS with the application that symbolic link will exist. But in our case, that symbolic link points to the wrong place. You can verify this by listing the contents in /Volumes/macOS Base System/System/Installation/.
What we are going to do is replace that symbolic link with the contents of the Packages directory we find in our InstallESD.dmg that we mounted in /Volumes/install_app. We can rm and then cp Or we can use the ditto(1) command.
We also need to copy some installer dependencies, i.e. BaseSystem.chunklist and BaseSystem.dmg, and add them to the image. We find those two files in /Applications/Install macOS Catalina.app/Contents/SharedSupport, let's list that directory:
Ok, now that we know where they are let's copy them to our image volume:
That's it. That is our image ready. Let's now unmount the images, clean up and prepare to convert our installation image to a read-only image.
Let's clean up and resize to free up any extra space. hdiutil can only resize filesystems of type HFS+. Again, read the man page for additional information on resize (hdiutil(1)). We can obtain the current size of the image using the -limit argument:
NOTE: as pointed out by quantum_libet we can simplify by using the size -min parameter, this is much more clean than my previous solution. Thanks for the tip!
Now we are ready to convert the image to a CD-R export image. You can see how to do this in the man page in the examples section (hdiutil(1), as you can see a lot of information in the man page).
And that's it. We now have the Catalina.iso. We can use our ISO in VirtualBox to create our macOS Virtual machine!
Alright, now for the ugly part. VirtualBox doesn't support APFS in their UEFI boot loader (you can read the discussion in VirtualBox's forum). Luckily Alexander Willner created a handy script with the name runMacOSinVirtualBox that automates the creation of a VirtualBox VM that works. The script creates a bootable partition that can be used to boot macOS while we wait for VirtualBox to support APFS.
His script can also be used to create an ISO, but we have already done that, so no need.
Ok clone or download his script and run it.
Now we can go to VirtualBox, and add our ISO image as an optical drive to the newly created VM.
Go to: Settings > Storage > Add New Optical Drive and select our newly created Catalina.iso.
Now you can start the VM. Now select Disk Utility and format your macOS partition using APFS (click Erase and select APF). After the formatting is complete, quit Disk Utility and click 'Reinstall macOS'. Follow and complete the installation process.
Congratulations! You now have Catalina running on VirtualBox :).
We could have copy/paste or cloned the scripts created by Alexander Willner, Jeff Greeling or Kedy Liu. But I believe it's helpful to understand any code we use so we can build on top of that knowledge.
We learned how to work with Disk Images. As you can see, it is a convenient topic to understand. For example, some people distribute their applications outside the AppStore using .dmgs.
And also you now have a VM that you can use to build your security lab or to explore Kernel Debugging.
Ok, that's it for this post. I hope that you found this post useful :).
I also would like to thank all the people that very openly share their knowledge and help us understand how things work. I think the metaphor standing on the shoulders of giants, is quite apt.
Thank you.
Most of the links I'll add here are already in the post but for convenience I'll add them again.
- A detailed article on the structure of the macOS Installation App - AFP548.
- Jeff Geerling's macOS VirtualBox VM script.
- Kedy Liu's article on macOS Kernel Debugging.
- Alexander Willner handy script runMacOSinVirtualBox.
- Documentation on Apple partition Map and GUID Partition Table.
