New
Elasticsearch-quality full text search on Postgres, now available with ParadeDB on Ubicloud. Try it now
We talked about encryption in transit in our Linux networking blog posts. After we published those posts, a few of our customers asked about encryption at rest. So, we wanted to write a blog post to document how Ubicloud encryption works for our block storage (currently non-replicated) service.
In short, Ubicloud leverages open source projects for block storage encryption. We use AES in XTS mode to avoid identical data blocks resulting in identical ciphertext; envelope encryption with two keys to defend against any one key getting compromised; and we enable key rotation. The following sections dive deeper into the details.
We use Linux KVM as our type-1 (bare-metal) hypervisor and Cloud Hypervisor as our virtual machine monitor (VMM). KVM implements virtual CPUs; and VMMs that run on KVM are responsible for emulating other devices, such as disks.
Cloud Hypervisor can provide basic file-backed disks for VMs. In fact, our block storage v0.1 implementation started with that approach. However, we quickly ran into limitations, such as lack of encryption, snapshots, or replication.
To overcome these limitations, we use the Storage Performance Development Kit (SPDK) to implement block storage for our VMs. SPDK uses a layered block device (bdev) framework, where each layer provides a specific function like file access, NVMe access, encryption, or compression. This layered approach organizes software modules into layers and enables creating flexible storage stacks.
SPDK already includes certain software modules. It’s also customizable and we can add more modules to enable new functionality for our block storage volumes. One such module we add is bdev_ubi, which enables us to provide copy-on-write (COW) image access on top of an existing OS image.
At a high level, our default SPDK setup looks like this:
In the above diagram, bdev_aio provides file access, vbdev_crypto adds encryption on top of that, and bdev_ubi adds COW image access from a base image file. This way, we can store multiple base OS images on the host. When the user creates a new VM and accesses the base image, we then create a disk.raw image that stores any modifications.
All user specific data is written through vbdev_crypto devices and is encrypted. In the future, we’ll also add encrypted image access. This is necessary if we want to provide support for custom user images.
We set up SPDK’s vbdev_crypto to use AES in XTS mode for encrypting data at rest. AES is a widely recognized and trusted encryption algorithm, and the XTS mode is specifically designed for encrypting data stored on block storage devices.
AES in XTS mode has been standardized in "IEEE P1619™/D16", see https://crossbowerbt.github.io/docs/crypto/pdf00086.pdf for more details.
AES-XTS effectively solves the issue of identical data blocks resulting in identical ciphertext, a vulnerability found in some traditional encryption methods. It achieves this by using two keys: one for standard AES block encryption and another to generate a unique, encrypted "tweak" for each block. This tweak, further modified by a mathematical function, ensures that even identical data blocks produce unique output (ciphertext).
There are other methods that produce unique blocks on identical plaintext, such as initialization vectors and chaining. These approaches however are prone to propagating errors across blocks during decryption. AES-XTS uses an algorithm that encrypts each block independently. This independence limits data loss from corruption to only the affected block.
We use a technique called “Key Wrapping” to protect encryption keys. Recommended by NIST, this technique is also used by other public cloud providers such as AWS and GCP. This approach uses two key types for each storage volume:
Data Encryption Keys (DEKs): These are the core encryption keys, specifically two 256-bit keys used by the AES-XTS algorithm to encrypt the actual disk data.
Key Encryption Key (KEK): DEKs are never stored in plaintext. Instead, they are encrypted using a KEK, ensuring their protection even if the storage host is compromised. We use the AES-256-GCM algorithm for this purpose, providing both confidentiality and integrity for the wrapped DEKs.
For example, if the customer disk is stored as a file named /var/storage/vm123456/0/disk.raw on the host filesystem, then the DEKs are stored in /var/storage/vm123456/0/data_encryption_key.json. The keys inside this json file are encrypted using KEK. The KEK itself is stored at our key repository, which is a database table that sits on a different physical host.
This architecture creates a robust security strategy. Compromising either the storage host (containing encrypted data and wrapped DEKs) or the key repository (containing the KEK) alone is insufficient to decrypt the data. Attackers would require simultaneous access to both, significantly increasing the difficulty of a successful attack.
To further enhance security, we enable rotation of KEKs. This limits the impact of any potential key compromise and minimizes the window of vulnerability.
Ubicloud applies encryption at the software layer and this naturally has performance implications. In our benchmarks, encrypted disks had a throughput roughly 20% lower than their unencrypted counterparts. Although the performance reduction is noticeable, we think it’s well worth it for the security benefits gained.
In conclusion, Ubicloud leverages open source for block storage encryption. We apply industry best practices to further enhance security, such as AES in XTS mode, envelope encryption, and key rotation. There’s a performance cost associated with encryption, but we think the benefits are well worth it.
The nice thing about open source is everything we've described here is available for you to look at. If you have any questions or feedback about our security practices, please start a discussion in our GitHub repository. If you have a security concern or believe you found a vulnerability, please send your report to security@ubicloud.com.
