Data Protection from Ransomware using Ceph S3

Data Protection from Ransomware using Ceph S3

Introduction

Ransomware attacks aren’t unheard of. Every few months there is a new malicious element on the rise that causes a new pool of unaware and aware users alike to be victimized by ransomware. All of their data is encrypted indefinitely with no promise of return until a certain sum is paid to the perpetrator. The payment itself also remains a gamble.

Here is a visualization by Information Is Beautiful from various sources.

As seen from the chart, that’s a non-trivial amount for any year.

However, with object data, there have been few attempts to tackle this sort of malware.

Traditional Solutions

The traditional solution to tackle situations like these include some sort of access control, either via restricting by buckets and/or users, by roles, by services, or by implementing boundaries. All of these solutions work by restriction rather than mitigation, i.e making sure that an attack won’t happen rather than making sure it cannot happen.

Object Lock and S3 Versioning

With Object Lock, the problem goes away because the objects themselves become immutable. A bucket with object-lock enabled thus becomes a descendant of the write-once-read-many (WORM) model.

Traditionally when an object is updated, the existing copy of the object is completely replaced by the newer upload. With object-lock enabled, the newer upload just gets another tag (versionId) while the older object remains in place, also identified by a tag. The newer upload becomes the current version of the object.

Ceph’s object gateway includes Ceph’s S3 API which is compatible with the Amazon S3 API.

With object-lock in place, every object can be locked either by a legal hold or a retention period or a combination of both.

Before getting into the details of what those are, let’s take a second to consider why you would want this setup:

  • Traditional solutions are reactive rather than proactive.
  • External solutions (such as ransomware scanners) need access to your data, have a price tag associated with them.
  • You are already using S3 and you prefer leveraging in-built functionality.

If you find yourself falling into those categories, the following information might be helpful.

Retention Period

Retention periods are a layer of protection that lock objects for a fixed amount of time as specified by the user. The object is locked according to the Retain until Date provided by the user. After the date has passed, the object becomes mutable again unless locked by a Legal Hold.

Legal Hold

Legal holds are another layer of protection that prevents the object from being deleted. They do not have a retention period associated with them. Legal holds lock the object until they are manually removed. They can be applied and removed by any user who has the s3:PutObjectLegalHold permission.

A combination of Legal Holds and Retention Modes can be set on an object.

Retention Modes

There are two retention modes with differing levels of protection that the user can choose between:

Governance Mode

Compliance Mode

Users cannot overwrite, delete or alter the object lock unless they are granted special permissions.
No user, including the root user, is allowed to overwrite, remove or alter the object lock status.

Some Real-Life Action

Object-lock only works with buckets that have versioning enabled. By extension, when a bucket is created with object lock enabled, it has versioning enabled by default.

The Setup

A Ceph cluster with 3 nodes is deployed with RADOS gateway and S3 being the only running services. The endpoint used is one of the RADOS gateway nodes.

In the following example, we are trying to find out how retention periods and legal holds work, which among them are more permissive. We already know that Compliance mode is the strictest of them all so let’s see that in practice in first.

Retention Periods

Compliance Mode

Create a bucket with object lock enabled:

$ aws --endpoint-url=http://172.31.117.5 s3api create-bucket --bucket test-bucket --object-lock-enabled-for-bucket

Add object lock configuration with either retention policy (compliance is chosen here)

$ aws --endpoint-url=http://172.31.117.5 s3api put-object-lock-configuration --bucket test-bucket --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":90}}}'

Add any object to the bucket

$ aws --endpoint-url=http://172.31.117.5 s3api put-object --bucket test-bucket --body some-object.png --key some-object.png 
{
    "ETag": "\"0700710d411b6bb8e62c48afbef55ab6\"",
    "VersionId": "vuKWwf0EJi70w5EYsLLh2eU9Sun2Wod"
}

Make sure that the object has object retention mode set to compliance

$ aws --endpoint-url=http://172.31.117.5 s3api get-object-retention --bucket test-bucket --key some-object.png 
{
    "Retention": {
        "Mode": "COMPLIANCE",
        "RetainUntilDate": "2022-10-26T07:48:57.740441+00:00"
    }
}

Delete the object (without the version id)

$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png
{
    "DeleteMarker": true,
    "VersionId": "OsXsNSWZe2oLisQ4.kEeDo1xjwklui0"
}

With the VersionId specified, it results in a permission denied error. It does not make much sense to delete a specific version of the object as the object when deleted isn’t permanently deleted but a “delete marker” is placed on the object which becomes the latest version of it.

List object versions to find out if the delete marker has been set

$ aws --endpoint-url=http://172.31.117.5 s3api list-object-versions --bucket test-bucket
{
    "Versions": [
        {
            "ETag": "\"0700710d411b6bb8e62c48afbef55ab6\"",
            "Size": 10485760,
            "StorageClass": "STANDARD",
            "Key": "some-object.png",
            "VersionId": "vuKWwf0EJi70w5EYsLLh2eU9Sun2Wod",
            "IsLatest": false,
            "LastModified": "2022-07-28T07:48:57.740000+00:00",
            "Owner": {
                "DisplayName": "test",
                "ID": "test"
            }
        }
    ],
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "test",
                "ID": "test"
            },
            "Key": "some-object.png",
            "VersionId": "OsXsNSWZe2oLisQ4.kEeDo1xjwklui0",
            "IsLatest": true,
            "LastModified": "2022-07-28T07:50:04.717000+00:00"
        }
    ]
}

Reissue the delete command on the object

$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png --version-id vuKWwf0EJi70w5EYsLLh2eU9Sun2Wod
An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock

We see that the object deletion is forbidden by the object lock we put in place. But what if we try to remove it?

$ aws --endpoint-url=http://172.31.117.5 s3api put-object-retention --bucket test-bucket --key some-object.png --version-id vuKWwf0EJi70w5EYsLLh2eU9Sun2Wod --retention '{ "Mode": "GOVERNANCE", "RetainUntilDate": "2025-01-01T00:00:00" }'
An error occurred (AccessDenied) when calling the PutObjectRetention operation: can't change retention mode from COMPLIANCE to GOVERNANCE

Okay but what if we try and reduce the retention period instead?

$ aws --endpoint-url=http://172.31.117.5 s3api put-object-retention --bucket test-bucket --key some-object.png --version-id vuKWwf0EJi70w5EYsLLh2eU9Sun2Wod --retention '{ "Mode": "COMPLIANCE", "RetainUntilDate": "2022-08-01T00:00:00" }'
An error occurred (AccessDenied) when calling the PutObjectRetention operation: proposed retain-until date shortens an existing retention period and governance bypass check failed

So it can be seen above that once compliance mode is set, there is no way to delete the object before the specified retention date has passed.

Governance Mode

Now let’s replicate all of the above with governance mode and see how it differs.

$ aws --endpoint-url=http://172.31.117.5 s3api create-bucket --bucket test-bucket --object-lock-enabled-for-bucket
$ aws --endpoint-url=http://172.31.117.5 s3api put-object-lock-configuration --bucket test-bucket --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":90}}}'
$ aws --endpoint-url=http://172.31.117.5 s3api put-object --bucket test-bucket --body some-object.png --key some-object.png
{
    "ETag": "\"0700710d411b6bb8e62c48afbef55ab6\"",
    "VersionId": "Ma6soN3GryqETDZsgz8Re3dPHf-caC9"
}

Once object creation is done, let’s now try to delete it.

$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png
{
    "DeleteMarker": true,
    "VersionId": "SfuC0rhMyBDjo.LO0wLcQVqaAy9A5za"
}
$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png --version-id Ma6soN3GryqETDZsgz8Re3dPHf-caC9
An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock
$ aws --endpoint-url=http://172.31.117.5 s3api get-object-retention --bucket test-bucket --key some-object.png --version-id Ma6soN3GryqETDZsgz8Re3dPHf-caC9
{
    "Retention": {
        "Mode": "GOVERNANCE",
        "RetainUntilDate": "2022-10-26T10:11:46.921668+00:00"
    }
}

Okay… so we know that we cannot delete the object, same as before but can we bypass governance mode?

$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png --version-id Ma6soN3GryqETDZsgz8Re3dPHf-caC9 --bypass-governance
{
    "VersionId": "Ma6soN3GryqETDZsgz8Re3dPHf-caC9"
}
$ aws --endpoint-url=http://172.31.117.5 s3api list-object-versions --bucket test-bucket {
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "test",
                "ID": "test"
            },
            "Key": "some-object.png",
            "VersionId": "SfuC0rhMyBDjo.LO0wLcQVqaAy9A5za",
            "IsLatest": true,
            "LastModified": "2022-07-28T10:12:50.432000+00:00"
        }
    ]
}

Yes, we can. But that’s only because the user test created the bucket and has full control. What if another I try to do it as another user who does not have the s3:BypassGovernanceRetention permission?

{
  "Version": "2022-07-28",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Action": "s3:BypassGovernanceRetention",
      "Resource": "*"
    }
  ]
}

After applying the above policy which denies that permission to all users, we create another object and then reattempt deleting it with the bypass.

$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png --version-id O6P4CX9Q8KDvQpRv.7Qse7a1.6xXJP6 --bypass-governance
An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock

We see here that even when a user has full control of a bucket, removing the s3:BypassGovernanceRetention permission protects the bucket from deletion.

Legal Holds

With legal holds, a similar story can be seen:

Create a bucket and put any object in it

$ aws --endpoint-url=http://172.31.117.5 s3api create-bucket --bucket test-bucket --object-lock-enabled-for-bucket
$ aws --endpoint-url=http://172.31.117.5 s3api put-object --bucket test-bucket --body some-object.png --key some-object.png
{
    "ETag": "\"0700710d411b6bb8e62c48afbef55ab6\"",
    "VersionId": "8jCB7AaXQoibKbVSrA3fnHqER-7QYrN"
}

Add a legal hold

$ aws --endpoint-url=http://172.31.117.5 s3api put-object-legal-hold --bucket test-bucket --key some-object.png --legal-hold Status=ON

Try to delete the object

$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png
{
    "DeleteMarker": true,
    "VersionId": "iYZ6Fm4fKmPQGqJ55uuIIm--6o5GOJX"
}
$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png --version-id 8jCB7AaXQoibKbVSrA3fnHqER-7QYrN
An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock

The legal hold forbids object deletion but in this case, we can disable it and go ahead with deletion.

$ $ aws --endpoint-url=http://172.31.117.5 s3api put-object-legal-hold --bucket test-bucket --key some-object.png --version-id 8jCB7AaXQoibKbVSrA3fnHqER-7QYrN --legal-hold Status=OFF
$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png --version-id 8jCB7AaXQoibKbVSrA3fnHqER-7QYrN
{
    "VersionId": "8jCB7AaXQoibKbVSrA3fnHqER-7QYrN"
}
$ aws --endpoint-url=http://172.31.117.5 s3api list-object-versions --bucket test-bucket
{
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "test",
                "ID": "test"
            },
            "Key": "some-object.png",
            "VersionId": "iYZ6Fm4fKmPQGqJ55uuIIm--6o5GOJX",
            "IsLatest": true,
            "LastModified": "2022-07-28T08:34:25.281000+00:00"
        }
    ]
}

Additionally, we can delete the delete marker too.

$ aws --endpoint-url=http://172.31.117.5 s3api delete-object --bucket test-bucket --key some-object.png --version-id iYZ6Fm4fKmPQGqJ55uuIIm--6o5GOJX
{
    "DeleteMarker": true,
    "VersionId": "iYZ6Fm4fKmPQGqJ55uuIIm--6o5GOJX"
}
$ aws --endpoint-url=http://172.31.117.5 s3api list-object-versions --bucket test-bucket

But what would happen if we deny the required permission?

{
  "Version": "2022-07-28",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": {
        "AWS": [
          "*"
        ]
      },
      "Action": "s3:PutObjectLegalHold",
      "Resource": "*"
    }
  ]
}

Once we apply the above bucket policy, we see the following happens:

$ aws --endpoint-url=http://172.31.117.5 s3api put-object-legal-hold --bucket test-bucket --key some-object.png --version-id VgtT3qRPjleP5tILYI8X0f7XUL7i2jL --legal-hold Status=OFF
An error occurred (AccessDenied) when calling the PutObjectLegalHold operation: Unknown

We notice that users without the s3:PutObjectLegalHold permission will not be able to remove a legal hold.

Unlike Governance mode, there is only single S3 permission that allows users to place and remove legal holds.

Thoughts and Conclusion

Object locking serves as a tremendous proactive measure against ransomware. However, as seen above, it might not be suitable in all cases. One such point is when objects are often updated and render the older ones invalid. If the object's lifespan is short-lived, an undesirable accumulation over time can be seen and since modes like compliance refuse any change at all, storage requirements may blow up.

In cases where data is sensitive or is known to be important, object locking brings in a high level of protection and lowers cortisol levels. When it comes to data security, object locking is a definite contender for another layer of protection.