22 steps to reduce your AWS bill (Part 3 — Storage)

A brief introduction to AWS cloud cost optimization techniques.

This is Part 3 from series of “22 steps to reduce you AWS bill series”. Please also check articles Part 1, Part 2, Part 4 and Part 5.

22 cloud cost optimization areas

The AWS cloud cost optimization is a very broad topic with many different components which are cost drivers adding-up to your total bill. The optimization process is not limited to the list presented below but these are the most common and most obvious sources. Depending on your AWS architecture and services used you should also try to explore additional methods that would support your AWS cost management and potentially lead to lower bills.

Storage

• Tidy up EBS volumes
• Delete obsolete snapshots
• Choose the right storage class
• Manage files lifecycle
• Compress files before storage
• Remove incomplete multipart uploads
• Optimize AWS S3 API calls

Tidy up EBS volumes

Another efficient method of lowering an AWS bill is to reduce the number of EBS volumes. You can do this in two ways. Depending on how long you have been working with AWS and how many EC2 instances you have been using so far there could be tens, hundreds, or even thousands of unattached EBS volumes. While decommissioning EC2 instance:

• someone can forget to delete an associated EBS volume
• EBS volume is left intentionally for future use but with time it becomes forgotten

These blocks although left ide still contributes to your monthly bill. You must also identify EBS volumes with low utilization e.g. activity lower than 1 IOPS per day for the last 7 days indicates that it is no longer in use. In both cases snapshotting and deleting EBS volumes will cut your storage costs.

To make things simpler there is the Amazon Data Lifecycle Manager that can help in automating the creation, retention, and deletion of EBS snapshots and EBS-backed AMIs.

Delete obsolete snapshots

As with EBS volumes, the possibility to make unlimited snapshots can lead to storing on an S3 bucket huge numbers of backups that are no longer needed. Probably it should be enough to keep snapshots for a couple of weeks depending on the frequency with which they are taken but still usually you’re only going to use the most recent one to restore your data. Although every single snapshot doesn’t cost much you can save thousands of dollars yearly by deleting those you no longer need.

Choose the right storage class

There are distinct levels of storage dedicated to different types of usage each having a different cost structure. You can pick one from six tires or use a mix of them. The cost for each storage tier comprises the actual storage amount, the volume of data transfer, the number of HTTP GET requests, and HTTP PUT requests. Determining which storage tier is most suitable for your data depends on the following factors:

• how much you store,
• how often data are being accessed,
• how quickly they should be accessible when needed.

For example, storing the most recent backups in S3 Standard class is a waste of money. They should be placed in S3 Standard — Infrequent Access or S3 One Zone — Infrequent Access. Historical backups, if needed, could be placed in S3 Glacier or S3 Glacier Deep Archive depending on the required retrieval time.

In the table below you can find a cost comparison of S3 storage classes (US East Ohio, Dec 17, 2020). Information about S3 classes and pricing is available at Amazon S3.

Manage files lifecycle

It is very easy to build terabytes large objects library when there are no capped limits for storage. Once stored, objects are very rarely deleted. After a few months, your AWS bill can show a significant change in storage cost compared to your expectations. Therefore infrequently accessed objects should be moved to a cheaper storage class. Instead of always using S3 Standard try S3 Analytics to manually analyze storage access patterns on the object data set and automate moving them into lower-cost storage tier with Life Cycle Policies or employ S3 Intelligent-Tiering to automatically move objects between four access tiers when access patterns change. This will substantially reduce long-term storage costs.

Compress files before storage

While working on files lifecycle you should also consider compressing data. It will reduce storage requirements subsequently reducing also its cost. It is especially true when moving objects to S3 Archive storage classes. One for the data transfer cost and two for the fixed storage cost.

Remove incomplete multipart uploads

In a petabyte large S3 object store there may be a lot of objects that are not complete. Multipart upload is a three-step process: initiation of the upload, uploading parts of an object, completion of the process. Upon receiving the complete multipart upload request, S3 constructs the object from the uploaded parts. Incomplete objects can appear when the uploading process was interrupted before finishing. You can set up S3 lifecycle policies to clean up incomplete multipart uploads or alternatively you can do it manually using AWS CLI, S3 API, and AWS Tools for Windows PowerShell.

Optimize AWS S3 API calls

Each time you make a request to an object stored in S3 you pay for that request based on its type: PUT, COPY, POST, LIST, GET, SELECT, lifecycle transition, and data retrievals. API calls are charged per object regardless of its size. It means that uploading one small file costs the same as uploading one set of 10GB files. Work with DEV and OPS to modify your applications so they use batches instead of transferring separately a large number of small files.