{"id":434,"date":"2010-03-23T14:13:59","date_gmt":"2010-03-23T19:13:59","guid":{"rendered":"http:\/\/www.webadminblog.com\/?p=434"},"modified":"2010-03-23T16:20:34","modified_gmt":"2010-03-23T21:20:34","slug":"amazon-ec2-ebs-instances-and-ephemeral-storage","status":"publish","type":"post","link":"https:\/\/www.webadminblog.com\/index.php\/2010\/03\/23\/amazon-ec2-ebs-instances-and-ephemeral-storage\/","title":{"rendered":"Amazon EC2 EBS Instances and Ephemeral Storage"},"content":{"rendered":"

Here’s a couple tidbits I’ve gleaned that are useful.<\/p>\n

When\u00a0 you start an “instance-store” Amazon EC2 instance, you get a certain amount of ephemeral storage allocated and mounted automatically.\u00a0 The amount of space varies by instance size and is defined here<\/a>.\u00a0 The storage location and format also varies by instance size and is defined here<\/a>.<\/p>\n

The upshot is that if you start an “instance-store” small Linux EC2 instance, it automagically has a free 150 GB \/mnt disk and a 1 GB swap partition up and runnin’ for ya.\u00a0 (mount points vary by image, but that’s where they are in the Amazon Fedora starter.)<\/p>\n

[root@domU-12-31-39-00-B2-01 ~]# df -k<\/pre>\n
Filesystem\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1K-blocks\u00a0\u00a0\u00a0\u00a0\u00a0 Used Available Use% Mounted on\r\n\/dev\/sda1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 10321208\u00a0\u00a0 1636668\u00a0\u00a0 8160252\u00a0 17% \/\r\n\/dev\/sda2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 153899044\u00a0\u00a0\u00a0 192072 145889348\u00a0\u00a0 1% \/mnt\r\nnone\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 873828\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 873828\u00a0\u00a0 0% \/dev\/shm<\/pre>\n
[root@domU-12-31-39-00-B2-01 ~]# free\r\ntotal\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 used\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 free\u00a0\u00a0\u00a0\u00a0 shared\u00a0\u00a0\u00a0 buffers\u00a0\u00a0\u00a0\u00a0 cached\r\nMem:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1747660\u00a0\u00a0\u00a0\u00a0\u00a0 84560\u00a0\u00a0\u00a0 1663100\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 4552\u00a0\u00a0\u00a0\u00a0\u00a0 37356\r\n-\/+ buffers\/cache:\u00a0\u00a0\u00a0\u00a0\u00a0 42652\u00a0\u00a0\u00a0 1705008\r\nSwap:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 917496\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 917496<\/pre>\n
But, you say, I am not old or insane!\u00a0 I use EBS-backed images, just as God intended.\u00a0 Well, that’s a good point.\u00a0 But when you pull up an EBS image, these ephemeral disk areas are not available to you.\u00a0 The good news is, that’s just by default.<\/p>\n
The ephemeral storage is still available and can be used (for free!) by an EBS-backed image.\u00a0 You just have to set the block devices up either explicitly when you run the instance or bake them into the image.<\/p>\n
Runtime:<\/strong><\/p>\n
You refer to the ephemeral chunks as “ephemeral0”, “ephemeral1”, etc. – they don’t tell you explicitly which is which but basically you just count up based on your instance type (review the doc<\/a>).\u00a0 For a small image, it has an ephemeral0 (ext3, 15 GB) and an ephemeral1 (swap, 1 GB).\u00a0 To add them to an EBS instance and mount them in the “normal” places, you do:<\/p>\n
ec2-run-instances <ami id> -k <your key> --block-device-mapping '\/dev\/sda2=ephemeral0'\r\n--block-device-mapping '\/dev\/sda3=ephemeral1'<\/pre>\n
On the instance you have to mount them – add these to \/etc\/fstab and mount -a or do whatever else it is you like to do:<\/p>\n
\/dev\/sda3\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 swap\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 swap\u00a0\u00a0\u00a0 defaults 0 0\r\n\/dev\/sda2\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/mnt\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ext3\u00a0\u00a0\u00a0 defaults 0 0<\/pre>\n
And if you want to turn the swap on immediately, “swapon \/dev\/sda3”.<\/p>\n
Image:<\/strong><\/p>\n
You can also bake them into an image.\u00a0 Add a fstab like the one above and when you create the image, do it like this, using the exact same –block-device-mapping flag:<\/p>\n
ec2-register -n <ami id> -d \"AMI Description\" --block-device-mapping  \/dev\/sda2=ephemeral0\r\n--block-device-mapping '\/dev\/sda3=ephemeral1' --snapshot your-snapname --architecture i386\r\n--kernel<aki id>\u00a0 --ramdisk <ari id><\/pre>\n
Ta da. Free storage that doesn’t persist.\u00a0 Very useful as \/tmp space.\u00a0 Opinion is split among the Linuxerati about whether you want swap space nowadays or not; some people say some mix of\u00a0 “if you’re using more than 1.8 GB of RAM you’re doing it wrong” and “swapping is horrid, just let bad procs die due to lack of memory and fix them.”\u00a0 YMMV.<\/p>\n
Ephemeral EBS?<\/strong><\/p>\n
As another helpful tip, let’s say you’re adding an EBS to an image that you don’t want to be persistent when the instance dies.\u00a0 By default, all EBSes are persistent and stick around muddying up your account till you clean them up.\u00a0\u00a0 If you don’t want certain EBS-backed drives to persist, what you do is of the form:<\/p>\n
ec2-modify-instance-attribute --block-device-mapping \"\/dev\/sdb=vol-f64c8e9f:true\" i-e2a0b08a<\/pre>\n
Where ‘true’ means “yes, please, delete me when I’m done.”\u00a0 This command throws a stack trace to the tune of<\/p>\n
Unexpected error: java.lang.ClassCastException: com.amazon.aes.webservices.client.InstanceBlockDeviceMappingDescription\r\ncannot be cast to com.amazon.aes.webservices.client.InstanceBlockDeviceMappingResponseDescription<\/pre>\n
But it works, that’s just a lame API tools bug.<\/p>\n","protected":false},"excerpt":{"rendered":"
Here’s a couple tidbits I’ve gleaned that are useful. When\u00a0 you start an “instance-store” Amazon EC2 instance, you get a certain amount of ephemeral storage allocated and mounted automatically.\u00a0 The amount of space varies by instance size and is defined here.\u00a0 The storage location and format also varies by instance size and is defined here. […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[82],"tags":[36,42,407,40,405,408,406],"class_list":["post-434","post","type-post","status-publish","format-standard","hentry","category-cloud-computing","tag-amazon","tag-aws","tag-ebs","tag-ec2","tag-ephemeral","tag-image","tag-storage"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pfI0c-70","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/434","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=434"}],"version-history":[{"count":6,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/434\/revisions"}],"predecessor-version":[{"id":436,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/434\/revisions\/436"}],"wp:attachment":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=434"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=434"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=434"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}