Amazon Simple Storage Service

lastmod 2026-08-20created 2020-07-0828 edits

Amazon S3 (Amazon S3, Simple Storage Service) is an online storage web service provided by Amazon Web Services. Amazon S3 provides storage through a web service interface. Amazon launched S3 and first made the web service publicly available in the United States in March 2006, and in Europe in November 2007. When the service first started, end users could use it by paying 0.15 US dollars per GB per month, along with network data transfer fees for sending and receiving data and per-request costs. From November 1, 2008, discounted prices were offered to users using 50 TB or more.

Categories

• boto3
• aiobotocore (asyncio)

Projects

• WeSQL - An innovative MySQL distribution that uses S3 as storage
• STU - Amazon S3 TUI explorer
• s3mini - An ultra-lightweight, ultra-fast S3 client for Node and edge platforms
• s3grep - An ultra-fast parallel grep tool for S3
• S3 Files

BYO S3

Used with Amazon S3, it stands for Bring Your Own S3. In other words, it means S3 where you bring your own device or license.

It refers to storing data or files by connecting your own Amazon S3 bucket (or compatible storage), owned and managed by the user, instead of the service provider's default storage space.

Rules for bucket naming

• Bucket restrictions and limitations

The following rules apply for naming S3 buckets:

• Bucket names must be between 3 and 63 characters long.
• Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).
• Bucket names must begin and end with a letter or number.
• Bucket names must not be formatted as an IP address (for example, 192.168.5.4).
• Bucket names can't begin with xn-- (for buckets created after February 2020).
• Bucket names must be unique within a partition. A partition is a grouping of Regions. AWS currently has three partitions: aws (Standard Regions), aws-cn (China Regions), and aws-us-gov (AWS GovCloud [US] Regions).
• Buckets used with Amazon S3 Transfer Acceleration can't have dots (.) in their names. For more information about transfer acceleration, see Amazon S3 Transfer Acceleration.

Uploading with a Presigned URL

• Three ways to upload files to S3
• When an AWS S3 Pre-signed URL expires after 15 minutes, :: Jflip's tech blog
• Sharing objects with presigned URLs - Amazon Simple Storage Service
• Working with presigned URLs - Amazon Simple Storage Service
• Uploading files to S3 using a Presigned URL
• Using Amazon S3 presigned URLs with Boto3 | Document translation collection

1. The client starts an image upload.
2. First, an authenticated URL for the upload is generated from the file name and type.
3. Sending a request with the image data attached to the generated URL uploads it to the corresponding S3.

How to stream video from S3

• See FFmpeg:Tool/FFmpeg:Example:HLS/HLS/S3.
• How to Serve HLS Video from an S3 Bucket | HTTP Live Streaming
• Adaptive Streaming with MPEG-DASH and HLS using AWS
• Building a CloudFront environment for HLS playback using AWS S3
• How to Upload FFmpeg Output Directly to Amazon S3 | Saturn Cloud Blog
• Creating a managed FFmpeg workflow for media processing using AWS Batch | AWS Tech Blog

List of Object Storage Softwares

• Cloud Service
  • Amazon Simple Storage Service (S3)
  • Cloudflare R2
• Self Hosted
  • minio
  • SeaweedFS
  • Ceph
  • Zenko
  • Riak S2
  • Triton Object Storage
  • LeoFS
  • HyperStore
  • RustFS
• Utilities
  • Mountpoint
  • Cyberduck

See also

• Object Storage
• AWS
• Copyparty - Open source that turns almost any device into a file server
• Filesystem
• Litestream VFS

Favorite site

• Wikipedia (en) Amazon S3
• A case-by-case guide to using S3 safely | Woowa Brothers Tech Blog
• How AWS S3 handles 1 petabyte per second on top of slow HDDs | GeekNews
  • [Original] How AWS S3 serves 1 petabyte per second on top of slow HDDs
  • AWS S3 is a large-scale multi-tenant object storage service that provides high availability and durability at low cost
  • It uses old, slow (≈120 IOPS) HDDs, but maximizes the economics of ultra-cheap, high-capacity storage media
  • S3 designs its internal storage layer (ShardStore) as a log-structured (LSM) design to fit the write path to sequential I/O, while the read path offsets performance problems with Erasure coding (5-of-9) and massive parallelization
  • Across the client, front end, and storage, it reduces tail latency and aggregates throughput with multipart uploads, byte-range GETs, connection pools, hedged requests, etc.
  • It avoids hotspots through random distributed placement and Power of Two Random Choices, continuous rebalancing, and decorrelation that flattens load as scale grows, ultimately achieving >1 PB/s-class throughput
• Building a storage cluster for pretraining with 30 petabytes of hard drives | GeekNews
• Self-hosting 10TB of S3 with a Framework laptop and disks | GeekNews
• How we saved $500K a year by building our own "S3" | GeekNews