advanced S3 and athena

Question 1

s3 access logs

Answer 1

• for audit purpose, log all accesses
• separate logging bucket
• warning: don’t set your logging bucket to be the same as the monitored bucket (infinite loop!)

Question 2

S3 replication (CRR & SRR)

Answer 2

• cross region and same region replication
  
  • asynchronous replication (must enable versioning)
  • buckets can be in different accounts
  • CRR - compliance, lower latency access,
  • SRR - disaster recovery, etc.
  • by default, delete markers are not replicated but there is a feature to do so
    
    • permanent deletes are NOT replicated
    • bucket must be empty before deleting
  • only NEW objects are replicated after enabling
    
    • otherwise use batch replication to replicate existing objects
  • no ‘chaining’ of replicaiton
    
    • bucket one replicates to bucket 2 which replicates to bucket 3, etc.

Question 3

S3 general purpose storage class

Answer 3

• 99.9999% availability
  
  • frequently accessed
  • low latency, high throughput
  • big data, mobile and gaming, content distribution

Question 4

S3 infrequent access storage class

Answer 4

• lower cost, cost on retrieval
• standard-IA
  
  • disaster recover, backups
• one zone-IA
  
  • store secondary copies of backups
  • less availability

Question 5

S3 Glacier storage class

Answer 5

• archive and backup
• pay per storage + retrieval cost
• instant retrieval
  
  • milisecond retrieval
  • minimum storage duration of 90 days
• Glacier Flexible Retrieval
  
  • expedited (retrieval in 1-5 minutes) standard (3-5 hrs) bulk (5-12 hrs)
  • minimum storage duration of 90 days
• Glacier Deep Archive
  
  • standard (12) hrs Bulk (48hrs)
  • 180 days minimum storage

Question 6

S3 intelligent tiering

Answer 6

• move between tiers based on usage
• higher cost
  
  • but no retrieval charges

Question 7

S3 lifecycle rules

Answer 7

• can automate moving between storage classes using lifecycle configuration
• define transition actions
  
  • i.e. move object to glacier after 6 months
• define expiration actions
  
  • delete object (or version of object) after set time
• rules can be created for certain prefixes i.e. (bucket/mp3/*)

Question 8

S3 performance

Answer 8

• automatically scales to high request rates, latency 100-200ms
• 3500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per prefix in bucket
• no limit to prefixes

Question 9

S3 multi-part upload

Answer 9

• recommended for files > 100MB
• must use for files > 5GB

Question 10

S3 Select and Glacier Select (server side filtering)

Answer 10

• filter rows and columns (simple SQL statements)
• less network, cpu
• only request what you need

Question 11

Athena

Answer 11

• serverless query service to perform analytics against S3 objects
• use SQL to query files
• supports CSV, JSON, etc.