New Event Notifications for Amazon #AWS #S3

Many AWS customers have been building applications that
use Amazon Simple Storage Service () for cost-efficient and highly scalable
persistent or temporary object storage. Some of them want to initiate
processing on the objects as they arrive; others want to capture
information about the objects and log it for tracking or security
purposes. These customers have been asking for a reliable and
scalable way to be notified when an S3 object is created or overwritten.

S3 Event Notifications
Today we are launching a new event notification feature for S3. The bucket owner
(or others, as permitted by an IAM policy) can now arrange for notifications to be
issued to Amazon Simple Queue Service (SQS) or Amazon Simple Notification Service (SNS) when a new object is added to the bucket or an
existing object is overwritten. Notifications can also be delivered to
AWS Lambda for processing by a Lambda function. Here’s the general flow:

Here’s what you need to do in order to start using this new feature with your application:

  1. Create the queue, topic, or Lambda function (which I’ll call the target for brevity) if necessary.
  2. Grant S3 permission to publish to the target or invoke the Lambda function.
    For SNS or SQS, you do this by applying an appropriate policy to the topic or the queue. For Lambda, you
    must create and supply an IAM role, then associate it with the Lambda function.
  3. Arrange for your application to be invoked in response to activity on the target. As you will see in a moment, you have several options here.
  4. Set the bucket’s Notification Configuration to point to the target.

From that point forward the events will be reliably delivered to the target as appropriate.

Notifications are configured at the bucket level and apply to all of the objects in the bucket (we plan to provide
control at a finer level at some point). You can elect to receive notification for any or all of the following events:

  • s3:ObjectCreated:Put – An object was created by an HTTP PUT operation.
  • s3:ObjectCreated:Post – An object was created by HTTP POST operation.
  • s3:ObjectCreated:Copy – An object was created an S3 copy operation.
  • s3:ObjectCreated:CompleteMultipartUpload – An object was created by the completion of a
    S3 multi-part upload.
  • s3:ObjectCreated:* – An object was created by one of the event types listed above
    or by a similar object creation event added in the future.
  • s3:ReducedRedundancyObjectLost – An S3 object stored with Reduced Redundancy has been
    lost.

Notification Details
Each notification is delivered as a JSON object with the following fields:

  • Region
  • Timestamp
  • Event Type (as listed above)
  • Request Actor Principal ID
  • Source IP of the request
  • Request ID
  • Host ID
  • Notification Configuration Destination ID
  • Bucket Name
  • Bucket ARN
  • Bucket Owner Principal ID
  • Object Key
  • Object Size
  • Object ETag
  • Object Version ID (if versioning is enabled on the bucket)

As you probably know already, S3 is eventually consistent. When you
receive an object creation notification, it is possible that the
new object has yet to be replicated and that a Get operation
will return an older copy. In order to allow you to detect and
handle this eventuality, the notification includes the
ETag of the
new object. Your code can Get the object and verify the ETag before
processing. If the ETags do not match, you can defer processing
by posting the message back to the SNS or SQS target.
Note that eventual consistency is a concern only
if your application allows existing objects to be overwritten.

Configuring Notifications Using the Console
Here is how to configure an event notification using the AWS Management Console. I have a bucket named jbarr-upload
and I want to send notifications to an SNS topic named jbarr-upload-notify. I have already configured
the topic to send an email to me. I start by granting permission for S3 to publish to the topic:

Then I configure the bucket to send notification to my topic:

I can use the menu to choose the event types that are of interest to me:

For testing purposes, I use the console to upload an object:

Here’s the resulting email notification (I’ve formatted the JSON for readability in order to get
my point across):

I’m confident that you can pick up from where I left off and start integrating this feature into your
own applications. You can, of course, use the AWS SDKs to configure and manage notifications.

Things to Know
Here are a couple of things to keep in mind as you start to think about the best way to use these
new notifications as part of your application:

  • Delivery Latency
    Notifications are delivered to the target in well under a second.
  • Cost – There is no
    charge for this feature. You will pay the usual messaging and
    execution charges for
    SQS, SNS, and Lambda (many applications can run within the AWS Free Tier).
  • Regions – The bucket
    and the target (SQS, SNS, or Lambda) must reside
    in the same Region.
  • Event Types – You can configure
    one notification per event type per bucket.
  • Delivery Reliability
    S3 is designed to deliver notifications with a very high degree of reliability. It
    includes built-in backoff and retry mechanisms to deal with momentary issues that
    might affect the deliverability of messages to any of the three types of targets.
  • Additional Event Types
    We expect to add additional event types over time and your feedback will
    help us to prioritize our work. Please feel free to tell us more about
    your needs in the S3 Forum.

Availability
This feature is available now and you can start using it today! I am looking forward to
hearing all about the interesting use cases that you come up with.

Jeff;

Related posts