AWS credentials, S3 configuration

AWS Credentials

3 values are required by AWS for our use of the aws-sdk-s3 library:

• AWS_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

Note: Keys are region-specific so these 3 values will typically be set all at the same time.

ENV

Our application expects these to be provided via ENV variables (which is the most common way to configure auth for AWS now).

Example:

AWS_ACCESS_KEY_ID=AKIAXYZ AWS_SECRET_ACCESS_KEY=/xx/xxx AWS_REGION=us-west-2 bundle exec rails console production

This is equivalent to:

export AWS_ACCESS_KEY_ID=AKIAXYZ
export AWS_SECRET_ACCESS_KEY=/xx/xxx 
export AWS_REGION=us-west-2 
bundle exec rails console production
# don't forget to exit the shell or unset the variables to avoid mistakenly including them later

Our application will also recognize the AWS_PROFILE ENV variable, which will distinguish one 'set' of key and region values, multiple sets of which can be defined in configuration files. Currently we use Puppet to provision those configuration files onto our worker machines.

Jobs Jobs Jobs

The component most sensitive to this configuration will be the "delivery jobs" specific to each endpoint (currently the only place S3 is actually used). Therefore the respective workers will be started via the command-line with their correct configurations, like:

AWS_ACCESS_KEY_ID=ABC AWS_SECRET_ACCESS_KEY=123 AWS_REGION=us-west-2 QUEUES=s3_endpoint_delivery bundle exec rake resque:work
AWS_ACCESS_KEY_ID=XYZ AWS_SECRET_ACCESS_KEY=987 AWS_REGION=us-east-1 QUEUES=s3_us_east_1_delivery bundle exec rake resque:work

The mechanism to accomplish that is effectively delegated to Ops.