Skip to content

Commit

Permalink
Merge pull request #1180 from guardian/an/pluto-message-ingestion-lam…
Browse files Browse the repository at this point in the history
…bda-remove-externals

pluto-message-ingestion: remove externals from ncc config
  • Loading branch information
andrew-nowak authored and bryophyta committed Dec 11, 2024
2 parents b57c0f2 + 7ef76fc commit 7b781e7
Show file tree
Hide file tree
Showing 2 changed files with 24 additions and 9 deletions.
29 changes: 24 additions & 5 deletions pluto-message-ingestion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,32 @@

An AWS Lambda that provides integration with [Pluto](https://github.com/guardian/pluto) via a Kinesis stream.

Pluto and MAM do not communicate directly as the AWS VPCs are not setup. Instead, they communicate via a Kinesis stream.
Pluto and MAM do not communicate directly as the AWS VPCs are not setup. Instead, they communicate via a Kinesis stream.
Pluto puts JSON messages on this stream and this Lambda picks them up.

The Lambda switches on `message.type` and calls the relevant MAM end-point.

## Message types processed

> [!INFO]
> To test the lambda in CODE, instead of sending messages on the stream
> you can create a test event in the Lambda console.
> Start by creating a message in one of the formats below, then base64-encode
> it, then insert that encoded message into an event that looks like:
> ```json
> {
> "Records": [
> {
> "kinesis": {
> "data": "[your encoded message]"
> }
> }
> ]
> }
> ```
> Your fake Pluto project and commission should now be sent to MAM and stored
> in the `media-atom-maker-CODE-pluto-[projects/commissions]-table`s
### `project-created`
Pluto will send a message to Kinesis when a new Project has been created. MAM will then store a copy of the metadata in Dynamo.
Expand Down Expand Up @@ -49,15 +68,15 @@ Pluto sends a JSON blob that looks like:

### Updating pre-existing projects
If an old Pluto Project is updated, a `project-updated` message still gets put on the Kinesis stream.
As MAM has not seen this project before, it does not know what to update and thus the message is discarded,
As MAM has not seen this project before, it does not know what to update and thus the message is discarded,
logging `attempted to update a project that does not exist`.

This means only projects created **after** the Pluto and MAM changes have gone out can be selected as a destination
for Direct Upload.
This means only projects created **after** the Pluto and MAM changes have gone out can be selected as a destination
for Direct Upload.

### Unknown message types
If a message is put on the Kinesis stream that is not recognised, it will be logged and the lambda will succeed.
This is because Kinesis blocks on failure; succeeding takes the message off the stream so we can continue with
This is because Kinesis blocks on failure; succeeding takes the message off the stream so we can continue with
remaining messages.

See [http://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html](http://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html) for more info.
4 changes: 0 additions & 4 deletions pluto-message-ingestion/ncc.config.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
{
"externals": {
"aws-sdk": "aws-sdk",
"/aws-sdk(/.*)/": "aws-sdk$1"
},
"minify": true
}

0 comments on commit 7b781e7

Please sign in to comment.