Sharing a database between microservices

[efekarakus]

A frequently asked question while creating a microservice architecture with AWS Copilot is how to share a database between these services.

We outlined solutions in #4273 (comment) and #2495 (comment), so I'm copying it over here in order to make it easier to discover the solution.

1. [preferred] Only one microservice should have read/write access to the database, remaining services access data through communication with service connect.
   One microservice should run copilot storage init to create the RDS cluster and have direct access to the database. This service can expose an API so that internal services can retrieve data from the database. For example, if the service with the database is called "warehouse" and exposes port 80 in image.build, then we can add a new API in the "warehouse" service for say "GET /inventory/{productId}". Then the backend service can make a request to that path:

   resp, err := http.Get("http://warehouse/inventory/1234") 

The tradeoff is that it's a bit more work upfront then just letting other services access the database. However, you gain the benefit of data encapsulation so you can always change the underlying data model safely. This architecture is preferred if you have several teams in your organization that need to work independently from each other.

2. An alternative proposal is to allow ingress from all the services in the environment by modifying the security group of the RDS cluster.
   There is a <App>-<Env>-EnvironmentSecurityGroup that's assigned to all services. After copilot generates the addon template with storage init, you can modify the CloudFormation template and add this ingress rule to the cluster's security group. Here is a snippet that may work for you:

   <clusterName>DBClusterSecurityGroup: # replace <clusterName> 
     Metadata:
       'aws:copilot:description': 'A security group for your DB cluster <cluster>' # replace <clusterName> 
     Type: AWS::EC2::SecurityGroup
     Properties:
       GroupDescription: The Security Group for the database cluster.
       SecurityGroupIngress:
         - ToPort: 5432
           FromPort: 5432
           IpProtocol: tcp
           Description: !Sub 'From the Aurora Security Group of the workload ${Name}.'
           SourceSecurityGroupId: !Ref backendclusterSecurityGroup
         - ToPort: 5432 # The additional ingress rule that you can add for other service to access the cluster
           FromPort: 5432
           IpProtocol: tcp
           Description: From Environment Security Group
           SourceSecurityGroupId: 
             Fn::ImportValue: 
               !Sub '${App}-${Env}-EnvironmentSecurityGroup'

   With this ingress, the cluster's security group will allow access from all services deployed in the environment.
   You'll need to also update the manifest of the remaining services with the database secret created in the addon stack.

   secrets:
     DBCREDENTIAL: <secret ARN>

   For additional ways of injecting a secret from AWS Secrets Manager see the following guide.

   The tradeoff is that we created tight coupling between our services because now they can impact each other. Services are all aware of the database schema, and a service A can modify data that it shouldn't. However, this setup is a lot more agile and easier to get going. This architecture is preferred if you're a smaller organization with a few teams and high trust.

Hope this helps!

[mlazar-endear]

Hi, I just want to share my use-case as a data point. I'm running a typical python web stack with two services (simplified for this example):

• A Load Balanced Web Service running nginx + gunicorn that serves the website & API.
• A Worker Service running celery that processes background tasks from the web service via SQS.

In this case, both services are running the exact same django codebase & docker container, so they share the same schema and are expected to use the same database. Solution 1 does not make sense and would be a huge anti-pattern for this model (Solution 2 is perfect though).