Secure CI deployment pipeline at AWS pt 1: Theory

Jenkins is security nightmare when you want to use it at deployment to production. If the Jenkins node does the deployment, the node has to have all access rights which are needed for deployment. So it might be related to infrastructure, IAM policies, access to AWS secret manager etc. Anyone who has modification access to the Jenkins job, has also full access to all of these. This might be even GDPR related risk, so building the secure deployment needs some thinking.

I'm expecting that every company has multiple accounts at their AWS Organization. The CI pipeline and other DevOps related systems are at the own AWS account. The production and test environments have own accounts. If we want Jenkins node to do the deployment, we'd have to give quite big access rights to production environment for it. That violates the least privilege approach.

So let's start to think about building blogs. AWS Lambda is nice way to run small functions, which can be triggered quite easily. So maybe that can solve part of the deployment process. AWS SNS is good way to start the Lambda. Now if the Jenkins node sends SNS message, which starts the Lambda which starts the deployment process the security is much better. But problem is that Lambda can't execute external applications like kubectl, terraform or some other fancy deployment tool. ECS is really nice for that. If it's executed at Fargate cluster the costs are minimal. But that needs also the feedback to Jenkins. SQS is perfect for that.

The basic architecture is like the diagram below. It's missing e.g. logging related components.

So what we have to program?

• The script which is executed by CI pipeline. It first sends the triggering message to SNS. Then it starts to listen the feedback queue. Only required cross account permissions are SNS sending and SQS reading.
• Lambda script which starts the deployment container. Only rights which it needs are starting the ECS container.
• Deployment container requires the script which submits all standard output and error from deployment process to the SQS. This container should have role which has all required permissions to do the deployment. 

So now we've got the components which are required for the secure cross account deployment. Jenkins can be replaced with almost any CI/CD tool. 

I'll continue this series as soon as I get some programming done.