Cloudiscovery helps you to analyze resources in your cloud (AWS/GCP/Azure/Alibaba/IBM) account. Now this tool only can check resources in AWS, but we are working to expand to other providers.
The tool consists of various commands to help you understand the cloud infrastructure.
Commands can generate diagrams. When modelling them, we try to follow the principle:
Graphical excellence is that which gives to the viewer the greatest number of ideas in the shortest time with the least ink in the smallest space.
Edward Tufte
The commands generate reports that can be used to further analyze resources.
- Run the cloudiscovery command with following options (if a region not pass, this script will try to get it from ~/.aws/credentials):
1.1 To detect AWS VPC resources (more on AWS VPC):
cloudiscovery aws-vpc [--vpc-id vpc-xxxxxxx] --region-name xx-xxxx-xxx [--profile-name profile] [--diagram [yes/no]] [--filter xxx] [--verbose]1.2 To detect AWS policy resources (more on AWS Policy):
cloudiscovery aws-policy [--profile-name profile] [--diagram [yes/no]] [--filter xxx] [--verbose]1.3 To detect AWS IoT resources (more on AWS IoT):
cloudiscovery aws-iot [--thing-name thing-xxxx] --region-name xx-xxxx-xxx [--profile-name profile] [--diagram [yes/no]] [--filter xxx] [--verbose]1.4 To detect all AWS resources (more on AWS All):
cloudiscovery aws-all --region-name xx-xxxx-xxx [--profile-name profile] [--services xxx,xxx] [--filter xxx] [--verbose]1.5 To check AWS limits per resource (more on AWS Limit):
cloudiscovery aws-limit --region-name xx-xxxx-xxx [--profile-name profile] [--services xxx,xxx] [--usage 0-100] [--verbose]1.6 To run AWS security controls (experimental feature):
cloudiscovery aws-security --region-name xx-xxxx-xxx [--profile-name profile] [--commands x] [--verbose]- For help use:
cloudiscovery [aws-vpc|aws-policy|aws-iot|aws-all|aws-limit] -hEnabling verbose mode, it is possible to debug all calls to the providers endpoints and check possible problems.
It's possible to filter resources by tags and resource type. To filter, add an option --filter <VALUE>, where <VALUE> can be:
Name=tags.costCenter;Value=20000- to filter resources by a tag namecostCenterand with value20000.Name=type;Value=aws_lambda_functionto only list lambda functions.
It's possible to pass multiple values, to be able to select a value from a set. Values are split by : sign. If a desired value has a : sign, wrap it in ' signs e.g. --filter="Name=tags.costCenter;Value=20000:'20001:1'.
It is possible to pass multiple filter options, just pass -f filter_1 -f filter_2. In that case, the tool will return resources that match either of the filters
Useful CF tags:
aws:cloudformation:stack-name- Stack nameaws:cloudformation:stack-id- Stack idaws:cloudformation:logical-id- Logical id defined in CF template
This tool has been written in Python3+ and AWS-CLI and it works on Linux, Windows and Mac OS.
Make sure the latest version of AWS-CLI is installed on your workstation, and other components needed, with Python pip already installed:
pip install -U cloudiscoveryOnce a while after installation, there can be some issues related with a cache from older version being used by a newer version. In that case, it's recommended to remove directory ./assets/.cache.
Make sure you have properly configured your AWS-CLI with a valid Access Key and Region:
aws configureMore on credentials configuration: Configuration basics
The configured credentials must be associated to a user or role with proper permissions to do all checks. If you want to use a role with narrowed set of permissions just to perform cloud discovery, use a role from the following the CF template maintained by our team.
To further increase security, you can add a block to check aws:MultiFactorAuthPresent condition in AssumeRolePolicyDocument. More on using IAM roles in the configuration file.
(Optional) If you want to be able to switch between multiple AWS credentials and settings, you can configure named profiles and later pass profile name when running the tool.
Cloudiscovery provides a CLI to easily perform desired actions.
Example of a diagram (diagrams.net supported):
Following resources are checked in VPC command:
- Autoscaling Group
- Classic/Network/Application Load Balancer
- Client VPN Endpoints
- CloudHSM
- DocumentDB
- Directory Service
- EC2 Instance
- ECS
- EFS
- ElastiCache
- Elasticsearch
- EKS
- EMR
- IAM Policy
- Internet Gateway (IGW)
- Lambda
- Media Connect
- Media Live
- Media Store Policy
- MSK
- NACL
- NAT Gateway
- Neptune
- QuickSight
- RDS
- REST Api Policy
- Route Table
- S3 Policy
- Sagemaker Notebook
- Sagemaker Training Job
- Sagemaker Model
- Security Group
- SQS Queue Policy
- Site-to-Site VPN Connections
- Subnet
- Synthetic Canary
- VPC Peering
- VPC Endpoint
- VPN Customer Gateways
- Virtual Private Gateways
- Workspace
The subnets are aggregated to simplify the diagram and hide infrastructure redundancies. There can be two types of subnet aggregates:
- Private* ones with a route
0.0.0.0/0to Internet Gateway - Public* ones without any route to IGW
If EC2 instances and ECS instances are part of an autoscaling group, those instances will be aggregated on a diagram.
More information: AWS WA, REL 2: How do you plan your network topology?
Example of a diagram:
Following resources are checked in Policy command:
- AWS Principal that are able to assume roles
- IAM Group
- IAM Group to policy relationship
- IAM Policy
- IAM Role
- IAM Role to policy relationship
- IAM User
- IAM User to group relationship
- IAM User to policy relationship
Some roles can be aggregated to simplify the diagram. If a role is associated with a principal and is not attached to any named policy, will be aggregated.
More information: AWS WA, SEC 3: How do you manage permissions for people and machines?
Example of a diagram:
Following resources are checked in IoT command:
- IoT Billing Group
- IoT Certificates
- IoT Jobs
- IoT Policies
- IoT Thing
- IoT Thing Type
A command to list ALL AWS resources.
Example of an HTML report:
The command calls all AWS services (200+) and operations with name Describe, Get... and List... (500+).
The operations must be allowed to be called by permissions described in AWS Permissions.
Types of resources mostly cover Terraform types. It is possible to narrow down scope of the resources to ones related with a given service with parameter -s e.g. -s ec2,ecs,cloudfront,rds.
More information: AWS WA, COST 2: How do you govern usage?
It's possible to check resources limits across various service in an account. This command implements over 60 limits checks.
Example of an HTML report:
With --services value,value,value parameter, you can narrow down checks to just services that you want to check.
With --threshold 0-100 option, you can customize a minimum percentage threshold to start reporting a warning.
- Services available
- Acm
- Amplify
- Apigateway
- Appmesh
- Appsync
- Autoscaling Plans
- Batch
- Chime
- Code Artifact
- Code Build
- Code Commit
- Code Deploy
- Codeguru Reviewer
- Codeguru Profiler
- Cognito Federated Identities
- Cloudformation
- Cloud Map
- CloudWatch Logs
- Dynamodb
- EBS
- EC2
- ECR
- ECS
- Elastic Inference
- Elastic Filesystem
- Elastic Beanstalk
- Elastic Loadbalancing
- Forecast
- Fraud Detector
- Gamelift
- Glue
- IAM
- Inspector
- Kendra
- KMS
- Media Connect
- Media Live
- Media Package
- Metwork Manager
- Polly
- Qldb
- Robomaker
- Route53
- Route53resolver
- RDS
- S3
- SES
- SNS
- SWF
- Transcribe
- Translate
- VPC
AWS has a default quota to all services. At the first time that an account is created, AWS apply this default quota to all services.
An administrator can ask to increase the quota value of a certain service via ticket. This command helps administrators detect those issues in advance.
More information: AWS WA, REL 1 How do you manage service limits?
This features is experimental, but now you can run commands to check and analyze some security issues. The following commands are available now:
- Access key age
- EBS Encryption enabled
- EC2 IMDSV2 Check
- DynamoDB PITR Enabled
- Incoming SSH Disabled
- Cloudtrail enabled
If you wish to analyze accounts in regions outside the main AWS partition (e.g. GovCloud or China), you should provide credentials (e.g. a profile) that are applicable to a given partition. It's not possible to analyze regions from multiple partitions.
To build docker container using Dockerfile
docker build -t cloudiscovery .After build container, you must start container using follow command. The run command will mount a filesystem with your actual aws cli credentials, then you won't need configure aws cli again.
docker run \
-it \
--mount type=bind,source=$HOME/.aws/,target=/root/.aws/,readonly \
cloudiscovery \
/bin/bash
- If you are using Diagram output and due to fact container is a slim image of Python image, you must run cloudiscovery with "--diagram no", otherwise you'll have an error about "xdg-open". The output file will be saved in "assets/diagrams".
This project support English and Portuguese (Brazil) languages. To contribute with a translation, follow this steps:
- Create a folder inside locales folder with prefix of new idiom with appropiate locale code. Copy "locales/messages.pot" to locales/newfolder/LC_MESSAGES/.
- To build ".mo" file running this command from project root folder:
python msgfmt.py -o locales/NEWFOLDER/LC_MESSAGES/messages.mo locales/NEWFOLDER/LC_MESSAGES/messagesIf you have improvements or fixes, we would love to have your contributions. Please use PEP 8 code style.
When developing, it's recommended to use venv.
In order to create a venv on macOS and Linux:
python3 -m venv envOn Windows:
py -m venv venv
OR
python -v venv venvOnce installed, you need to activate the virtual environment. Activation will put specific paths for python and pip commands.
On macOS and Linux call:
source venv/bin/activateOn Windows:
.\venv\Scripts\activateMake sure you have installed pre-commit.
Install development requirements:
pip install -U -r requirements.txt -r requirements-dev.txtAdd precommit hooks:
pre-commit installTo run pre-commit hooks, you can issue the following command:
pre-commit run --all-filesRunning cloudiscovery in development mode:
python cloudiscovery/__init__.py OPTIONS
To add new resources to check limit, please remove "assets/.cache/cache.db"
- Update the version in cloudiscovery/__init__.py and create a new git tag with
git tag $VERSION. - Once you push the tag to GitHub with
git push --tags, a new CircleCI build is triggered.
- mingrammer/diagrams - library being used to draw diagrams
- Lucidchart Cloud Insights - commercial extension to Lucidchart
- Cloudcraft - commercial visualization tool
