ELB Log Analyzer

Tool for analyzing ELB logs for automating steps to retreive details of ip's user agent, total request count, to which urls requests were made along with their total count, and http methods in json format.

S3 Bucket Log Downloader

Downloads S3 bucket objects that we created in specified time window.

Installation

• Using Pip

  python3 -m pip install elb-log-analyzer

AWS configuration

• Create IAM policy with below configuration

  {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "S3ListSpecificDirectory",
          "Effect": "Allow",
          "Action": "s3:ListBucket",
          "Resource": "arn:aws:s3:::alb-log-bucket-name"
      },
      {
          "Sid": "S3GetSpecificDirectory",
          "Effect": "Allow",
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*"
      }
  ]
  }

  Note: above policy will allow user to list all contents in the bucket but download objects only from s3://alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/*

• Create AWS access keys

• Use aws cli to configure access key for boto3

  aws configure

S3 Bucket Log Downloader Usage

• Print Help Menu.

  python3 -m elb_log_analyzer.s3_log -h

• Download all log files generated in 10 hours from now.

  python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -H 10

• Download all log files generated in 40 mins from now.

  python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -m 40

• Download all log files generated in 20 secs from now.

  python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -s 20

• Download all log files generated in 10 hours, 40 mins and 20 secs from now and store in a directory.

  python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' --hours 10 --minutes 40 --seconds 20 -o './logs/downloads'

Analyzer

Analyzes downloaded log files.

Analyzer Usage

• Print Help Menu

  python3 -m elb_log_analyzer -h

• Print json data on console

  python3 -m elb_log_analyzer -i [INPUT_LOG_FILE_PATH]

• Store json data in a file

  python3 -m elb_log_analyzer -i [INPUT_LOG_FILE_PATH] -o [OUTPUT_FILE_PATH]

  Note: INPUT_LOG_FILE_PATH can be log file or a directory containing all log files ending with .log extension

• Get IP details from IPAbuseDB

  python3 -m elb_log_analyzer -i [LOG_FILE_PATH] -t [REQUESTS_THRESHOLD_VALUE] -k [IP_ABUSE_DB_API_KEY] -o [OUTPUT_FILE_PATH]

Alerts

Send alert to slack channel with abusive ip details.

Usage

• Send alert from analyzed file

  python elb_log_analyzer.alerts -w [SLACK_WEBHOOK] -f [ANALYZED_LOG_FILE_LOCATION]

Dashboard

Dashboard to visualize data.

Dashboard Installation

• Install requirements

  python3 -m pip install dashboard/requirements.txt

Usage

• Start App

  streamlit run dashboard/app.py

• Enter Log File/Directory Path

Publish package to pypi

• Using poetry

  python3 -m poetry publish --build --username [PYPI_USERNAME] --password [PYPI_PASSWORD]

Usage Summary

• Download log files

  python3 -m elb_log_analyzer.s3_log -b elb-log-bucket -p 'alb-log-bucket-name/AWSLogs/XXXXXXXXXXXX/elasticloadbalancing/aws-region/' -H [HOURS] -o logs

• Analyze Log Files

  python3 -m elb_log_analyzer -i logs -o log.json -t [REQUEST_THRESHOLD] -k [IP_ABUSE_API_KEY] 

• Send Alert to slack with client ips having total number of requests greater than threshold requests

  python -m elb_log_analyzer.alerts -w [SLACK_WEBHOOK] -f [ANALYZED_LOG_FILE_LOCATION]

• Visualize Analyzed Logs using Dashboard

  streamlit run dashboard/app.py

Docker WorkFlow

• Pull image

  docker pull dmdhrumilmistry/elb-log-analyzer

• Create an .env file

  # bucket configuration
  BUCKET_NAME='elb-logs-bucket-name'
  BUCKET_PREFIX='AWSLogs/XXXXXXXX/elasticloadbalancing/eu-west-2/'
  
  # SECRETS conf
  REQUESTS_THRESHOLD=400
  IP_ABUSE_DB_API_KEY='UPDATE_HERE'
  SLACK_WEBHOOK='UPDATE_HERE'
  
  # consts
  DATE_SUFFIX="$(date '+%Y/%m/%d')"
  LOG_ANALYSIS_INTERVAL=5

• Start Container

  docker run --env-file .env --rm dmdhrumilmistry/elb-log-analyzer