Skip to content

AWS Summit 2023 ASEAN --- COM301 Implementing Observability using AWS Distro for OpenTelemetry (ADOT). Usecase Implementation for Simple Golang RESTful API Application (Bookstore)

License

Notifications You must be signed in to change notification settings

devopscorner/golang-adot

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Golang Bookstore ADOT

ADOT (AWS Distro for OpenTelemetry) Implementation for Simple Golang RESTful API Application (Bookstore)

goreport all contributors tags docker pulls download all download latest view clone issues pull requests forks stars license


Available Tags

Alpine GOLANG

Image name Size
devopscorner/bookstore-adot:latest docker image size default-latest latest
devopscorner/bookstore-adot:alpine docker image size latest
devopscorner/bookstore-adot:alpine-latest docker image size
devopscorner/bookstore-adot:alpine-3.17 docker image size

CICD Alpine-3.17

Image name Size
devopscorner/bookstore-adot:cicd-alpine docker image size
devopscorner/bookstore-adot:cicd-alpine-latest docker image size
devopscorner/bookstore-adot:cicd-alpine-3.17 docker image size

CICD CodeBuild-4.0

Image name Size
devopscorner/bookstore-adot:cicd-codebuild docker image size default-cicd-codebuild cicd-codebuild
devopscorner/bookstore-adot:cicd-codebuild-latest docker image size
devopscorner/bookstore-adot:cicd-codebuild-4.0 docker image size
devopscorner/bookstore-adot:cicd-latest docker image size cicd-latest

Documentation

  • Index Documentation, go to this link
  • Build, Tag & Push container image to DockerHub, go to this link
  • Build, Tag & Push container image to Amazon ECR (Elastic Container Registry), go to this link
  • Workflow CI/CD Pipeline, go to this link
  • Deployments:
    • AWS Developer Tools (AWS CodeCommit, AWS CodeBuild & AWS CodePipeline), detail here link
    • Jenkins CI/CD, detail here link
    • Terraform AWS CodeBuild, AWS CodePipeline & Amazon SNS, detail here link

GO Repository Pattern

  • Folder Structure
.
├── .env
├── .env.example
├── config
│   ├── config.go
│   ├── config_test.go
│   ├── const.go
│   ├── logger.go
│   └── value.go
├── controller
│   ├── book_controller.go
│   ├── book_controller_test.go
│   ├── login_controller.go
│   └── login_controller_test.go
├── driver
│   ├── db.go
│   ├── dynamodb.go
│   ├── mysql.go
│   ├── psql.go
│   └── sqlite.go
├── go.mod
├── go.sum
├── main.go
├── main_test.go
├── middleware
│   ├── auth_middleware.go
│   └── auth_middleware_test.go
├── migrate_book.go.example
├── migrate_book_dynamo.go.example
├── model
│   └── book.go
├── observability
│   ├── metrics.go
│   ├── provider.go
│   ├── tracing-otel.go
│   └── tracing-xray.go
├── repository
│   └── book_repository.go
├── routes
│   ├── book_routes.go
│   ├── main_routes.go
│   ├── telemetry_routes.go
│   └── tracing_routes.go
└── view
    ├── book_view.go
    ├── error_view.go
    └── login_view.go

10 directories, 37 files

Coverages:

AWS Services

  • Amazon Elastic Container Registry (ECR)
  • Amazon Managed Service for Prometheus (AMP)
  • Amazon Managed Service for Grafana (AMG)
  • AWS Distro for OpenTelemetry (ADOT)
  • AWS CloudWatch
  • AWS X-Ray
  • AWS Systems Manager Parameter Store
  • AWS Developer Tools (AWS CodeCommit, AWS CodePipeline, AWS CodeBuild)
  • Amazon EKS
  • Amazon OpenSearch -- (Optional)
  • Amazon DynamoDB

Others:

  • Docker
  • Docker-Compose
  • Golang
  • HelmChart
  • Postman
  • Terraform

Environment Variables

  • Default Value

    GIN_MODE=release
    APP_URL=http://localhost
    APP_PORT=8080
    DB_CONNECTION=dynamo
    DB_HOST=localhost
    DB_PORT=
    DB_DATABASE=dynamodb-golang-adot
    DB_USERNAME=root
    DB_PASSWORD=
    JWT_AUTH_USERNAME=devopscorner
    JWT_AUTH_PASSWORD=DevOpsCorner2023
    JWT_SECRET=s3cr3t
    
  • Multi Driver Connection

    # default
    DB_CONNECTION=dynamo
    ---
    Available for:
    - sqlite
    - mysql
    - postgres
    - dynamo
    
  • DynamoDB Connection

    DB_CONNECTION=dynamo
    ---
    DB_DATABASE       --> Dynamo Table
    AWS_REGION        --> AWS Region (Dynamo Region)
    AWS_ACCESS_KEY    --> AWS Access Key
    AWS_SECRET_KEY_ID --> AWS Secret Key Id
    
  • AWS X-Ray (AWS Distribution Tracing System)

    • XRAY_VERSION: Daemon Version X-Ray

      • latest: Use the latest version of the AWS X-Ray daemon available.
      • 3.x: Use version 3.x of the AWS X-Ray daemon.
      • 2.x: Use version 2.x of the AWS X-Ray daemon.
      • 1.x: Use version 1.x of the AWS X-Ray daemon.
    • `XRAY_DAEMON_ENDPOINT: Daemon Endpoint of X-Ray

      XRAY_DAEMON_ENDPOINT=http://localhost:2000
      
    • `XRAY_DAEMON_PORT: Daemon Endpoint Port of X-Ray

      XRAY_DAEMON_PORT=2000
      
  • Prometheus

    PROMETHEUS_ENDPOINT=http://localhost:9090
    PROMETHEUS_PORT=9090
    
  • Grafana

    GRAFANA_ENDPOINT=http://localhost:3000
    GRAFANA_API_KEY=YOUR_GRAFANA_API_KEY
    
  • OpenTelemetry

    OTEL_INSTRUMENTATION_METRIC_ENABLED=true
    OTEL_INSTRUMENTATION_TRACE_ENABLED=true
    OTEL_INSTRUMENTATION_LOG_ENABLED=true
    
    # Trace Type: xray / jaeger
    OTEL_INSTRUMENTATION_TRACE_NAME=xray
    
    OTEL_SERVICE_NAME=bookstore-adot
    OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
    OTEL_EXPORTER_OTLP_PORT=4317
    OTEL_EXPORTER_OTLP_INSECURE=true
    OTEL_EXPORTER_OTLP_HEADERS=
    OTEL_RESOURCE_ATTRIBUTES=
    
  • Jaeger Environment

    • JAEGER_SERVICE_NAME: The name of the service being instrumented (JAEGER_SERVICE_NAME = OTEL_SERVICE_NAME).
    • JAEGER_AGENT_HOST: The host name or IP address of the Jaeger agent to use for sending trace data.
    • JAEGER_AGENT_PORT: The port number of the Jaeger agent to use for sending trace data.
    • JAEGER_SAMPLER_TYPE: The type of sampling to use for tracing (e.g. const, probabilistic, rateLimiting, remote).
    • JAEGER_SAMPLER_PARAM: The parameter value to use for the selected sampler type.
    • JAEGER_SAMPLER_MANAGER_HOST_PORT: The host name and port of the remote sampling manager to use (if using remote sampling).
    • JAEGER_REPORTER_LOG_SPANS: Whether to log spans instead of sending them to a Jaeger agent (e.g. true or false).
    • JAEGER_REPORTER_MAX_QUEUE_SIZE: The maximum number of spans that can be queued before they are sent to a Jaeger agent.
    • JAEGER_REPORTER_FLUSH_INTERVAL: The interval at which to flush the span queue and send spans to a Jaeger agent (e.g. 1 * time.Second).
    • JAEGER_REPORTER_LOCAL_AGENT_HOST_PORT: The host and port of the local Jaeger agent, if using UDP instead of HTTP.
    • JAEGER_REPORTER_COLLECTOR_ENDPOINT: The endpoint URL of the Jaeger collector (e.g. http://localhost:14268/api/traces).
    • JAEGER_REPORTER_COLLECTOR_USER: The username for authenticating with the Jaeger collector, if required.
    • JAEGER_REPORTER_COLLECTOR_PASSWORD: The password for authenticating with the Jaeger collector, if required.
    • JAEGER_TAGS: Additional tags to attach to the trace (e.g. key1=value1,key2=value2).

Reproduce Testing

  • RESTful API Testing, go to this link

Reproduce Provisioning

  1. Install Terraform Modules, detail here
  2. Provisioning HelmChart Template, detail here
  3. Provisioning TFState DB & Bucket, detail here
  4. Provisioning Infra Core, detail here
  5. Provisioning Infra CI/CD, detail here
  6. Provisioning Resources Amazon EKS, detail here
  7. Provisioning Resources Managed Services Prometheus (AMP), detail here
  8. Provisioning Resources Managed Services Grafana (AMG), detail here
  9. Provisioning Resources Managed Services OpenSearch, detail here
  10. Provisioning Resources EKS ADOT Collector, detail here

Snapshots

  • Grafana Dashboard golang-adot-001 golang-adot-002

  • XRay golang-adot-003 golang-adot-004

Tested Environment

Versioning

  • Docker version

    docker version
    
    Client:
      Cloud integration: v1.0.22
      Version:           20.10.17-rd
      API version:       1.41
      Go version:        go1.17.11
      Git commit:        c2e4e01
      Built:             Fri Jul 22 18:31:17 2022
      OS/Arch:           darwin/amd64
      Context:           default
      Experimental:      true
    
  • Docker-Compose version

    docker-compose -v
    ---
    Docker Compose version v2.11.1
    
  • AWS Cli

    aws --version
    ---
    aws-cli/2.8.7 Python/3.9.11 Darwin/21.6.0 exe/x86_64 prompt/off
    
  • Terraform Cli

    terraform version
    ---
    Terraform v1.3.5
    on darwin_amd64
    - provider registry.terraform.io/hashicorp/aws v3.74.3
    - provider registry.terraform.io/hashicorp/local v2.1.0
    - provider registry.terraform.io/hashicorp/null v3.1.0
    - provider registry.terraform.io/hashicorp/random v3.1.0
    - provider registry.terraform.io/hashicorp/time v0.7.2
    
  • Terraform Environment Cli

    tfenv -v
    ---
    tfenv 2.2.2
    
  • Golang Version Manager (GVM)

    gvm version
    ---
    Go Version Manager v1.0.22 installed at /Users/devopscorner/.gvm
    
  • Golang Version

    go version
    ---
    go version go1.19.6 darwin/arm64
    

Security Check

Make sure that you didn't push sensitive information in this repository

  • AWS Credentials (AWS_ACCESS_KEY, AWS_SECRET_KEY)
  • AWS Account ID
  • AWS Resources ARN
  • Username & Password
  • Private (id_rsa) & Public Key (id_rsa.pub)
  • DNS Zone ID
  • APP & API Key

Copyright

  • Author: Dwi Fahni Denni (@zeroc0d3)
  • Vendor: DevOps Corner Indonesia (devopscorner.id)
  • License: Apache v2