Introduction

This project is the back-end prototype of a crypto wallet application I developed with the Go programming language during my internship at NetAdım company.

I would also like to mention that I learned the Go programming language alongside this project. Additionally, this project allowed me to learn various DevOps concepts.

When I started writing the project, I had searched for many such examples, but the examples I found always had shortcomings. I am sure that there are many aspects of this project that can be improved. However, I believe it is an excellent example project because I designed it with this year's most commonly used technologies and best practices.

Introduction
Getting Started
- Compose Setup
  - Requirements
  - Installation
- Kubernetes Setup
  - Kubectl
    - Lens
Project Design
Crypto API
Auto Documentation
- Swagger
- Swaggo
Monitoring
- Prometheus
- Grafana
Logging
- Traditional Logging
- Loki
Load Test
- Using K6
- Monitoring Test Results
API
Websocket
Copyright

To-do List

While working on this project, you can find below the expectations that the company I interned at had from me:

Implement the following endpoints:
- User registration, login and me endpoints.
- An endpoint for listing crypto exchanges
- Endpoints for selling & buying crypto.
- Create a websocket endpoint to live update crypto exchanges.
Respond HTTP requests with Fiber
Provide a better database interface with GORM.
Dockerize the project to avoid version conflicts.
Use the following database relationships:
- HasMany,
- HasOne,
- BelongsTo,
- ManyToMany
Use JWT to authenticate user.
Use GORM to migrate models.
Documentize the API with Swagger
Test the app
- Unit tests.
- Mock repository layer for unit tests.
Monitor the app
- Monitoring with Prometheus and Grafana
- Profile application with load test
- Monitor logs with Loki and Grafana.
Create a micro service
- Create a mail verification system for "Go Crypto".
- Create a mail sender micro service named "Notifier" to send verification mails.
- Connect "Notifier" and "Go Crypto" with Kafka
- Use Apache ZooKeeper with Kafka

Developer Tips

File "dockerfile.dev" is in use for development purposes. You can change "build" property of "gocrypto" container from docker-compose.yml
You can set "HOST" environment variable to listen to a specific domain with Fiber.
Credentials for Grafana:
- Username: admin
- Password: root

Getting Started

As you probably know, running a single Go file is not so hard. But when it comes to running a project with multiple dependencies, it gets a little bit complicated. And Docker comes to the rescue in this situation. You can run the project with a single command using Docker. You also need Docker for deployment purposes.

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and deploy it as one package.

Docker can get your project up and running in a few minutes. But that's not enough for a project like this. You need to monitor your project, test it, and more. And for that you need to install some other tools like Prometheus, Grafana, K6, Postgres.

Compose Setup: Used for development purposes.
Kubernetes Setup: Used for production purposes.

Note

Both ways uses Docker Engine to run the project. You can install Docker Engine from here.

Compose Setup

Compose Setup uses Docker Compose to run the project. Which is a tool designed to run multi-container applications. You need to configure a file named "docker-compose.yml" to install all the containers needed and run the project. You can Click here to learn more about Docker Compose.

Go Crypto runs with the following containers:

Container	Description
Go-Crypto	Runs the main project. Docker file can be found at project directory.
Notifier	Runs the micro service that sends the mails.
K6	Runs a script that simulates connection load to test servers.
Postgres	Database to store user data.
Prometheus	Database to store metric data.
Loki	Database to store logs.
Grafana	Analytics visualizer interface.
ZooKeeper	Micro service manager.
Kafka	Queue for micro services.
Kafka UI	Interface for kafka. Used for development purposes

These containers all work together to run the project.

Requirements

You need to install the following tools to run the project with Docker Compose:

git-cli: Used to download the project to your device.
docker-compose: Used to quickly install required images (eg. PostgreSQL, Loki, Grafana).

Installation

Clone repository: git clone https://www.github.com/ByPikod/go-crypto.git
Run docker-compose: docker-compose up
- Optionally add "-d" arg to run at background: docker-compose up -d

Kubernetes Setup

Kubernetes is an open-source container orchestration platform used for automating the deployment, scaling, and management of containerized applications.

Kubernetes can automatically scale your applications based on demand, ensuring that you have the right amount of resources to handle your workload.

To understand what is Horizontally Scaling, first you should know what is Verical Scaling.

Vertical scaling, aka. scaling up, is a method used to increase the capacity and performance of a single server or machine by adding more resources to it (eg. RAM, CPU).

And Horizontal scaling, aka. scaling out, is a method used to increase the capacity and performance of a system by adding more machines or nodes to a network or cluster.

You can scale out Go-Crypto by the following steps:

Install Kubarnetes
Apply configuration files: kubectl apply -f .\kubernetes\

And that's it, you are ready to go! Kubernetes will automatically boot up the containers and scale them based on demand. Altho you will still need to run the other containers with docker-compose.

Here is the list of containers that will be running with Kubernetes:

Go-Crypto
Notifier
Postgres

Beside these containers, you will still need to run the rest of the containers with docker-compose.

Kubectl

Kubectl is a command-line tool that allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.

You can find some useful commands below:

Listing
- kubectl get pods - List all pods
- kubectl get services - List all services
- kubectl get deployments - List all deployments
- kubectl get nodes - List all nodes
Deleting Pods
- kubectl delete <type> <pod-name> - Delete pods, services, deployments (Used to restart pods)
- kubectl delete <type> -l label=<label-value> - Delete all pods, services, deployments, etc. by a label.
- kubectl delete <type> --all - Delete all pods, services, deployments, etc of a type.
Logs
- kubectl logs <pod-name> - Get logs of a pod.
Executing commands
- kubectl exec -it <pod-name> -- <command> - Execute a command in a pod.

Lens

Lens is a powerful IDE for Kubernetes. It is a standalone application for MacOS, Windows and Linux operating systems. You can use Lens to monitor your Kubernetes cluster.

Project Design

This project is a REST API project, and it has been built using the Multitier Architecture design. The Multitier Architecture design is a REST API pattern consisting of layers named Controller (Presentation), Service (Business), and Repository (Persistence).

Technologies

You can find the technologies used in this project below:

Go: Programming language.
Libraries/Frameworks
- Gorn: Database management
- Fiber: HTTP library.
- JWT: Auth standard.
Documentation
- Swagger: Rest API documentation.
- Swaggo: Auto config generator for Swagger
Database
- Postgres: Database
- Prometheus: Analytics collector.
Tools
- Grafana: Analytics visualizer.
- Air: Live reload tool for go projects.
- Git: Version control system
- Docker: Containerization platform.

Ports

The ports exposed by the project are:

Name	Port	Description
Fiber	80	Rest API itself.
Swagger	8080	Auto generated documentation for API.
Grafana	3000	Monitoring interface.
Postgres	5432	Database
Prometheus	9090	Metric database.
Loki	3100	Log database.
ZooKeeper	2181	Micro services.
Kafka	9092	Micro services.

Some ports are exposed on development purposes.

Folder Structure

This project follows common designs used in the back-end of web applications. Here is the project tree with comments explaining the modules, files, and their purposes:

.
├── controllers # Endpoints (aka Presentation layer)
│   ├── exchanges.go
│   ├── user.go
│   └── wallet.go
├── core # Core components
│   ├── config.go # Retrieve environment variables
│   └── database.go # Initialize database connection
├── helpers # Utilities
│   ├── database.go # Database utilities
│   ├── errors.go # HTTP Errors (e.g 404, 403, 400)
│   ├── password.go # Password hashing, comparing
│   ├── token.go # JWT utilities
│   └── validate.go # Payload validations
├── log
│   ├── local.go # Logging functions for local logs.
│   └── log.go # Logging functions for Loki.
├── main.go
├── middleware
│   ├── auth.go # Authorization with JWT
│   ├── json.go # Adds header accepts "application/json"
│   ├── metrics.go # Monitoring
│   └── websocket.go # Return error if websocket request missing upgrade header.
├── models # Database & API models
│   ├── exchanges.go
│   ├── transaction.go
│   ├── user.go
│   └── wallet.go
├── repositories # Repository layer (aka Persistance)
│   ├── exchanges.go
│   ├── user.go
│   └── wallet.go
├── router # Routes
│   └── router.go
└── services # Service layer (aka Bussiness layer)
    ├── exchanges.go
    ├── user.go
    └── wallet.go

Models

The module called "Models" is the boilerplate that represents data structures used in the background of web applications. Models are used to introduce raw data into the programming language being used. Here are the models for this project:

User: Holds user data. Password encrypted with bcrypt.

  type User struct {
    gorm.Model
    Name     string   `json:"name" gorm:"not null"`
    Lastname string   `json:"lastName" gorm:"not null"`
    Mail     string   `json:"mail" gorm:"index;not null;unique"`
    Password string   `json:"password" gorm:"not null"`
    Wallets  []Wallet `gorm:"foreignKey:UserID"` // HasMany
  }

Wallet: User can have multiple wallets with different currencies for each one. These wallets can have transactions histories.

type Wallet struct {
    gorm.Model
    Currency    string        `json:"currency" gorm:"not null;index"`
    Balance     float64       `json:"balance" gorm:"default:0;not null"`
    UserID      uint          `json:"userID" gorm:"not null;index"`
    User        User          `gorm:"foreignKey:UserID"` // BelongsTo
    Transaction []Transaction `gorm:"foreignKey:WalletID"` // HasMany
}

Transaction: Transaction history holds the history of transactions as the name describes.

type Transaction struct {
    gorm.Model
    Type     int8    `json:"type" gorm:"not null"`
    Change   float64 `json:"change" gorm:"not null"`
    Balance  float64 `json:"balance" gorm:"not null"`
    WalletID uint    `json:"walletID" gorm:"not null;index"`
    Wallet   Wallet  `gorm:"foreignKey:WalletID"` // BelongsTo
}

Exchanges: This model holds the exchange data received from API.

type ExchangeRates struct {
    Currency string             `json:"currency"`
    Rates    map[string]float64 `json:"rates"`
}

Crypto API

Coinbase's free to use public API (v2) is used to fetch crypto exchange data. Here you can find the endpoint below:

https://api.coinbase.com/v2/exchange-rates?currency=BTC

Auto Documentation

Swagger

Swagger is a tool that helps developers to create document and test APIs (Application Programming Interfaces) for their software applications.

Swaggo

Swagger needs a configuration file to create documents. This configuration is automatically generated by Swaggo from your comment lines.

You can read the documentation from "localhost:8080" once you started docker.

Monitoring

Prometheus and Grafana are tools used for monitoring and analyzing metrics of a web application. With Grafana and Prometheus, you can analyze a wide range of data, from sales metrics to resource utilization and more.

Prometheus

Prometheus is a software that collects "metric" data from the http servers by requesting a specific endpoint at target servers at intervals you specified. The metric data is stored chronologically by the Prometheus. Data can be accessed via web interface or Rest API that Prometheus provides.

Grafana

And Grafana is an open source analytics monitoring tool that provides bunch of visual components like (e.g charts, gauges). Grafana can have multiple data sources and Prometheus is one of them. Grafana can request to API of the Prometheus and visualize your chronologically stored metrics data.

I've configured Prometheus to gather default Go Metrics from Go Fiber and visualized some of those metrics in Grafana as can be seen in the picture below:

Logging

This project uses two different methods for logging. One is for local logging with simple print functions and the other is for advanced logging for customers.

Traditional Logging

I've used traditional local logging functions for the stuff that only concerns developers.

Loki

Loki is an open-source log aggregation system developed by Grafana Labs. It is designed to help you collect, store, and query log data from various sources in a scalable and efficient manner.

In this project, I used Loki for advanced logging. And I used zap-loki library for establishing the connection between Go and Loki.

Monitoring Loki data source in Grafana:

Load Test

The term load testing is used in different ways in the professional software testing community. Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program concurrently. (https://en.wikipedia.org/wiki/Load_testing)

I've used "Grafana K6" to load test this project. K6 has a very simple interface.

Using K6

First, you should create a JavaScript file for K6, as mentioned, named "scripts/loadtest.js".

This script, using the framework provided by K6, allows you to quickly send load requests to your web application. You can also perform checks on responses using the functions provided by K6.

K6 is originally designed to export metrics to a data source called InfluxDB. However, you can obtain output for Prometheus using the experimental Prometheus Remote Write module.

Monitoring Test Results

The output obtained from Prometheus can be visualized using the Grafana interface, as explained in the "Monitoring" section.

API

Endpoints /api/
User Endpoints /api/user/
User Wallet Endpoints: /api/user/wallet/

Endpoints

Uncategorised Endpoints

GET /api/exchange-rates/

Description

Lists the crypto currency exchange rates.

Response

{
    "currency": "USD",
    "rates": {
        "00": 13.651877133105803,
        "1INCH": 3.898635477582846,
        "AAVE": 0.0159936025589764,
        "ABT": 13.708019191226867,
        "ACH": 64.1148938898506,
        ...
}

User Endpoints

POST /api/user/login/

Description

Returns auth token if matching credentials provided.

Parameters

Name	Type	Description
mail	string	Mail address
password	string	Password

Response

{
    "status": true,
    "message": "OK!",
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJVc2VySUQiOjF9.HBNfNTMv3Jd9Wf-m3v6buHgGLQL0Srl8zwGro8JHcO4"
}

POST /api/user/register/

Description

Creates a new account if provided details are appropriate. If verification code is not provided, it will try to send a verification code to the mail address provided. If the bool named "verificationSent" in response data is true, you must have received the mail address.

Parameters

Name	Type	Description
name	string	First name
lastName	string	Last name
mail	string	Mail address
password	string	Password
verification	string	Verification code

Response

{
    "status": true,
    "message": "OK!",
}

GET /api/user/me/

Description

Returns user information.

Parameters

Auth required

Response

{
    "id": 1,
    "lastname": "Batulu",
    "mail": "admin@yahyabatulu.com",
    "name": "Yahya"
}

User Wallet Endpoints

GET /api/user/wallet/balance/

Description

Returns user balance.

Parameters

Auth required

Response

{
    "BTC": 40990.47869000058,
    "USD": 995270.5766880848
}

POST /api/user/wallet/deposit/

Description

A money deposit endpoint. Virtual POS not implemented. It's just a prototype.

Parameters

Auth required

Name	Type	Description
amount	float	Amount of money to deposit.

Response

{
    "status": true,
    "newBalance": 64.05993807839195,
    "message": "OK",
}

POST /api/user/wallet/withdraw/

Description

A money withdraw endpoint.

Parameters

Auth required

Name	Type	Description
amount	float	Amount of money to withdraw.

Response

{
    "status": true,
    "newBalance": 64.05993807839195,
    "message": "OK",
}

POST /api/user/wallet/buy/

Description

Performs a crypto purchase and returns success or failure depending on the result.

Parameters

Auth required

Name	Type	Description
amount	float	Amount of crypto to buy.
currency	string	Currency to buy

Response

{
    "message": "OK!",
    "status": true, // Success state
    "Balance": { // New balance
        "BTC": 1.0053999999999998,
        "USD": 64.05993807839195
    },
    "sold_amount": 135.72802500006378,
    "sold_currency": "USD",
    "bought_amount": 0.005,
    "bought_currency": "BTC",
}

POST /api/user/wallet/sell/

Description

Performs a crypto selling and returns success or failure depending on the result.

Parameters

Auth required

Name	Type	Description
amount	float	Amount of crypto to sell.
currency	string	Currency to sell.

Response

{
    "message": "OK!",
    "status": true,
    "Balance": {
        "BTC": 995270.5766880848,
        "USD": 40990.47869000058
    },
    "bought_amount": 135.17072499994828,
    "bought_currency": "USD",
    "sold_amount": 0.005,
    "sold_currency": "BTC",
}

Websocket

WS /ws/exchange-rates

Description

Returns exchange-rates as it changed.

Response

{
    "currency": "USD",
    "rates": {
        "00": 13.651877133105803,
        "1INCH": 3.898635477582846,
        "AAVE": 0.0159936025589764,
        "ABT": 13.708019191226867,
        "ACH": 64.1148938898506,
        ...
}

Copyright

This project is licensed under the terms of the MIT License.

You are free to use this project in compliance with the MIT License. If you decide to use, modify, or redistribute this software, you must include a copy of the original license and copyright notice in all copies or substantial portions of the software.

For more information about the MIT License, visit: MIT License.

Files

README.md

Latest commit

History

README.md

File metadata and controls

Introduction

Table of Contents

To-do List

Developer Tips

Getting Started

Compose Setup

Requirements

Installation

Kubernetes Setup

Kubectl

Lens

Project Design

Technologies

Ports

Folder Structure

Models

Crypto API

Auto Documentation

Swagger

Swaggo

Monitoring

Prometheus

Grafana

Logging

Traditional Logging

Loki

Load Test

Using K6

Monitoring Test Results

API

Endpoints

Uncategorised Endpoints

Description

Response

User Endpoints

Description

Parameters

Response

Description

Parameters

Response

Description

Parameters

Response

User Wallet Endpoints

Description

Parameters

Response

Description

Parameters

Response

Description

Parameters

Response

Description

Parameters

Response

Description

Parameters

Response

Websocket

Description

Response

Copyright