Skip to content

Latest commit

 

History

History
572 lines (485 loc) · 31.4 KB

README.md

File metadata and controls

572 lines (485 loc) · 31.4 KB
Raw

Ortelius in a box

Welcome to Ortelius in a Box

Ortelius in a Box!

Project Description

The goal is to open up the Microservices world and give visibility to developers so that it does not feel like you are developing with a blind fold on.

Ortelius and its supporting Microservices are deployed into a Kind cluster using Terraform.

DBeaver or any suitable database client for Postgresql can be used to access the Postgresql database. The Postgres database is persisted using volume mounts.

Devspace is the tool for deploying and connecting VS Code to the Dev Container inside the Ortelius namespace. The Dev Container will contain all the tools required for development of the Ortelius technology stack.

Kubeshark can be used by the developer for Microservice API troubleshooting.

The LocalStack extension in your Docker Desktop opens up the cloud world for you to develop against AWS services running locally on your machine.

Leapp is the one ring to rule them all when it comes to cloud provider authentication and role switching.

Thunderclient is a powerful API tester inside VS Code.

The great thing is that this is immutable and transportable to any operating system that supports Docker, Kubectl, Helm, Kind and Terraform.

Whilst I have tried to cover every scenario in the documentation I welcome feedback for any ambiguous or missing instructions.

Future Endeavours

Pre-requisite checklist

Developer Tools

Pre Flight Checks & Troubleshooting Tips

  • If you have Helm Charts installed run helm repo update
  • Kube config is expected to be in the default location $KUBECONFIG
  • If you get an AJAX error when trying to login then logout first with http://localhost:8080/dmadminweb/Logout or you could use incognito mode (Its a cookie issue)
  • If you get incorrect username or password the database is probably borked, destroy and re-deploy
  • If everything goes completely haywire manually remove everything such as *.tfstate and the Ortelius Docker containers representing the Kind K8s nodes and execute terraform plan -auto-approve and terraform apply -auto-approve
  • Please make sure you have a folder in your root /tmp/postgres and please make sure you add this mount point to your Docker Desktop under Settings --> Resources --> File Sharing (/tmp is sufficient)
  • If you are already using ports 8080 or 5432 in your environment you will need to change the ports in main.tf here
      # ortelius http port | change port 8080 to something else if it clashes with your environment
      extra_port_mappings {
        container_port = 31000
        host_port      = 8080 <-- make the port change here
        listen_address = "0.0.0.0"
      }
      # postgresql port | change port 5432 to something else if it clashes with your environment
      extra_port_mappings {
        container_port = 31316
        host_port      = 5432 <-- make the port change here
        listen_address = "0.0.0.0"
      }
    }

Terraform Structure

Architecture

Ortelius | Supply Chain Catalog & DevOps Intelligence

Ortelius is central catalog of supply chain and DevOps intelligence. It is designed to track and version composition details for every component of your software supply chain along with all consuming logical applications. With Ortelius, you can easily view your logical application's SBOMs, CVEs, service dependencies, and inventory based on versions, even in a decoupled microservices architecture. Ortelius aggregates DevOps, security and supply chain data for each independent component moving through the pipeline. It is particularly useful in cloud-native, microservices architectures where the logical application becomes ambiguous. Ortelius tracks who is consuming shared components, versions them when they are updated and then creates new release candidates for every logical application that is impacted by a component change. It then aggregates that data to the logical application level so you don't have to. The latest version of Ortelius is maintained by the Ortelius Community managed by the Continuous Delivery Foundation (Linux Foundation). It was originally created by DeployHub and OpenMake Software. Our mission is to simplify the adoption of modern architecture through a world-class microservice catalog driven by a supportive and diverse global open source community.

Ortelius

  • Ortelius on ArtifactHUB is here
  • Helm Charts are here
  • Chart.yaml is here
  • Ortelius architecture is here
  • Ortelius architecture diagram is here

How to access Ortelius

How to access PostgreSQL with DBeaver

  • Free multi-platform database tool for developers, database administrators, analysts and all people who need to work with databases.
  • Supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, MS Access, Teradata, Firebird, Apache Hive, Phoenix, Presto, etc.
  • You can use DBeaver Community Edition which is available here
  • Supports ChatGPT integration for smart completion and code generation as an optional extension
  • Apache License 2.0
  • PostgreSQL localhost mount Postgresql Mount

Connection settings

  • Password
postgres

Connection Settings

  • Ortelius Database Tables

Ortelius Database Tables

Kind Ortelius Cluster Details

Context kind-ortelius

  • List all the contexts
kubectl config get context
  • Switch to the kind-ortelius context
kubectl config use-context kind-ortelius
  • Help with contexts
kubectl config -h

Namespaces ortelius

  • List all namespaces
kubectl get namespace -A

Nodes

  • Control Plane ortelius-control-plane
  • Worker ortelius-worker
kubectl get nodes -A

Docker.com | Application Containerisation Platform

GitHub Docker

Docker is a platform for developing, shipping, and running applications. It uses containers, which are lightweight, standalone, executable packages of software that include everything needed to run the application, including the code, runtime, libraries, environment variables, and system tools. Containers provide a consistent, isolated environment for applications to run, making it easier to develop, test, and deploy applications. Docker allows developers to automate the deployment of applications into containers and manage containers as a single unit. It simplifies the process of deploying applications, making it easier to scale applications, and enabling organizations to adopt a microservices architecture.

Terraform | Provisioning & Managing IaC

GitHub Terraform

Terraform is an open-source tool for provisioning and managing infrastructure as code. It provides a simple, declarative syntax for defining infrastructure resources, such as virtual machines, DNS entries, and databases. Terraform can manage popular service providers as well as custom in-house solutions. By describing infrastructure as code, Terraform enables versioning, testing, and collaboration of infrastructure changes. Terraform can create, update, and delete resources in parallel, while minimizing the risk of conflicts and errors. Terraform also provides a state management system that tracks changes to infrastructure over time, making it easier to roll back changes if necessary. With Terraform, organizations can automate their infrastructure management processes, improve reliability, and increase efficiency.

Terraform README

Lets get going

  • Clone the repo
  • Navigate to /
  • Run the following
terraform init

terraform plan
  • Open videos in a new tab

Time the deployment

time terraform apply --auto-approve

Don't time the deployment

terraform apply --auto-approve
  • Open videos in a new tab

Destroy the deployment

terraform destroy --auto-approve
  • Open videos in a new tab

Logs

In total, there 5 log levels which can be used for debugging purposes:

  • TRACE one of the most descriptive log levels, if you set the log level to TRACE, Terraform will write every action and step into the log file.
  • DEBUG a little bit more sophisticated logging which is used by developers at critical or more complex pieces of code to reduce debugging time.
  • INFO the info log level is useful when needing to log some informative instructions or readme type instructions.
  • WARN used when something is not critical but would be nice to include in the form of a log so that the developer can make adjustments later.
  • ERROR as the name suggests, this is used if something is terribly wrong and is a blocker.
export TF_LOG="DEBUG"

export TF_LOG_PATH="/abraham/terraform-debug.log"

KinD | Kubernetes in Docker

GitHub KinD

Kind (Kubernetes in Docker) is a tool for running local Kubernetes clusters using Docker containers as nodes. It provides an easy-to-use environment for testing and developing applications that run on a Kubernetes cluster. Kind allows developers to run a full Kubernetes cluster on their development machine, eliminating the need for remote clusters and making it easier to test and debug applications. Kind creates a multi-node cluster by launching multiple Docker containers on a single host, making it possible to test complex scenarios and network configurations. Kind is also useful for testing Kubernetes plugins and extensions, and for developing and testing operators, custom controllers, and other Kubernetes-related software.

Why KinD?

  • kind supports multi-node (including HA) clusters
  • kind supports building Kubernetes release builds from source
  • support for make / bash or docker, in addition to pre-published builds
  • kind supports Linux, macOS and Windows
  • kind is a CNCF certified conformant Kubernetes installer

KinD Nodes!

Get Nodes

kind get nodes ortelius

Kubeconfig

kind get kubeconfig -n ortelius

Get Clusters

kind get clusters

Logs | Great for debugging & troubleshooting

kind export logs -n ortelius

Container Runtime Interface (CRI) CLI | Container Troubleshooting Tool

GitHub Crictl

Crictl is a command line tool for interacting with a containerd-based container runtime. It provides a simple, human-readable interface for performing common container operations such as pulling images, starting and stopping containers, and viewing logs. Crictl also supports advanced features such as executing commands inside containers and managing network configurations. It is designed to be a fast and flexible alternative to other container runtime management tools, and can be used in production or development environments. Crictl is a component of the containerd project, which is a lightweight, high-performance runtime for managing containers and is used by many popular container orchestration platforms such as Kubernetes.

  • Download crictl here

List containers and check the container is in a created state

$ crictl ps -a
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT
3e025dd50a72d       busybox             32 seconds ago      Created             busybox             0

Start container

$ crictl start 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60
3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60

$ crictl ps
CONTAINER ID        IMAGE               CREATED              STATE               NAME                ATTEMPT
3e025dd50a72d       busybox             About a minute ago   Running             busybox             0

Exec a command in a container

crictl exec -i -t 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60 ls
bin   dev   etc   home  proc  root  sys   tmp   usr   var

Kubernetes.io | The Conductor of Containerised Applications

Kubernetes is an open-source platform for automating deployment, scaling, and management of containerised applications. It provides a unified API for defining and managing containers, enabling organizations to simplify the deployment and scaling of applications. Kubernetes automates the distribution of containers across a cluster of machines and monitors the health of the containers, ensuring that they are always running. It can automatically replace failed containers and ensure that the desired number of replicas are running. Kubernetes also provides built-in service discovery and load balancing, making it easier to connect microservices and ensure that traffic is distributed evenly across the cluster. Kubernetes has become the de facto standard for container orchestration and is widely adopted by organizations of all sizes.

GitHub Kubernetes

Kubectl

  • Install kubectl the command line tool here

  • Use the kubectl cheat sheet here

  • Use Devdocs and the official documentation here

  • Add the aliases & auto complete which are in the cheat sheet

  • Real time logging of a pod

kubectl logs ms-nginx-6ccbb5f95c-9gjg7 -n ortelius -f
  • Remote into a pod
kubectl -n ortelius exec -it ms-nginx-6ccbb5f95c-9gjg7 -n ortelius -c ms-nginx -- sh

Combine Kubens & Kubectx with Fuzzy Finder | K8s Context & Namespace Switching

  • kubectx is a tool to switch between contexts (clusters) on kubectl faster.
  • kubens is a tool to switch between Kubernetes namespaces (and configure them for kubectl) easily.
  • Download Kubens & Kubectx here
  • Licence

FzF

  • Is a general-purpose command-line fuzzy finder.
  • Is an interactive Unix filter for command-line that can be used with any list; files, command history, processes, hostnames, bookmarks, git commits, etc.
  • Download Fuzzy Finder here
  • Licence

FzF in action

Fuzzy Finder!

Kubectx for switching context

kubectx kind-ortelius

Kubens for switching namespaces

kubens ortelius

KubeShark Realtime visibility into K8's

GitHub

Real-time visibility into K8s' internal network, capturing, dissecting and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters.

  • Install here
  • Documentation is here
  • Licence
  • Tap Ortelius kubeshark tap -n ortelius

DevSpace | Dev Container inside K8s

GitHub DevSpace

  • What is DevSpace
  • Devspace could be used for the local Terraform environment whereas Codespaces is more suited for a developer on the repos themselves
  • Licence

How

  • Download the CLI here
  • Init DevSpace in your repo and use the existing Dockerfile and devspace.yaml by following the prompts.
  • Currently it is using the Microsoft Universal Dev Container which I have built and pushed to my DockerHub
  • The Microsoft Universersal Dev Container has the following language platforms Python, Node.js, JavaScript, TypeScript, C++, Java, C#, F#, .NET Core, PHP, Go, Ruby, Conda
  • Licence
devspace init
  • run the UI
devspace ui
  • start coding
devspace dev -n ortelius

warn Are you using the correct namespace?
warn Current namespace: 'default'
warn Last    namespace: 'devspace'

? Which namespace do you want to use? devspace
info Using namespace 'devspace'
info Using kube context 'kind-ortelius'
info Created namespace: devspace
deploy:app Deploying chart component-chart (app) with helm...
deploy:app Deployed helm chart (Release revision: 1)
deploy:app Successfully deployed app with helm
dev:app Waiting for pod to become ready...
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app DevSpace is waiting, because Pod app-devspace-6dcdc9bf56-trrrc has status: ContainerCreating
dev:app Selected pod app-devspace-6dcdc9bf56-trrrc
dev:app ports Port forwarding started on: 2345 -> 2345
dev:app sync  Sync started on: ./ <-> /app
dev:app sync  Waiting for initial sync to complete
dev:app sync  Initial sync completed
dev:app ssh   Port forwarding started on: 10479 -> 8022
dev:app proxy Port forwarding started on: 11920 <- 10567
dev:app ssh   Use 'ssh app.dev-env-setup.devspace' to connect via SSH
dev:app term  Opening shell to container-0:app-devspace-6dcdc9bf56-trrrc (pod:container)
  • On the command line switch to the bash shell
devspace ./app # bash

root@app-devspace-6dcdc9bf56-6snnf:/# cd app

root@app-devspace-6dcdc9bf56-6snnf:/app# ls
Dockerfile  README.md     app            devspace_start.sh  localstack.yaml  module.tf  outputs.tf  plan.out      service-nginx.yaml  variables.tf
LICENSE     TF-README.md  devspace.yaml  images             main.tf          ortelius   plan.json   providers.tf  terraform.tfstate   wazuh

Connect VS Code

VS Code Extensions

  • Install the Remote-SSH extension here
  • Click on the Remote Explorer icon

VS Code Remote Explorer!

  • SSH Settings configured by Devspace
# DevSpace Start app.dev-env-setup.devspace
Host app.dev-env-setup.devspace
  HostName localhost
  LogLevel error
  Port 10479
  IdentityFile "/Users/tvl/.devspace/ssh/id_devspace_ecdsa"
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  User devspace
# DevSpace End app.dev-env-setup.devspace
  • Install PostgreSQL Database Manager for database admin from inside your vscode here
  • The extension supports many database engines so don't be misled by the name

VS Code PostgreSQL!

Component Helm Chart

The component chart allows you to define application components (e.g. a database, an API server, a webserver with static files) and deploy them using Helm. Helm is the package manager for Kubernetes and allows you to manage these components (e.g. upgrading).

Why?

Compared to manually creating Helm charts, the component chart allows you to define your application components using a unified Helm chart.

This provides the following benefits:

  • 70% less YAML to maintain (only values.yaml for chart)
  • Highly flexible configuration via values.yaml
  • Fast and easy definition of Kubernetes resources
  • Kubernetes best practices (e.g. recommended annotations and labels)

GitHub Codespaces & Dev Containers | Devcontainers for your GitHub repos

Leapp | Cross-Platform Cloud Access Application

GitHub

Leapp is a Cross-Platform Cloud access App, built on top of Electron. The App is designed to manage and secure Cloud Access in multi-account environments, and it is available for MacOS, Windows, and Linux.

  • Cloud credentials generation in 1 click
  • Data stored locally encrypted in the OS System Vault
  • Multiple Cloud-Access supported strategies
  • Automatic short-lived credentials rotation
  • Automatic provisioning of Sessions from AWS Single Sign-on
  • Open multiple AWS console from different AWS accounts in Firefox and Chrome web extensions!
  • Connect to EC2 instances straight away
  • Managing Leapp with its CLI
  • Create your own Leapp plugin to customize the App functionalities from the template
  • Download here
  • Licence

Localstack | AWS Cloudstack on your local machine with Docker extensions

GitHub LocalStack

LocalStack is a fully functional local AWS cloud stack that enables developers to develop and test their cloud applications offline. It provides an easy-to-use test/mocking framework for developing cloud applications, eliminating the need for a live AWS environment.

LocalStack VS Code Extension

  • Commandeer extension here

Commandeer

LocalStack S3 Endpoint

LocalStack S3 Endpoint!

AWS CLI Local

  • This package provides the awslocal command, which is a thin wrapper around the aws command line interface for use with LocalStack.
awslocal --endpoint-url=http://localhost:4566 kinesis list-streams
StreamNames: []

Thunderclient | API Testing Inside VS Code

GitHub

A new way to test APIs inside VS Code

  • Simple & Easy to Use CLI tool
  • Seamless Integration with Thunder Client Extension
  • Run Requests/Collections and View Test results
  • Syntax Highlighting and VSCode Themes
  • Open Request/Collection UI from the terminal
  • Enhances CURL commands
  • CI/CD Integration
  • Please refer here for more detail
  • Licence