Skip to content

Latest commit

 

History

History
90 lines (64 loc) · 2.78 KB

README.md

File metadata and controls

90 lines (64 loc) · 2.78 KB
Raw

Pluto Restore Assets

Overview

Pluto Restore Assets is a service designed to manage and monitor the restoration of assets from AWS S3 Glacier storage. It consists of two main components: an API server and a worker service, both running in Kubernetes.

Features

  • API Server: RESTful service listening on port 9000 for restore requests
  • Worker Service: Handles the actual restoration process with AWS S3
  • Kubernetes Integration: Runs as containerized services with proper RBAC
  • AWS S3 Integration: Manages asset restoration from Glacier
  • Logging: Comprehensive request and operation logging

Environment Variables

The application requires the following environment variables:

  • KUBE_NAMESPACE: Kubernetes namespace (default: "default")
  • ASSET_BUCKET_LIST: Comma-separated list of asset buckets
  • MANIFEST_BUCKET: S3 bucket for storing manifests
  • AWS_ROLE_ARN: AWS role ARN for permissions
  • AWS_ACCESS_KEY_ID: AWS access key ID
  • AWS_SECRET_ACCESS_KEY: AWS secret access key
  • AWS_DEFAULT_REGION: AWS region
  • BASE_PATH: Base path for restored assets

API Endpoints

  • POST /api/v1/restore: Create a new restore job
    • Required fields: id, user, path
  • GET /api/v1/restore/{id}: Get status of a restore job
  • GET /health: Health check endpoint

Code Structure

API Server (cmd/api/)

  • main.go: Server initialization and routing
  • handlers/: Request handlers and interfaces
    • restore.go: Main restore endpoint logic
    • interfaces.go: Interface definitions
    • restore_test.go: Handler unit tests

Worker Service (cmd/worker/)

  • main.go: Worker process implementation
  • Handles AWS S3 interactions and restore operations

Internal Packages

  • internal/s3utils/: AWS S3 utility functions
  • internal/types/: Shared type definitions
  • pkg/kubernetes/: Kubernetes integration

Testing

The project includes comprehensive tests:

  • API Handler Tests: cmd/api/handlers/restore_test.go
  • S3 Utility Tests: internal/s3utils/monitor_test.go
  • Manifest Generation Tests: internal/s3utils/manifest_test.go

Building and Running Locally

make deploy-latest

Kubernetes Configuration

The service uses several Kubernetes resources:

Deployments

  • API Server deployment
  • Worker deployment for handling restore jobs

Service

Exposes the API server on port 9000

RBAC

  • job-creator-role.yaml: Defines permissions
  • job-creator-rolebinding.yaml: Binds role to service account

AWS S3 Glacier Restore Costs

Summary for 1000 objects totaling 1TB:

Retrieval Option Retrieval Time Total Cost
Expedited 1–5 minutes $40.72
Standard 3–5 hours $10.29
Bulk 5–12 hours $2.59