Merge pull request #1 from cristian-rincon/feature/initial-scaffolding

Feature/initial scaffolding

.gitignore

@@ -160,3 +160,46 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Created by https://www.toptal.com/developers/gitignore/api/terraform
+# Edit at https://www.toptal.com/developers/gitignore?templates=terraform
+
+### Terraform ###
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Exclude all .tfvars files, which are likely to contain sensitive data, such as
+# password, private keys, and other secrets. These should not be part of version
+# control as they are data points which are potentially sensitive and subject
+# to change depending on the environment.
+*.tfvars
+*.tfvars.json
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Include override files you do wish to add to version control using negated pattern
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
+
+# End of https://www.toptal.com/developers/gitignore/api/terraform
+
+.DS_Store

CHANGELOG.md

@@ -0,0 +1,5 @@
+### Changelog
+
+All notable changes to this project will be documented in this file. Dates are displayed in UTC.
+
+Generated by [`auto-changelog`](https://github.com/CookPete/auto-changelog).

README.md

@@ -1 +1,65 @@
-# latam-challenge
+# latam-challenge
+
+## Setup
+
+Apis to be enabled:
+
+- Billing API
+- Cloud Functions API
+- Pub/Sub API
+- Bigquery API
+- IAM API
+
+## 1. Deploy infrastructure
+
+### 1.1 Initialization
+
+> **Constraint:** tested using Terraform v1.8.1
+
+```bash
+cd infra
+terraform init
+```
+
+Create a new file called terraform.tfvars with the following information:
+
+```md
+project_id=<YOUR_GCP_PROJECT_ID>
+region=<YOUR_GCP_REGION>
+```
+
+### 1.2 Plan
+
+```bash
+terraform plan -var-file=terraform.tfvars
+```
+
+### 1.3 Apply
+
+```bash
+terraform apply -var-file=terraform.tfvars
+```
+
+## 2. Test the deployed infrastructure
+
+### 2.1 Ingest data
+
+```bash
+curl -m 310 -X POST https://<fn_ingest_data_url> \
+-H "Authorization: bearer $(gcloud auth print-identity-token)" -H "Content-Type: application/json" \
+-d '{
+    "product_id": "45678",
+    "product_name": "Laptop 2",
+    "category": "Electronics",
+    "unit_price": 999.99,
+    "supplier": "TechSupplier Inc."
+}'
+```
+
+### 2.2 Query data
+
+```bash
+curl -m 310 -X POST https://<fn_fetch_data_url> \
+-H "Authorization: bearer $(gcloud auth print-identity-token)" -H "Content-Type: application/json" \
+-d '{}'
+```

infra/README.md

@@ -0,0 +1,55 @@
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_google"></a> [google](#requirement\_google) | 6.2.0 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_archive"></a> [archive](#provider\_archive) | 2.6.0 |
+| <a name="provider_google"></a> [google](#provider\_google) | 6.2.0 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [google_bigquery_dataset.ecommerce_analytics](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/bigquery_dataset) | resource |
+| [google_bigquery_table.product_sales_table](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/bigquery_table) | resource |
+| [google_cloudfunctions2_function.fn_fetch_data](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/cloudfunctions2_function) | resource |
+| [google_cloudfunctions2_function.fn_ingest_data](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/cloudfunctions2_function) | resource |
+| [google_project_iam_member.data_engineering_bigquery_data_editor](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/project_iam_member) | resource |
+| [google_project_iam_member.data_engineering_bigquery_user](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/project_iam_member) | resource |
+| [google_project_iam_member.data_engineering_cloudfunctions](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/project_iam_member) | resource |
+| [google_project_iam_member.data_engineering_pubsub](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/project_iam_member) | resource |
+| [google_pubsub_subscription.product_sales_subscription](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/pubsub_subscription) | resource |
+| [google_pubsub_topic.product_sales_topic](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/pubsub_topic) | resource |
+| [google_service_account.data_engineering](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/service_account) | resource |
+| [google_storage_bucket.staging_bucket](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/storage_bucket) | resource |
+| [google_storage_bucket_object.fetch_data_source_object](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/storage_bucket_object) | resource |
+| [google_storage_bucket_object.ingest_data_source_object](https://registry.terraform.io/providers/hashicorp/google/6.2.0/docs/resources/storage_bucket_object) | resource |
+| [archive_file.fetch_data_source_code_archive](https://registry.terraform.io/providers/hashicorp/archive/latest/docs/data-sources/file) | data source |
+| [archive_file.ingestion_source_code_archive](https://registry.terraform.io/providers/hashicorp/archive/latest/docs/data-sources/file) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_environment"></a> [environment](#input\_environment) | The environment | `string` | n/a | yes |
+| <a name="input_project_id"></a> [project\_id](#input\_project\_id) | The project ID | `string` | n/a | yes |
+| <a name="input_region"></a> [region](#input\_region) | The region | `string` | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_ecommerce_analytics_dataset_id"></a> [ecommerce\_analytics\_dataset\_id](#output\_ecommerce\_analytics\_dataset\_id) | n/a |
+| <a name="output_fn_ingest_data_url"></a> [fn\_ingest\_data\_url](#output\_fn\_ingest\_data\_url) | n/a |
+| <a name="output_product_sales_subscription_name"></a> [product\_sales\_subscription\_name](#output\_product\_sales\_subscription\_name) | n/a |
+| <a name="output_product_sales_table_id"></a> [product\_sales\_table\_id](#output\_product\_sales\_table\_id) | n/a |
+| <a name="output_product_sales_topic_name"></a> [product\_sales\_topic\_name](#output\_product\_sales\_topic\_name) | n/a |

infra/bigquery.tf

@@ -0,0 +1,57 @@
+// Bigquery dataset "ecommerce_analytics"
+resource "google_bigquery_dataset" "ecommerce_analytics" {
+  dataset_id                  = "ecommerce_analytics"
+  project                     = var.project_id
+  location                    = var.region
+  default_table_expiration_ms = "2592000000" # 30 days in milliseconds
+  labels = {
+    env = var.environment
+  }
+}
+
+# Bigquery table "ecommerce_analytics.sales"
+resource "google_bigquery_table" "product_sales_table" {
+  dataset_id          = google_bigquery_dataset.ecommerce_analytics.dataset_id
+  table_id            = "product_sales"
+  deletion_protection = true
+  time_partitioning {
+    type = "DAY"
+  }
+
+  // Bigquery table schema
+  // product_id: STRING
+  // product_name: STRING
+  // category: STRING
+  // unit_price: FLOAT
+  // supplier: STRING
+
+  schema = <<EOF
+  [
+    {
+      "name": "product_id",
+      "type": "STRING",
+      "mode": "REQUIRED"
+    },
+    {
+      "name": "product_name",
+      "type": "STRING",
+      "mode": "REQUIRED"
+    },
+    {
+      "name": "category",
+      "type": "STRING",
+      "mode": "REQUIRED"
+    },
+    {
+      "name": "unit_price",
+      "type": "FLOAT",
+      "mode": "REQUIRED"
+    },
+    {
+      "name": "supplier",
+      "type": "STRING",
+      "mode": "REQUIRED"
+    }
+  ]
+    EOF
+}

infra/iam_roles.tf

@@ -0,0 +1,31 @@
+// Service account for the resources
+resource "google_service_account" "data_engineering" {
+  account_id   = "data-engineering"
+  display_name = "Data Engineering Service Account"
+  project      = var.project_id
+}
+
+// Grant the service account the necessary roles to use pub/sub, cloud functions and bigquery
+resource "google_project_iam_member" "data_engineering_pubsub" {
+  project = var.project_id
+  role    = "roles/pubsub.editor"
+  member  = "serviceAccount:${google_service_account.data_engineering.email}"
+}
+
+resource "google_project_iam_member" "data_engineering_bigquery_user" {
+  project = var.project_id
+  role    = "roles/bigquery.user"
+  member  = "serviceAccount:${google_service_account.data_engineering.email}"
+}
+resource "google_project_iam_member" "data_engineering_bigquery_data_editor" {
+  project = var.project_id
+  role    = "roles/bigquery.dataEditor"
+  member  = "serviceAccount:${google_service_account.data_engineering.email}"
+}
+
+resource "google_project_iam_member" "data_engineering_cloudfunctions" {
+  project = var.project_id
+  role    = "roles/cloudfunctions.developer"
+  member  = "serviceAccount:${google_service_account.data_engineering.email}"
+}
+