diff --git a/cost/google/schedule_instance/CHANGELOG.md b/cost/google/schedule_instance/CHANGELOG.md index c92e45ede7..61b1efa6ea 100644 --- a/cost/google/schedule_instance/CHANGELOG.md +++ b/cost/google/schedule_instance/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## v3.0 + +- Several parameters altered to be more descriptive and human-readable +- Added ability to specify custom label keys for tracking instance schedules +- Added ability to filter resources by project +- Added ability to filter resources by region +- Added ability to filter resources by multiple label key:value pairs +- Added ability for user to start and stop instances directly +- Normalized incident export to be consistent with other policies +- Added additional fields to incident export for additional context +- Streamlined code for better readability and faster execution +- Policy action error logging modernized and now works as expected in EU/APAC +- Added logic required for "Meta Policy" use-cases +- To facilitate "Meta Policy" use-cases, policy now requires a Flexera credential + ## v2.5 - Improved logging, and error capture/handling diff --git a/cost/google/schedule_instance/README.md b/cost/google/schedule_instance/README.md index 5f303ee418..dca7de2847 100644 --- a/cost/google/schedule_instance/README.md +++ b/cost/google/schedule_instance/README.md @@ -1,76 +1,78 @@ # Google Schedule Instance -## What it does +## What It Does -This Policy Template allows you to schedule start and stop times for your Google instance, along with the option to terminate instance, update and delete schedule. +This policy schedules Google VM instances to start and stop at specific times based on a configuration stored in the instance's labels. The user can also perform a variety of ad hoc actions on the instance from the incident page. -## How to Use +## How To Use -This policy relies on a label with format 'schedule' to stop and start instances based on a schedule. The label value defines the schedule with a start time(start hour and start minute), stop time(stop hour and stop minute), days of the week and timezone. The start and stop time are in 24 hour format, and the days of the week are two character abbreviation for example: mo, tu, we. See full example below.. Use a Timezone TZ value to indicate a timezone to stop/start the instance(s) +This policy uses the schedule label value (default key: schedule) for scheduling the instance. The appropriate value should be added to as a label to every instance you want to manage via this policy. -## schedule Label Example +This value is a string consisting of 3 underscore-separated substrings: -Since google label supports only `-`, `_`, lowercase characters, numbers and International characters, The special characters in timezone should be replaced like `/` with `-`, `+` with `p` and `-`(minus) with `m` and all characters should be lowercase. -For example, the timezone `Etc/Gmt+10` should be used as `etc-gmtp10`, `Etc/GMT-4` as `etc-gmtm4`, `America/North_Dakota/New_Salem` as `america-north_dakota-new_salem`, `America/Port-au-Prince` as `america-port-au-prince` etc. +- *Hours* - Start and stop hours are in 4-digit 24-hour format without any colons or other separators. For example, a value of `0815-1730` will start instances at 8:15 and stop them at 17:30 (5:30 pm). If the minute field is left blank, the minute value of `00` will be assumed. +- *Days of the Week* - Hyphen-separated list of days indicated by their two-letter abbreviation value from the following list: su,mo,tu,we,th,fr,sa. For example, a value of `mo-tu-we-th-fr` will start and stop the instances on weekdays but not on weekends. +- *Timezone* - Timezone in [tz database format](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) with the `/` character replaced with a hyphen and all characters converted to lowercase. For example, a value of `america-new_york` would specify US Eastern Time. Defaults to UTC if no Timezone field is provided. -Start and Stop time are 24 hour format: for example 0830-1715 is start at 8:30am, and stop at 5:15pm. +**Example Value:** 0815-1730_mo-tu-we-th-fr_america-new_york -Days of the week: su-mo-tu-we-th-fr-sa +- Starts instances at 8:15am +- Stops instance at 5:30pm +- Monday - Friday, US Eastern Time. -Timezone: Use the TZ database name from the [timezone list](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) in the above mentioned format. - -Example: schedule=0830-1715_mo-tu-we-th-fr_america-new_york. Stops instances at 5:15pm, starts instance at 8:30am, Monday - Friday, Eastern Time(America/New_York). -Please note that `_` is being used for separating the start-stop time, days of the week and the timezone. - -Instances are off during the weekend and start back up on Monday morning at 8:30am and are off at 5:15pm every weekday. Times are UTC unless the Timezone field is provided. - -Note: Please note that for this policy to work, the time should be in 24 hour format and both hours and minutes must have 2 digits: `0800`for `8am` or `2130` for `9:30pm`. -Please refer the [formatted timezones list](https://github.com/flexera-public/policy_templates/blob/master/data/tz_database/timezones_list.json) having timezone in the above mentioned format as key and corresponding TZ database timezone as value. +In the above example, instances are off during the weekend and start back up on Monday morning at 8:15am and are off at 5:30pm every weekday. Times are UTC unless the Timezone field is provided. ## Input Parameters This policy has the following input parameters required when launching the policy. -- *Email addresses* - A list of email addresses to notify -- *Exclusion Tags* - A list of Google tags to ignore instances. Format: Key=Value. -- *Automatic Action(s)* -(Optional) When this value is set, this policy will automatically take the selected action(s) +- *Email Addresses* - Email addresses of the recipients you wish to notify when new incidents are created. +- *Schedule Label Key* - Label key that schedule information is stored in. Default is recommended for most use cases. +- *Next Start Label Key* - Label key to use for scheduling instance to start. Default is recommended for most use cases. +- *Next Stop Label Key* - Label key to use for scheduling instance to stop. Default is recommended for most use cases. +- *Allow/Deny Projects* - Whether to treat Allow/Deny Projects List parameter as allow or deny list. Has no effect if Allow/Deny Projects List is left empty. +- *Allow/Deny Projects List* - Filter results by project ID/name, either only allowing this list or denying it depending on how the above parameter is set. Leave blank to consider all projects +- *Allow/Deny Regions* - Whether to treat Allow/Deny Regions List parameter as allow or deny list. Has no effect if Allow/Deny Regions List is left empty. +- *Allow/Deny Regions List* - Filter results by region, either only allowing this list or denying it depending on how the above parameter is set. Leave blank to consider all the regions. +- *Exclusion Labels (Key:Value)* - Google labels to ignore resources that you don't want to produce recommendations for. Use Key:Value format for specific label key/value pairs, and Key:\* format to match any resource with a particular key, regardless of value. Examples: env:production, DO_NOT_DELETE:\* +- *Automatic Actions* - When this value is set, this policy will automatically take the selected action(s). ## Policy Actions The following policy actions are taken on any resources found to be out of compliance. -- Send an email report -- stop - stop a selected instance -- start - start a selected instance -- terminate - terminates or deletes the selected instance. -- update schedule - change existing schedule tag. input to provide a new stop/start schedule -- delete schedule - removes the schedule tag +- Send an email report. +- *Execute Schedules* - Start or stop the resources as needed based on their schedules +- *Update Schedules* - Update the schedule tag on the resources with a new schedule +- *Delete Schedules* - Delete all schedule tags from the resource so that it is no longer powered on or off by this policy +- *Start Instances* - Start the resources if they are not currently running. +- *Stop Instances* - Stop the resources if they are currently running. +- *Delete Instances* - Delete the resources. ## Prerequisites -### Schedule Label Format - -This policy uses `schedule` label value for scheduling the instance. The format should be like `0800-1715_mo-tu-we-th-fr_america-new_york`. Please refer to `Schedule Label Example` section for more details. -Please note that for this policy to work, the time should be in 24 hour format and both hours and minutes must have 2 digits: `0800`for `8am` or `2130` for `9:30pm`. - -This policy uses [credentials](https://docs.flexera.com/flexera/EN/Automation/ManagingCredentialsExternal.htm) for connecting to the cloud -- in order to apply this policy you must have a credential registered in the system that is compatible with this policy. If there are no credentials listed when you apply the policy, please contact your cloud admin and ask them to register a credential that is compatible with this policy. The information below should be consulted when creating the credential. +This Policy Template requires that several APIs be enabled in your Google Cloud environment: -### Credential configuration +- [Cloud Resource Manager API](https://console.cloud.google.com/flows/enableapi?apiid=cloudresourcemanager.googleapis.com) +- [Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=compute.googleapis.com) -For administrators [creating and managing credentials](https://docs.flexera.com/flexera/EN/Automation/ManagingCredentialsExternal.htm) to use with this policy, the following information is needed: +This Policy Template uses [Credentials](https://docs.flexera.com/flexera/EN/Automation/ManagingCredentialsExternal.htm) for authenticating to datasources -- in order to apply this policy you must have a Credential registered in the system that is compatible with this policy. If there are no Credentials listed when you apply the policy, please contact your Flexera Org Admin and ask them to register a Credential that is compatible with this policy. The information below should be consulted when creating the credential(s). -Provider tag value to match this policy: `gce` +- [**Google Cloud Credential**](https://docs.flexera.com/flexera/EN/Automation/ProviderCredentials.htm#automationadmin_4083446696_1121577) (*provider=gce*) which has the following: + - Permissions + - `compute.instances.list` + - `compute.instances.get` + - `compute.instances.delete` + - `compute.instances.setLabels` + - `compute.instances.start` + - `compute.instances.stop` + - `compute.zones.list` + - `resourcemanager.projects.get` -Required permissions in the provider: +- [**Flexera Credential**](https://docs.flexera.com/flexera/EN/Automation/ProviderCredentials.htm) (*provider=flexera*) which has the following roles: + - `billing_center_viewer` -- The `compute.instances.list` permission -- The `compute.instances.get` permission -- The `compute.instances.delete` permission -- The `compute.instances.setLabels` permission -- The `compute.instances.start` permission -- The `compute.instances.stop` permission -- The `resourcemanager.projects.get` permission -- The `compute.zones.list` permission +The [Provider-Specific Credentials](https://docs.flexera.com/flexera/EN/Automation/ProviderCredentials.htm) page in the docs has detailed instructions for setting up Credentials for the most common providers. ## Supported Clouds diff --git a/cost/google/schedule_instance/google_schedule_instance.pt b/cost/google/schedule_instance/google_schedule_instance.pt index b099a46ce9..49e7895f54 100644 --- a/cost/google/schedule_instance/google_schedule_instance.pt +++ b/cost/google/schedule_instance/google_schedule_instance.pt @@ -1,16 +1,16 @@ name "Google Schedule Instance" rs_pt_ver 20180301 type "policy" -short_description "This Policy Template allows you to schedule start and stop times for your instance, along with the option to terminate instance, update and delete schedule. See the [README](https://github.com/flexera-public/policy_templates/tree/master/cost/google/schedule_instance/) and [docs.flexera.com/flexera/EN/Automation](https://docs.flexera.com/flexera/EN/Automation/AutomationGS.htm) to learn more." +short_description "Schedules Google VM instances to start and stop at specific times based on a configuration stored in the instance's labels. See the [README](https://github.com/flexera-public/policy_templates/tree/master/cost/google/schedule_instance/) and [docs.flexera.com/flexera/EN/Automation](https://docs.flexera.com/flexera/EN/Automation/AutomationGS.htm) to learn more." long_description "" category "Cost" severity "low" default_frequency "daily" info( - version: "2.5", - provider: "GCE", + version: "3.0", + provider: "Google", service: "Compute", - policy_set:"Schedule Instance" + policy_set: "Schedule Instance" ) ############################################################################### @@ -19,23 +19,85 @@ info( parameter "param_email" do type "list" - label "Email addresses" - description "A list of email addresses to notify" + category "Policy Settings" + label "Email Addresses" + description "A list of email addresses to notify." default [] end -parameter "param_tags_to_exclude" do +parameter "param_label_schedule" do + type "string" + category "Label Keys" + label "Schedule Label Key" + description "Label key that schedule information is stored in. Default is recommended for most use cases." + default "schedule" +end + +parameter "param_label_next_start" do + type "string" + category "Label Keys" + label "Next Start Label Key" + description "Label key to use for scheduling instance to start. Default is recommended for most use cases." + default "next_start" +end + +parameter "param_label_next_stop" do + type "string" + category "Label Keys" + label "Next Stop Label Key" + description "Label key to use for scheduling instance to stop. Default is recommended for most use cases." + default "next_stop" +end + +parameter "param_projects_allow_or_deny" do + type "string" + category "Filters" + label "Allow/Deny Projects" + description "Allow or Deny entered Projects. See the README for more details." + allowed_values "Allow", "Deny" + default "Allow" +end + +parameter "param_projects_list" do + type "list" + category "Filters" + label "Allow/Deny Projects List" + description "A list of allowed or denied Subscription IDs/names. See the README for more details." + default [] +end + +parameter "param_regions_allow_or_deny" do + type "string" + category "Filters" + label "Allow/Deny Regions" + description "Allow or Deny entered regions. See the README for more details." + allowed_values "Allow", "Deny" + default "Allow" +end + +parameter "param_regions_list" do type "list" - label "Exclusion Tags" - description "A list of Google tags to ignore instances. Format: Key=Value" + category "Filters" + label "Allow/Deny Regions List" + description "A list of allowed or denied regions. See the README for more details." + default [] +end + +parameter "param_exclusion_labels" do + type "list" + category "Filters" + label "Exclusion Labels (Key:Value)" + description "Google labels to ignore resources that you don't want to produce recommendations for. Use Key:Value format for specific label key/value pairs, and Key:* format to match any resource with a particular key, regardless of value. Examples: env:production, DO_NOT_DELETE:*" + allowed_pattern /(^$)|[\w]*\:.*/ default [] end parameter "param_automatic_action" do type "list" - label "Automatic Action(s)" - description "When this value is set, this policy will automatically take the selected action(s)" - allowed_values ["Schedule Instances"] + category "Actions" + label "Automatic Actions" + description "When this value is set, this policy will automatically take the selected action." + allowed_values ["Execute Schedules"] default [] end @@ -43,18 +105,25 @@ end # Authentication ############################################################################### -credentials "google_auth" do +credentials "auth_google" do schemes "oauth2" label "Google" description "Select the Google Cloud Credential from the list." tags "provider=gce" end +credentials "auth_flexera" do + schemes "oauth2" + label "Flexera" + description "Select Flexera One OAuth2 credentials" + tags "provider=flexera" +end + ############################################################################### # Pagination ############################################################################### -pagination "google_pagination" do +pagination "pagination_google" do get_page_marker do body_path "nextPageToken" end @@ -64,621 +133,1003 @@ pagination "google_pagination" do end ############################################################################### -# Datasources +# Datasources & Scripts ############################################################################### -datasource "ds_get_projectId" do +# Table to derive region from zone +datasource "ds_zone_to_region" do + run_script $js_zone_to_region +end + +script "js_zone_to_region", type:"javascript" do + result "result" + code <<-EOS + result = { + "us-east1-b": "us-east1", + "us-east1-c": "us-east1", + "us-east1-d": "us-east1", + "us-east4-c": "us-east4", + "us-east4-b": "us-east4", + "us-east4-a": "us-east4", + "us-central1-c": "us-central1", + "us-central1-a": "us-central1", + "us-central1-f": "us-central1", + "us-central1-b": "us-central1", + "us-west1-b": "us-west1", + "us-west1-c": "us-west1", + "us-west1-a": "us-west1", + "europe-west4-a": "europe-west4", + "europe-west4-b": "europe-west4", + "europe-west4-c": "europe-west4", + "europe-west1-b": "europe-west1", + "europe-west1-d": "europe-west1", + "europe-west1-c": "europe-west1", + "europe-west3-c": "europe-west3", + "europe-west3-a": "europe-west3", + "europe-west3-b": "europe-west3", + "europe-west2-c": "europe-west2", + "europe-west2-b": "europe-west2", + "europe-west2-a": "europe-west2", + "asia-east1-b": "asia-east1", + "asia-east1-a": "asia-east1", + "asia-east1-c": "asia-east1", + "asia-southeast1-b": "asia-southeast1", + "asia-southeast1-a": "asia-southeast1", + "asia-southeast1-c": "asia-southeast1", + "asia-northeast1-b": "asia-northeast1", + "asia-northeast1-c": "asia-northeast1", + "asia-northeast1-a": "asia-northeast1", + "asia-south1-c": "asia-south1", + "asia-south1-b": "asia-south1", + "asia-south1-a": "asia-south1", + "australia-southeast1-b": "australia-southeast1", + "australia-southeast1-c": "australia-southeast1", + "australia-southeast1-a": "australia-southeast1", + "southamerica-east1-b": "southamerica-east1", + "southamerica-east1-c": "southamerica-east1", + "southamerica-east1-a": "southamerica-east1", + "asia-east2-a": "asia-east2", + "asia-east2-b": "asia-east2", + "asia-east2-c": "asia-east2", + "asia-northeast2-a": "asia-northeast2", + "asia-northeast2-b": "asia-northeast2", + "asia-northeast2-c": "asia-northeast2", + "asia-northeast3-a": "asia-northeast3", + "asia-northeast3-b": "asia-northeast3", + "asia-northeast3-c": "asia-northeast3", + "asia-south2-a": "asia-south2", + "asia-south2-b": "asia-south2", + "asia-south2-c": "asia-south2", + "asia-southeast2-a": "asia-southeast2", + "asia-southeast2-b": "asia-southeast2", + "asia-southeast2-c": "asia-southeast2", + "australia-southeast2-a": "australia-southeast2", + "australia-southeast2-b": "australia-southeast2", + "australia-southeast2-c": "australia-southeast2", + "europe-central2-a": "europe-central2", + "europe-central2-b": "europe-central2", + "europe-central2-c": "europe-central2", + "europe-north1-a": "europe-north1", + "europe-north1-b": "europe-north1", + "europe-north1-c": "europe-north1", + "europe-southwest1-a": "europe-southwest1", + "europe-southwest1-b": "europe-southwest1", + "europe-southwest1-c": "europe-southwest1", + "europe-west6-a": "europe-west6", + "europe-west6-b": "europe-west6", + "europe-west6-c": "europe-west6", + "northamerica-northeast1-a": "northamerica-northeast1", + "northamerica-northeast1-b": "northamerica-northeast1", + "northamerica-northeast1-c": "northamerica-northeast1", + "northamerica-northeast2-a": "northamerica-northeast2", + "northamerica-northeast2-b": "northamerica-northeast2", + "northamerica-northeast2-c": "northamerica-northeast2", + "us-west2-a": "us-west2", + "us-west2-b": "us-west2", + "us-west2-c": "us-west2", + "us-west3-a": "us-west3", + "us-west3-b": "us-west3", + "us-west3-c": "us-west3", + "us-west4-a": "us-west4", + "us-west4-b": "us-west4", + "us-west4-c": "us-west4", + "us-west5-a": "us-west5", + "us-west5-b": "us-west5", + "us-west5-c": "us-west5", + "us-west6-a": "us-west6", + "us-west6-b": "us-west6", + "us-west6-c": "us-west6", + "us-west7-a": "us-west7", + "us-west7-b": "us-west7", + "us-west7-c": "us-west7" + } +EOS +end + +# Get applied policy metadata for use later +datasource "ds_applied_policy" do request do - auth $google_auth - verb "GET" - host "cloudresourcemanager.googleapis.com" - path "/v1/projects" - header "Accept", "application/json" - query "filter", "lifecycleState=ACTIVE" - end - result do - encoding"json" - collect jmes_path(response, "projects[*]") do - field "projectId", jmes_path(col_item, "projectId") - end + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/applied_policies/", policy_id]) + header "Api-Version", "1.0" end end -datasource "ds_get_zones" do - iterate $ds_get_projectId +datasource "ds_google_projects" do request do - auth $google_auth - pagination $google_pagination - verb "GET" - host "compute.googleapis.com" - path join(["/compute/v1/projects/", val(iter_item, "projectId"), "/zones"]) - header "Accept", "application/json" + auth $auth_google + pagination $pagination_google + host "cloudresourcemanager.googleapis.com" + path "/v1/projects/" + query "filter", "(lifecycleState:ACTIVE)" + # Header X-Meta-Flexera has no affect on datasource query, but is required for Meta Policies + # Forces `ds_is_deleted` datasource to run first during policy execution + header "Meta-Flexera", val($ds_is_deleted, "path") end result do encoding "json" - collect jmes_path(response, "items[*]") do - field "projectId", val(iter_item, "projectId") - field "zone", jmes_path(col_item, "name") + collect jmes_path(response, "projects[*]") do + field "number", jmes_path(col_item, "projectNumber") + field "id", jmes_path(col_item, "projectId") + field "name", jmes_path(col_item, "name") end end end -datasource "ds_get_all_instances" do - iterate $ds_get_zones +datasource "ds_google_projects_filtered" do + run_script $js_google_projects_filtered, $ds_google_projects, $param_projects_allow_or_deny, $param_projects_list +end + +script "js_google_projects_filtered", type: "javascript" do + parameters "ds_google_projects", "param_projects_allow_or_deny", "param_projects_list" + result "result" + code <<-EOS + if (param_projects_list.length > 0) { + result = _.filter(ds_google_projects, function(project) { + include_project = _.contains(param_projects_list, project['id']) || _.contains(param_projects_list, project['name']) || _.contains(param_projects_list, project['number']) + + if (param_projects_allow_or_deny == "Deny") { + include_project = !include_project + } + + return include_project + }) + } else { + result = ds_google_projects + } +EOS +end + +datasource "ds_get_instances" do + iterate $ds_google_projects_filtered request do - auth $google_auth - pagination $google_pagination - verb "GET" + auth $auth_google host "compute.googleapis.com" - path join(["/compute/v1/projects/", val(iter_item, "projectId"), "/zones/", val(iter_item, "zone"), "/instances"]) - query "filter", "labels.schedule!=null" + path join(["/compute/v1/projects/", val(iter_item, 'id'), "/aggregated/instances"]) + header "User-Agent", "RS Policies" + header "Content-Type", "application/json" + ignore_status [403, 404] end result do encoding "json" - collect jmes_path(response, "items[*]") do - field "id", jmes_path(col_item,"id") - field "state", jmes_path(col_item,"status") - field "labels", jmes_path(col_item, "labels") - field "zone", val(iter_item,"zone") - field "selfLink", jmes_path(col_item, "selfLink") - end + field "instances", jq(response, "[ .items | to_entries | map_values(.value) | map(select(has(\"instances\"))) | .[].instances | .[] | {id,name,selfLink,status,tags,zone,kind,hostname,cpuPlatform,labels,description,machineType}]") + field "projectId", val(iter_item, "id") + field "projectName", val(iter_item, "name") + field "projectNumber", val(iter_item, "number") end end -datasource "ds_filter_resources" do - run_script $js_filter_resources, $ds_get_all_instances, $param_tags_to_exclude +datasource "ds_instances" do + run_script $js_instances, $ds_get_instances, $ds_zone_to_region end -############################################################################### -# Scripts -############################################################################### +script "js_instances", type: "javascript" do + parameters "ds_get_instances", "ds_zone_to_region" + result "result" + code <<-EOS + result = [] + + _.each(ds_get_instances, function(response) { + _.each(response['instances'], function(instance) { + zone = instance['zone'].split('/')[8] + region = ds_zone_to_region[zone] + + result.push({ + id: instance['id'], + name: instance['name'], + description: instance['description'], + status: instance['status'], + selfLink: instance['selfLink'], + cpuPlatform: instance['cpuPlatform'], + machineType: instance['machineType'], + hostname: instance['hostname'], + kind: instance['kind'], + labels: instance['labels'], + projectId: response['projectId'], + projectName: response['projectName'], + projectNumber: response['projectNumber'], + zone: zone, + region: region + }) + }) + }) +EOS +end +datasource "ds_instances_label_filtered" do + run_script $js_instances_label_filtered, $ds_instances, $param_label_schedule, $param_exclusion_labels +end -script "js_filter_resources", type: "javascript" do - parameters "ds_get_all_instances", "param_tags_to_exclude" +script "js_instances_label_filtered", type: "javascript" do + parameters "ds_instances", "param_label_schedule", "param_exclusion_labels" result "result" code <<-EOS - var result=[]; - for(var i=0;i next_start || now > next_stop || next_start==null || next_stop==null){ - result.push({ - "id":instance.id, - "state": instance.state, - "zone": instance.zone, - "schedule":schedule, - "labels":all_labels.slice(2), - "next_start":next_start_iso, - "next_stop":next_stop_iso, - "selfLink": instance.selfLink - }) + + exclude_instance = false + + _.each(param_exclusion_labels, function(exclusion_label) { + if (_.contains(labels, exclusion_label)) { + exclude_instance = true } - } - } + }) + + return exclude_instance + }) + } else { + result = ds_instances } - result=_.sortBy(result, 'id'); - result=_.sortBy(result, 'zone'); - EOS -end -############################################################################### -# Escalations -############################################################################### + // Filter out any VMs that do not have the schedule label key + result = _.filter(result, function(vm) { + label_keys = [] + if (typeof(vm['labels']) == 'object') { label_keys = _.keys(vm['labels']) } -escalation "esc_email" do - automatic true - label "Sends Email" - description "Sends Incident Email" - email $param_email + return _.contains(label_keys, param_label_schedule) + }) +EOS end -escalation "esc_schedule_instance" do - automatic contains($param_automatic_action, "Schedule Instances") - label "Schedule" - description "Stop or Start the Instance" - run "schedule_instance", data, rs_optima_host +datasource "ds_instances_region_filtered" do + run_script $js_instances_region_filtered, $ds_instances_label_filtered, $param_regions_allow_or_deny, $param_regions_list end -escalation "esc_terminate_instance" do - automatic false - label "Terminate Instance" - description "Terminate or delete the Instance" - run "terminate_instance", data, rs_optima_host +script "js_instances_region_filtered", type: "javascript" do + parameters "ds_instances_label_filtered", "param_regions_allow_or_deny", "param_regions_list" + result "result" + code <<-EOS + if (param_regions_list.length > 0) { + result = _.filter(ds_instances_label_filtered, function(instance) { + include_instance = _.contains(param_regions_list, instance['region']) + + if (param_regions_allow_or_deny == "Deny") { + include_instance = !include_instance + } + + return include_instance + }) + } else { + result = ds_instances_label_filtered + } +EOS end -escalation "esc_update_schedule" do - automatic false - label "Update Schedule" - description "Update the existing schedule Label" - parameter "param_schedule" do - type "string" - label "New Schedule" - description "Provide Schedule Value" - end - run "update_schedule", data, $param_schedule, rs_optima_host +datasource "ds_instances_to_schedule" do + run_script $js_instances_to_schedule, $ds_instances_region_filtered, $ds_applied_policy, $param_label_schedule, $param_label_next_start, $param_label_next_stop end -escalation "esc_delete_schedule" do - automatic false - label "Delete Schedule" - description "Delete Schedule Tag" - run "delete_schedule", data, rs_optima_host +script "js_instances_to_schedule", type: "javascript" do + parameters "ds_instances_region_filtered", "ds_applied_policy", "param_label_schedule", "param_label_next_start", "param_label_next_stop" + result "result" + code <<-EOS + result = [] + now = new Date() + + _.each(ds_instances_region_filtered, function(instance) { + schedule = null + next_start = null + next_stop = null + + labels = [] + + if (typeof(instance['labels']) == 'object') { + _.each(_.keys(instance['labels']), function(key) { + value = instance['labels'][key] + + labels.push(key + '=' + value) + + if (key == param_label_schedule) { schedule = value } + if (key == param_label_next_start) { next_start = value } + if (key == param_label_next_stop) { next_stop = value } + }) + } + + if (next_start != null) { next_start = new Date(next_start) } + if (next_stop != null) { next_stop = new Date(next_stop) } + + if (now > next_start || now > next_stop || next_start == null || next_stop == null) { + if (next_start != null) { next_start = next_start.toISOString() } + if (next_stop != null) { next_stop = next_stop.toISOString() } + + result.push({ + resourceID: instance['id'], + resourceName: instance['name'], + description: instance['description'], + status: instance['status'], + selfLink: instance['selfLink'], + platform: instance['cpuPlatform'], + resourceType: instance['machineType'], + hostname: instance['hostname'], + kind: instance['kind'], + accountID: instance['projectId'], + accountName: instance['projectName'], + projectNumber: instance['projectNumber'], + zone: instance['zone'], + region: instance['region'], + tags: labels.join(', '), + policy_name: ds_applied_policy['name'], + next_start: next_start, + next_stop: next_stop, + schedule: schedule, + service: "Compute Engine" + }) + } + }) + + result = _.sortBy(result, 'resourceID') + result = _.sortBy(result, 'zone') + result = _.sortBy(result, 'region') + result = _.sortBy(result, 'accountID') +EOS end + ############################################################################### # Policy ############################################################################### -policy "policy_schedule_instance" do - validate $ds_filter_resources do - summary_template "Google Schedule Instance list" - hash_exclude "id", "labels" +policy "pol_schedule_instance" do + validate_each $ds_instances_to_schedule do + summary_template "{{ with index data 0 }}{{ .policy_name }}{{ end }}: {{ len data }} Google Scheduled VM Instances" + # Policy check fails and incident is created only if data is not empty and the Parent Policy has not been terminated + check logic_or($ds_parent_policy_terminated, eq(val(item, "resourceID"), "")) escalate $esc_email - escalate $esc_schedule_instance - escalate $esc_terminate_instance - escalate $esc_update_schedule - escalate $esc_delete_schedule - check eq(size(data),0) + escalate $esc_execute_schedules + escalate $esc_update_schedules + escalate $esc_delete_schedules + escalate $esc_start_instances + escalate $esc_stop_instances + escalate $esc_delete_instances + hash_exclude "tags" export do resource_level true + field "accountID" do + label "Project ID" + end + field "accountName" do + label "Project Name" + end + field "projectNumber" do + label "Project Number" + end + field "resourceID" do + label "Resource ID" + end + field "resourceName" do + label "Resource Name" + end + field "resourceType" do + label "Resource Type" + end field "zone" do label "Zone" end - field "id" do - label "Instance ID" + field "region" do + label "Region" + end + field "hostname" do + label "Hostname" end - field "state" do - label "Instance State" + field "platform" do + label "Platform" + end + field "tags" do + label "Labels" + end + field "service" do + label "Service" + end + field "status" do + label "Status" + end + field "selfLink" do + label "Resource Link" end field "schedule" do label "Schedule" end field "next_start" do - label "Next Start Time" + label "Next Start" end field "next_stop" do - label "Next Stop Time" - end - field "labels" do - label "Labels" + label "Next Stop" end - field "selfLink" do - label "SelfLink" + field "id" do + label "ID" + path "resourceID" end end end end ############################################################################### -# Cloud Workflow +# Escalations ############################################################################### -# CWF function to handle errors -define handle_error() do - if !$$errors - $$errors = [] +escalation "esc_email" do + automatic true + label "Send Email" + description "Send incident email" + email $param_email +end + +escalation "esc_execute_schedules" do + automatic contains($param_automatic_action, "Execute Schedules") + label "Execute Schedules" + description "Approval to start or stop all selected instances based on schedule" + run "execute_schedules", data, $param_label_schedule, $param_label_next_start, $param_label_next_stop +end + +escalation "esc_update_schedules" do + automatic false + label "Update Schedules" + description "Approval to update the schedule labels on all selected instances" + parameter "param_schedule" do + type "string" + category "Policy Actions" + label "New Schedule" + description "Enter a new value for the schedule label. See README for more details" end - $$errors << $_error["type"] + ": " + $_error["message"] - # We check for errors at the end, and raise them all together - # Skip errors handled by this definition - $_error_behavior = "skip" + run "update_schedules", data, $param_schedule, $param_label_schedule, $param_label_next_start, $param_label_next_stop end -# CWF function to capture and output logs -define log_message($message) do - # Instantiate log messages array doesn't exist - if !$$log_messages - $$log_messages = [] - end - # Append timestamp to message - $now = strftime(now(), "%Y/%m/%d %H:%M:%S") - $log = $now + ": " + $message - # Append to log messages array - $$log_messages << $log - # Update task label with current message - task_label($log) - # Aggregate all log messages into a single string which is easier to grab from UI and interpret - $$log_messages_string = join($$log_messages, "\n\n") -end - -define schedule_instance($data, $$rs_optima_host) return $all_response do - $all_response=[] - foreach $item in $data do - $rule = split($item['schedule'],'_')[1] - $rule= split($rule, "-") - $rule= join($rule, ",") - $rule= upcase($rule) - $time_range = split($item['schedule'],'_')[0] - $start_time = split($time_range,'-')[0] - $start_time = insert($start_time, 2, ":") - $start_hour = split($start_time, ':')[0] - $start_minute = split($start_time, ':')[1] - if $start_minute==null - $start_minute='00'; - end - $stop_time = split($time_range,'-')[1] - $stop_time= insert($stop_time, 2, ":") - $stop_hour = split($stop_time, ':')[0] - $stop_minute = split($stop_time, ':')[1] - if $stop_minute==null - $stop_minute='00'; - end - $start_rule = join(["FREQ=WEEKLY;BYDAY=",$rule]) - $stop_rule = join(["FREQ=WEEKLY;BYDAY=",$rule]) - $timezone = split($item['schedule'],'_')[2] - if ! $timezone - $timezone = "UTC" - end - $size=size(split($item['schedule'], "_")) - if $size > 3 - $count = 3 - $arr_val=split($item['schedule'], "_") - while $count < $size do - $timezone=$timezone+"_"+$arr_val[$count] - $count=$count+1 - end - end - call get_tzlist() retrieve $timezones - $timezones=from_json($timezones) - $timezone=$timezones[$timezone] - # checking if the timezone is having required format as mentioned in the readme - if $timezone != null - call window_active($start_hour, $start_minute, $start_rule, $stop_hour, $stop_minute, $stop_rule, $timezone) retrieve $window_active , $next_start, $next_stop - - call log_message('window_active='+to_s($window_active)) - call log_message('next_start='+to_s($next_start)) - call log_message('next_stop='+to_s($next_stop)) - - if ($window_active) - call log_message($item['schedule'] + ' schedule window is currently active: Instances may be started.') - else - call log_message($item['schedule'] + ' schedule window is currently in-active: Instances may be stopped.') - end +escalation "esc_delete_schedules" do + automatic false + label "Delete Schedules" + description "Approval to delete the schedule labels on all selected instances" + run "delete_schedules", data, $param_label_schedule, $param_label_next_start, $param_label_next_stop +end - $stoppable = /^(RUNNING|STAGING)$/ - $startable = /^(STOPPING|TERMINATED)$/ - - $next_start=split($next_start, "T") - $next_start[1]=split($next_start[1],"-") - $next_start[1]=join($next_start[1], "m") - $next_start[1]=split($next_start[1],":") - $next_start[1]=join($next_start[1], "-") - $next_start[1]=split($next_start[1], "+") - $next_start[1]=join($next_start[1], "p") - $next_start=join($next_start, "t") - - $next_stop=split($next_stop,"T") - $next_stop[1]=split($next_stop[1],"-") - $next_stop[1]=join($next_stop[1], "m") - $next_stop[1]=split($next_stop[1],":") - $next_stop[1]=join($next_stop[1], "-") - $next_stop[1]=split($next_stop[1], "+") - $next_stop[1]=join($next_stop[1], "p") - $next_stop=join($next_stop, "t") - - call get_labelfingerprint($item) retrieve $labelFingerprint, $labels_obj - - $labels_obj['next_start']=$next_start - $labels_obj['next_stop']=$next_stop - $labels_obj['schedule']=$item['schedule'] - - $body = { - "labels" : $labels_obj, - "labelFingerprint" : $labelFingerprint - } - $response= http_post( - auth: $$google_auth, - url: join([$item['selfLink'], "/setLabels"]), - headers:{ - "cache-control": "no-cache", - "content-type": "application/json" - }, - body: $body - ) - call log_message('req=POST '+join([$item['selfLink'], "/setLabels"])+' body='+to_json($body)+' response='+to_json($response)) - if($window_active) - if($item['state']=~$startable) - call log_message('> ' + $item['id'] + ': Starting ... ' + to_s($item)) - sub on_error: handle_error($response) do - $response= http_post( - auth: $$google_auth, - url : join([$item['selfLink'], "/start"]), - headers:{ - "cache-control": "no-cache", - "content-type": "application/json" - } - ) - call log_message('req=POST '+join([$item['selfLink'], "/start"])+' response='+to_json($response)) - $all_response << $response - end - else - call log_message('> ' + $item['id'] + ': No action - Instance state is ' + $item['state']) - end - else - if($item['state'] =~ $stoppable) - call log_message('> ' + $item['id'] + ': Stopping ...' + to_s($item)) - sub on_error: handle_error() do - $response= http_post( - auth: $$google_auth, - url: join([$item['selfLink'], "/stop"]), - headers:{ - "cache-control": "no-cache", - "content-type": "application/json" - } - ) - call log_message('req=POST '+join([$item['selfLink'], "/stop"])+' response='+to_json($response)) - end - else - call log_message('> ' + $item['id'] + ': No action - Instance state is ' + $item['state']) - end - end - # generating log for incorrect timezone format - else - call log_message('Incorrect Timezone in schedule label ' + to_s($item['schedule'])) +escalation "esc_start_instances" do + automatic false + label "Start Instances" + description "Approval to start all selected instances" + run "start_instances", data +end + +escalation "esc_stop_instances" do + automatic false + label "Stop Instances" + description "Approval to stop all selected instances" + run "stop_instances", data +end + +escalation "esc_delete_instances" do + automatic false + label "Delete Instances" + description "Approval to delete all selected instances" + run "delete_instances", data +end + +############################################################################### +# Cloud Workflow +############################################################################### + +# Core CWF functions for iterating through items +define execute_schedules($data, $param_label_schedule, $param_label_next_start, $param_label_next_stop) return $all_responses do + $$all_responses = [] + + foreach $instance in $data do + sub on_error: handle_error() do + call execute_schedule($instance, $param_label_schedule, $param_label_next_start, $param_label_next_stop) end end - # If we encountered any errors, use `raise` to mark the CWF process as errored if inspect($$errors) != "null" raise join($$errors,"\n") end end -define terminate_instance($data, $$rs_optima_host) return $all_response do - $all_response=[] - foreach $item in $data do - call log_message('> ' + $item['id'] + ': Terminating ... ' + to_s($item)) +define update_schedules($data, $param_schedule, $param_label_schedule, $param_label_next_start, $param_label_next_stop) return $all_responses do + $$all_responses = [] + + foreach $instance in $data do sub on_error: handle_error() do - $response= http_delete( - auth: $$google_auth, - url: $item['selfLink'], - headers:{ - "cache-control": "no-cache", - "content-type": "application/json" - } - ) - call log_message('req=DELETE '+$item['selfLink']+' response='+to_json($response)) + call update_schedule($instance, $param_schedule, $param_label_schedule, $param_label_next_start, $param_label_next_stop) retrieve $update_response, $window_active end end - # If we encountered any errors, use `raise` to mark the CWF process as errored if inspect($$errors) != "null" raise join($$errors,"\n") end end -define delete_schedule($data, $$rs_optima_host) return $all_response do - $all_response=[] - foreach $item in $data do - call get_labelfingerprint($item) retrieve $labelFingerprint, $labels_obj - $new_label={} - $labels_obj=to_a($labels_obj) - foreach $label in $labels_obj do - if $label[0]!='schedule' && $label[0]!='next_start' && $label[0]!='next_stop' - $new_label[$label[0]]=$label[1]; - end - end - call log_message('> ' + $item['id'] + ': Deleting schedule Tag ... ' + to_s($item)) +define delete_schedules($data, $param_label_schedule, $param_label_next_start, $param_label_next_stop) return $all_responses do + $$all_responses = [] + foreach $instance in $data do sub on_error: handle_error() do - $body = { - "labels" : $new_label, - "labelFingerprint" : $labelFingerprint - } - $response= http_post( - auth: $$google_auth, - url: join([$item['selfLink'], "/setLabels"]), - headers:{ - "cache-control": "no-cache", - "content-type": "application/json" - }, - body: $body - ) - call log_message('req=POST '+join([$item['selfLink'], "/setLabels"])+' body='+to_json($body)+' response='+to_json($response)) + call delete_schedule($instance, $param_label_schedule, $param_label_next_start, $param_label_next_stop) end end - # If we encountered any errors, use `raise` to mark the CWF process as errored if inspect($$errors) != "null" raise join($$errors,"\n") end end -define update_schedule($data, $param, $$rs_optima_host) return $all_response do - $all_response=[] - foreach $item in $data do - $rule = split($param,'_')[1] - $rule= split($rule, "-") - $rule= join($rule, ",") - $rule= upcase($rule) - $time_range = split($param,'_')[0] - $start_time = split($time_range,'-')[0] - $start_time = insert($start_time, 2, ":") - $start_hour = split($start_time, ':')[0] - $start_minute = split($start_time, ':')[1] - if $start_minute==null - $start_minute='00'; - end - $stop_time = split($time_range,'-')[1] - $stop_time= insert($stop_time, 2, ":") - $stop_hour = split($stop_time, ':')[0] - $stop_minute = split($stop_time, ':')[1] - if $stop_minute==null - $stop_minute='00'; - end - $start_rule = join(["FREQ=WEEKLY;BYDAY=",$rule]) - $stop_rule = join(["FREQ=WEEKLY;BYDAY=",$rule]) - $timezone = split($param,'_')[2] - if ! $timezone - $timezone = "UTC" +define start_instances($data) return $all_responses do + $$all_responses = [] + + foreach $instance in $data do + sub on_error: handle_error() do + call start_instance($instance) end - $size=size(split($param, "_")) - if $size > 3 - $count = 3 - $arr_val=split($param, "_") - while $count < $size do - $timezone=$timezone+"_"+$arr_val[$count] - $count=$count+1 - end + end + + if inspect($$errors) != "null" + raise join($$errors,"\n") + end +end + +define stop_instances($data) return $all_responses do + $$all_responses = [] + + foreach $instance in $data do + sub on_error: handle_error() do + call stop_instance($instance) end - call get_tzlist() retrieve $timezones - $timezones=from_json($timezones) - $timezone=$timezones[$timezone] - # checking if the timezone is having required format as mentioned in the readme - if $timezone != null - call window_active($start_hour, $start_minute, $start_rule, $stop_hour, $stop_minute, $stop_rule, $timezone) retrieve $window_active , $next_start, $next_stop - - $next_start=split($next_start, "T") - $next_start[1]=split($next_start[1],"-") - $next_start[1]=join($next_start[1], "m") - $next_start[1]=split($next_start[1],":") - $next_start[1]=join($next_start[1], "-") - $next_start[1]=split($next_start[1], "+") - $next_start[1]=join($next_start[1], "p") - $next_start=join($next_start, "t") - - $next_stop=split($next_stop,"T") - $next_stop[1]=split($next_stop[1],"-") - $next_stop[1]=join($next_stop[1], "m") - $next_stop[1]=split($next_stop[1],":") - $next_stop[1]=join($next_stop[1], "-") - $next_stop[1]=split($next_stop[1], "+") - $next_stop[1]=join($next_stop[1], "p") - $next_stop=join($next_stop, "t") - - call get_labelfingerprint($item) retrieve $labelFingerprint, $labels_obj - - $labels_obj['schedule']=$param - $labels_obj['next_start']=$next_start - $labels_obj['next_stop']=$next_stop - call log_message('> ' + $item['id'] + ': Updating schedule Tag ...' + to_s($item)) - sub on_error: handle_error() do - $body = { - "labels" : $labels_obj, - "labelFingerprint" : $labelFingerprint - } - $response= http_post( - auth: $$google_auth, - url: join([$item['selfLink'], "/setLabels"]), - headers:{ - "cache-control": "no-cache", - "content-type": "application/json" - }, - body: $body - ) - call log_message('req=POST '+join([$item['selfLink'], "/setLabels"])+' body='+to_json($body)+' response='+to_json($response)) - end - # generating log for incorrect timezone format - else - call log_message('Error: Unexpected Timezone for schedule label. ' + to_s($param)) + end + + if inspect($$errors) != "null" + raise join($$errors,"\n") + end +end + +define delete_instances($data) return $all_responses do + $$all_responses = [] + + foreach $instance in $data do + sub on_error: handle_error() do + call delete_instance($instance) end end - # If we encountered any errors, use `raise` to mark the CWF process as errored if inspect($$errors) != "null" raise join($$errors,"\n") end end +# Secondary CWF functions for taking action on individual instances +define execute_schedule($instance, $param_label_schedule, $param_label_next_start, $param_label_next_stop) return $response do + call update_schedule($instance, $instance['schedule'], $param_label_schedule, $param_label_next_start, $param_label_next_stop) retrieve $update_response, $window_active + + if $window_active + call start_instance($instance) retrieve $start_response + else + call stop_instance($instance) retrieve $stop_response + end +end + +define update_schedule($instance, $schedule, $param_label_schedule, $param_label_next_start, $param_label_next_stop) return $response, $window_active do + # Time + $schedule_time = split($schedule, '_')[0] + + $start_time = split($schedule_time, '-')[0] + $start_time = insert($start_time, 2, ":") + $start_hour = split($start_time, ':')[0] + $start_minute = split($start_time, ':')[1] + + if ! $start_minute + $start_minute = '00' + end + + $stop_time = split($schedule_time, '-')[1] + $stop_time = insert($stop_time, 2, ":") + $stop_hour = split($stop_time, ':')[0] + $stop_minute = split($stop_time, ':')[1] + + if ! $stop_minute + $stop_minute = '00' + end + + # Days + $schedule_days = split($schedule, '_')[1] + $days = gsub(capitalize($schedule_days), "-", ",") + + $start_rule = join(["FREQ=WEEKLY;BYDAY=", $days]) + $stop_rule = join(["FREQ=WEEKLY;BYDAY=", $days]) + + # Timezone + $schedule_timezone = split($schedule, '_')[2] + + if split($schedule, '_')[3] + $schedule_timezone = $schedule_timezone + split($schedule, '_')[3] + end + + if split($schedule, '_')[4] + $schedule_timezone = $schedule_timezone + split($schedule, '_')[4] + end + + $timezone = "UTC" + + if $schedule_timezone + call get_tzlist() retrieve $get_tzlist_response, $timezones + $timezone = $timezones[$schedule_timezone] + end + + call window_active($start_hour, $start_minute, $start_rule, $stop_hour, $stop_minute, $stop_rule, $timezone) retrieve $window_active, $next_start, $next_stop + + $next_start = split($next_start, "T") + $next_start[1] = split($next_start[1], "-") + $next_start[1] = join($next_start[1], "m") + $next_start[1] = split($next_start[1], ":") + $next_start[1] = join($next_start[1], "-") + $next_start[1] = split($next_start[1], "+") + $next_start[1] = join($next_start[1], "p") + $next_start = join($next_start, "t") + + $next_stop = split($next_stop, "T") + $next_stop[1] = split($next_stop[1], "-") + $next_stop[1] = join($next_stop[1], "m") + $next_stop[1] = split($next_stop[1], ":") + $next_stop[1] = join($next_stop[1], "-") + $next_stop[1] = split($next_stop[1], "+") + $next_stop[1] = join($next_stop[1], "p") + $next_stop = join($next_stop, "t") + + call get_instance_info($instance) retrieve $get_instance_info_response, $instance_state, $new_labels, $fingerprint + + $new_labels[$param_label_schedule] = $schedule + $new_labels[$param_label_next_start] = $next_start + $new_labels[$param_label_next_stop] = $next_stop + + $url = $instance['selfLink'] + "/setLabels" + task_label("POST " + $url) + + $response = http_post( + auth: $$auth_google, + url: $url, + headers: { + "cache-control": "no-cache", + "content-type": "application/json" + }, + body: { + "labels": $new_labels, + "labelFingerprint": $fingerprint + } + ) + + task_label("Update Google VM instance labels response: " + $instance["resourceName"] + " " + to_json($response)) + $$all_responses << to_json({"req": "POST " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response updating Google VM instance labels: " + $instance["resourceName"] + " " + to_json($response) + else + task_label("Updating Google VM instance labels successful: " + $instance["resourceName"]) + end +end + +define delete_schedule($instance, $param_label_schedule, $param_label_next_start, $param_label_next_stop) return $response do + call get_instance_info($instance) retrieve $get_instance_info_response, $instance_state, $labels, $fingerprint + + $new_labels = {} + + foreach $label_key in keys($labels) do + if $label_key != $param_label_next_start && $label_key != $param_label_next_stop && $label_key != $param_label_schedule + $new_labels[$label_key] = $labels[$label_key] + end + end + + $url = $instance['selfLink'] + "/setLabels" + task_label("POST " + $url) + + $response = http_post( + auth: $$auth_google, + url: $url, + headers: { + "cache-control": "no-cache", + "content-type": "application/json" + }, + body: { + "labels": $new_labels, + "labelFingerprint": $fingerprint + } + ) + + task_label("Delete Google VM instance labels response: " + $instance["resourceName"] + " " + to_json($response)) + $$all_responses << to_json({"req": "POST " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response deleting Google VM instance labels: " + $instance["resourceName"] + " " + to_json($response) + else + task_label("Deleting Google VM instance labels successful: " + $instance["resourceName"]) + end +end + +define delete_instance($instance) return $response do + $url = $instance['selfLink'] + task_label("DELETE " + $url) + + $response = http_delete( + auth: $$auth_google, + url: $url, + headers: { + "cache-control": "no-cache", + "content-type": "application/json" + } + ) + + task_label("Delete Google VM instance response: " + $instance["resourceName"] + " " + to_json($response)) + $$all_responses << to_json({"req": "DELETE " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response deleting Google VM instance: "+ $instance["resourceName"] + " " + to_json($response) + else + task_label("Delete Google VM instance successful: " + $instance["resourceName"]) + end +end + +# Tertiary CWF functions for specific tasks define window_active($start_hour, $start_minute, $start_rule, $stop_hour, $stop_minute, $stop_rule, $tz) return $window_active, $next_start, $next_stop do - $params = { - verb: 'post', - host: 'bjlaftw4kh.execute-api.us-east-1.amazonaws.com', + $host = "bjlaftw4kh.execute-api.us-east-1.amazonaws.com" + $href = "/production" + $url = $host + $href + task_label("POST " + $url) + + $response = http_request( https: true, - href: '/production', - headers:{ - 'content-type': 'application/json' - }, + verb: "post", + href: $href, + host: $host, + headers: { "content-type": "application/json" }, body: { - 'start_hour': $start_hour, - 'start_minute': $start_minute, - 'start_rule': $start_rule, - 'stop_minute': $stop_minute, - 'stop_hour': $stop_hour, - 'stop_rule': $stop_rule, - 'tz': $tz + "start_hour": $start_hour, + "start_minute": $start_minute, + "start_rule": $start_rule, + "stop_minute": $stop_minute, + "stop_hour": $stop_hour, + "stop_rule": $stop_rule, + "tz": $tz } - } - $response = http_request($params) - call log_message('req=POST bjlaftw4kh.execute-api.us-east-1.amazonaws.com/production body='+to_json($body)+' response='+to_json($response)) - $body = $response['body'] - call log_message('window active $body='+to_s($body)) + ) + + task_label("Post AWS API Gateway response: " + $host + " " + to_json($response)) + $$all_responses << to_json({"req": "POST " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response posting AWS API Gateway: " + $host + " " + to_json($response) + else + task_label("Post AWS API Gateway successful: " + $host) + $window_active = to_b($response['body']['event_active']) + $next_start = $response['body']['next_start'] + $next_stop = $response['body']['next_stop'] + end +end + +define start_instance($instance) return $response do + $url = $instance['selfLink'] + '/start' + task_label("POST " + $url) + + $response = http_post( + auth: $$auth_google, + url: $url, + headers: { + "cache-control": "no-cache", + "content-type": "application/json" + } + ) + + task_label("Start Google VM instance response: " + $instance["resourceName"] + " " + to_json($response)) + $$all_responses << to_json({"req": "POST " + $url, "resp": $response}) - $window_active = to_b($body['event_active']) - $next_start = $body['next_start'] - $next_stop = $body['next_stop'] + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response starting Google VM instance: "+ $instance["resourceName"] + " " + to_json($response) + else + task_label("Start Google VM instance successful: " + $instance["resourceName"]) + end end -define get_labelfingerprint($instance) return $labelFingerprint, $labels_obj do +define stop_instance($instance) return $response do + $url = $instance['selfLink'] + '/stop' + task_label("POST " + $url) + + $response = http_post( + auth: $$auth_google, + url: $url, + headers: { + "cache-control": "no-cache", + "content-type": "application/json" + } + ) + + task_label("Stop Google VM instance response: " + $instance["resourceName"] + " " + to_json($response)) + $$all_responses << to_json({"req": "POST " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response stopping Google VM instance: "+ $instance["resourceName"] + " " + to_json($response) + else + task_label("Stop Google VM instance successful: " + $instance["resourceName"]) + end + + sleep(5) + call get_instance_info($instance) retrieve $state_response, $current_state, $labels, $fingerprint + sleep(5) + + while $current_state != "STOPPED" && $current_state != "TERMINATED" do + call get_instance_info($instance) retrieve $state_response, $current_state, $labels, $fingerprint + sleep(5) + end +end + +define get_instance_info($instance) return $response, $instance_state, $labels, $fingerprint do + $url = $instance['selfLink'] + task_label("GET " + $url) + $response = http_get( - auth: $$google_auth, - url: $instance['selfLink'], - headers:{ + auth: $$auth_google, + url: $url, + headers: { "cache-control": "no-cache", "content-type": "application/json" } ) - call log_message('req=GET '+$instance['selfLink']+' response='+to_json($response)) - $body = $response['body'] - $labelFingerprint = $body['labelFingerprint'] - $labels_obj= $body['labels'] + + task_label("Get Google VM instance response: " + $instance["resourceName"] + " " + to_json($response)) + $$all_responses << to_json({"req": "GET " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response getting Google VM instance: "+ $instance["resourceName"] + " " + to_json($response) + else + task_label("Get Google VM instance successful: " + $instance["resourceName"]) + $instance_state = $response["body"]["status"] + $labels = $response["body"]["labels"] + $fingerprint = $response["body"]["labelFingerprint"] + end end -define get_tzlist() return $timezones do - $params = { - verb: 'get', - host: 'raw.githubusercontent.com', +define get_tzlist() return $response, $tzlist do + $host = "raw.githubusercontent.com" + $href = "/rightscale/policy_templates/master/data/tz_database/timezones_list.json" + $url = $host + $href + task_label("GET " + $url) + + $response = http_request( https: true, - href: '/rightscale/policy_templates/master/data/tz_database/timezones_list.json', + verb: "get", + host: $host, + href: $href, headers:{ 'User-Agent': 'RS Policies' } + ) + + task_label("Get Github Timezone List response: " + $url + " " + to_json($response)) + $$all_responses << to_json({"req": "POST " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response getting Github Timezone List: "+ $url + " " + to_json($response) + else + task_label("Get Github Timezone List successful: " + $url) + $tzlist = from_json($response['body']) + end +end + +define handle_error() do + if !$$errors + $$errors = [] + end + $$errors << $_error["type"] + ": " + $_error["message"] + # We check for errors at the end, and raise them all together + # Skip errors handled by this definition + $_error_behavior = "skip" +end + +############################################################################### +# Meta Policy [alpha] +# Not intended to be modified or used by policy developers +############################################################################### + +# If the meta_parent_policy_id is not set it will evaluate to an empty string and we will look for the policy itself, +# if it is set we will look for the parent policy. +datasource "ds_get_policy" do + request do + auth $auth_flexera + host rs_governance_host + ignore_status [404] + path join(["/api/governance/projects/", rs_project_id, "/applied_policies/", switch(ne(meta_parent_policy_id,""), meta_parent_policy_id, policy_id) ]) + header "Api-Version", "1.0" + end + result do + encoding "json" + field "id", jmes_path(response, "id") + end +end + +datasource "ds_parent_policy_terminated" do + run_script $js_decide_if_self_terminate, $ds_get_policy, policy_id, meta_parent_policy_id +end + +# If the policy was applied by a meta_parent_policy we confirm it exists if it doesn't we confirm we are deleting +# This information is used in two places: +# - determining whether or not we make a delete call +# - determining if we should create an incident (we don't want to create an incident on the run where we terminate) +script "js_decide_if_self_terminate", type: "javascript" do + parameters "found", "self_policy_id", "meta_parent_policy_id" + result "result" + code <<-EOS + var result + if (meta_parent_policy_id != "" && found.id == undefined) { + result = true + } else { + result = false } - $response = http_request($params) - call log_message('req=GET raw.githubusercontent.com/rightscale/policy_templates/master/data/tz_database/timezones_list.json response_code='+to_s($response['code'])+' response_size='+to_s(size($response['body']))) - $timezones = $response['body'] + EOS +end + +# Two potentials ways to set this up: +# - this way and make a unneeded 'get' request when not deleting +# - make the delete request an interate and have it iterate over an empty array when not deleting and an array with one item when deleting +script "js_make_terminate_request", type: "javascript" do + parameters "should_delete", "policy_id", "rs_project_id", "rs_governance_host" + result "request" + code <<-EOS + + var request = { + auth: 'auth_flexera', + host: rs_governance_host, + path: "/api/governance/projects/" + rs_project_id + "/applied_policies/" + policy_id, + headers: { + "API-Version": "1.0", + "Content-Type":"application/json" + }, + } + + if (should_delete) { + request.verb = 'DELETE' + } + EOS +end + +datasource "ds_terminate_self" do + request do + run_script $js_make_terminate_request, $ds_parent_policy_terminated, policy_id, rs_project_id, rs_governance_host + end +end + +datasource "ds_is_deleted" do + run_script $js_check_deleted, $ds_terminate_self +end + +# This is just a way to have the check delete request connect to the farthest leaf from policy. +# We want the delete check to the first thing the policy does to avoid the policy erroring before it can decide whether or not it needs to self terminate +# Example a customer deletes a credential and then terminates the parent policy. We still want the children to self terminate +# The only way I could see this not happening is if the user who applied the parent_meta_policy was offboarded or lost policy access, the policies who are impersonating the user +# would not have access to self-terminate +# It may be useful for the backend to enable a mass terminate at some point for all meta_child_policies associated with an id. +script "js_check_deleted", type: "javascript" do + parameters "response" + result "result" + code <<-EOS + result = {"path":"/"} + EOS end diff --git a/cost/google/schedule_instance/google_schedule_instance_meta_parent.pt b/cost/google/schedule_instance/google_schedule_instance_meta_parent.pt new file mode 100644 index 0000000000..df31f0039c --- /dev/null +++ b/cost/google/schedule_instance/google_schedule_instance_meta_parent.pt @@ -0,0 +1,1322 @@ +name "Google Schedule Instance Meta Parent" +rs_pt_ver 20180301 +type "policy" +short_description "Applies and manages \"child\" [Google Schedule Instance](https://github.com/flexera-public/policy_templates/tree/master/cost/google/schedule_instance) Policies." +severity "low" +category "Cost" +tenancy "single" +default_frequency "15 minutes" +info( + provider: "Google", + version: "3.0", # This version of the Meta Parent Policy Template should match the version of the Child Policy Template as it appears in the Catalog for best reliability + publish: "false" +) + +############################################################################## +# Parameters +############################################################################## + +## Meta Parent Parameters +## These are params specific to the meta parent policy. +parameter "param_combined_incident_email" do + type "list" + label "Email addresses for combined incident" + description "A list of email addresses to notify with the consolidated child policy incident." + default [] +end + +parameter "param_dimension_filter_includes" do + type "list" + label "Dimension Include Filters" + description <<-EOS + Filters [`dimension_name=dimension_value` and `dimension_name=~dimension_value` pairs] to determine which Google Projects returned by the Flexera Bill Analysis API to **INCLUDE** and be applied to. + Use = to match the entire value and =~ to match a substring contained in the value. + During each run this policy will select Google Projects who match **all** the filters defined and apply a child policy for each. + If no include filters are provided, then all Google Projects are included by default. + Most of the dimensions in Flexera can be used [default dimensions, custom tag dimensions, rule-based dimensions]. Full list of available dimensions documented in the [Bill Analysis API Docs](https://reference.rightscale.com/bill_analysis/). + EOS + default [] +end + +parameter "param_dimension_filter_excludes" do + type "list" + label "Dimension Exclude Filters" + description <<-EOS + Filters [`dimension_name=dimension_value` and `dimension_name=~dimension_value` pairs] to determine which Google Projects returned by the Flexera Bill Analysis API to **EXCLUDE** and *not* have policy applied to. + Use = to match the entire value and =~ to match a substring contained in the value. + During each run this policy will select Google Projects who match **all** the filters defined here and excludes them from results. + Can be used to exclude specific Google Projects [`vendor_account=123456789012`] + Most of the dimensions in Flexera can be used [default dimensions, custom tag dimensions, rule-based dimensions]. Full list of available dimensions documented in the [Bill Analysis API Docs](https://reference.rightscale.com/bill_analysis/). + EOS + default [] +end + +parameter "param_policy_schedule" do + type "string" + label "Child Policy Schedule" + description "The interval at which the child policy checks for conditions and generates incidents." + default "weekly" + allowed_values "daily", "weekly", "monthly" +end + +parameter "param_template_source" do + type "string" + label "Child Policy Template Source" + description "By default, will use the \"Google Schedule Instance\" Policy Template from Catalog. Optionally, you can use the \"Google Schedule Instance\" Policy Template uploaded in the current Flexera Project." + default "Published Catalog Template" + allowed_values "Published Catalog Template", "Uploaded Template" +end + +## Child Policy Parameters +parameter "param_label_schedule" do + type "string" + category "Label Keys" + label "Schedule Label Key" + description "Label key that schedule information is stored in. Default is recommended for most use cases." + default "schedule" +end + +parameter "param_label_next_start" do + type "string" + category "Label Keys" + label "Next Start Label Key" + description "Label key to use for scheduling instance to start. Default is recommended for most use cases." + default "next_start" +end + +parameter "param_label_next_stop" do + type "string" + category "Label Keys" + label "Next Stop Label Key" + description "Label key to use for scheduling instance to stop. Default is recommended for most use cases." + default "next_stop" +end + +parameter "param_regions_allow_or_deny" do + type "string" + category "Filters" + label "Allow/Deny Regions" + description "Allow or Deny entered regions. See the README for more details." + allowed_values "Allow", "Deny" + default "Allow" +end + +parameter "param_regions_list" do + type "list" + category "Filters" + label "Allow/Deny Regions List" + description "A list of allowed or denied regions. See the README for more details." + default [] +end + +parameter "param_exclusion_labels" do + type "list" + category "Filters" + label "Exclusion Labels (Key:Value)" + description "Google labels to ignore resources that you don't want to produce recommendations for. Use Key:Value format for specific label key/value pairs, and Key:* format to match any resource with a particular key, regardless of value. Examples: env:production, DO_NOT_DELETE:*" + allowed_pattern /(^$)|[\w]*\:.*/ + default [] +end + +parameter "param_automatic_action" do + type "list" + category "Actions" + label "Automatic Actions" + description "When this value is set, this policy will automatically take the selected action." + allowed_values ["Execute Schedules"] + default [] +end + +############################################################################### +# Authentication +############################################################################### +credentials "auth_google" do + schemes "oauth2" + label "Google" + description "Select the Google Cloud Credential from the list." + tags "provider=gce" +end + +credentials "auth_flexera" do + schemes "oauth2" + label "Flexera" + description "Select Flexera One OAuth2 credentials" + tags "provider=flexera" +end + +############################################################################### +# Pagination +############################################################################### + +pagination "pagination_google" do + get_page_marker do + body_path "nextPageToken" + end + set_page_marker do + query "pageToken" + end +end + +############################################################################### +# Datasources & Scripts +############################################################################### + +# Get Applied Parent Policy Details +datasource "ds_self_policy_information" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/applied_policies/", policy_id]) + header "Api-Version", "1.0" + end + result do + encoding "json" + field "name", jmes_path(response, "name") + field "creator_id", jmes_path(response, "created_by.id") + field "credentials", jmes_path(response, "credentials") + field "options", jmes_path(response, "options") + end +end + +datasource "ds_child_policy_options" do + run_script $js_child_policy_options, $ds_self_policy_information +end + +script "js_child_policy_options", type: "javascript" do + parameters "ds_self_policy_information" + result "options" + code <<-EOS + // Filter Options that are not appropriate for Child Policy + var options = _.map(ds_self_policy_information.options, function(option){ + // param_combined_incident_email, param_dimension_filter_includes, param_dimension_filter_excludes", param_policy_schedule are exclusion to Meta Parent Policy Parameters + if (!_.contains(["param_combined_incident_email", "param_dimension_filter_includes", "param_dimension_filter_excludes", "param_policy_schedule", "param_template_source"], option.name)) { + return { "name": option.name, "value": option.value }; + } + }); + // Explicitly add param_email which is disabled/does not exist in meta parent policy + options.push({ + "name": "param_email", + "value": [] + }); + EOS +end + +datasource "ds_child_policy_options_map" do + run_script $js_child_policy_options_map, $ds_child_policy_options +end + +script "js_child_policy_options_map", type: "javascript" do + parameters "ds_child_policy_options" + result "options" + code <<-EOS + function format_options_keyvalue(options) { + var options_keyvalue_map = {}; + _.each(options, function(option) { + options_keyvalue_map[option.name] = option.value; + }); + return options_keyvalue_map; + } + var options = format_options_keyvalue(ds_child_policy_options) + EOS +end + +datasource "ds_format_self" do + run_script $js_format_self, $ds_self_policy_information, $ds_child_policy_options_map +end + +script "js_format_self", type: "javascript" do + parameters "ds_self_policy_information", "ds_child_policy_options_map" + result "formatted" + code <<-EOS + var formatted = { + "name": ds_self_policy_information["name"], + "creator_id": ds_self_policy_information["creator_id"], + "credentials": ds_self_policy_information["credentials"], + "options": ds_child_policy_options_map + }; + EOS +end + +# Get Pulished Policy Details +datasource "ds_published_child_policy_information" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/orgs/", rs_org_id, "/published_templates"]) + header "Api-Version", "1.0" + end + result do + encoding "json" + # Select the published policy that is published by "support@flexera.com" and matches the name of the child policy template + collect jq(response, '.items[] | select(.name == "Google Schedule Instance" and .created_by.email == "support@flexera.com")' ) do + field "name", jmes_path(col_item, "name") + field "href", jmes_path(col_item, "href") + field "short_description", jmes_path(col_item, "short_description") + end + end +end + +# Get Uploaded Policy Details +datasource "ds_project_child_policy_information" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/policy_templates"]) + header "Api-Version", "1.0" + end + result do + encoding "json" + # Select the uploaded policy that matches the name of the child policy template + collect jq(response, '.items[] | select(.name == "Google Schedule Instance")' ) do + field "name", jmes_path(col_item, "name") + field "href", jmes_path(col_item, "href") + field "short_description", jmes_path(col_item, "short_description") + end + end +end + +datasource "ds_get_billing_centers" do + request do + auth $auth_flexera + host rs_optima_host + path join(["/analytics/orgs/",rs_org_id,"/billing_centers"]) + header "Api-Version", "1.0" + header "User-Agent", "RS Policies" + query "view", "allocation_table" + ignore_status [403] + end + result do + encoding "json" + # Select the Billing Centers that have "parent_id" undefined or "" (i.e. top-level Billing Centers) + collect jq(response, '.[] | select(.parent_id == null)' ) do + field "href", jq(col_item,".href") + field "id", jq(col_item,".id") + field "name", jq(col_item,".name") + field "parent_id", jq(col_item,".parent_id") + end + end +end + +# Get the Google projects +datasource "ds_get_google_projects" do + request do + run_script $js_make_billing_center_request, rs_org_id, rs_optima_host, $ds_get_billing_centers, $param_dimension_filter_includes, $param_dimension_filter_excludes + end + result do + encoding "json" + collect jmes_path(response,"rows[*]") do + field "projectID", jmes_path(col_item,"dimensions.vendor_account") + field "projectName", jmes_path(col_item,"dimensions.vendor_account_name") + end + end +end + +script "js_make_billing_center_request", type: "javascript" do + parameters "rs_org_id", "rs_optima_host", "billing_centers_unformatted", "param_dimension_filter_includes", "param_dimension_filter_excludes" + result "request" + code <<-EOS + + billing_centers_formatted = [] + + for (x=0; x< billing_centers_unformatted.length; x++) { + billing_centers_formatted.push(billing_centers_unformatted[x]["id"]) + } + + finish = new Date() + finishFormatted = finish.toJSON().split("T")[0] + start = new Date() + start.setDate(start.getDate() - 30) + startFormatted = start.toJSON().split("T")[0] + + // Default dimensions and filter expressions required for meta parent policy + var dimensions = ["vendor_account", "vendor_account_name"]; + var filter_expressions = [ + { dimension: "vendor", type: "equal", value: "GCP" } + ] + + // Append to default dimensions and filter expressions using parent policy params + _.each(param_dimension_filter_includes, function (v) { + // split key=value string + if (v.indexOf('=~') == -1) { + var split = v.split("="); + var type = "equal" + } else { + var split = v.split("=~"); + var type = "substring" + } + + var k = split[0]; + var v = split[1]; + + // append to lists + dimensions.push(k); + + if (type == "equal") { + filter_expressions.push({ dimension: k, type: "equal", value: v }); + } else { + filter_expressions.push({ dimension: k, type: "substring", substring: v }); + } + }); + + // Append to filter expressions using exclude policy params + _.each(param_dimension_filter_excludes, function (v) { + // split key=value string + if (v.indexOf('=~') == -1) { + var split = v.split("="); + var type = "equal" + } else { + var split = v.split("=~"); + var type = "substring" + } + + var k = split[0]; + var v = split[1]; + + // append to lists + dimensions.push(k); + + if (type == "equal") { + filter_expressions.push({ "type": "not", "expression": { "dimension": k, "type": "equal", "value": v } }); + } else { + filter_expressions.push({ "type": "not", "expression": { "dimension": k, "type": "substring", "substring": v } }); + } + }); + + // Produces a duplicate-free version of the array + dimensions = _.uniq(dimensions); + + var body = { + "dimensions": dimensions, + "granularity":"day", + "start_at": startFormatted, + "end_at": finishFormatted, + "metrics":["cost_amortized_unblended_adj"], + "billing_center_ids": billing_centers_formatted, + "filter": + { + "type": "and", + "expressions": filter_expressions + }, + "summarized": true + } + var request = { + auth: 'auth_flexera', + host: rs_optima_host, + scheme: 'https', + verb: 'POST', + path: "/bill-analysis/orgs/"+ rs_org_id + "/costs/aggregated", + headers: { + "API-Version": "1.0", + "Content-Type":"application/json" + }, + body: JSON.stringify(body) + } + EOS +end + +# Get Child policies +datasource "ds_get_existing_policies" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/applied_policies"]) + header "Api-Version", "1.0" + query "meta_parent_policy_id", policy_id + end + result do + collect jq(response, '.items[]?') do + field "name", jq(col_item, ".name") + field "applied_policy_id", jq(col_item, ".id") + field "options", jq(col_item, ".options") + field "updated_at", jq(col_item, ".updated_at") + field "status", jq(col_item, ".status") + end + end +end + +# Get Child policies incidents +datasource "ds_get_existing_policies_incidents" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/incidents"]) + header "Api-Version", "1.0" + query "meta_parent_policy_id", policy_id + end + result do + collect jq(response, '.items[]?') do + field "incident_id", jq(col_item, ".id") + field "applied_policy_id", jq(col_item, ".applied_policy.id") + field "state", jq(col_item, ".state") + field "violation_data_count", jq(col_item, ".violation_data_count") + field "updated_at", jq(col_item, ".updated_at") + field "meta_parent_policy_id", jq(col_item, ".meta_parent_policy_id") + end + end +end + +datasource "ds_format_incidents" do + run_script $js_format_existing_policies_incidents, $ds_get_existing_policies_incidents +end + +script "js_format_existing_policies_incidents", type: "javascript" do + parameters "unformatted" + result "formatted" + code <<-EOS + formatted={} + for (x=0;x newDate) { + duplicates.push({ + "applied_policy_id":ds_get_existing_policies[x]["applied_policy_id"], + "applied_policy_name":ds_get_existing_policies[x]["name"], + "status":ds_get_existing_policies[x]["status"], + "updated_at":ds_get_existing_policies[x]["updated_at"], + "incident": ds_format_incidents[ds_get_existing_policies[x]["applied_policy_id"]] + }) + } else { + duplicates.push({ + "applied_policy_id":current["applied_policy_id"], + "applied_policy_name":current["applied_policy_name"], + "status":current["status"], + "updated_at":current["updated_at"], + "incident": current["incident"] + }) + formatted[projectID] = { + "applied_policy_id":ds_get_existing_policies[x]["applied_policy_id"], + "applied_policy_name":ds_get_existing_policies[x]["name"], + "status":ds_get_existing_policies[x]["status"], + "updated_at":ds_get_existing_policies[x]["updated_at"], + "incident": ds_format_incidents[ds_get_existing_policies[x]["applied_policy_id"]], + "options": options + } + + } + } + } + + result.formatted=formatted + result.duplicates=duplicates + result.tracking=tracking + EOS +end + +datasource "ds_take_in_parameters" do + run_script $js_take_in_parameters, $ds_get_google_projects, $ds_format_self, first($ds_published_child_policy_information), first($ds_project_child_policy_information), $ds_format_existing_policies, $ds_child_policy_options, $ds_child_policy_options_map, $param_template_source, $param_policy_schedule, policy_id, f1_app_host, rs_org_id, rs_project_id +end + +# hardcode template href with id from catalog +# catalog policies show in customer's published templates with their org id +# "template_href": "/api/governance/orgs/" + rs_org_id + "/published_templates/62618616e3dff80001572bf0" +# update logic: the only reason we're going to update the child policies for is changes to options +# and only some options, email is always blank and projectID is tied to the idenity of each policy, so: new project creation, removal of project: termination +# param_automatic_action is a list with only one action, unless the person is applying using an API and putting the same value multiple times this should either be a length of 0 or 1 +# param_log_to_cm_audit_entries is a String of Yes or No +# param_exclude_tags and param_allowed_regions are arrays. I'm doing an update on the order changing but the values remaining the same. +# If we only want to do an update on the values changing we could sort before doing the equality check. +script "js_take_in_parameters", type: "javascript" do + parameters "ds_get_google_projects", "ds_format_self", "ds_published_child_policy_information", "ds_project_child_policy_information", "ds_format_existing_policies", "ds_child_policy_options", "ds_child_policy_options_map", "param_template_source", "param_policy_schedule", "meta_parent_policy_id", "f1_app_host", "rs_org_id", "rs_project_id" + result "grid_and_cwf" + code <<-EOS + + // Set Child Policy Information based on param_template_source value + if (param_template_source == "Published Catalog Template") { + child_policy_information = ds_published_child_policy_information + } else { + child_policy_information = ds_project_child_policy_information + } + + max_actions = 50; + + grid_and_cwf={grid:[], to_create:[], to_update:[], to_delete:[], parent_policy:ds_format_self}; + + should_keep = ds_format_existing_policies.tracking; + + // Construct UI URL prefixes for policy template summary + ui_url_prefix = "https://" + f1_app_host + "/orgs/" + rs_org_id; + applied_policy_url_prefix = ui_url_prefix + "/automation/applied-policies/projects/" + rs_project_id + "?noIndex=1&policyId="; + incident_url_prefix = ui_url_prefix + "/automation/incidents/projects/" + rs_project_id + "?noIndex=1&incidentId="; + + function add_to_grid(ep, action) { + policy_status={ + "policy_name": ep["applied_policy_name"], + "policy_link": applied_policy_url_prefix + ep["applied_policy_id"], + "meta_policy_status": action, + "policy_status": ep["status"], + "policy_last_update": ep["updated_at"], + }; + if (ep.incident != null) { + policy_status["incident_link"] = incident_url_prefix + ep.incident.incident_id; + policy_status["incident_state"] = ep.incident.state; + policy_status["incident_violation_data_count"] = ep.incident.violation_data_count; + policy_status["incident_last_update"] = ep.incident.updated_at; + } + grid_and_cwf.grid.push(policy_status); + } + + for (x=0; x -1) { + _.each(incident["violation_data"], function(violation) { + violation["incident_id"] = incident["id"]; + result.push(violation); + }); + } + }); +EOS +end + + +# Escalation for Execute Schedules +escalation "esc_execute_schedules" do + automatic false # Do not automatically action from meta parent. the child will handle automatic escalations if param is set + label "Execute Schedules" + description "Approval to start or stop all selected instances based on schedule" + run "esc_execute_schedules", data, rs_governance_host, rs_project_id +end +define esc_execute_schedules($data, $governance_host, $rs_project_id) do + call child_run_action($data, $governance_host, $rs_project_id, "Execute Schedules") +end + +# Escalation for Update Schedules +escalation "esc_update_schedules" do + automatic false # Do not automatically action from meta parent. the child will handle automatic escalations if param is set + label "Update Schedules" + description "Approval to update the schedule labels on all selected instances" + run "esc_update_schedules", data, rs_governance_host, rs_project_id +end +define esc_update_schedules($data, $governance_host, $rs_project_id) do + call child_run_action($data, $governance_host, $rs_project_id, "Update Schedules") +end + +# Escalation for Delete Schedules +escalation "esc_delete_schedules" do + automatic false # Do not automatically action from meta parent. the child will handle automatic escalations if param is set + label "Delete Schedules" + description "Approval to delete the schedule labels on all selected instances" + run "esc_delete_schedules", data, rs_governance_host, rs_project_id +end +define esc_delete_schedules($data, $governance_host, $rs_project_id) do + call child_run_action($data, $governance_host, $rs_project_id, "Delete Schedules") +end + +# Escalation for Start Instances +escalation "esc_start_instances" do + automatic false # Do not automatically action from meta parent. the child will handle automatic escalations if param is set + label "Start Instances" + description "Approval to start all selected instances" + run "esc_start_instances", data, rs_governance_host, rs_project_id +end +define esc_start_instances($data, $governance_host, $rs_project_id) do + call child_run_action($data, $governance_host, $rs_project_id, "Start Instances") +end + +# Escalation for Stop Instances +escalation "esc_stop_instances" do + automatic false # Do not automatically action from meta parent. the child will handle automatic escalations if param is set + label "Stop Instances" + description "Approval to stop all selected instances" + run "esc_stop_instances", data, rs_governance_host, rs_project_id +end +define esc_stop_instances($data, $governance_host, $rs_project_id) do + call child_run_action($data, $governance_host, $rs_project_id, "Stop Instances") +end + +# Escalation for Delete Instances +escalation "esc_delete_instances" do + automatic false # Do not automatically action from meta parent. the child will handle automatic escalations if param is set + label "Delete Instances" + description "Approval to delete all selected instances" + run "esc_delete_instances", data, rs_governance_host, rs_project_id +end +define esc_delete_instances($data, $governance_host, $rs_project_id) do + call child_run_action($data, $governance_host, $rs_project_id, "Delete Instances") +end + + +# Summary and a conditional incident which will show up if any policy is being applied, updated or deleted. +# Minimum of 1 incident, max of four +# Could swap the summary to only showing running +# Could also just have one incident and use meta_status to determine which escalation happens +policy "policy_scheduled_report" do + # Consolidated Incident Check(s) + # Consolidated incident for Google Scheduled VM Instances + validate $ds_instances_to_schedule_combined_incidents do + summary_template "Consolidated Incident: {{ len data }} Google Scheduled VM Instances" + escalate $esc_email + escalate $esc_execute_schedules + escalate $esc_update_schedules + escalate $esc_delete_schedules + escalate $esc_start_instances + escalate $esc_stop_instances + escalate $esc_delete_instances + check eq(size(data), 0) + export do + resource_level true + field "accountID" do + label "Project ID" + end + field "accountName" do + label "Project Name" + end + field "projectNumber" do + label "Project Number" + end + field "resourceID" do + label "Resource ID" + end + field "resourceName" do + label "Resource Name" + end + field "resourceType" do + label "Resource Type" + end + field "zone" do + label "Zone" + end + field "region" do + label "Region" + end + field "hostname" do + label "Hostname" + end + field "platform" do + label "Platform" + end + field "tags" do + label "Labels" + end + field "service" do + label "Service" + end + field "status" do + label "Status" + end + field "selfLink" do + label "Resource Link" + end + field "schedule" do + label "Schedule" + end + field "next_start" do + label "Next Start" + end + field "next_stop" do + label "Next Stop" + end + field "id" do + label "ID" + end + field "incident_id" do + label "Child Incident ID" + end + end + end + + # Status Incident Check + validate $ds_take_in_parameters do + summary_template "{{ data.parent_policy.name }}: Status of Child Policies" + detail_template <<-EOS +The current status of Child Policies for **{{ data.parent_policy.name }}**: + +Total Child Applied Policies: {{ len data.grid }} + +| Applied Policy | Meta Child Policy Status | Policy Status | Policy Last Update | Incident | Incident State | Violation Count | Incident Last Update | +| -------------- | ------------------------ | ------------- | ------------------ | -------- | -------------- | --------------- | -------------------- | +{{ range data.grid -}} +| {{- if .policy_link }} [{{ .policy_name }}]({{ .policy_link }}) {{ else }} {{ .policy_name }} {{- end }} | {{- if .meta_policy_status }} {{ .meta_policy_status }} {{ else }} No value {{- end }} | {{- if .policy_status }} {{ .policy_status }} {{ else }} No value {{- end }} | {{- if .policy_last_update }} {{ .policy_last_update }} {{ else }} No value {{- end }} | {{- if .incident_link }} [Incident]({{ .incident_link }}) {{ else }} No Incident {{- end }} | {{- if .incident_state }} {{ .incident_state }} {{ else }} No Incident {{- end }} | {{- if .incident_last_update }} {{ .incident_violation_data_count }} {{ else }} No Incident {{- end }} | {{- if .incident_last_update }} {{ .incident_last_update }} {{ else }} No Incident {{- end }} | +{{ end -}} + +EOS + check false + # Export Table disabled for now until UI can support URL / linking + # Without linking the `policy_link`, `incident_link` values are not usable directly from UI + # export "grid" do + # resource_level true + # field "id" do + # label "Applied Policy ID" + # end + # field "policy_name" do + # label "Applied Policy Name" + # end + # field "policy_link" do + # label "Applied Policy Link" + # end + # field "meta_policy_status" do + # label "Meta Child Policy Status" + # end + # field "policy_status" do + # label "Policy Status" + # end + # field "policy_last_update" do + # label "Policy Last Update" + # end + # field "incident_link" do + # label "Incident Link" + # end + # field "incident_state" do + # label "Incident State" + # end + # field "incident_violation_data_count" do + # label "Incident Violation Count" + # end + # field "incident_last_update" do + # label "Incident Last Update" + # end + # end + end + + # Create Child Policies Incident Check + validate $ds_to_create do + summary_template "Policies being created" + detail_template <<-EOS + Policies Being Created: + + | Applied Policy | + | --------------- | + {{ range data -}} + | {{ .name }} | + {{ end -}} + EOS + escalate $create_policies + check eq(size(data),0) + end + + # Update Child Policies Incident Check + validate $ds_to_update do + summary_template "Policies being updated" + detail_template <<-EOS + Policies Being Updated: + + | Applied Policy | + | --------------- | + {{ range data -}} + | {{ .name }} | + {{ end -}} + EOS + escalate $update_policies + check eq(size(data),0) + end + + # Delete Child Policies Incident Check + validate $ds_to_delete do + summary_template "Policies being deleted" + detail_template <<-EOS + Policies being Deleted: + + | Applied Policy | + | --------------- | + {{ range data -}} + | {{ .name }} | + {{ end -}} + EOS + escalate $delete_policies + check eq(size(data),0) + end +end + +# Begin Shared Functions for Child Actions from Consolidated Incident +define groupByIncidentID($data) return $incidents do + # Empty hash to store incidents is incident_id + $incidents = {} + + task_label("Grouping items by Incident ID") + $index = 1 + foreach $item in $data do + task_label("Grouping items by Incident ID. "+to_s($index)+"/"+to_s(size($data))) + if !$incidents[$item["incident_id"]] + #task_label("Grouping items by Incident ID. "+to_s($index)+"/"+to_s(size($data))". New Incident: "+$item["incident_id"]) + $incidents[$item["incident_id"]] = {"id": $item["incident_id"], "resource_ids": []} + end + #task_label("Grouping items by Incident ID. "+to_s($index)+"/"+to_s(size($data))". Appending Resource: "+$item["id"]) + # Append resource id to the list for the incident + $incidents[$item["incident_id"]]["resource_ids"] = $incidents[$item["incident_id"]]["resource_ids"] + [$item["id"]] + end +end + +define child_run_action($data, $governance_host, $rs_project_id, $action_label) do + # Empty global array for log strings, helpful for debugging + $$debug = [] + + # Group Resources by Incident ID + # This reduces the number of requests made to the Flexera API + call groupByIncidentID($data) retrieve $incidents + $$debug_incidents = to_json($incidents) + + call runActions($incidents, $action_label, $governance_host, $rs_project_id) + + # If we encountered any errors, use `raise` to mark the CWF process as errored + if inspect($$errors) != "null" + raise join($$errors,"\n") + end + + # If we made it here, all actions completed successfully + # Celebrate Success! + task_label("All \""+$action_label+"\" actions completed successfully!") +end + +define runActions($incidents, $action_label, $governance_host, $rs_project_id) do + foreach $id in keys($incidents) do + sub on_error: handle_error() do + $incident = $incidents[$id] + task_label("Triggering action \""+$action_label+"\" on "+size($incident["resource_ids"])+" count resources via incident "+$incident["id"]) + $request = { + auth: $$auth_flexera, + verb: "get", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/incidents/", $incident["id"]]), + headers: { "Api-Version": "1.0" }, + query_strings: { "view": "extended" } + } + $response = http_request($request) + $$debug << to_json({ + "request": $request, + "response": $response + }) + $action_id = "" + foreach $action in $response["body"]["available_actions"] do + # If we have not already found the action id, and the label matches, set the action id + # The first check is to prevent looking through the entire list if we already have the id + if $action["label"] == $action_label + $action_id = $action["id"] + end + end + if $action_id == "" + raise "Could not find action id for \""+$action_label+"\" response="+to_json($response) + end + # Now we are reach to trigger the action + $request = { + auth: $$auth_flexera, + verb: "post", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/incidents/", $incident["id"],"/actions/", $action_id,"/run_action"]), + headers: { "Api-Version": "1.0" }, + body: { "options":[{ "name": "ids", "value": $incident["resource_ids"] }] } + } + $response = http_request($request) + $$debug << to_json({ + "request": $request, + "response": $response + }) + # Get the action status from response header + $action_location = $response["headers"]["Location"] + + # Setup some variables for the wait loop + $action_status = "" + $loop_count = 0 + $loop_endtime = now() + (3600*2) # 2 hours from now + # [ queued, aborted, pending, running, completed, failed, denied ] + while ($action_status !~ /^(aborted|completed|failed|denied)/) && (now() <= $loop_endtime) do + # Using Loop Count to slowly increment the sleep time + # This is to prevent the loop from hammering our APIs + $loop_count = $loop_count + 1 + task_label("action_status=\""+$action_status+"\" Sleeping for "+to_s($loop_count)+" seconds") + sleep($loop_count) + task_label("action_status=\""+$action_status+"\" Getting action status") + $request = { + auth: $$auth_flexera, + verb: "get", + https: true, + host: $governance_host, + href: $action_location, + headers: { "Api-Version": "1.0" }, + query_strings: { "view": "extended" } + } + $response = http_request($request) + $$debug << to_json({ + "request": $request, + "response": $response + }) + $action_status = $response["body"]["status"] + end + if ($action_status != "completed") + # Check if we are out of time first + if (now() > $loop_endtime) + raise "action_status=\""+$action_status+"\" Action did not complete in time. Aborting to prevent endless loop. action_status_json="+to_json($response) + else + # If not, then it was aborted, failed or denied + raise "action_status=\""+$action_status+"\" Action did not complete as expected. action_status_json="+to_json($response) + end + end + # If we made it here, the action completed successfully + task_label("action_status=\""+$action_status+"\" Action completed successfully") + end + end +end +# End Shared Functions for Child Actions from Consolidated Incident + +# CWF function to handle errors +define handle_error() do + if !$$errors + $$errors = [] + end + $$errors << $_error["type"] + ": " + $_error["message"] + # We check for errors at the end, and raise them all together + # Skip errors handled by this definition + $_error_behavior = "skip" +end + +# Used only for emailing the combined child incident if so desired +escalation "esc_email" do + automatic true + label "Send Email" + description "Send incident email" + email $param_combined_incident_email +end + +escalation "create_policies" do + run "create_applied_policies", data, rs_governance_host, rs_project_id +end + +# if name !=null +define create_applied_policies($data, $governance_host, $rs_project_id) return $responses do + $responses = [] + $$debug = [] + $item_index = 0 + $item_total = size($data) + foreach $item in $data do + $item_index = $item_index + 1 + $status = to_s("("+$item_index+"/"+$item_total+")") + task_label($status+" Creating Applied Policy with Options: " + to_json($item["options"])) + $response = http_request( + auth: $$auth_flexera, + verb: "post", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/applied_policies"]), + headers: { "Api-Version": "1.0" }, + body: { + "name": $item["name"], + "description": $item["description"], + "template_href": $item["template_href"], + "frequency": $item["frequency"], + "options": $item["options"], + "credentials": $item["credentials"], + "meta_parent_policy_id": $item["meta_parent_policy_id"] + } + ) + $responses << $response + $$debug << to_json({ + "response": $response, + "item": $item, + "governance_host": $governance_host + }) + end +end + +escalation "update_policies" do + run "update_applied_policies", data, rs_governance_host, rs_project_id +end + +define update_applied_policies($data, $governance_host, $rs_project_id) return $responses do + $responses = [] + $$debug = [] + $item_index = 0 + $item_total = size($data) + foreach $item in $data do + $item_index = $item_index + 1 + $status = to_s("("+$item_index+"/"+$item_total+")") + task_label($status+" Updating Applied Policy with Options: " + to_json($item["options"])) + $response = http_request( + auth: $$auth_flexera, + verb: "patch", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/applied_policies/", $item["applied_policy_id"]]), + headers: { "Api-Version": "1.0" }, + body: { + "options": $item["options"] + } + ) + $responses << $response + $$debug << to_json({ + "response": $response, + "item": $item, + "governance_host": $governance_host + }) + end +end + +escalation "delete_policies" do + run "delete_applied_policies", data, rs_governance_host, rs_project_id +end + +define delete_applied_policies($data, $governance_host, $rs_project_id) return $responses do + $responses = [] + $$debug = [] + $item_index = 0 + $item_total = size($data) + foreach $item in $data do + $item_index = $item_index + 1 + $status = to_s("("+$item_index+"/"+$item_total+")") + task_label($status+" Deleting Applied Policy: " + $item["id"]) + $response = http_request( + auth: $$auth_flexera, + verb: "delete", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/applied_policies/", $item["id"]]), + headers: { "Api-Version": "1.0" } + ) + $responses << $response + $$debug << to_json({ + "response": $response, + "item": $item, + "governance_host": $governance_host + }) + end +end diff --git a/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb b/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb index 3ba064c760..5719dc226a 100644 --- a/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb +++ b/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb @@ -65,6 +65,7 @@ "../../cost/google/idle_ip_address_recommendations/google_idle_ip_address_recommendations.pt", "../../cost/google/idle_persistent_disk_recommendations/google_idle_persistent_disk_recommendations.pt", "../../cost/google/rightsize_vm_recommendations/google_rightsize_vm_recommendations.pt", + "../../cost/google/schedule_instance/google_schedule_instance.pt", "../../cost/google/cud_recommendations/google_committed_use_discount_recommendations.pt", "../../cost/google/old_snapshots/google_delete_old_snapshots.pt" ]