Creating a dashboard that displays details at the endpoint level. #4643
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Terraform plan for meta No changes. Your infrastructure matches the configuration.📝 Plan generated in Pull Request Checks #4188 |
|
Terraform plan for dev Plan: 2 to add, 0 to change, 1 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.dev.module.cors.null_resource.cors_header must be replaced
-/+ resource "null_resource" "cors_header" {
!~ id = "*******************" -> (known after apply)
!~ triggers = { # forces replacement
!~ "always_run" = "2025-01-15T20:24:56Z" -> (known after apply)
}
}
# module.dev.module.newrelic.newrelic_one_dashboard_json.fac_monitoring will be created
+ resource "newrelic_one_dashboard_json" "fac_monitoring" {
+ account_id = (known after apply)
+ guid = (known after apply)
+ id = (known after apply)
+ json = jsonencode(
{
+ description = "FAC Monitoring"
+ name = "FAC Monitoring - Dev"
+ pages = [
+ {
+ description = null
+ name = "At A Glance"
+ widgets = [
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 2
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM SyntheticCheck SELECT percentage(count(result), WHERE result = 'SUCCESS') AS 'Uptime' WHERE monitorName = 'Fac Ping Dev' SINCE 1 hour ago"
},
]
}
+ title = "Uptime (last hour)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 3
+ height = 3
+ row = 1
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "Uptime (%)"
+ precision = "4"
+ type = "decimal"
},
]
+ linkedEntityGuids = []
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM SyntheticCheck SELECT percentage(count(result), WHERE result = 'SUCCESS') WHERE monitorName = 'Fac Ping Dev' SINCE 7 days ago"
},
]
+ thresholds = [
+ {
+ alertSeverity = "WARNING"
+ value = 0.99
},
+ {
+ alertSeverity = "CRITICAL"
+ value = 0.95
},
]
+ vizcoConfiguration = {
+ billboard = {
+ thresholds = [
+ {
+ limit = 0.99
+ type = "warning"
},
+ {
+ limit = 0.95
+ type = "critical"
},
]
}
+ dataFormatters = [
+ {
+ name = "Uptime (%)"
+ precision = "4"
+ type = "decimal"
},
]
}
}
+ title = "Uptime Target: Above 95%"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 5
+ height = 3
+ row = 1
+ width = 6
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM SyntheticCheck SELECT percentage(count(result), WHERE result = 'SUCCESS') WHERE monitorName = 'Fac Ping Dev' SINCE 14 days AGO TIMESERIES"
},
]
}
+ title = "Uptime"
+ visualization = {
+ id = "viz.line"
}
},
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 4
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT count(*) AS 'Transactions' WHERE appName = 'gsa-fac-dev' SINCE 1 hour ago"
},
]
}
+ title = "Transactions (last hour)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 3
+ height = 3
+ row = 4
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "Transactions Per Hour"
+ precision = "0"
+ type = "decimal"
},
]
+ linkedEntityGuids = []
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT rate(count(*), 1 hour) AS 'Transactions (per hour)' WHERE appName ='gsa-fac-dev' SINCE 7 days ago"
},
]
+ thresholds = [
+ {
+ alertSeverity = "WARNING"
+ value = 300
},
+ {
+ alertSeverity = "CRITICAL"
+ value = 100
},
]
+ vizcoConfiguration = {
+ billboard = {
+ thresholds = [
+ {
+ limit = 300
+ type = "warning"
},
+ {
+ limit = 100
+ type = "critical"
},
]
}
+ dataFormatters = [
+ {
+ name = "Transactions Per Hour"
+ precision = "0"
+ type = "decimal"
},
]
}
}
+ title = "Transactions Target: 100 (average per hour/last week)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 5
+ height = 3
+ row = 4
+ width = 6
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT count(*) AS 'Transactions' WHERE appName = 'gsa-fac-dev' SINCE 14 days ago TIMESERIES"
},
]
}
+ title = "Transactions"
+ visualization = {
+ id = "viz.line"
}
},
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 7
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM TransactionError SELECT count(*) AS 'Errors (last hour)' SINCE 1 hour ago"
},
]
}
+ title = "Errors (last hour)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 3
+ height = 3
+ row = 7
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "Success Rate (%)"
+ precision = "4"
+ type = "decimal"
},
]
+ linkedEntityGuids = []
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT 100 - percentage(count(*), WHERE error is true) AS 'Success Rate' WHERE appName ='gsa-fac-dev' SINCE 7 days ago "
},
]
+ thresholds = [
+ {
+ alertSeverity = "WARNING"
+ value = 0.985
},
+ {
+ alertSeverity = "CRITICAL"
+ value = 0.975
},
]
+ vizcoConfiguration = {
+ billboard = {
+ thresholds = [
+ {
+ limit = 0.985
+ type = "warning"
},
+ {
+ limit = 0.975
+ type = "critical"
},
]
}
+ dataFormatters = [
+ {
+ name = "Success Rate (%)"
+ precision = "4"
+ type = "decimal"
},
]
}
}
+ title = "Success Rate Target: Above 97.5%"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 5
+ height = 3
+ row = 7
+ width = 6
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM TransactionError SELECT count(*) AS 'Errors (last hour)' WHERE (appName = 'gsa-fac-dev') SINCE 14 days ago TIMESERIES"
},
]
}
+ title = "Errors"
+ visualization = {
+ id = "viz.line"
}
},
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 10
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "p95"
+ type = "decimal"
},
+ {
+ name = "p50"
+ type = "decimal"
},
+ {
+ name = "Average"
+ type = "decimal"
},
]
+ facet = {
+ showOtherSeries = false
}
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT average(convert(duration, 's', 'ms')) AS 'Average', percentile(convert(duration, 's', 'ms'), 50) AS 'p50', percentile(convert(duration, 's', 'ms'), 95) AS 'p95' WHERE appName = 'gsa-fac-dev' SINCE 1 hour ago"
},
]
+ platformOptions = {
+ ignoreTimeRange = false
}
}
+ title = "Latency (last hour)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 3
+ height = 3
+ row = 10
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "Average Latency (ms)"
+ precision = "0"
+ type = "decimal"
},
]
+ linkedEntityGuids = []
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT average(convert(duration, 's', 'ms')) AS 'Average' WHERE (appName = 'gsa-fac-dev') SINCE 7 days ago "
},
]
+ thresholds = [
+ {
+ alertSeverity = "CRITICAL"
+ value = 1000
},
+ {
+ alertSeverity = "WARNING"
+ value = 800
},
]
+ vizcoConfiguration = {
+ billboard = {
+ thresholds = [
+ {
+ limit = 1000
+ type = "critical"
},
+ {
+ limit = 800
+ type = "warning"
},
]
}
+ dataFormatters = [
+ {
+ name = "Average Latency (ms)"
+ precision = "0"
+ type = "decimal"
},
]
}
}
+ title = "Average Latency Target (last week): 1000ms"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 5
+ height = 3
+ row = 10
+ width = 6
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ facet = {
+ showOtherSeries = false
}
+ legend = {
+ enabled = true
}
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT average(duration) AS 'Average', percentile(duration, 50) AS 'p50', percentile(duration, 95) AS 'p95' WHERE appName = 'gsa-fac-dev' SINCE 14 days ago TIMESERIES"
},
]
+ platformOptions = {
+ ignoreTimeRange = false
}
+ thresholds = {
+ isLabelVisible = true
}
+ units = {
+ unit = "MS"
}
+ yAxisLeft = {
+ zero = true
}
+ yAxisRight = {
+ zero = true
}
}
+ title = "Latency"
+ visualization = {
+ id = "viz.line"
}
},
+ {
+ layout = {
+ column = 1
+ height = 1
+ row = 13
+ width = 10
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ text = "# Host Metrics"
}
+ title = ""
+ visualization = {
+ id = "viz.markdown"
}
},
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 14
+ width = 5
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Metric SELECT average(apm.service.cpu.usertime.utilization) * 100 as cpuUsage WHERE appName = 'gsa-fac-dev' FACET `host` LIMIT 20 SINCE 7 days ago TIMESERIES"
},
]
}
+ title = "Per-host CPU usage"
+ visualization = {
+ id = "viz.line"
}
},
+ {
+ layout = {
+ column = 6
+ height = 3
+ row = 14
+ width = 5
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "SELECT (average(apm.service.memory.physical) * rate(count(apm.service.instance.count), 1 minute)) / 1000 AS memoryUsage FROM Metric WHERE appName = 'gsa-fac-dev' FACET `host` LIMIT 20 SINCE 7 days ago TIMESERIES"
},
]
}
+ title = "Per-host memory usage"
+ visualization = {
+ id = "viz.line"
}
},
]
},
+ {
+ description = null
+ name = "Endpoint Details"
+ widgets = [
+ {
+ layout = {
+ column = 1
+ height = 1
+ row = 1
+ width = 10
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ text = "# UEI Validation"
}
+ title = ""
+ visualization = {
+ id = "viz.markdown"
}
},
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 2
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT count(*) AS 'Transactions (last hour)' WHERE (appName = 'gsa-fac-dev') AND (request.uri LIKE '/api/sac/ueivalidation%') SINCE 1 hour ago"
},
]
}
+ title = "Transactions (last hour)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ description = "Average number of transactions per hour over the past week"
+ layout = {
+ column = 3
+ height = 3
+ row = 2
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "Transactions Per Hour"
+ precision = "0"
+ type = "decimal"
},
]
+ linkedEntityGuids = []
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT rate(count(*), 1 hour) AS 'Transactions (per hour)' WHERE appName ='gsa-fac-dev' AND request.uri LIKE '/api/sac/ueivalidation%' SINCE 7 days ago"
},
]
+ thresholds = [
+ {
+ alertSeverity = "WARNING"
+ value = 5
},
+ {
+ alertSeverity = "CRITICAL"
+ value = 1
},
]
+ vizcoConfiguration = {
+ billboard = {
+ thresholds = [
+ {
+ limit = 5
+ type = "warning"
},
+ {
+ limit = 1
+ type = "critical"
},
]
}
+ dataFormatters = [
+ {
+ name = "Transactions Per Hour"
+ precision = "0"
+ type = "decimal"
},
]
}
}
+ title = "Transactions Target: 1 (per hour/past week)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 5
+ height = 3
+ row = 2
+ width = 6
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountId = 3919076
+ query = "FROM Transaction SELECT count(*) AS 'Transactions' WHERE appName = 'gsa-fac-dev' and request.uri LIKE '/api/sac/ueivalidation%' SINCE 14 days ago TIMESERIES"
},
]
}
+ title = "Transactions"
+ visualization = {
+ id = "viz.line"
}
},
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 5
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM TransactionError SELECT count(*) AS 'Errors (last hour)' WHERE (appName = 'gsa-fac-dev') AND (request.uri LIKE '/api/sac/ueivalidation%') SINCE 1 hour ago"
},
]
}
+ title = "Errors (last hour)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 3
+ height = 3
+ row = 5
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "Success Rate (%)"
+ precision = "4"
+ type = "decimal"
},
]
+ linkedEntityGuids = []
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT 100 - percentage(count(*), WHERE error is true) AS 'Success Rate' WHERE appName ='gsa-fac-dev' AND request.uri LIKE '/api/sac/ueivalidation%' SINCE 7 days ago "
},
]
+ thresholds = [
+ {
+ alertSeverity = "WARNING"
+ value = 0.985
},
+ {
+ alertSeverity = "CRITICAL"
+ value = 0.975
},
]
+ vizcoConfiguration = {
+ billboard = {
+ thresholds = [
+ {
+ limit = 0.985
+ type = "warning"
},
+ {
+ limit = 0.975
+ type = "critical"
},
]
}
+ dataFormatters = [
+ {
+ name = "Success Rate (%)"
+ precision = "4"
+ type = "decimal"
},
]
}
}
+ title = "Success Rate Target: Above 97.5%"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 5
+ height = 3
+ row = 5
+ width = 6
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ nrqlQueries = [
+ {
+ accountId = 3919076
+ query = "FROM TransactionError SELECT count(*) AS 'Errors (last hour)' WHERE (appName = 'gsa-fac-dev') AND (request.uri LIKE '/api/sac/ueivalidation%') SINCE 14 days ago TIMESERIES"
},
]
}
+ title = "Errors"
+ visualization = {
+ id = "viz.line"
}
},
+ {
+ layout = {
+ column = 1
+ height = 3
+ row = 8
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "p95"
+ type = "decimal"
},
+ {
+ name = "p50"
+ type = "decimal"
},
+ {
+ name = "Average"
+ type = "decimal"
},
]
+ facet = {
+ showOtherSeries = false
}
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT average(convert(duration, 's', 'ms')) AS 'Average', percentile(convert(duration, 's', 'ms'), 50) AS 'p50', percentile(convert(duration, 's', 'ms'), 95) AS 'p95' WHERE (appName = 'gsa-fac-dev') AND (request.uri LIKE '/api/sac/ueivalidation%') SINCE 1 hour ago"
},
]
+ platformOptions = {
+ ignoreTimeRange = false
}
}
+ title = "Latency (last hour)"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 3
+ height = 3
+ row = 8
+ width = 2
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ dataFormatters = [
+ {
+ name = "Average Latency (ms)"
+ precision = "0"
+ type = "decimal"
},
]
+ linkedEntityGuids = []
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
+ query = "FROM Transaction SELECT average(convert(duration, 's', 'ms')) AS 'Average' WHERE (appName = 'gsa-fac-dev') AND (request.uri LIKE '/api/sac/ueivalidation%') SINCE 7 days ago "
},
]
+ thresholds = [
+ {
+ alertSeverity = "CRITICAL"
+ value = 1000
},
+ {
+ alertSeverity = "WARNING"
+ value = 800
},
]
+ vizcoConfiguration = {
+ billboard = {
+ thresholds = [
+ {
+ limit = 1000
+ type = "critical"
},
+ {
+ limit = 800
+ type = "warning"
},
]
}
+ dataFormatters = [
+ {
+ name = "Average Latency (ms)"
+ precision = "0"
+ type = "decimal"
},
]
}
}
+ title = "Average Latency Target (last week): 1000ms"
+ visualization = {
+ id = "viz.billboard"
}
},
+ {
+ layout = {
+ column = 5
+ height = 3
+ row = 8
+ width = 6
}
+ linkedEntityGuids = null
+ rawConfiguration = {
+ facet = {
+ showOtherSeries = false
}
+ legend = {
+ enabled = true
}
+ nrqlQueries = [
+ {
+ accountIds = [
+ 3919076,
]
Plan is too large to fit in a PR comment. See the full plan in the workflow log.📝 Plan generated in Pull Request Checks #4188 |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
The goal of this PR is to add additional dashboarding into new relic. This will generate a new dashboard called "FAC Monitoring - ". It contains a high level dashboard for a quick overview of FAC as a whole, then the second page has details for individual endpoints.
Note: This is for monitoring only, there is no alarming configured.
Issue
Testing
I ran this in preview and overrode the environment to production. You can see the dashboard it created by looking in new relic for "FAC Monitoring - Production"
What's Next?
Add more endpoints. Consider breaking them up across multiple pages (e.g. Audit, Dissemination, Search, ect.)
Consider adding additional alarming using the same endpoint configurations.
Consider making the endpoint configuration environment specific (e.g. dev will likely always be "red" for transactions)
PR Checklist: Submitter
maininto your branch shortly before creating the PR. (You should also be mergingmaininto your branch regularly during development.)git status | grep migrations. If there are any results, you probably need to add them to the branch for the PR. Your PR should have only one new migration file for each of the component apps, except in rare circumstances; you may need to delete some and re-runpython manage.py makemigrationsto reduce the number to one. (Also, unless in exceptional circumstances, your PR should not delete any migration files.)PR Checklist: Reviewer
make docker-clean; make docker-first-run && docker compose up; then rundocker compose exec web /bin/bash -c "python manage.py test"The larger the PR, the stricter we should be about these points.
Pre Merge Checklist: Merger
-/+ resource "null_resource" "cors_header"should be destroying and recreating its self and~ resource "cloudfoundry_app" "clamav_api"might be updating itssha256for thefac-file-scannerandfac-av-${ENV}by default.main.