Monitoring and Alerting Workflow

✅ Fresh -- Updated March 2026 from official Google Cloud documentation.

Overview

Set up comprehensive monitoring for your Google Cloud API usage, including dashboards, alerts, and incident response.

Monitoring Architecture

Step 1: API Dashboard

The API Dashboard is available in the Cloud Console with zero configuration:

https://console.cloud.google.com/apis/dashboard?project=PROJECT_ID

It shows:

Traffic: Total requests over time
Errors: Error rate by response code
Latency: Request latency percentiles (p50, p95, p99)

Step 2: Set Up Cloud Monitoring

Enable the Monitoring API

bash

gcloud services enable monitoring.googleapis.com

Key Metrics to Monitor

Metric	What It Tells You
`api/request_count`	Total API requests (broken down by method, response code)
`api/request_latencies`	How long requests take (distribution)
`api/request_sizes`	Request payload sizes
`api/response_sizes`	Response payload sizes

Create a Custom Dashboard

bash

# Using gcloud (or create in Console)
gcloud monitoring dashboards create \
  --config-from-file=dashboard.json

Example dashboard JSON structure:

json

{
  "displayName": "API Health Dashboard",
  "gridLayout": {
    "widgets": [
      {
        "title": "API Request Rate",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "metric.type=\"serviceruntime.googleapis.com/api/request_count\"",
                "aggregation": {
                  "perSeriesAligner": "ALIGN_RATE",
                  "alignmentPeriod": "60s"
                }
              }
            }
          }]
        }
      }
    ]
  }
}

Step 3: Configure Alerts

Alert for High Error Rate

Create an alert policy via gcloud:

bash

gcloud monitoring policies create \
  --notification-channels="projects/PROJECT/notificationChannels/CHANNEL_ID" \
  --display-name="High API Error Rate" \
  --condition-display-name="Error rate > 5%" \
  --condition-filter='metric.type="serviceruntime.googleapis.com/api/request_count" AND resource.type="consumed_api" AND metric.labels.response_code_class="4xx" OR metric.labels.response_code_class="5xx"' \
  --condition-threshold-value=0.05 \
  --condition-threshold-comparison=COMPARISON_GT \
  --condition-threshold-duration=300s

Common Alert Policies

Alert	Condition	Threshold
High error rate	Error count / total count	> 5% for 5 min
High latency (p99)	request_latencies p99	> 2000ms for 10 min
Quota near limit	allocation_usage / limit	> 80%
Traffic drop	request_count rate	< 50% of baseline for 15 min
Zero traffic	request_count	= 0 for 30 min (during business hours)

Step 4: Set Up Log-Based Monitoring

Enable Audit Logs

bash

# Enable Data Access audit logs (Admin Activity is on by default)
gcloud projects get-iam-policy PROJECT_ID --format=json > policy.json
# Edit policy.json to add auditConfigs, then:
gcloud projects set-iam-policy PROJECT_ID policy.json

Query Logs for API Errors

bash

gcloud logging read \
  'resource.type="consumed_api" AND severity>=ERROR' \
  --limit=50 \
  --format="table(timestamp, resource.labels.service, severity, textPayload)"

Create a Log-Based Metric

Track specific error patterns:

bash

gcloud logging metrics create api_permission_denied \
  --description="Count of PERMISSION_DENIED errors" \
  --log-filter='resource.type="consumed_api" AND jsonPayload.status.code=7'

Step 5: Quota Monitoring

Check quota usage:

bash

gcloud services list --enabled --format="table(name)"
# Then check quota in Console:
# https://console.cloud.google.com/iam-admin/quotas?project=PROJECT_ID

Step 6: Incident Response Flow

Verification Checklist

[ ] API Dashboard is accessible and showing data
[ ] Custom monitoring dashboard is created
[ ] Alert policies are configured for error rate, latency, and quota
[ ] Notification channels are set up (email, Slack, PagerDuty)
[ ] Audit logs are enabled for sensitive APIs
[ ] Incident response runbook is documented
[ ] Team knows how to acknowledge and resolve alerts

Monitoring and Alerting Workflow ​

Overview ​

Monitoring Architecture ​

Step 1: API Dashboard ​

Step 2: Set Up Cloud Monitoring ​

Enable the Monitoring API ​

Key Metrics to Monitor ​

Create a Custom Dashboard ​

Step 3: Configure Alerts ​

Alert for High Error Rate ​

Common Alert Policies ​

Step 4: Set Up Log-Based Monitoring ​

Enable Audit Logs ​

Query Logs for API Errors ​

Create a Log-Based Metric ​

Step 5: Quota Monitoring ​

Step 6: Incident Response Flow ​

Verification Checklist ​

See Also ​

Monitoring and Alerting Workflow

Overview

Monitoring Architecture

Step 1: API Dashboard

Step 2: Set Up Cloud Monitoring

Enable the Monitoring API

Key Metrics to Monitor

Create a Custom Dashboard

Step 3: Configure Alerts

Alert for High Error Rate

Common Alert Policies

Step 4: Set Up Log-Based Monitoring

Enable Audit Logs

Query Logs for API Errors

Create a Log-Based Metric

Step 5: Quota Monitoring

Step 6: Incident Response Flow

Verification Checklist

See Also