Manage Secrets

This guide covers managing sensitive data like API keys, database passwords, and tokens using AWS Secrets Manager.

Overview

MenoTime stores all secrets in AWS Secrets Manager instead of environment variables or configuration files. This provides:

• ✅ Centralized secret management
• ✅ Encryption at rest
• ✅ Audit logging of access
• ✅ Automatic rotation capabilities
• ✅ Fine-grained IAM access control

Secrets are automatically loaded into the application at startup.

How Secrets are Loaded

When a task starts in ECS Fargate, AWS automatically populates environment variables from Secrets Manager:

# app/config.py
import os
from dotenv import load_dotenv

# For local dev, load from .env.local
load_dotenv('.env.local')

# For production, AWS ECS populates environment variables from Secrets Manager
DATABASE_URL = os.getenv('DATABASE_URL')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
JWT_SECRET_KEY = os.getenv('JWT_SECRET_KEY')

The ECS task definition includes a secretsManager block:

{
  "containerDefinitions": [
    {
      "name": "menotime-api",
      "secrets": [
        {
          "name": "DATABASE_URL",
          "valueFrom": "arn:aws:secretsmanager:us-west-1:ACCOUNT_ID:secret:menotime/database/url"
        },
        {
          "name": "JWT_SECRET_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-west-1:ACCOUNT_ID:secret:menotime/jwt/secret"
        }
      ]
    }
  ]
}

Viewing Existing Secrets

List all secrets:

aws secretsmanager list-secrets \
  --region us-west-1 \
  --query 'SecretList[?contains(Name, `menotime`)].Name' \
  --output table

View a specific secret:

aws secretsmanager get-secret-value \
  --secret-id menotime/database/url \
  --region us-west-1

The output includes:

{
  "ARN": "arn:aws:secretsmanager:us-west-1:ACCOUNT_ID:secret:menotime/database/url",
  "Name": "menotime/database/url",
  "VersionId": "abc123def456",
  "SecretString": "postgresql://user:password@host:5432/db"
}

List Secrets by Environment

# Development secrets
aws secretsmanager list-secrets --filters "Key=name,Values=menotime/dev/" --region us-west-1

# Staging secrets
aws secretsmanager list-secrets --filters "Key=name,Values=menotime/staging/" --region us-west-1

# Production secrets
aws secretsmanager list-secrets --filters "Key=name,Values=menotime/prod/" --region us-west-1

Adding a New Secret

Step 1: Create the Secret in AWS

aws secretsmanager create-secret \
  --name menotime/prod/sendgrid-api-key \
  --description "SendGrid API key for production email service" \
  --secret-string "sg_live_xxxxxxxxxxxxxxxxxxxxx" \
  --region us-west-1

For more complex secrets, use a JSON format:

aws secretsmanager create-secret \
  --name menotime/prod/oauth-credentials \
  --description "OAuth provider credentials" \
  --secret-string '{
    "client_id": "12345abcde",
    "client_secret": "secret-key-here",
    "provider": "google"
  }' \
  --region us-west-1

Step 2: Update the ECS Task Definition

Add the secret to the task definition:

aws ecs describe-task-definition \
  --task-definition menotime-api-prod \
  --region us-west-1 > task-def.json

# Edit task-def.json to add the secret

Edit the JSON file and add to the secrets array:

{
  "name": "SENDGRID_API_KEY",
  "valueFrom": "arn:aws:secretsmanager:us-west-1:ACCOUNT_ID:secret:menotime/prod/sendgrid-api-key"
}

Register the updated task definition:

aws ecs register-task-definition \
  --cli-input-json file://task-def.json \
  --region us-west-1

Step 3: Update the Application

Use the secret in your code:

# app/config.py
import os
import json

# For simple string secrets
SENDGRID_API_KEY = os.getenv('SENDGRID_API_KEY')

# For JSON secrets, parse them
OAUTH_CREDS_RAW = os.getenv('OAUTH_CREDENTIALS', '{}')
OAUTH_CREDS = json.loads(OAUTH_CREDS_RAW)

CLIENT_ID = OAUTH_CREDS.get('client_id')
CLIENT_SECRET = OAUTH_CREDS.get('client_secret')

Step 4: Test Locally

For local testing, add the secret to .env.local:

SENDGRID_API_KEY=sg_test_xxxxxxxxxxxxxxxxxxxxx
OAUTH_CREDENTIALS={"client_id": "12345abcde", "client_secret": "secret", "provider": "google"}

Test your code:

python -c "from app.config import SENDGRID_API_KEY; print(SENDGRID_API_KEY)"

Step 5: Deploy

Deploy to staging first:

# Push changes to develop
git add app/config.py
git commit -m "feat: add SendGrid API integration"
git push origin feature/sendgrid-integration

# Create PR, merge, and deploy to staging
# Test in staging environment

After staging verification, deploy to production following the production deployment guide.

Rotating Secrets

Rotate secrets periodically for security. AWS can automate this process.

Automatic Rotation

Enable automatic rotation for a secret:

aws secretsmanager rotate-secret \
  --secret-id menotime/prod/database/password \
  --rotation-rules AutomaticallyAfterDays=30 \
  --region us-west-1

This requires a Lambda function to handle the rotation. AWS provides templates for common services like RDS.

Manual Rotation

For secrets that can't be auto-rotated:

# Get the current secret
aws secretsmanager get-secret-value \
  --secret-id menotime/prod/jwt-secret \
  --region us-west-1

# Update with new value
aws secretsmanager update-secret \
  --secret-id menotime/prod/jwt-secret \
  --secret-string "new-secret-value-here" \
  --region us-west-1

After Rotating

1. Update all references in code if needed
2. Deploy the application
3. Verify in staging first
4. Deploy to production
5. Monitor logs for any issues

Accessing Secrets in ECS Tasks

View Secret in Running Task

# Get a running task ID
TASK_ID=$(aws ecs list-tasks \
  --cluster production \
  --service-name menotime-api \
  --region us-west-1 \
  --query 'taskArns[0]' \
  --output text | awk -F'/' '{print $NF}')

# Exec into the container
aws ecs execute-command \
  --cluster production \
  --task $TASK_ID \
  --container menotime-api \
  --interactive \
  --command "/bin/bash" \
  --region us-west-1

# Inside the container, view the secret (it's loaded as env var)
echo $DATABASE_URL

Using AWS CLI Inside Container

# Inside the ECS task container
aws secretsmanager get-secret-value \
  --secret-id menotime/prod/database/password \
  --region us-west-1

This requires the task execution role to have Secrets Manager permissions (usually already configured).

Accessing Secrets for Local Development

Option 1: Use Local .env File

Create .env.local with local secret values:

DATABASE_URL=postgresql://user:password@localhost:5432/menotime
JWT_SECRET_KEY=dev-secret-key-change-in-production
SENDGRID_API_KEY=sg_test_xxx_not_real_xxx

Load in your app:

from dotenv import load_dotenv
load_dotenv('.env.local')

Option 2: Fetch from AWS Secrets Manager

For development against staging/production databases:

# app/secrets.py
import boto3
import json
import os
from functools import lru_cache

secretsmanager = boto3.client('secretsmanager', region_name='us-west-1')

@lru_cache(maxsize=100)
def get_secret(secret_name: str) -> str:
    """Fetch and cache a secret from AWS Secrets Manager."""
    try:
        response = secretsmanager.get_secret_value(secret_id=secret_name)
        return response['SecretString']
    except Exception as e:
        print(f"Error fetching secret {secret_name}: {e}")
        raise

# Usage
def get_database_url():
    # Try local .env first
    url = os.getenv('DATABASE_URL')
    if url:
        return url

    # Fall back to Secrets Manager
    return get_secret('menotime/prod/database/url')

DATABASE_URL = get_database_url()

Secret Naming Conventions

Follow a consistent naming convention:

menotime/{environment}/{service}/{secret-type}

Examples:
- menotime/dev/database/url
- menotime/staging/database/password
- menotime/prod/database/url
- menotime/prod/jwt/secret
- menotime/prod/email/sendgrid-api-key
- menotime/prod/auth/oauth-google-credentials

Security Best Practices

✅ DO

• ✅ Store all secrets in AWS Secrets Manager
• ✅ Rotate secrets every 30-90 days
• ✅ Use different secrets for each environment
• ✅ Restrict IAM access to secrets
• ✅ Enable audit logging
• ✅ Use encryption at rest (default in Secrets Manager)
• ✅ Keep .env.local out of git (add to .gitignore)
• ✅ Use strong, random values for secrets
• ✅ Monitor secret access in CloudTrail

❌ DON'T

• ❌ Never commit secrets to git
• ❌ Never hardcode secrets in code
• ❌ Never share secrets in Slack, email, or tickets
• ❌ Never log secrets (they're redacted in CloudWatch)
• ❌ Never use the same secret across environments
• ❌ Never store secrets in Docker images
• ❌ Never send secrets in query parameters
• ❌ Never expose secrets in error messages
• ❌ Never grant broad Secrets Manager access

Example of Secret Leakage to Avoid

# ❌ BAD - secret in code
JWT_SECRET = "my-super-secret-key-12345"

# ❌ BAD - secret in logs
logger.info(f"Connecting with password: {password}")

# ❌ BAD - secret in environment variable exposed
print(os.environ)

# ✅ GOOD - load from Secrets Manager
from app.config import JWT_SECRET  # Loaded from environment at startup

# ✅ GOOD - log without sensitive data
logger.info("Connecting to database")

IAM Permissions

Ensure your ECS task execution role has these permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-1:ACCOUNT_ID:secret:menotime/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:kms:us-west-1:ACCOUNT_ID:key/*"
      ],
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.us-west-1.amazonaws.com"
        }
      }
    }
  ]
}

Check the task execution role:

aws iam get-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-name SecretsManagerAccess

Troubleshooting

Secret Not Found Error

# Check if secret exists
aws secretsmanager describe-secret \
  --secret-id menotime/prod/jwt-secret \
  --region us-west-1

# Check task execution role has permissions
aws iam list-attached-role-policies --role-name ecsTaskExecutionRole

Task Fails to Start

# Check CloudWatch logs
aws logs tail /ecs/menotime-api-prod --follow --region us-west-1

# Look for: "SecretNotFoundException" or "AccessDeniedException"

Can't Access Secret Locally

# Verify AWS credentials are configured
aws sts get-caller-identity

# Check you're using the right region
export AWS_REGION=us-west-1

# Try fetching the secret directly
aws secretsmanager get-secret-value --secret-id menotime/dev/database/url

Audit and Monitoring

View Secret Access Logs

# List CloudTrail events for a secret
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=menotime/prod/database/password \
  --region us-west-1 \
  --max-results 10

Set Up Alerts

Create CloudWatch alarms for suspicious secret access:

aws cloudwatch put-metric-alarm \
  --alarm-name "SecretAccessAnomaly" \
  --alarm-description "Alert on unusual secret access" \
  --metric-name UnauthorizedOperationCount \
  --namespace AWS/SecretsManager \
  --statistic Sum \
  --period 300 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --region us-west-1

Related Guides

• Local Development Setup
• Deploy to Staging
• Deploy to Production
• Troubleshooting