Best Practices for Implementing Configuration Class in Python

Managing configurations for development (dev), user acceptance testing (uat), and production (prod) environments is a critical part of building scalable and secure Python applications. The configuration needs to be environment-specific, secure, and easy to maintain. Pydantic's BaseSettings provides a robust way to handle configurations with type validation, .env file support, and seamless integration with secret management systems like AWS Secrets Manager.

In this blog, we will cover:

1. Benefits of using Pydantic.BaseSettings for configuration management.
2. Best practices for implementing a unified configuration class for different environments.
3. Secure integration with AWS Secrets Manager for sensitive fields.
4. A complete working example with environment-based switching.

Why Use Pydantic’s BaseSettings for Configuration?

Pydantic.BaseSettings makes configuration management:

• Easy: Reads values from environment variables, .env files, or secret management systems.
• Secure: Supports sensitive fields (e.g., API keys, passwords) using SecretStr.
• Type-Safe: Ensures that configuration values have the correct types (e.g., integers for ports, URLs for database URIs).
• Maintainable: A single class can handle different environments, reducing boilerplate.

1. Configuration Class Using Pydantic’s BaseSettings

We’ll use a single configuration class that dynamically switches its values based on the environment (dev, uat, prod). The class will:

• Define default values for shared configurations.
• Override environment-specific fields using environment variables or secrets.
• Use SecretStr for secure fields.
• Fetch sensitive values from AWS Secrets Manager for production.

2. Code Example: Unified Config Class

Step 1: Install Dependencies

First, ensure pydantic and boto3 (for AWS Secrets Manager) are installed:

pip install pydantic boto3 python-dotenv

Step 2: Implement the Configuration Class

Here is a Python configuration class that handles dev, uat, and prod environments:

import os
from typing import Optional
from pydantic import BaseSettings, SecretStr
import boto3
from botocore.exceptions import ClientError
import json

class AppConfig(BaseSettings):
    """
    Unified configuration class using Pydantic BaseSettings.
    """
    # Common settings
    ENV: str = "dev"  # Default to "dev" environment
    DEBUG: bool = False
    DATABASE_URI: str
    SECRET_KEY: SecretStr = SecretStr("default-secret-key")
    AWS_SECRET_NAME: Optional[str] = None  # Name of the AWS secret for production
    AWS_REGION: str = "us-east-1"  # Default AWS region for Secrets Manager

    # Method to fetch secrets from AWS Secrets Manager
    def load_aws_secrets(self):
        if self.AWS_SECRET_NAME:
            try:
                client = boto3.client("secretsmanager", region_name=self.AWS_REGION)
                secret_response = client.get_secret_value(SecretId=self.AWS_SECRET_NAME)
                secrets = json.loads(secret_response.get("SecretString", "{}"))

                # Override sensitive fields from AWS Secrets Manager
                if "DATABASE_PASSWORD" in secrets:
                    db_uri_parts = self.DATABASE_URI.split("@")
                    user_info, db_host = db_uri_parts[0], db_uri_parts[1]
                    user_info = f"{secrets.get('DB_USER', 'user')}:{secrets.get('DATABASE_PASSWORD', '')}"
                    self.DATABASE_URI = f"{user_info}@{db_host}"

                if "SECRET_KEY" in secrets:
                    self.SECRET_KEY = SecretStr(secrets["SECRET_KEY"])

                print("Secrets loaded from AWS Secrets Manager.")
            except ClientError as e:
                print(f"Error fetching secrets from AWS: {e}")

    class Config:
        env_file = ".env"  # Load local environment variables from a .env file (for development)

3. Environment-Specific Configuration Example

The configuration values for different environments (dev, uat, prod) can be set using environment variables.

Sample .env File for dev Environment:

ENV=dev
DEBUG=True
DATABASE_URI=postgresql://dev_user:password@localhost/dev_db

Environment Variables for prod (without .env):

export ENV=prod
export DATABASE_URI=postgresql://prod_user:password@prod-db-server/prod_db
export AWS_SECRET_NAME=myapp-prod-secret
export AWS_REGION=us-east-1

4. Full Example Usage

Here’s an example script that loads the configuration dynamically based on the ENV variable:

def main():
    # Instantiate configuration
    config = AppConfig()

    print(f"Running in {config.ENV.upper()} mode")
    print(f"Database URI (before secrets): {config.DATABASE_URI}")
    print(f"Debug Mode: {config.DEBUG}")

    # Load AWS secrets for production
    if config.ENV == "prod":
        config.load_aws_secrets()

    print(f"Database URI (after secrets): {config.DATABASE_URI}")
    print(f"Secret Key: {config.SECRET_KEY.get_secret_value()}")

if __name__ == "__main__":
    main()

Explanation of the Code:

Instantiating the Config Class:

• The configuration is initialized with values from the environment.
• If ENV is prod, load_aws_secrets() is called to fetch and override sensitive values from AWS Secrets Manager.

Database URI Update:

• The AWS secret is expected to return sensitive data like DB_USER, DATABASE_PASSWORD, and SECRET_KEY.

Secrets Manager Integration:

• If secrets are not available or there’s an error, the application gracefully prints an error message.

5. Example AWS Secrets Manager JSON Secret:

In AWS Secrets Manager, the secret can be stored as JSON:

{
  "DB_USER": "prod_user",
  "DATABASE_PASSWORD": "prod_password_secure",
  "SECRET_KEY": "super_secure_key_for_app"
}

6. Running the Application:

Development Mode:

export ENV=dev
python app.py

Production Mode:

export ENV=prod
export AWS_SECRET_NAME=myapp-prod-secret
export AWS_REGION=us-east-1
python app.py

Sample output:

Running in PROD mode
Database URI (before secrets): postgresql://prod_user:password@prod-db-server/prod_db
Secrets loaded from AWS Secrets Manager.
Database URI (after secrets): postgresql://prod_user:prod_password_secure@prod-db-server/prod_db
Secret Key: super_secure_key_for_app

Best Practices for Pydantic Configuration with Secrets:

Avoid Hardcoding Sensitive Information:

• Use .env files or AWS Secrets Manager to avoid hardcoding sensitive values in the code.

Enable AWS Secret Rotation:

• AWS Secrets Manager supports automatic secret rotation. Enable this for database credentials to improve security.

Environment-Specific .env Files:

• Maintain separate .env files for dev, uat, and prod, and switch using the ENV variable.

Use SecretStr for Secure Fields:

• For fields like SECRET_KEY, use pydantic.SecretStr to ensure values are not accidentally exposed in logs or printed output.

Cache AWS Secrets Locally:

• To avoid repeated API calls to AWS, you can use lru_cache or store the secrets locally during the lifetime of the application.

Conclusion

Using Pydantic.BaseSettings for configuration provides type-safe, environment-based settings with minimal boilerplate. By integrating with AWS Secrets Manager, you can securely handle sensitive values like DATABASE_PASSWORD and SECRET_KEY for production. This approach improves security, maintainability, and flexibility while simplifying your configuration setup.

For most applications, this unified approach with BaseSettings is a best practice that scales across dev, uat, and prod environments.