Skip to main content
Data Storage Modules 1.3.0Last updated in version 1.3.0

DMS Module

View Source Release Notes

This module creates AWS Database Migration Service (DMS) resources for database migration and replication.

What This Module Creates​

  • DMS replication instance for classic DMS tasks
  • DMS serverless replication config when task_compute_config is set
  • Source and target endpoints
  • Replication tasks or replication configs
  • Required IAM roles and policies
  • CloudWatch log groups
  • Security groups for network access

Supported Engines​

Currently supports:

  • MySQL
  • MariaDB
  • PostgreSQL
  • Aurora MySQL
  • Aurora PostgreSQL
  • Amazon S3

(As source and target)

Migration Types​

The module supports three migration types via migration_type variable:

  • full-load - One-time migration
  • cdc - Ongoing replication only
  • full-load-and-cdc - Full migration plus ongoing replication

Usage​

module "dms" {
  source = "../modules/dms"

  name       = "my-database-migration"
  vpc_id     = "vpc-12345678"
  subnet_ids = ["subnet-abc123", "subnet-def456"]

  # Classic DMS replication instance mode
  instance_type               = "dms.t3.medium"
  instance_allocated_storage  = 100
  task_migration_type         = "full-load-and-cdc"
  task_table_mappings         = file("${path.module}/table_mappings.json")
  task_start_replication_task = false

  source_endpoint_engine_name   = "mysql"
  source_endpoint_server_name   = "source.example.com"
  source_endpoint_port          = 3306
  source_endpoint_username      = var.source_username
  source_endpoint_password      = var.source_password
  source_endpoint_database_name = "app"

  target_endpoint_engine_name   = "postgres"
  target_endpoint_server_name   = "target.example.com"
  target_endpoint_port          = 5432
  target_endpoint_username      = var.target_username
  target_endpoint_password      = var.target_password
  target_endpoint_database_name = "app"
}

If the account-level DMS roles already exist, set create_iam_roles = false and let AWS DMS use the existing dms-vpc-role, dms-cloudwatch-logs-role, and dms-access-for-endpoint roles.

Migration Guide​

This module now manages the classic DMS resources with count, which changes their Terraform addresses from aws_dms_replication_instance.this and aws_dms_replication_task.this to indexed addresses ending in [0].

If you already manage these resources in state, move them before applying to avoid a destroy and recreate:

terraform state mv 'module.dms.aws_dms_replication_instance.this' 'module.dms.aws_dms_replication_instance.this[0]'
terraform state mv 'module.dms.aws_dms_replication_task.this' 'module.dms.aws_dms_replication_task.this[0]'

If your module path is not module.dms, adjust the address prefix accordingly.

Configuration​

  • See the root README for instructions on using Terraform modules.
  • See the variables.tf for all the variables you can set on this module.
  • See the dms-mysql examples folder for instruction on how to setup the module to migrate data from one AWS RDS MySQL instance to another using a classic DMS replication instance.
  • See the dms-aurora examples folder for instructions on how to setup the module to migrate data between Aurora clusters using a classic DMS replication instance.
  • See the dms-serverless example folder for instructions on how to run the module in DMS serverless mode.

Key Variables​

  • name - Name prefix for all DMS resources
  • instance_type - Instance size for classic DMS mode (e.g., dms.t3.medium)
  • instance_allocated_storage - Storage in GB
  • vpc_id - VPC for deployment
  • subnet_ids - Subnets for the replication instance or serverless subnet group
  • task_migration_type - full-load, cdc, or full-load-and-cdc
  • task_compute_config - Serverless compute settings. When set, the module creates aws_dms_replication_config instead of aws_dms_replication_task. If create_subnet_group = false, include replication_subnet_group_id
  • create_iam_roles - Whether this module should create the shared AWS-required DMS IAM roles

Common Issues​

  • Network: Ensure replication instance can reach both databases
  • Storage: Allocate enough for your data volume
  • Primary keys: Required for CDC performance
  • LOB columns: May need special handling
  • IAM roles: AWS DMS expects the shared account-level roles dms-vpc-role, dms-cloudwatch-logs-role, and dms-access-for-endpoint. Set create_iam_roles = false if they already exist in the account

Outputs​

  • replication_instance_arn - ARN of replication instance when classic mode is used
  • endpoints - Source and target endpoints created by the module
  • replication_tasks - Classic DMS replication tasks
  • replication_configs - DMS serverless replication configs

Sample Usage​

main.tf

# ------------------------------------------------------------------------------------------------------
# DEPLOY GRUNTWORK'S DMS MODULE
# ------------------------------------------------------------------------------------------------------

module "dms" {

  source = "git::git@github.com:gruntwork-io/terraform-aws-data-storage.git//modules/dms?ref=v1.3.0"

  # ----------------------------------------------------------------------------------------------------
  # REQUIRED VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # The name used to namespace all resources created by these templates,
  # including the DB instance (e.g. drupaldb). Must be unique for this region.
  # May contain only lowercase alphanumeric characters, hyphens, underscores,
  # periods, and spaces.
  name = <string>

  # The migration type. Can be one of `full-load` | `cdc` | `full-load-and-cdc`.
  task_migration_type = <string>

  # An escaped JSON string that contains the table mappings. For information on
  # table mapping see
  # http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html
  task_table_mappings = <string>

  # ----------------------------------------------------------------------------------------------------
  # OPTIONAL VARIABLES
  # ----------------------------------------------------------------------------------------------------

  # If true, create the account-level IAM roles required by AWS DMS
  # (`dms-vpc-role`, `dms-cloudwatch-logs-role`, and `dms-access-for-endpoint`).
  # Set to false when these shared roles already exist or are managed outside
  # this module.
  create_iam_roles = true

  # If true, create an `aws_dms_replication_instance` for non-serverless
  # replication task mode. This is automatically ignored when
  # `task_compute_config` is set for serverless mode.
  create_replication_instance = true

  # If false, the DMS instance will bind to `instance_subnet_group_id` variable.
  create_subnet_group = true

  # A map of custom tags to apply to the DMS Instance and the Security Group
  # created for it. The key is the tag name and the value is the tag value.
  custom_tags = {}

  # The amount of storage (in gigabytes) to be initially allocated for the
  # replication instance. Min: 5, Max: 6144, Default: 50
  instance_allocated_storage = null

  # Indicates that major version upgrades are allowed
  instance_allow_major_version_upgrade = true

  # Indicates whether the changes should be applied immediately or during the
  # next maintenance window
  instance_apply_immediately = null

  # Indicates that minor engine upgrades will be applied automatically to the
  # replication instance during the maintenance window
  instance_auto_minor_version_upgrade = true

  # The EC2 Availability Zone that the replication instance will be created in
  instance_availability_zone = null

  # The [engine
  # version](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReleaseNotes.html)
  # number of the replication instance
  instance_engine_version = null

  # The replication instance identifier. This parameter is stored as a lowercase
  # string
  instance_id = null

  # The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt
  # the connection parameters
  instance_kms_key_arn = null

  # Specifies if the replication instance is a multi-az deployment. You cannot
  # set the `availability_zone` parameter if the `multi_az` parameter is set to
  # `true`
  instance_multi_az = null

  # The weekly time range during which system maintenance can occur, in
  # Universal Coordinated Time (UTC)
  instance_preferred_maintenance_window = null

  # Specifies the accessibility options for the replication instance
  instance_publicly_accessible = null

  # An existing subnet group to associate with the replication instance
  instance_subnet_group_id = null

  # A map of additional tags to apply to the replication instance
  instance_tags = {}

  # A map of timeouts for replication instance create/update/delete operations
  instance_timeouts = {}

  # The compute and memory capacity of the replication instance as specified by
  # the replication instance class. Required only for non-serverless replication
  # task mode.
  instance_type = null

  # A list of VPC security group IDs to be used with the replication instance
  instance_vpc_security_group_ids = null

  # Name of the endpoint database
  source_endpoint_database_name = null

  # Type of engine for the endpoint.
  source_endpoint_engine_name = "mysql"

  # Additional attributes associated with the connection. For available
  # attributes for a source Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
  # available attributes for a target Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
  source_endpoint_extra_connection_attributes = null

  # ARN for the KMS key that will be used to encrypt the connection parameters.
  # If you do not specify a value for kms_key_arn, then AWS DMS will use your
  # default encryption key. AWS KMS creates the default encryption key for your
  # AWS account. Your AWS account has a different default encryption key for
  # each AWS region
  source_endpoint_kms_key_arn = null

  # Password to be used to login to the endpoint database.
  source_endpoint_password = null

  #  Port used by the endpoint database.
  source_endpoint_port = null

  # (Required if using non-s3 source) Host name of the server.
  source_endpoint_server_name = null

  # SSL mode to use for the connection. Valid values are none, require,
  # verify-ca, verify-full
  source_endpoint_ssl_mode = "none"

  # User name to be used to login to the endpoint database.
  source_endpoint_username = null

  # (Required if using s3) S3 bucket name.
  source_s3_bucket_name = null

  # (Required if using s3 for CDC; otherwise, Optional) Folder path of CDC
  # files. If cdc_path is set, AWS DMS reads CDC files from this path and
  # replicates the data changes to the target endpoint. Supported in AWS DMS
  # versions 3.4.2 and later.
  source_s3_cdc_path = null

  # (Required if using s3) JSON document that describes how AWS DMS should
  # interpret the data.
  source_s3_external_table_definition = null

  # ARN of the IAM role with permissions to the S3 Bucket. Default one will be
  # created if not specified
  source_s3_service_access_role_arn = null

  # The description of the aws_dms_replication_subnet_group that is created.
  # Defaults to 'Subnet group for the var.name DB' if not specified.
  subnet_group_description = null

  # The name of the aws_dms_replication_subnet_group that is created, or an
  # existing one to use if create_subnet_group is false. Defaults to var.name if
  # not specified.
  subnet_group_name = null

  # A list of subnet ids where the Replication Instance should be deployed. In
  # the standard Gruntwork VPC setup, these should be the private persistence
  # subnet ids. This is ignored if create_subnet_group=false.
  subnet_ids = []

  # Name of the endpoint database
  target_endpoint_database_name = null

  # Type of engine for the endpoint. 
  target_endpoint_engine_name = "mysql"

  # Additional attributes associated with the connection. For available
  # attributes for a source Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For
  # available attributes for a target Endpoint, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.
  target_endpoint_extra_connection_attributes = null

  # ARN for the KMS key that will be used to encrypt the connection parameters.
  # If you do not specify a value for kms_key_arn, then AWS DMS will use your
  # default encryption key. AWS KMS creates the default encryption key for your
  # AWS account. Your AWS account has a different default encryption key for
  # each AWS region
  target_endpoint_kms_key_arn = null

  # Password to be used to login to the endpoint database.
  target_endpoint_password = null

  #  Port used by the endpoint database.
  target_endpoint_port = null

  # (Required if using non-s3 target) Host name of the server.
  target_endpoint_server_name = null

  # SSL mode to use for the connection. Valid values are none, require,
  # verify-ca, verify-full
  target_endpoint_ssl_mode = "none"

  # User name to be used to login to the endpoint database.
  target_endpoint_username = null

  # (Required if using s3) S3 bucket name.
  target_s3_bucket_name = null

  # Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from
  # this path and replicates the data changes to the target endpoint. Supported
  # in AWS DMS versions 3.4.2 and later.
  target_s3_cdc_path = null

  # JSON document that describes how AWS DMS should interpret the data.
  target_s3_external_table_definition = null

  # ARN of the IAM role with permissions to the S3 Bucket. Default one will be
  # created if not specified
  target_s3_service_access_role_arn = null

  # (Conflicts with task_cdc_start_time) Indicates when you want a change data
  # capture (CDC) operation to start. The value can be in date, checkpoint, or
  # LSN/SCN format depending on the source engine. For more information, see
  # https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.StartPoint.Native.
  task_cdc_start_position = null

  # (Conflicts with task_cdc_start_position) The Unix timestamp integer for the
  # start of the Change Data Capture (CDC) operation.
  task_cdc_start_time = null

  # Optional compute configuration for serverless replication mode. When set,
  # the module creates `aws_dms_replication_config` instead of
  # `aws_dms_replication_task`.
  task_compute_config = null

  # An escaped JSON string that contains the task settings. For a complete list
  # of task settings, see
  # http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html.
  task_settings = null

  # Whether to run or stop the replication task.
  task_start_replication_task = false

  # The id of the VPC in which this Replication Instance should be deployed.
  vpc_id = null

}


Reference​

Required​

namestringrequired

The name used to namespace all resources created by these templates, including the DB instance (e.g. drupaldb). Must be unique for this region. May contain only lowercase alphanumeric characters, hyphens, underscores, periods, and spaces.

task_migration_typestringrequired

The migration type. Can be one of full-load | cdc | full-load-and-cdc.

task_table_mappingsstringrequired

An escaped JSON string that contains the table mappings. For information on table mapping see http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html

Optional​

create_iam_rolesbooloptional

If true, create the account-level IAM roles required by AWS DMS (dms-vpc-role, dms-cloudwatch-logs-role, and dms-access-for-endpoint). Set to false when these shared roles already exist or are managed outside this module.

true

If true, create an aws_dms_replication_instance for non-serverless replication task mode. This is automatically ignored when task_compute_config is set for serverless mode.

true

If false, the DMS instance will bind to instance_subnet_group_id variable.

true
custom_tagsmap(string)optional

A map of custom tags to apply to the DMS Instance and the Security Group created for it. The key is the tag name and the value is the tag value.

{}

The amount of storage (in gigabytes) to be initially allocated for the replication instance. Min: 5, Max: 6144, Default: 50

null

Indicates that major version upgrades are allowed

true

Indicates whether the changes should be applied immediately or during the next maintenance window

null

Indicates that minor engine upgrades will be applied automatically to the replication instance during the maintenance window

true

The EC2 Availability Zone that the replication instance will be created in

null

The engine version number of the replication instance

null
instance_idstringoptional

The replication instance identifier. This parameter is stored as a lowercase string

null
instance_kms_key_arnstringoptional

The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt the connection parameters

null
instance_multi_azbooloptional

Specifies if the replication instance is a multi-az deployment. You cannot set the availability_zone parameter if the multi_az parameter is set to true

null

The weekly time range during which system maintenance can occur, in Universal Coordinated Time (UTC)

null

Specifies the accessibility options for the replication instance

null

An existing subnet group to associate with the replication instance

null
instance_tagsmap(string)optional

A map of additional tags to apply to the replication instance

{}
instance_timeoutsmap(string)optional

A map of timeouts for replication instance create/update/delete operations

{}
instance_typestringoptional

The compute and memory capacity of the replication instance as specified by the replication instance class. Required only for non-serverless replication task mode.

null

A list of VPC security group IDs to be used with the replication instance

null

Name of the endpoint database

null

Type of engine for the endpoint.

"mysql"

Additional attributes associated with the connection. For available attributes for a source Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For available attributes for a target Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.

null

ARN for the KMS key that will be used to encrypt the connection parameters. If you do not specify a value for kms_key_arn, then AWS DMS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS region

null

Password to be used to login to the endpoint database.

null
source_endpoint_portnumberoptional

Port used by the endpoint database.

null

(Required if using non-s3 source) Host name of the server.

null

SSL mode to use for the connection. Valid values are none, require, verify-ca, verify-full

"none"

User name to be used to login to the endpoint database.

null
source_s3_bucket_namestringoptional

(Required if using s3) S3 bucket name.

null
source_s3_cdc_pathstringoptional

(Required if using s3 for CDC; otherwise, Optional) Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from this path and replicates the data changes to the target endpoint. Supported in AWS DMS versions 3.4.2 and later.

null

(Required if using s3) JSON document that describes how AWS DMS should interpret the data.

null

ARN of the IAM role with permissions to the S3 Bucket. Default one will be created if not specified

null

The description of the aws_dms_replication_subnet_group that is created. Defaults to 'Subnet group for the name DB' if not specified.

null
subnet_group_namestringoptional

The name of the aws_dms_replication_subnet_group that is created, or an existing one to use if create_subnet_group is false. Defaults to name if not specified.

null
subnet_idslist(string)optional

A list of subnet ids where the Replication Instance should be deployed. In the standard Gruntwork VPC setup, these should be the private persistence subnet ids. This is ignored if create_subnet_group=false.

[]

Name of the endpoint database

null

Type of engine for the endpoint.

"mysql"

Additional attributes associated with the connection. For available attributes for a source Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html. For available attributes for a target Endpoint, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html.

null

ARN for the KMS key that will be used to encrypt the connection parameters. If you do not specify a value for kms_key_arn, then AWS DMS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS region

null

Password to be used to login to the endpoint database.

null
target_endpoint_portnumberoptional

Port used by the endpoint database.

null

(Required if using non-s3 target) Host name of the server.

null

SSL mode to use for the connection. Valid values are none, require, verify-ca, verify-full

"none"

User name to be used to login to the endpoint database.

null
target_s3_bucket_namestringoptional

(Required if using s3) S3 bucket name.

null
target_s3_cdc_pathstringoptional

Folder path of CDC files. If cdc_path is set, AWS DMS reads CDC files from this path and replicates the data changes to the target endpoint. Supported in AWS DMS versions 3.4.2 and later.

null

JSON document that describes how AWS DMS should interpret the data.

null

ARN of the IAM role with permissions to the S3 Bucket. Default one will be created if not specified

null

(Conflicts with task_cdc_start_time) Indicates when you want a change data capture (CDC) operation to start. The value can be in date, checkpoint, or LSN/SCN format depending on the source engine. For more information, see https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.StartPoint.Native.

null
task_cdc_start_timestringoptional

(Conflicts with task_cdc_start_position) The Unix timestamp integer for the start of the Change Data Capture (CDC) operation.

null
task_compute_configobject(…)optional

Optional compute configuration for serverless replication mode. When set, the module creates aws_dms_replication_config instead of aws_dms_replication_task.

object({
    availability_zone            = optional(string)
    dns_name_servers             = optional(string)
    kms_key_id                   = optional(string)
    max_capacity_units           = optional(number)
    min_capacity_units           = optional(number)
    multi_az                     = optional(bool)
    preferred_maintenance_window = optional(string)
    replication_subnet_group_id  = optional(string)
    vpc_security_group_ids       = optional(list(string))
  })
null
task_settingsstringoptional

An escaped JSON string that contains the task settings. For a complete list of task settings, see http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html.

null

Whether to run or stop the replication task.

false
vpc_idstringoptional

The id of the VPC in which this Replication Instance should be deployed.

null