Terraform stuck on creating resource
Running the elasticsearch alarms example is timing out locally after ~1h. It gets stuck waiting for the elasticsearch cluster to be created `aws_elasticsearch_domain.cluster: Still creating... [XXmXXs elapsed]` and then fails with a timeout error. Increasing timeout on `go tests` didn't make a difference because there's a 1h creation timeout limit on the terraform resource itself, which isn't configurable on provider version < 4.8.0. (https://github.com/hashicorp/terraform-provider-aws/pull/17513) Additional context: the elastic search domain does take a while to be created, but creation time should be around ~20min for this example. I am using `aws-vault` v4.4.1 for [AWS authentication](https://blog.gruntwork.io/authenticating-to-aws-with-environment-variables-e793d6f6d02e) to our sandbox account through the `allow-full-access-from-other-accounts` iam role.
After running the example with TF LOG trace I found the issue: an expired token exception (see below). It seems that `aws-vault` has a timeout to assume roles of 15 minutes (see `--help` output below). Adjusting the `--assume-role-ttl` flag solved the issue. If I run it like this, example tests now run without issues: `aws-vault exec sandbox --assume-role-ttl=1h -- go test -v -timeout 120m -parallel 128 -run TestElasticsearchAlarms` ``` TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: -----------------------------------------------------: timestamp=2022-03-29T15:21:01.150+0200 TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: 2022-03-29T15:21:01.774+0200 [INFO] provider.terraform-provider-aws_v3.75.1_x5: 2022/03/29 15:21:01 [DEBUG] [aws-sdk-go] DEBUG: Response es/DescribeElasticsearchDomain Details: TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: ---[ RESPONSE ]-------------------------------------- TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: HTTP/1.1 403 TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: Connection: close TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: Content-Length: 67 TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: Content-Type: application/json TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: Date: Tue, 29 Mar 2022 13:21:01 GMT TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: Keep-Alive: timeout=5 TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: X-Amzn-Errortype: ExpiredTokenException TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: -----------------------------------------------------: timestamp=2022-03-29T15:21:01.773+0200 TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: 2022-03-29T15:21:01.774+0200 [INFO] provider.terraform-provider-aws_v3.75.1_x5: 2022/03/29 15:21:01 [DEBUG] [aws-sdk-go]: timestamp=2022-03-29T15:21:01.773+0200 TestElasticsearchAlarms 2022-03-29T15:21:01+02:00 logger.go:66: 2022-03-29T15:21:01.774+0200 [INFO] provider.terraform-provider-aws_v3.75.1_x5: 2022/03/29 15:21:01 [DEBUG] [aws-sdk-go] DEBUG: Validate Response es/DescribeElasticsearchDomain failed, attempt 12/25, error ExpiredTokenException ``` ``` aws-vault exec --help usage: aws-vault exec [<flags>] <profile> [<cmd>] [<args>...] Executes a command with AWS credentials in the environment Flags: --help Show context-sensitive help (also try --help-long and --help-man). --version Show application version. --debug Show debugging output --backend=BACKEND Secret backend to use [keychain file] --prompt=terminal Prompt driver to use [terminal osascript] --keychain="aws-vault" Name of macOS keychain to use, if it doesn't exist it will be created -n, --no-session Use root credentials, no session created -t, --session-ttl=4h Expiration time for aws session --assume-role-ttl=15m Expiration time for aws assumed role -m, --mfa-token=MFA-TOKEN The mfa token to use -s, --server Run the server in the background for credentials Args: <profile> Name of the profile [<cmd>] Command to execute [<args>] Command arguments ```