Certificates pending when trying to use parent and child route53 modules
Wondering if gruntworks has seen this before, or has some ideas around best practices. We use the terraform-aws-service-catalog/networking/route53 module to first create "parent route53 resources" and then we use terragrunt, with a dependency to create child route53 resources using those parents. Our pipelines apply the parent module fine. However when it goes to apply the child. it ALWAYS fails with below. We think we need some sort of wait or something inbetween these but haven't been able to pin point where it goes or "how long". I have tried a null resource to run after the parent is created, but that doesn't seem to do the trick. Also once it fails, if I apply again on the child route53 module it always passes ``` module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["rds.XXX"]: Still creating... [44m50s elapsed] module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["clb.XXX"]: Still creating... [44m50s elapsed] module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["clb.XXX"]: Still creating... [44m50s elapsed] module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["sys.XXX"]: Still creating... [45m0s elapsed] ╷ │ Error: Error describing created certificate: Expected certificate to be issued but was in state PENDING_VALIDATION │ │ with module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["clb.XXX"], │ on .terraform/modules/route_53_zone_creation.acm-tls-certificates/modules/acm-tls-certificate/main.tf line 125, in resource "aws_acm_certificate_validation" "cert": │ 125: resource "aws_acm_certificate_validation" "cert" { │ ``` --- <ins datetime="2022-08-09T12:44:55Z"> <p><a href="https://support.gruntwork.io/hc/requests/109114">Tracked in ticket #109114</a></p> </ins>
Hi Lorelei, When I've encountered this issue in the past, it was always due to some subtle mis-configuration in my code or my actual Route 53 domains, nameservers and Route53 Public Hosted Zones. I'll link [the official AWS doc on how DNS-based certificate validation works](https://docs.aws.amazon.com/acm/latest/userguide/dns-validation.html) which is worth reading if you haven't already. In addition, here's a couple of the things that I've needed to ensure in the past to make sure programmatic DNS validation worked properly: 1. The domain you're issuing ACM certificate requests against must be a valid, registered domain that you control. 2. If the domain name you're trying to generate a certificate for was registered via Route53, ensure that the name servers of _the domain itself_ match the name server records in the Route53 hosted zone that gets created to match it. In other words, it's possible, and a common source of this error, to register example.com via Route53, which will have a set of NS records randomly assigned to it, and then create a public Route 53 Hosted Zone named example.com _which then has a different set of NS records randomly assigned to it_. If you do not reconcile these NS records, public resolution will fail, and so programmatic validation via DNS will fail within AWS. The NS records set on the domain entry itself in Route 53 must match those in the Public Hosted Zone associated with the domain. 3. You can test the programmatic DNS validation process "out of band" by running `dig` queries against the `CNAME` validation challenge records that get inserted in your Route53 Public Hosted Zone. You should see records similar to `random_value>.acm-validations.aws` being inserted by our Terraform module. Importantly, _these records must publicly resolve over the internet_as that is how AWS tests them during validation. If the dig queries fail from your machine, they're going to fail during the validation process, and you'll get stuck at this validation pending step until the timeouts are reached. You can also use a tool such as `https://www.whatsmydns.net/`, setting your record type to `CNAME` and entering the validation challenge record directly in the search bar. When a few resolvers are able to respond with your records - this is a sign that validation will likely eventually succeed. Conversely, if no resolvers return your record after a few minutes, it's likely you have a Route53 misconfiguration issue preventing your domain (and its newly inserted records) from resolving correctly over the public internet. 4. Mis-configured requests for wilcard certificate requests might not match the common name of the domain - be careful if you're requesting wildcard domains in your tests, for example. Hope this helps and gives you something more to dig into!