Error assuming role allow-ops-admin-access-from-other-accounts.
A customer asked: > We are trying to assume a role allow-ops-admin-access-from-other-accounts using the IAM user from security account that is in the group access-all-external-account . The group has permission for various roles including allow-ops-admin-access-from-other-accounts We executed aws-auth --role-arn "arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts" --role-duration-seconds 3600 but we are getting User: arn:aws:iam::XXXXXXXXXXXX:user/XXXXXXXX not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts But when we assume the other role allow-auto-deploy-from-other-accounts which is also an existing permission within the group, we can assume it successfully. Any thought on this? Do we miss something?
## Debug log ### Confirming the IAM user, groups and permissions in question In diagnosing the issue, we first ensured that we understood which group the IAM user that the customer was referencing was in. We confirmed it was the `-access-all-external-accounts` group that was defined in both `vars/autogen/common_vars.yml`: ``` ... create_access_keys: false create_login_profile: true groups: - access-all-external-accounts ... ``` and `security/_global/account-baseline/users.yml` ``` {customer-user-name}: create_access_keys: false create_login_profile: true groups: - access-all-external-accounts - iam-admin - ssh-grunt-sudo-users pgp_key: keybase:{customer-user-name} ... ``` At this point we knew that the user wanted to assumed `allow-ops-admin-access-from-other-accounts` which is defined in their `cross_account_groups.yml` and that their intended IAM user _should_ be able to assume it (because it is a part of the `access-all-external-accounts` group), however the error encountered suggested the user did _not_ have sufficient access to assume the role. Next, we double-checked the `access-all-external-accounts` IAM policy, which is attached to the `access-all-external-accounts` group and found there all of the expected permissions to assume the various roles defined in each of the various AWS accounts. ### Reproducing the users's error with aws-auth Next, we [installed `aws-auth`](https://github.com/gruntwork-io/terraform-aws-security/tree/master/modules/aws-auth#aws-auth-helper) and rolled new security credentials for the intended IAM user who was attempting to assume the role. We then issued the following command on behalf of the target IAM user to attempt to reproduce the reported problem: ``` aws-auth --role-arn "arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts" --role-duration-seconds 3600 2021-08-19 12:24:04 [INFO] [aws-auth] Assuming role arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts. These creds will expire after 3600 seconds.) ``` and we were able to reproduce the reported issue: ``` An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::XXXXXXXXXXXX:user/{customer-iam-user} is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts ``` ### Confirming trust relationships and MFA settings We also confirmed that the target account (dev) had a valid trust relationship configured with the security account. At this point we also saw the MFA setting: ``` Condition Key Value Bool aws:MultiFactorAuthPresent true ``` ### Discovering the root cause This means that the successful assumption of the target IAM role would require presenting a valid MFA token! Next, we configured virtual MFA using Google Authenticator within the security account for the target IAM user and retried our same `aws-auth` call again, this time passing both the `--serial-number` from the IAM user (same as their user ID) and the `--token-code` flag with the One Time Password (OTP) from Google Authenticator : ``` aws-auth --role-arn "arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts" --role-duration-seconds 3600 --serial-number arn:aws:iam::123456789011:mfa/jondoe --token-code 123456 ... Success! ```