Terraform State Error 시 해결 방법

Overview

Terraform 실행시에 State 상태에서 멈춰있을 때 해결방법에 대해서 알아본다.

 

 

---

 

Terraform State Error

 

Terraform 실행시에 State 상태에서 멈춰서 동작이 하지 않을때가 있다. Terraform은 `TF_LOG` 환경 변수를 통해 로깅을 제공한다. 따라서 해당명령어로 디버깅할 수 있다.

export TF_LOG=DEBUG
terraform apply -var-file="devqa.tfvars"

 

그러면 에러가 나는 부분을 쉽게 찾을 수 있다.

2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: ---[ REQUEST ]---------------------------------------
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: GET /storage/v1/b/cdn.somaz.link?alt=json&prettyPrint=false HTTP/1.1
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Host: storage.googleapis.com
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.6.2 (+https://www.terraform.io) Terraform-Plugin-SDK/2.31.0 terraform-provider-google/5.21.0
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: X-Goog-Api-Client: gl-go/1.20.14 gdcl/0.167.0
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Accept-Encoding: gzip
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5
2024-05-13T18:55:35.041+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: -----------------------------------------------------
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: 2024/05/13 18:55:35 [DEBUG] Google API Response Details:
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: ---[ RESPONSE ]--------------------------------------
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: HTTP/2.0 404 Not Found
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Content-Length: 171
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Content-Type: application/json; charset=UTF-8
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Date: Mon, 13 May 2024 09:55:35 GMT
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Pragma: no-cache
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Server: UploadServer
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Vary: Origin
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: Vary: X-Origin
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: X-Guploader-Uploadid: ABPtcPpV7JnWcV3-s71tZ-ssfJlp25m2X7jjt2HXd6nVCGthglQX3mUfXIvR5vIX_NWFmP-PXANThKl1Xw
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: {
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:  "error": {
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:   "code": 404,
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:   "message": "The specified bucket does not exist.",
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:   "errors": [
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:    {
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:     "message": "The specified bucket does not exist.",
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:     "domain": "global",
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:     "reason": "notFound"
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:    }
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:   ]
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5:  }
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: }
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: -----------------------------------------------------
2024-05-13T18:55:35.410+0900 [DEBUG] provider.terraform-provider-google_v5.21.0_x5: 2024/05/13 18:55:35 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found

 

 

전체코드를 보면 404 Not Found 에러가 발생한다. 즉, 특정 리소스가 수동으로 삭제되어 `Terraform State` 와 상태가 동일하지 않아서 에러가 발생하는 것이고, Terraform에서는 해당 리소스를 게속 찾는 에러가 나는것이다.

GET /storage/v1/b/cdn.somaz.link?alt=json&prettyPrint=false HTTP/1.1
...
HTTP/2.0 404 Not Found

 

따라서 에러 해결을 위해서는 `Terraform state` 에서 수동으로 삭제된 리소스를 찾은뒤에, 수동으로 삭제해줘야 한다.

data.google_client_config.default
google_compute_address.bastion_ip
google_compute_backend_bucket.dev_sm_cdn_bucket_backend
google_compute_global_address.dev1_adam_lb_ip
google_compute_global_address.dev1_multipath_lb_ip
google_compute_global_address.dev_sm_cdn_lb_ip
google_compute_global_address.qa1_adam_lb_ip
google_compute_global_address.qa1_multipath_lb_ip
google_compute_global_address.qa_sm_cdn_lb_ip
google_compute_global_address.review_multipath_lb_ip
google_compute_global_forwarding_rule.dev_sm_cdn_http_forwarding_rule
google_compute_global_forwarding_rule.dev_sm_cdn_https_forwarding_rule
google_compute_instance.bastion
google_compute_managed_ssl_certificate.dev_sm_cdn_lb_certificate
google_compute_target_http_proxy.dev_sm_cdn_http_proxy
google_compute_target_https_proxy.dev_sm_cdn_https_proxy
google_compute_url_map.dev_sm_cdn_http_url_map
google_compute_url_map.dev_sm_cdn_https_url_map
google_dns_record_set.beta_multipath_record
google_dns_record_set.dev1_adam_record
google_dns_record_set.dev1_multipath_record
google_dns_record_set.dev_sm_cdn_record
google_dns_record_set.develop_sm_cdn_record_io_record
google_dns_record_set.qa1_adam_record
google_dns_record_set.qa1_multipath_record
google_dns_record_set.qa_sm_cdn_record
google_dns_record_set.review_multipath_record
google_storage_bucket.dev_sm_cdn_luxon_run
google_storage_bucket.develop_sm_cdn_luxon_run
google_storage_bucket_iam_binding.public_read_develop_sm_cdn_luxon_run
google_storage_bucket_iam_member.public_read_dev_sm_cdn_luxon_run
google_storage_bucket_iam_member.public_read_qa_sm_cdn_luxon_run
module.gcs_buckets.google_storage_bucket.buckets["devqa-sm-terraform-remote-tfstate"]
module.gcs_buckets.random_id.bucket_suffix
module.gke_autopilot.data.google_compute_subnetwork.gke_subnetwork[0]
module.gke_autopilot.data.google_container_engine_versions.region
module.gke_autopilot.data.google_container_engine_versions.zone
module.gke_autopilot.google_compute_firewall.intra_egress[0]
module.gke_autopilot.google_compute_firewall.master_webhooks[0]
module.gke_autopilot.google_container_cluster.primary
module.gke_autopilot.google_project_iam_member.cluster_service_account-artifact-registry["devqa-sm"]
module.gke_autopilot.google_project_iam_member.cluster_service_account-gcr["devqa-sm"]
module.gke_autopilot.google_project_iam_member.cluster_service_account-nodeService_account[0]
module.gke_autopilot.google_service_account.cluster_service_account[0]
module.gke_autopilot.random_string.cluster_service_account_suffix
module.memorystore.google_redis_instance.default
module.memorystore_qa1_sm.google_redis_instance.default
module.memorystore_review_sm.google_redis_instance.default
module.mysql_public_dev1_sm.google_sql_database.default[0]
module.mysql_public_dev1_sm.google_sql_database_instance.default
module.mysql_public_dev1_sm.google_sql_user.additional_users["somaz_db"]
module.mysql_public_dev1_sm.google_sql_user.default[0]
module.mysql_public_dev1_sm.null_resource.module_depends_on
module.mysql_public_dev1_sm.random_password.additional_passwords["somaz_db"]
module.mysql_public_dev1_sm.random_password.user-password
module.mysql_public_dev1_sm.google_sql_database.additional_databases["auth"]
module.mysql_public_dev1_sm.google_sql_database.additional_databases["log"]
module.mysql_public_dev1_sm.google_sql_database.default[0]
module.mysql_public_dev1_sm.google_sql_database_instance.default
module.mysql_public_dev1_sm.google_sql_user.additional_users["somaz_db"]
module.mysql_public_dev1_sm.google_sql_user.default[0]
module.mysql_public_dev1_sm.null_resource.module_depends_on
module.mysql_public_dev1_sm.random_password.additional_passwords["somaz_db"]
module.mysql_public_dev1_sm.random_password.user-password
module.secret_manager.google_secret_manager_secret.secrets["apple-store-key"]
module.secret_manager.google_secret_manager_secret.secrets["devqa-sm-database"]
module.secret_manager.google_secret_manager_secret.secrets["devqa-sm-mongo-database"]
module.secret_manager.google_secret_manager_secret_version.secret-version["apple-store-key"]
module.secret_manager.google_secret_manager_secret_version.secret-version["devqa-sm-database"]
module.secret_manager.google_secret_manager_secret_version.secret-version["devqa-sm-mongo-database"]
module.memorystore.module.enable_apis.google_project_service.project_services["redis.googleapis.com"]
module.memorystore_qa1_sm.module.enable_apis.google_project_service.project_services["redis.googleapis.com"]
module.memorystore_review_sm.module.enable_apis.google_project_service.project_services["redis.googleapis.com"]

 

`google_storage_bucket.dev_sm_cdn_luxon_run` 리소스가 수동으로 삭제 된걸 확인하였다.

아래의 명령어로 지워준다.

terraform state rm google_storage_bucket.dev_sm_cdn_luxon_run

 

마지막으로 다시 명령을 실행해주면 정상적으로 동작할 것이다.

terraform apply -var-file="devqa.tfvars"

 

또한 특정 모듈이나 리소스의 의존성 문제가 있을 수 있으므로 `terraform graph` 명령어를 사용해 의존성 그래프를 검사할 수 있다.

terraform graph

terraform graph | dot -Tsvg > graph.svg

 

 

 

---

Reference

none